Documentation/x86: Improve the AMX documentation
v5 cover Documentation/x86: Improve the AMX documentation
Hi all,
The change from the last version [1]:
* Update the purpose part (patch1) by rewriting the altstack story and having it as the only reason.
Thanks Dave Hansen for the feedback.
=== Cover Letter ===
When the AMX feature was supported in Linux, the dynamic feature enabling model was introduced. The following documentation changes were considered to help users to adopt this new enabling model:
(1) The AMX-enabling code example is expected to clarify the steps.
(2) Along with that, a couple of ABI constants may be useful for users.
(3) Also, describing the motiviation will provide the context of this.
(4) Lastly, the description of new guest options are added as missing.
If they are acceptable, the arch_prctl(2) manual page [2] will be followed up separately with something similar to the kernel documentation.
These patches can be also found in this repository:
git://github.com/intel/amx-linux.git amx-doc
And the compiled preview is available here:
https://htmlpreview.github.io/?https://github.com/intel/amx-linux/doc-web/x86/xstate.html
Thanks,
Chang
[1] https://lore.kernel.org/lkml/[email protected]/
[2] arch_prctl(2): https://man7.org/linux/man-pages/man2/arch_prctl.2.html
Chang S. Bae (4):
Documentation/x86: Explain the purpose for dynamic features x86/arch_prctl: Add AMX feature numbers as ABI constants Documentation/x86: Add the AMX enabling example Documentation/x86: Explain the state component permission for guests
Documentation/x86/xstate.rst | 100 ++++++++++++++++++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 3 + 2 files changed, 103 insertions(+)
base-commit: 08ed00508bc1fa0a0bc6dd2420f982b55051de23
Signature · 1 line
2.17.1
PATCH v5 1/4 Documentation/x86: Explain the purpose for dynamic features
This summary will help to guide the proper use of the enabling model.
---
Changes from v4:
* Re-write about the sigaltstack sizing (Dave Hansen).
* Drop the second point as the case is not clear yet.
Changes from v3:
* Add as a new patch (Tony Luck).
--- Documentation/x86/xstate.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Documentation/x86/xstate.rst +16 -0
@@ -11,6 +11,22 @@ are enabled by XCR0 as well, but the first use of related instruction is trapped by the kernel because by default the required large XSTATE buffers are not allocated automatically. +The purpose for dynamic features +-------------------------------- + +Legacy userspace libraries often have hard-coded, static sizes for +alternate signal stacks, often using MINSIGSTKSZ which is typically 2KB. +That stack must be able to store at *least* the signal frame that the +kernel sets up before jumping into the signal handler. That signal frame +must include an XSAVE buffer defined by the CPU. + +However, that means that the size of signal stacks is dynamic, not static, +because different CPUs have differently-sized XSAVE buffers. A compiled-in +size of 2KB with existing applications is too small for new CPU features +like AMX. Instead of universally requiring larger stack, with the dynamic +enabling, the kernel can enforce userspace applications to have +properly-sized altstacks. + Using dynamically enabled XSTATE features in user space applications --------------------------------------------------------------------
Signature · 1 line
2.17.1
PATCH v5 2/4 x86/arch_prctl: Add AMX feature numbers as ABI constants
In reply to Chang S. BaeAMX state is dynamically enabled by the architecture-specific prctl(). Expose the state components as ABI constants. They become handy not to be looked up from the architecture specification.
---
Changes from v2:
* Add as a new patch (Tony Luck).
--- arch/x86/include/uapi/asm/prctl.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/uapi/asm/prctl.h +3 -0
@@ -16,6 +16,9 @@#define ARCH_GET_XCOMP_GUEST_PERM 0x1024#define ARCH_REQ_XCOMP_GUEST_PERM 0x1025 +#define ARCH_XCOMP_TILECFG 17 +#define ARCH_XCOMP_TILEDATA 18 +#define ARCH_MAP_VDSO_X32 0x2001#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003
Signature · 1 line
2.17.1
PATCH v5 3/4 Documentation/x86: Add the AMX enabling example
In reply to Chang S. BaeExplain steps to enable the dynamic feature with a code example.
---
Changes from v1:
* Update the description without mentioning CPUID & XGETBV (Dave Hansen).
Changes from v2:
* Massage sentences (Bagas Sanjaya).
* Adjust the example with the (future) prctl.h.
--- Documentation/x86/xstate.rst | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
Documentation/x86/xstate.rst +55 -0
@@ -80,6 +80,61 @@ the handler allocates a larger xstate buffer for the task so the large state can be context switched. In the unlikely cases that the allocation fails, the kernel sends SIGSEGV. +AMX TILE_DATA enabling example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Below is the example of how userspace applications enable +TILE_DATA dynamically: + + 1. The application first needs to query the kernel for AMX + support:: + + #include <asm/prctl.h> + #include <sys/syscall.h> + #include <stdio.h> + #include <unistd.h> + + #ifndef ARCH_GET_XCOMP_SUPP + #define ARCH_GET_XCOMP_SUPP 0x1021 + #endif + + #ifndef ARCH_XCOMP_TILECFG + #define ARCH_XCOMP_TILECFG 17 + #endif + + #ifndef ARCH_XCOMP_TILEDATA + #define ARCH_XCOMP_TILEDATA 18 + #endif + + #define MASK_XCOMP_TILE ((1 << ARCH_XCOMP_TILECFG) | \ + (1 << ARCH_XCOMP_TILEDATA)) + + unsigned long features; + long rc; + + ... + + rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features); + + if (!rc && (features & MASK_XCOMP_TILE) == MASK_XCOMP_TILE) + printf("AMX is available.\n"); + + 2. After that, determining support for AMX, an application must + explicitly ask permission to use it:: + + #ifndef ARCH_REQ_XCOMP_PERM + #define ARCH_REQ_XCOMP_PERM 0x1023 + #endif + + ... + + rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, ARCH_XCOMP_TILEDATA); + + if (!rc) + printf("AMX is ready for use.\n"); + +Note this example does not include the sigaltstack preparation. + Dynamic features in signal frames ---------------------------------
Signature · 1 line
2.17.1
PATCH v5 4/4 Documentation/x86: Explain the state component permission for guests
In reply to Chang S. BaeCommit 980fe2fddcff ("x86/fpu: Extend fpu_xstate_prctl() with guest permissions") extends a couple of arch_prctl(2) options for VCPU threads. Add description for them.
---
Changes from v1:
* Add the reason for the guest options (Dave Hansen).
* Add a note to allude some VMM policy, i.e. KVM_X86_XCOMP_GUEST_SUPP.
* Move it in the separate section.
Note the correspondent attributes were also proposed for the KVM API. But, it was seen as inessential:
https://lore.kernel.org/lkml/[email protected]/ --- Documentation/x86/xstate.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
Documentation/x86/xstate.rst +29 -0
@@ -143,3 +143,32 @@ entry if the feature is in its initial configuration. This differs from non-dynamic features which are always written regardless of their configuration. Signal handlers can examine the XSAVE buffer's XSTATE_BV field to determine if a features was written. + +Dynamic features for virtual machines +------------------------------------- + +The permission for the guest state component needs to be managed separately +from the host, as they are exclusive to each other. A coupled of options +are extended to control the guest permission: + +-ARCH_GET_XCOMP_GUEST_PERM + + arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features); + + ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it + provides the same semantics and functionality but for the guest + components. + +-ARCH_REQ_XCOMP_GUEST_PERM + + arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr); + + ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. It has the + same semantics for the guest permission. While providing a similar + functionality, this comes with a constraint. Permission is frozen when the + first VCPU is created. Any attempt to change permission after that point + is going to be rejected. So, the permission has to be requested before the + first VCPU creation. + +Note that some VMMs may have already established a set of supported state +components. These options are not presumed to support any particular VMM.
Signature · 1 line
2.17.1
v5 cover Documentation/x86: Improve the AMX documentation
In reply to Chang S. BaeHi,
This is a resend of the previous posting [1]. They are rebased on the current tip master.
Looks like the below changes are still helpful for users as I had to reference the pending series to some folks:
(1) The AMX-enabling code example is expected to clarify the steps.
(2) Along with that, a couple of ABI constants may be useful for users.
(3) Also, describing the motivation will provide the context of this.
(4) Lastly, the description of new guest options is added as missing.
If they are acceptable, I will follow up on the arch_prctl(2) documentation
[2] to update based on this.
These patches can be also found in this repository:
git://github.com/intel/amx-linux.git amx-doc
And the compiled preview is available here:
https://htmlpreview.github.io/?https://github.com/intel/amx-linux/doc-web/x86/xstate.html
Thanks,
Chang
[1] https://lore.kernel.org/lkml/[email protected]/
[2] arch_prctl(2): https://man7.org/linux/man-pages/man2/arch_prctl.2.html
Chang S. Bae (4):
Documentation/x86: Explain the purpose for dynamic features x86/arch_prctl: Add AMX feature numbers as ABI constants Documentation/x86: Add the AMX enabling example Documentation/x86: Explain the state component permission for guests
Documentation/x86/xstate.rst | 100 ++++++++++++++++++++++++++++++ arch/x86/include/uapi/asm/prctl.h | 3 + 2 files changed, 103 insertions(+)
base-commit: 1642cea9d9ab82b596857eea14bf6e04806a6c5a
Signature · 1 line
2.17.1
PATCH RESEND v5 1/4 Documentation/x86: Explain the purpose for dynamic features
This summary will help to guide the proper use of the enabling model.
---
Changes from v4:
* Re-write about the sigaltstack sizing (Dave Hansen).
* Drop the second point as the case is not clear yet.
Changes from v3:
* Add as a new patch (Tony Luck).
--- Documentation/x86/xstate.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Documentation/x86/xstate.rst +16 -0
@@ -11,6 +11,22 @@ are enabled by XCR0 as well, but the first use of related instruction is trapped by the kernel because by default the required large XSTATE buffers are not allocated automatically. +The purpose for dynamic features +-------------------------------- + +Legacy userspace libraries often have hard-coded, static sizes for +alternate signal stacks, often using MINSIGSTKSZ which is typically 2KB. +That stack must be able to store at *least* the signal frame that the +kernel sets up before jumping into the signal handler. That signal frame +must include an XSAVE buffer defined by the CPU. + +However, that means that the size of signal stacks is dynamic, not static, +because different CPUs have differently-sized XSAVE buffers. A compiled-in +size of 2KB with existing applications is too small for new CPU features +like AMX. Instead of universally requiring larger stack, with the dynamic +enabling, the kernel can enforce userspace applications to have +properly-sized altstacks. + Using dynamically enabled XSTATE features in user space applications --------------------------------------------------------------------
Signature · 1 line
2.17.1
PATCH RESEND v5 2/4 x86/arch_prctl: Add AMX feature numbers as ABI constants
In reply to Chang S. BaeAMX state is dynamically enabled by the architecture-specific prctl(). Expose the state components as ABI constants. They become handy not to be looked up from the architecture specification.
---
Changes from v2:
* Add as a new patch (Tony Luck).
--- arch/x86/include/uapi/asm/prctl.h | 3 +++ 1 file changed, 3 insertions(+)
arch/x86/include/uapi/asm/prctl.h +3 -0
@@ -16,6 +16,9 @@#define ARCH_GET_XCOMP_GUEST_PERM 0x1024#define ARCH_REQ_XCOMP_GUEST_PERM 0x1025 +#define ARCH_XCOMP_TILECFG 17 +#define ARCH_XCOMP_TILEDATA 18 +#define ARCH_MAP_VDSO_X32 0x2001#define ARCH_MAP_VDSO_32 0x2002#define ARCH_MAP_VDSO_64 0x2003
Signature · 1 line
2.17.1
PATCH RESEND v5 3/4 Documentation/x86: Add the AMX enabling example
In reply to Chang S. BaeExplain steps to enable the dynamic feature with a code example.
---
Changes from v1:
* Update the description without mentioning CPUID & XGETBV (Dave Hansen).
Changes from v2:
* Massage sentences (Bagas Sanjaya).
* Adjust the example with the (future) prctl.h.
--- Documentation/x86/xstate.rst | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+)
Documentation/x86/xstate.rst +55 -0
@@ -80,6 +80,61 @@ the handler allocates a larger xstate buffer for the task so the large state can be context switched. In the unlikely cases that the allocation fails, the kernel sends SIGSEGV. +AMX TILE_DATA enabling example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Below is the example of how userspace applications enable +TILE_DATA dynamically: + + 1. The application first needs to query the kernel for AMX + support:: + + #include <asm/prctl.h> + #include <sys/syscall.h> + #include <stdio.h> + #include <unistd.h> + + #ifndef ARCH_GET_XCOMP_SUPP + #define ARCH_GET_XCOMP_SUPP 0x1021 + #endif + + #ifndef ARCH_XCOMP_TILECFG + #define ARCH_XCOMP_TILECFG 17 + #endif + + #ifndef ARCH_XCOMP_TILEDATA + #define ARCH_XCOMP_TILEDATA 18 + #endif + + #define MASK_XCOMP_TILE ((1 << ARCH_XCOMP_TILECFG) | \ + (1 << ARCH_XCOMP_TILEDATA)) + + unsigned long features; + long rc; + + ... + + rc = syscall(SYS_arch_prctl, ARCH_GET_XCOMP_SUPP, &features); + + if (!rc && (features & MASK_XCOMP_TILE) == MASK_XCOMP_TILE) + printf("AMX is available.\n"); + + 2. After that, determining support for AMX, an application must + explicitly ask permission to use it:: + + #ifndef ARCH_REQ_XCOMP_PERM + #define ARCH_REQ_XCOMP_PERM 0x1023 + #endif + + ... + + rc = syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, ARCH_XCOMP_TILEDATA); + + if (!rc) + printf("AMX is ready for use.\n"); + +Note this example does not include the sigaltstack preparation. + Dynamic features in signal frames ---------------------------------
Signature · 1 line
2.17.1
PATCH RESEND v5 4/4 Documentation/x86: Explain the state component permission for guests
In reply to Chang S. BaeCommit 980fe2fddcff ("x86/fpu: Extend fpu_xstate_prctl() with guest permissions") extends a couple of arch_prctl(2) options for VCPU threads. Add description for them.
---
Changes from v1:
* Add the reason for the guest options (Dave Hansen).
* Add a note to allude some VMM policy, i.e. KVM_X86_XCOMP_GUEST_SUPP.
* Move it in the separate section.
Note the correspondent attributes were also proposed for the KVM API. But, it was seen as inessential:
https://lore.kernel.org/lkml/[email protected]/ --- Documentation/x86/xstate.rst | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
Documentation/x86/xstate.rst +29 -0
@@ -143,3 +143,32 @@ entry if the feature is in its initial configuration. This differs from non-dynamic features which are always written regardless of their configuration. Signal handlers can examine the XSAVE buffer's XSTATE_BV field to determine if a features was written. + +Dynamic features for virtual machines +------------------------------------- + +The permission for the guest state component needs to be managed separately +from the host, as they are exclusive to each other. A coupled of options +are extended to control the guest permission: + +-ARCH_GET_XCOMP_GUEST_PERM + + arch_prctl(ARCH_GET_XCOMP_GUEST_PERM, &features); + + ARCH_GET_XCOMP_GUEST_PERM is a variant of ARCH_GET_XCOMP_PERM. So it + provides the same semantics and functionality but for the guest + components. + +-ARCH_REQ_XCOMP_GUEST_PERM + + arch_prctl(ARCH_REQ_XCOMP_GUEST_PERM, feature_nr); + + ARCH_REQ_XCOMP_GUEST_PERM is a variant of ARCH_REQ_XCOMP_PERM. It has the + same semantics for the guest permission. While providing a similar + functionality, this comes with a constraint. Permission is frozen when the + first VCPU is created. Any attempt to change permission after that point + is going to be rejected. So, the permission has to be requested before the + first VCPU creation. + +Note that some VMMs may have already established a set of supported state +components. These options are not presumed to support any particular VMM.
Signature · 1 line
2.17.1