arm64: mpam: Document when to set arm64.nompam
Subsystems (3)
PATCH v2 1/1 arm64: mpam: Document when to set arm64.nompam
arm64.nompam skips the MPAM2_EL2 and MPAMHCR_EL2 writes in finalise_el2_state and leaves the ARM64_MPAM cpucap unset. Where EL3 firmware has cleared MPAM3_EL3.TRAPLOWER, the EL2 trap controls are left unwritten, at reset values that are UNKNOWN, and KVM still hides MPAM from the guest but no longer enables the traps that stop a guest from using it. KVM never saves or restores MPAM0_EL1 and MPAM1_EL1, so what one guest writes reaches the next guest and the host. Nor can the kernel make the EL2 writes unconditionally: they trap to EL3 on the firmware the option exists for, and TRAPLOWER cannot be read below EL3.
Say so in mpam.rst, and add the rule and a pointer to the kernel-parameters entry.
---
Notes:
Changes since v1, the first three from Ben Horgan's review:
- Put arm64.nompam under a new "Command line parameters" section.
- Separate MPAM3_EL3.MPAMEN from TRAPLOWER, which v1 conflated.
- Add that KVM does not save or restore MPAM0_EL1 and MPAM1_EL1.
- Drop "No functional change intended." from the commit message.
Bradley Morgan's Reviewed-by on v1 is not carried over: both paragraphs, the section heading and the commit message changed.
v1: https://lore.kernel.org/all/[email protected]/
.../admin-guide/kernel-parameters.txt | 3 ++- Documentation/arch/arm64/mpam.rst | 25 +++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +2 -1
@@ -575,7 +575,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension support
Documentation/arch/arm64/mpam.rst +25 -0
@@ -87,6 +87,31 @@ The supported features are: MBWU monitors can be exposed to the user after support for more monitoring scopes is added to resctrl. +Command line parameters +======================= + +arm64.nompam +------------ +Firmware controls MPAM through two bits of MPAM3_EL3. MPAMEN enables +it: while set, the PARTID and PMG in the MPAMn_ELx registers label the +CPU's memory requests. TRAPLOWER, which resets to 1, traps accesses to +the MPAM system registers from the lower exception levels to EL3. +Firmware must clear it, or handle the trap and emulate MPAM as disabled. +Where it does neither, the CPUs still advertise MPAM in the ID registers, +the kernel's MPAM register accesses trap to EL3, and the boot fails. +``arm64.nompam`` exists for that firmware: it makes the kernel treat the +CPUs as not implementing MPAM, so no MPAM system register is accessed. +Set it only on a machine that does not boot without it. + +It is not a way to turn MPAM off. Where firmware has cleared TRAPLOWER, +the option leaves the trap controls in MPAM2_EL2 and MPAMHCR_EL2 +unwritten, and their reset values are UNKNOWN. KVM still hides MPAM from +guests but no longer enables the traps that stop a guest from using it, +so a guest may be able to read and write MPAM0_EL1 and MPAM1_EL1. KVM +does not save or restore them, so what one guest writes is still there +when the next guest runs on that CPU, and when the host does. With MPAMEN +set, EL0 and EL1 requests then carry that PARTID and PMG. + Reporting Bugs ============== If you are not seeing the counters or controls you expect please share the
Signature · 1 line
2.39.5
Quoted 25 lines · Fuad Tabba
arm64.nompam skips the
MPAM2_EL2andMPAMHCR_EL2writes infinalise_el2_stateand leaves theARM64_MPAMcpucap unset. Where EL3 firmware has cleared MPAM3_EL3.TRAPLOWER, the EL2 trap controls are left unwritten, at reset values that are UNKNOWN, and KVM still hides MPAM from the guest but no longer enables the traps that stop a guest from using it. KVM never saves or restoresMPAM0_EL1andMPAM1_EL1, so what one guest writes reaches the next guest and the host. Nor can the kernel make the EL2 writes unconditionally: they trap to EL3 on the firmware the option exists for, and TRAPLOWER cannot be read below EL3.Say so in
mpam.rst, and add the rule and a pointer to the kernel-parameters entry.Suggested-by Ben Horgan <[email protected]>Link https://lore.kernel.org/all/CA+EHjTxeWxZiuSmnKLGLxTBXP4oJT7-LuffbPAyCSZZ5TW=5Ew@mail.gmail.com/ open hereSigned-off-by Fuad Tabba <[email protected]>---
Notes:
Changes since v1, the first three from Ben Horgan's review:
- Put arm64.nompam under a new "Command line parameters" section.
- Separate MPAM3_EL3.MPAMEN from TRAPLOWER, which v1 conflated.
- Add that KVM does not save or restoreMPAM0_EL1andMPAM1_EL1.
- Drop "No functional change intended." from the commit message.
Quoted 2 lines
Bradley Morgan's Reviewed-by on v1 is not carried over: both paragraphs, the section heading and the commit message changed.
Still LGTM, the only arm device I own is pixel 7, and also a modern arm test-peoples-crap box. So from what I recall, my tag stands
Quoted 54 lines
v1: https://lore.kernel.org/all/[email protected]/
.../admin-guide/kernel-parameters.txt| 3 ++-Documentation/arch/arm64/mpam.rst| 25 +++++++++++++++++++2 files changed, 27 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +2
-1@@ -575,7 +575,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension supportDocumentation/arch/arm64/mpam.rst +25
-0@@ -87,6 +87,31 @@ The supported features are: MBWU monitors can be exposed to the user after support for more monitoring scopes is added to resctrl. +Command line parameters +======================= + +arm64.nompam +------------ +Firmware controls MPAM through two bits of MPAM3_EL3. MPAMEN enables +it: while set, the PARTID and PMG in the MPAMn_ELx registers label the +CPU's memory requests. TRAPLOWER, which resets to 1, traps accesses to +the MPAM system registers from the lower exception levels to EL3. +Firmware must clear it, or handle the trap and emulate MPAM as disabled. +Where it does neither, the CPUs still advertise MPAM in the ID registers, +the kernel's MPAM register accesses trap to EL3, and the boot fails. +``arm64.nompam`` exists for that firmware: it makes the kernel treat the +CPUs as not implementing MPAM, so no MPAM system register is accessed. +Set it only on a machine that does not boot without it. + +It is not a way to turn MPAM off. Where firmware has cleared TRAPLOWER, +the option leaves the trap controls in MPAM2_EL2 and MPAMHCR_EL2 +unwritten, and their reset values are UNKNOWN. KVM still hides MPAM from +guests but no longer enables the traps that stop a guest from using it, +so a guest may be able to read and write MPAM0_EL1 and MPAM1_EL1. KVM +does not save or restore them, so what one guest writes is still there +when the next guest runs on that CPU, and when the host does. With MPAMEN +set, EL0 and EL1 requests then carry that PARTID and PMG. +
Quoted 5 lines
Reporting Bugs
==============
If you are not seeing the counters or controls you expect please share the
Ben Horgan
maintainerHi Fuad,
Quoted 67 lines · Fuad Tabba
arm64.nompam skips the
MPAM2_EL2andMPAMHCR_EL2writes infinalise_el2_stateand leaves theARM64_MPAMcpucap unset. Where EL3 firmware has cleared MPAM3_EL3.TRAPLOWER, the EL2 trap controls are left unwritten, at reset values that are UNKNOWN, and KVM still hides MPAM from the guest but no longer enables the traps that stop a guest from using it. KVM never saves or restoresMPAM0_EL1andMPAM1_EL1, so what one guest writes reaches the next guest and the host. Nor can the kernel make the EL2 writes unconditionally: they trap to EL3 on the firmware the option exists for, and TRAPLOWER cannot be read below EL3.Say so in
mpam.rst, and add the rule and a pointer to the kernel-parameters entry.Suggested-by Ben Horgan <[email protected]>Link https://lore.kernel.org/all/CA+EHjTxeWxZiuSmnKLGLxTBXP4oJT7-LuffbPAyCSZZ5TW=5Ew@mail.gmail.com/ open hereSigned-off-by Fuad Tabba <[email protected]>---
Notes:
Changes since v1, the first three from Ben Horgan's review:
- Put arm64.nompam under a new "Command line parameters" section.
- Separate MPAM3_EL3.MPAMEN from TRAPLOWER, which v1 conflated.
- Add that KVM does not save or restoreMPAM0_EL1andMPAM1_EL1.
- Drop "No functional change intended." from the commit message.Bradley Morgan's Reviewed-by on v1 is not carried over: both paragraphs, the section heading and the commit message changed.
v1: https://lore.kernel.org/all/[email protected]/
.../admin-guide/kernel-parameters.txt | 3 ++- Documentation/arch/arm64/mpam.rst | 25 +++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +2
-1@@ -575,7 +575,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension supportDocumentation/arch/arm64/mpam.rst +11
-0@@ -87,6 +87,31 @@ The supported features are: MBWU monitors can be exposed to the user after support for more monitoring scopes is added to resctrl. +Command line parameters +======================= + +arm64.nompam +------------ +Firmware controls MPAM through two bits of MPAM3_EL3. MPAMEN enables +it: while set, the PARTID and PMG in the MPAMn_ELx registers label the +CPU's memory requests. TRAPLOWER, which resets to 1, traps accesses to +the MPAM system registers from the lower exception levels to EL3. +Firmware must clear it, or handle the trap and emulate MPAM as disabled. +Where it does neither, the CPUs still advertise MPAM in the ID registers,
Documentation/arch/arm64/mpam.rst The supported features are: · line 110
Quoted 10 lines
Documentation/arch/arm64/mpam.rst +10
-0+the kernel's MPAM register accesses trap to EL3, and the boot fails. +``arm64.nompam`` exists for that firmware: it makes the kernel treat the +CPUs as not implementing MPAM, so no MPAM system register is accessed. +Set it only on a machine that does not boot without it. + +It is not a way to turn MPAM off. Where firmware has cleared TRAPLOWER, +the option leaves the trap controls in MPAM2_EL2 and MPAMHCR_EL2 +unwritten, and their reset values are UNKNOWN. KVM still hides MPAM from +guests but no longer enables the traps that stop a guest from using it, +so a guest may be able to read and write MPAM0_EL1 and MPAM1_EL1. KVM
As you are listing registers, MPAMSM_EL1 can go here too.
Generally, I think this documentation change looks good now but holding off on a tag for the moment in case anything comes up from your MPAM trapping change [1].
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Thanks,
Ben
Quoted 7 lines
Documentation/arch/arm64/mpam.rst +4
-0+does not save or restore them, so what one guest writes is still there +when the next guest runs on that CPU, and when the host does. With MPAMEN +set, EL0 and EL1 requests then carry that PARTID and PMG. + Reporting Bugs ============== If you are not seeing the counters or controls you expect please share the
Hi Ben,
Quoted 75 lines · Ben Horgan
Hi Fuad,
Quoted 71 lines · Fuad Tabba
arm64.nompam skips the
MPAM2_EL2andMPAMHCR_EL2writes infinalise_el2_stateand leaves theARM64_MPAMcpucap unset. Where EL3 firmware has cleared MPAM3_EL3.TRAPLOWER, the EL2 trap controls are left unwritten, at reset values that are UNKNOWN, and KVM still hides MPAM from the guest but no longer enables the traps that stop a guest from using it. KVM never saves or restoresMPAM0_EL1andMPAM1_EL1, so what one guest writes reaches the next guest and the host. Nor can the kernel make the EL2 writes unconditionally: they trap to EL3 on the firmware the option exists for, and TRAPLOWER cannot be read below EL3.Say so in
mpam.rst, and add the rule and a pointer to the kernel-parameters entry.Suggested-by Ben Horgan <[email protected]>Link https://lore.kernel.org/all/CA+EHjTxeWxZiuSmnKLGLxTBXP4oJT7-LuffbPAyCSZZ5TW=5Ew@mail.gmail.com/ open hereSigned-off-by Fuad Tabba <[email protected]>---
Notes:
Changes since v1, the first three from Ben Horgan's review:
- Put arm64.nompam under a new "Command line parameters" section.
- Separate MPAM3_EL3.MPAMEN from TRAPLOWER, which v1 conflated.
- Add that KVM does not save or restoreMPAM0_EL1andMPAM1_EL1.
- Drop "No functional change intended." from the commit message.Bradley Morgan's Reviewed-by on v1 is not carried over: both paragraphs, the section heading and the commit message changed.
v1: https://lore.kernel.org/all/[email protected]/
.../admin-guide/kernel-parameters.txt | 3 ++- Documentation/arch/arm64/mpam.rst | 25 +++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-)Documentation/admin-guide/kernel-parameters.txt +2
-1@@ -575,7 +575,8 @@ Kernel parameters Set instructions support arm64.nompam [ARM64] Unconditionally disable Memory Partitioning And - Monitoring support + Monitoring support. Only for a machine that does not + boot without it. See Documentation/arch/arm64/mpam.rst arm64.nomte [ARM64] Unconditionally disable Memory Tagging Extension supportDocumentation/arch/arm64/mpam.rst +15
-0@@ -87,6 +87,31 @@ The supported features are: MBWU monitors can be exposed to the user after support for more monitoring scopes is added to resctrl. +Command line parameters +======================= + +arm64.nompam +------------ +Firmware controls MPAM through two bits of MPAM3_EL3. MPAMEN enables +it: while set, the PARTID and PMG in the MPAMn_ELx registers label the +CPU's memory requests. TRAPLOWER, which resets to 1, traps accesses to +the MPAM system registers from the lower exception levels to EL3. +Firmware must clear it, or handle the trap and emulate MPAM as disabled. +Where it does neither, the CPUs still advertise MPAM in the ID registers, +the kernel's MPAM register accesses trap to EL3, and the boot fails. +``arm64.nompam`` exists for that firmware: it makes the kernel treat the +CPUs as not implementing MPAM, so no MPAM system register is accessed. +Set it only on a machine that does not boot without it.
Quoted 8 lines
Documentation/arch/arm64/mpam.rst The supported features are: · line 110
Quoted 6 lines
Documentation/arch/arm64/mpam.rst +6
-0+ +It is not a way to turn MPAM off. Where firmware has cleared TRAPLOWER, +the option leaves the trap controls in MPAM2_EL2 and MPAMHCR_EL2 +unwritten, and their reset values are UNKNOWN. KVM still hides MPAM from +guests but no longer enables the traps that stop a guest from using it, +so a guest may be able to read and write MPAM0_EL1 and MPAM1_EL1. KVMAs you are listing registers,
MPAMSM_EL1can go here too.
I can respin a V3 with this added.
Quoted 4 lines
Generally, I think this documentation change looks good now but holding off on a tag for the moment in case anything comes up from your MPAM trapping change [1].
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
The trapping respin doesn't reach the nompam path: the override skips the finalise_el2_state writes whether they clear the traps or set them, so nothing here needs changing.
Cheers,
/fuad
Quoted 13 lines
Thanks,
Ben
Quoted 7 lines
Documentation/arch/arm64/mpam.rst +4
-0+does not save or restore them, so what one guest writes is still there +when the next guest runs on that CPU, and when the host does. With MPAMEN +set, EL0 and EL1 requests then carry that PARTID and PMG. + Reporting Bugs ============== If you are not seeing the counters or controls you expect please share the