PKS/PMEM: Add Stray Write Protection
Subsystems (23)
v9 cover PKS/PMEM: Add Stray Write Protection
From: Ira Weiny <[email protected]>
I'm looking for Intel acks on the series prior to submitting to maintainers. Most of the changes from V8 to V9 was in getting the tests straightened out. But there are some improvements in the actual code.
Changes for V9
Review and update all commit messages.
Update cover letter below
PKS Core
Separate user and supervisor pkey code in the headers
create linux/pks.h for supervisor calls
This facilitated making the pmem code more efficient
Completely rearchitect the test code
[After Dave Hansen and Rick Edgecombe found issues in the test
code it was easier to rearchitect the code completely rather than attempt to fix it.]
Remove pks_test_callback in favor of using fault hooks
Fault hooks also isolate the fault callbacks from being false positives if non-test consumers are running
Make additional PKS_TEST_RUN_ALL Kconfig option which is
mutually exclusive to any non-test PKS consumer PKS_TEST_RUN_ALL takes over all pkey callbacks
Ensure that each test runs within it's own context and is
mutually exclusive from running while any other test is running.
Ensure test session and context memory is cleaned up on file
close
Use pr_debug() and dynamic debug for in kernel debug messages Enhance test_pks selftest
Add the ability to run all tests not just the context
switch test
Standardize output [PASS][FAIL][SKIP]
Add '-d' option enables dynamic debug to see the kernel
debug messages
Incorporate feedback from Rick Edgecombe
Update all pkey types to u8
Fix up test code barriers
Move patch declaring PKS_INIT_VALUE ahead of the patch which enables
PKS so that PKS_INIT_VALUE can be used when pks_setup() is first created
From Dan Williams
Use macros instead of an enum for a pkey allocation scheme
which is predicated on the config options of consumers This almost worked perfectly. It required a bit of tweeking to be able to allocate all of the keys.
From Dave Hansen
Reposition some code to be near/similar to user pkeys
s/pks_write_current/x86_pkrs_load s/pks_saved_pkrs/pkrs
Update Documentation s/PKR_{RW,AD,WD}_KEY/PKR_{RW,AD,WD}_MASK Consistently use lower case for pkey Update commit messages
Add Acks
PMEM Stray Write
Building on the change to the pks_mk_*() function rename
s/pgmap_mk_*/pgmap_set_*/
s/dax_mk_*/dax_set_*/
From Dan Williams
Avoid adding new dax operations by teaching dax_device about pgmap Remove pgmap_protection_flag_invalid() patch (Just let
kmap'ings fail)
PKS/PMEM Stray write protection
===============================
This series is broken into 2 parts.
1) Introduce Protection Key Supervisor (PKS), testing, and documentation
2) Use PKS to protect PMEM from stray writes
Introduce Protection Key Supervisor (PKS)
-----------------------------------------
PKS enables protections on 'domains' of supervisor pages to limit supervisor mode access to pages beyond the normal paging protections. PKS works in a similar fashion to user space pkeys, PKU. As with PKU, supervisor pkeys are checked in addition to normal paging protections. And page mappings are assigned to a domain by setting a 4 bit pkey in the PTE of that mapping.
Unlike PKU, permissions are changed via a MSR update. This update avoids TLB flushes making this an efficient way to alter protections vs PTE updates.
Also, unlike PTE updates PKS permission changes apply only to the current processor. Therefore changing permissions apply only to that thread and not any other cpu/process. This allows protections to remain in place on other cpus for additional protection and isolation.
Even though PKS updates are thread local, XSAVE is not supported for the PKRS MSR. Therefore this implementation saves and restores the MSR across context switches and during exceptions within software. Nested exceptions are supported by each exception getting a new PKS state.
For consistent behavior with current paging protections, pkey 0 is reserved and configured to allow full access via the pkey mechanism, thus preserving the default paging protections because PTEs naturally have a pkey value of 0.
Other keys, (1-15) are statically allocated by kernel consumers when configured. This is done by adding the appropriate PKS_NEW_KEY and PKS_DECLARE_INIT_VALUE macros to pks-keys.h.
Two PKS consumers, PKS_TEST and PMEM stray write protection, are included in this series. When the number of users grows larger the sharing of keys will need to be resolved depending on the needs of the users at that time. Many methods have been contemplated but the number of kernel users and use cases envisioned is still quite small, much less than the 15 available keys.
To summarize, the following are key attributes of PKS.
1) Fast switching of permissions
1a) Prevents access without page table manipulations 1b) No TLB flushes required
2) Works on a per thread basis, thus allowing protections to be preserved on threads which are not actively accessing data through the mapping.
PKS is available with 4 and 5 level paging. For this and simplicity of implementation, the feature is restricted to x86_64.
Use PKS to protect PMEM from stray writes
-----------------------------------------
DAX leverages the direct-map to enable 'struct page' services for PMEM. Given that PMEM capacity may be an order of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Such a stray write becomes a silent data corruption bug.
Stray pointers to System RAM may result in a crash or other undesirable behavior which, while unfortunate, are usually recoverable with a reboot. Stray writes to PMEM are permanent in nature and thus are more likely to result in permanent user data loss. Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, direct-I/O, and any properly kmap'ed page), it is amenable to PKS protection.
Set up an infrastructure for extra device access protection. Then implement the protection using the new Protection Keys Supervisor (PKS) on architectures which support it.
Because PMEM pages are all associated with a struct dev_pagemap and flags in struct page are valuable the flag of protecting memory can be stored in struct dev_pagemap. All PMEM is protected by the same pkey. So a single flag is all that is needed in each dev_pagemap to indicate protection.
General access in the kernel is supported by modifying the kmap infrastructure which can detect if a page is pks protected and enable access until the corresponding unmap is called.
Because PKS is a thread local mechanism and because kmap was never really intended to create a long term mapping, this implementation does not support the kmap()/kunmap() calls. Calling kmap() on a PMEM protected page is allowed but accessing that mapping will cause a fault.
Originally this series modified many of the kmap call sites to indicate they were thread local.[1] And an attempt to support kmap()[2] was made. But now that kmap_local_page() has been developed[3] and in more wide spread use, kmap() can safely be left unsupported.
How the fault is handled is configurable via a new module parameter memremap.pks_fault_mode. Two modes are supported.
'relaxed' (default) -- WARN_ONCE, disable the protection and allow
access
'strict' -- prevent any unguarded access to a protected dev_pagemap
range
This 'safety valve' feature has already been useful in the development of this feature.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/
----------------------------------------------------------------------------
Changes for V8
Feedback from Thomas
* clean up noinstr mess
* Fix static PKEY allocation mess
* Ensure all functions are consistently named.
* Split up patches to do 1 thing per patch
* pkey_update_pkval() implementation
* Streamline the use of pks_write_pkrs() by not disabling preemption
- Leave this to the callers who require it.
- Use documentation and lockdep to prevent errors
* Clean up commit messages to explain in detail _why_ each patch is there.
Feedback from Dave H.
* Leave out pks_mk_readonly() as it is not used by the PMEM use case
Feedback from Peter Anvin
* Replace pks_abandon_pkey() with pks_update_exception() This is an even greater simplification in that it no longer attempts to shield users from faults. As the main use case for abandoning a key was to allow a system to continue running even with an error. This should be a rare event so the performance should not be an issue.
* Simplify ARCH_ENABLE_SUPERVISOR_PKEYS
* Update PKS Test code
- Add default value test
- Split up the test code into patches which follow each feature addition
- simplify test code processing
- ensure consistent reporting of errors.
* Ensure all entry points to the PKS code are protected by cpu_feature_enabled(X86_FEATURE_PKS)
- At the same time make sure non-entry points or sub-functions to the PKS code are not _unnecessarily_ protected by the feature check
* Update documentation
- Use kernel docs to place the docs with the code for easier internal developer use
* Adjust the PMEM use cases for the core changes
* Split the PMEM patches up to be 1 change per patch and help clarify review
* Review all header files and remove those no longer needed
* Review/update/clarify all commit messages
Fenghua Yu (1):
mm/pkeys: Define PKS page table macros
Ira Weiny (43):
entry: Create an internal irqentry_exit_cond_resched() call Documentation/protection-keys: Clean up documentation for User Space pkeys
x86/pkeys: Clarify PKRU_AD_KEY macro
x86/pkeys: Make PKRU macros generic
x86/fpu: Refactor arch_set_user_pkey_access()
mm/pkeys: Add Kconfig options for PKS
x86/pkeys: Add PKS CPU feature bit
x86/fault: Adjust WARN_ON for pkey fault Documentation/pkeys: Add initial PKS documentation mm/pkeys: Provide for PKS key allocation
x86/pkeys: Enable PKS on cpus which support it
mm/pkeys: PKS testing, add initial test code x86/selftests: Add test_pks
x86/pkeys: Introduce pks_write_pkrs()
x86/pkeys: Preserve the PKS MSR on context switch mm/pkeys: Introduce pks_set_readwrite()
mm/pkeys: Introduce pks_set_noaccess()
mm/pkeys: PKS testing, add a fault call back
mm/pkeys: PKS testing, add pks_set_*() tests
mm/pkeys: PKS testing, test context switching
x86/entry: Add auxiliary pt_regs space
entry: Split up irqentry_exit_cond_resched()
entry: Add calls for save/restore auxiliary pt_regs x86/entry: Define arch_{save|restore}_auxiliary_pt_regs() x86/pkeys: Preserve PKRS MSR across exceptions
x86/fault: Print PKS MSR on fault
mm/pkeys: PKS testing, Add exception test
mm/pkeys: Introduce pks_update_exception()
mm/pkeys: PKS testing, test pks_update_exception() mm/pkeys: PKS testing, add test for all keys
mm/pkeys: Add pks_available()memremap_pages: Add Kconfig for DEVMAP_ACCESS_PROTECTIONmemremap_pages: Introduce pgmap_protection_available()memremap_pages: Introduce a PGMAP_PROTECTION flagmemremap_pages: Introduce devmap_protected()memremap_pages: Reserve a PKS pkey for eventual use by PMEMmemremap_pages: Set PKS pkey in PTEs if requestedmemremap_pages: Define pgmap_set_{readwrite|noaccess}() callsmemremap_pages: Add memremap.pks_fault_mode
kmap: Make kmap work for devmap protected pages
dax: Stray access protection for dax_direct_access() nvdimm/pmem: Enable stray access protection
devdax: Enable stray access protection
Rick Edgecombe (1):
mm/pkeys: Introduce PKS fault callbacks
.../admin-guide/kernel-parameters.txt | 12 + Documentation/core-api/protection-keys.rst | 130 ++- arch/x86/Kconfig | 6 +arch/x86/entry/calling.h | 20 + arch/x86/entry/common.c | 2 +- arch/x86/entry/entry_64.S | 22 + arch/x86/entry/entry_64_compat.S | 6 + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/entry-common.h | 15 + arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/pgtable_types.h | 22 + arch/x86/include/asm/pkeys.h | 2 + arch/x86/include/asm/pkeys_common.h | 18 + arch/x86/include/asm/pkru.h | 20 +- arch/x86/include/asm/pks.h | 46 ++ arch/x86/include/asm/processor.h | 15 +- arch/x86/include/asm/ptrace.h | 21 + arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/asm-offsets_64.c | 15 + arch/x86/kernel/cpu/common.c | 2 + arch/x86/kernel/dumpstack.c | 32 +- arch/x86/kernel/fpu/xstate.c | 22 +- arch/x86/kernel/head_64.S | 6 + arch/x86/kernel/process_64.c | 3 + arch/x86/mm/fault.c | 17 +-arch/x86/mm/pkeys.c | 320 +++++++- drivers/dax/device.c | 2 +drivers/dax/super.c | 59 ++ drivers/md/dm-writecache.c | 8 +- drivers/nvdimm/pmem.c | 26 +fs/dax.c | 8 +fs/fuse/virtio_fs.c | 2 +include/linux/dax.h | 5 + include/linux/entry-common.h | 15 +- include/linux/highmem-internal.h | 4 + include/linux/memremap.h | 1 + include/linux/mm.h | 72 ++ include/linux/pgtable.h | 4 + include/linux/pks-keys.h | 92 +++ include/linux/pks.h | 73 ++ include/linux/sched.h | 7 + include/uapi/asm-generic/mman-common.h | 1 + init/init_task.c | 3 +kernel/entry/common.c | 44 +- kernel/sched/core.c | 40 +-lib/Kconfig.debug | 33 +
lib/Makefile | 3 +lib/pks/Makefile | 3 +lib/pks/pks_test.c | 755 ++++++++++++++++++ mm/Kconfig | 32 +mm/memremap.c | 132 +++ tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/test_pks.c | 514 ++++++++++++
54 files changed, 2617 insertions(+), 109 deletions(-)
create mode 100644 arch/x86/include/asm/pkeys_common.h create mode 100644 arch/x86/include/asm/pks.h create mode 100644 include/linux/pks-keys.h create mode 100644 include/linux/pks.h
create mode 100644 lib/pks/Makefile
create mode 100644 lib/pks/pks_test.c
create mode 100644 tools/testing/selftests/x86/test_pks.c
--
2.35.1
PATCH v9 1/45 entry: Create an internal irqentry_exit_cond_resched() call
From: Ira Weiny <[email protected]>
The static call to irqentry_exit_cond_resched() was not properly being overridden when called from xen_pv_evtchn_do_upcall().
Define __irqentry_exit_cond_resched() as the static call and place the override logic in irqentry_exit_cond_resched().
---
Changes for V9
Update the commit message a bit
Because this was found via code inspection and it does not actually fix any seen bug I've not added a fixes tag.
But for reference:
--- include/linux/entry-common.h | 5 ++++- kernel/entry/common.c | 23 +++++++++++++-------- kernel/sched/core.c | 40 ++++++++++++++++++------------------ 3 files changed, 38 insertions(+), 30 deletions(-)
include/linux/entry-common.h +4 -1
@@ -455,10 +455,13 @@ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs);* Conditional reschedule with additional sanity checks. */ void irqentry_exit_cond_resched (void ); + +void __irqentry_exit_cond_resched (void );#ifdef CONFIG_PREEMPT_DYNAMIC -DECLARE_STATIC_CALL (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +DECLARE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif +/** * irqentry_exit - Handle return from exception that used irqentry_enter() * @regs: Pointer to pt_regs (exception entry regs)
kernel/entry/common.c +14 -9
@@ -380,7 +380,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)return ret; } -void irqentry_exit_cond_resched (void ) +void __irqentry_exit_cond_resched (void ) {if (!preempt_count ()) {/* Sanity check RCU and thread stack */
@@ -392,9 +392,20 @@ void irqentry_exit_cond_resched(void) } }#ifdef CONFIG_PREEMPT_DYNAMIC -DEFINE_STATIC_CALL (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +DEFINE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif +void irqentry_exit_cond_resched (void ) +{ +if (IS_ENABLED (CONFIG_PREEMPTION )) { +#ifdef CONFIG_PREEMPT_DYNAMIC +static_call (__irqentry_exit_cond_resched)(); +#else +__irqentry_exit_cond_resched (); +#endif + } +} + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) {lockdep_assert_irqs_disabled ();
@@ -420,13 +431,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) }instrumentation_begin (); -if (IS_ENABLED (CONFIG_PREEMPTION )) { -#ifdef CONFIG_PREEMPT_DYNAMIC -static_call (irqentry_exit_cond_resched)(); -#else -irqentry_exit_cond_resched (); -#endif - } +irqentry_exit_cond_resched ();/* Covers both tracing and lockdep */ trace_hardirqs_on ();instrumentation_end ();
kernel/sched/core.c +20 -20
@@ -6571,29 +6571,29 @@ EXPORT_STATIC_CALL_TRAMP(preempt_schedule_notrace);* SC:might_resched * SC:preempt_schedule * SC:preempt_schedule_notrace -* SC:irqentry_exit_cond_resched +* SC:__irqentry_exit_cond_resched * * * NONE: -* cond_resched <- __cond_resched -* might_resched <- RET0 -* preempt_schedule <- NOP -* preempt_schedule_notrace <- NOP -* irqentry_exit_cond_resched <- NOP +* cond_resched <- __cond_resched +* might_resched <- RET0 +* preempt_schedule <- NOP +* preempt_schedule_notrace <- NOP +* __irqentry_exit_cond_resched <- NOP * * VOLUNTARY: -* cond_resched <- __cond_resched -* might_resched <- __cond_resched -* preempt_schedule <- NOP -* preempt_schedule_notrace <- NOP -* irqentry_exit_cond_resched <- NOP +* cond_resched <- __cond_resched +* might_resched <- __cond_resched +* preempt_schedule <- NOP +* preempt_schedule_notrace <- NOP +* __irqentry_exit_cond_resched <- NOP * * FULL: -* cond_resched <- RET0 -* might_resched <- RET0 -* preempt_schedule <- preempt_schedule -* preempt_schedule_notrace <- preempt_schedule_notrace -* irqentry_exit_cond_resched <- irqentry_exit_cond_resched +* cond_resched <- RET0 +* might_resched <- RET0 +* preempt_schedule <- preempt_schedule +* preempt_schedule_notrace <- preempt_schedule_notrace +* __irqentry_exit_cond_resched <- __irqentry_exit_cond_resched */ enum {
@@ -6629,7 +6629,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, __cond_resched);static_call_update (preempt_schedule, __preempt_schedule_func);static_call_update (preempt_schedule_notrace, __preempt_schedule_notrace_func); -static_call_update (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +static_call_update (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);switch (mode) {case preempt_dynamic_none:
@@ -6637,7 +6637,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, (void *)&__static_call_return0);static_call_update (preempt_schedule,NULL );static_call_update (preempt_schedule_notrace,NULL ); -static_call_update (irqentry_exit_cond_resched,NULL ); +static_call_update (__irqentry_exit_cond_resched,NULL );pr_info ("Dynamic Preempt: none\n" );break ;
@@ -6646,7 +6646,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, __cond_resched);static_call_update (preempt_schedule,NULL );static_call_update (preempt_schedule_notrace,NULL ); -static_call_update (irqentry_exit_cond_resched,NULL ); +static_call_update (__irqentry_exit_cond_resched,NULL );pr_info ("Dynamic Preempt: voluntary\n" );break ;
@@ -6655,7 +6655,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, (void *)&__static_call_return0);static_call_update (preempt_schedule, __preempt_schedule_func);static_call_update (preempt_schedule_notrace, __preempt_schedule_notrace_func); -static_call_update (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +static_call_update (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);pr_info ("Dynamic Preempt: full\n" );break ; }
Signature · 1 line
2.35.1
Re: [PATCH V9 01/45] entry: Create an internal irqentry_exit_cond_resched() call
Quoted 1 line · Ira
From: Ira Weiny <[email protected]>
Rebasing to 5.18-rc1 revealed a different fix has been applied for this work.[1]
Please disregard this patch.
Ira
[1] 4624a14f4daa ("sched/preempt: Simplify irqentry_exit_cond_resched() callers")
Quoted 183 lines
The static call to
irqentry_exit_cond_resched()was not properly being overridden when called fromxen_pv_evtchn_do_upcall().Define
__irqentry_exit_cond_resched()as the static call and place the override logic inirqentry_exit_cond_resched().Cc Peter Zijlstra (Intel) <[email protected]>Signed-off-by Ira Weiny <[email protected]>---
Changes for V9
Update the commit message a bitBecause this was found via code inspection and it does not actually fix any seen bug I've not added a fixes tag.
But for reference:
Fixes 40607ee97e4e ("preempt/dynamic: Provide irqentry_exit_cond_resched() static call")--- include/linux/entry-common.h | 5 ++++- kernel/entry/common.c | 23 +++++++++++++-------- kernel/sched/core.c | 40 ++++++++++++++++++------------------ 3 files changed, 38 insertions(+), 30 deletions(-)include/linux/entry-common.h +4
-1@@ -455,10 +455,13 @@ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs);* Conditional reschedule with additional sanity checks. */ void irqentry_exit_cond_resched (void ); + +void __irqentry_exit_cond_resched (void );#ifdef CONFIG_PREEMPT_DYNAMIC -DECLARE_STATIC_CALL (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +DECLARE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif +/** * irqentry_exit - Handle return from exception that used irqentry_enter() * @regs: Pointer to pt_regs (exception entry regs) kernel/entry/common.c +14
-9@@ -380,7 +380,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)return ret; } -void irqentry_exit_cond_resched (void ) +void __irqentry_exit_cond_resched (void ) {if (!preempt_count ()) {/* Sanity check RCU and thread stack */ @@ -392,9 +392,20 @@ void irqentry_exit_cond_resched(void) } }#ifdef CONFIG_PREEMPT_DYNAMIC -DEFINE_STATIC_CALL (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +DEFINE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif +void irqentry_exit_cond_resched (void ) +{ +if (IS_ENABLED (CONFIG_PREEMPTION )) { +#ifdef CONFIG_PREEMPT_DYNAMIC +static_call (__irqentry_exit_cond_resched)(); +#else +__irqentry_exit_cond_resched (); +#endif + } +} + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) {lockdep_assert_irqs_disabled ();@@ -420,13 +431,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) }instrumentation_begin (); -if (IS_ENABLED (CONFIG_PREEMPTION )) { -#ifdef CONFIG_PREEMPT_DYNAMIC -static_call (irqentry_exit_cond_resched)(); -#else -irqentry_exit_cond_resched (); -#endif - } +irqentry_exit_cond_resched ();/* Covers both tracing and lockdep */ trace_hardirqs_on ();instrumentation_end ();kernel/sched/core.c +20
-20@@ -6571,29 +6571,29 @@ EXPORT_STATIC_CALL_TRAMP(preempt_schedule_notrace);* SC:might_resched * SC:preempt_schedule * SC:preempt_schedule_notrace -* SC:irqentry_exit_cond_resched +* SC:__irqentry_exit_cond_resched * * * NONE: -* cond_resched <- __cond_resched -* might_resched <- RET0 -* preempt_schedule <- NOP -* preempt_schedule_notrace <- NOP -* irqentry_exit_cond_resched <- NOP +* cond_resched <- __cond_resched +* might_resched <- RET0 +* preempt_schedule <- NOP +* preempt_schedule_notrace <- NOP +* __irqentry_exit_cond_resched <- NOP * * VOLUNTARY: -* cond_resched <- __cond_resched -* might_resched <- __cond_resched -* preempt_schedule <- NOP -* preempt_schedule_notrace <- NOP -* irqentry_exit_cond_resched <- NOP +* cond_resched <- __cond_resched +* might_resched <- __cond_resched +* preempt_schedule <- NOP +* preempt_schedule_notrace <- NOP +* __irqentry_exit_cond_resched <- NOP * * FULL: -* cond_resched <- RET0 -* might_resched <- RET0 -* preempt_schedule <- preempt_schedule -* preempt_schedule_notrace <- preempt_schedule_notrace -* irqentry_exit_cond_resched <- irqentry_exit_cond_resched +* cond_resched <- RET0 +* might_resched <- RET0 +* preempt_schedule <- preempt_schedule +* preempt_schedule_notrace <- preempt_schedule_notrace +* __irqentry_exit_cond_resched <- __irqentry_exit_cond_resched */ enum {@@ -6629,7 +6629,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, __cond_resched);static_call_update (preempt_schedule, __preempt_schedule_func);static_call_update (preempt_schedule_notrace, __preempt_schedule_notrace_func); -static_call_update (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +static_call_update (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);switch (mode) {case preempt_dynamic_none:@@ -6637,7 +6637,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, (void *)&__static_call_return0);static_call_update (preempt_schedule,NULL );static_call_update (preempt_schedule_notrace,NULL ); -static_call_update (irqentry_exit_cond_resched,NULL ); +static_call_update (__irqentry_exit_cond_resched,NULL );pr_info ("Dynamic Preempt: none\n" );break ;@@ -6646,7 +6646,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, __cond_resched);static_call_update (preempt_schedule,NULL );static_call_update (preempt_schedule_notrace,NULL ); -static_call_update (irqentry_exit_cond_resched,NULL ); +static_call_update (__irqentry_exit_cond_resched,NULL );pr_info ("Dynamic Preempt: voluntary\n" );break ;@@ -6655,7 +6655,7 @@ void sched_dynamic_update(int mode)static_call_update (might_resched, (void *)&__static_call_return0);static_call_update (preempt_schedule, __preempt_schedule_func);static_call_update (preempt_schedule_notrace, __preempt_schedule_notrace_func); -static_call_update (irqentry_exit_cond_resched, irqentry_exit_cond_resched); +static_call_update (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);pr_info ("Dynamic Preempt: full\n" );break ; }Signature · 1 line
2.35.1
PATCH v9 2/45 Documentation/protection-keys: Clean up documentation for User Space pkeys
In reply to [email protected]From: Ira Weiny <[email protected]>
The documentation for user space pkeys was a bit dated including things such as Amazon and distribution testing information which is irrelevant now.
Update the documentation. This also streamlines adding the Supervisor pkey documentation later on.
---
Changes for V9:
use pkey
Change information on which CPU's have PKU
--- Documentation/core-api/protection-keys.rst | 44 +++++++++++----------- 1 file changed, 21 insertions(+), 23 deletions(-)
Documentation/core-api/protection-keys.rst +21 -23
@@ -4,31 +4,29 @@ Memory Protection Keys ====================== -Memory Protection Keys for Userspace (PKU aka PKEYs) is a feature -which is found on Intel's Skylake (and later) "Scalable Processor" -Server CPUs. It will be available in future non-server Intel parts -and future AMD processors. - -For anyone wishing to test or use this feature, it is available in -Amazon's EC2 C5 instances and is known to work there using an Ubuntu -17.04 image. - -Memory Protection Keys provides a mechanism for enforcing page-based -protections, but without requiring modification of the page tables -when an application changes protection domains. It works by -dedicating 4 previously ignored bits in each page table entry to a -"protection key", giving 16 possible keys. - -There is also a new user-accessible register (PKRU) with two separate -bits (Access Disable and Write Disable) for each key. Being a CPU -register, PKRU is inherently thread-local, potentially giving each +Memory Protection Keys provide a mechanism for enforcing page-based +protections, but without requiring modification of the page tables when an +application changes protection domains. + +Pkeys Userspace (PKU) is a feature which can be found on: + * Intel server CPUs, Skylake and later + * Intel client CPUs, Tiger Lake (11th Gen Core) and later + * Future AMD CPUs + +Pkeys work by dedicating 4 previously Reserved bits in each page table entry to +a "protection key", giving 16 possible keys. + +Protections for each key are defined with a per-CPU user-accessible register +(PKRU). Each of these is a 32-bit register storing two bits (Access Disable +and Write Disable) for each of 16 keys. + +Being a CPU register, PKRU is inherently thread-local, potentially giving each thread a different set of protections from every other thread. -There are two new instructions (RDPKRU/WRPKRU) for reading and writing -to the new register. The feature is only available in 64-bit mode, -even though there is theoretically space in the PAE PTEs. These -permissions are enforced on data access only and have no effect on -instruction fetches. +There are two instructions (RDPKRU/WRPKRU) for reading and writing to the +register. The feature is only available in 64-bit mode, even though there is +theoretically space in the PAE PTEs. These permissions are enforced on data +access only and have no effect on instruction fetches. Syscalls ========
Signature · 1 line
2.35.1
PATCH v9 3/45 x86/pkeys: Clarify PKRU_AD_KEY macro
In reply to [email protected]From: Ira Weiny <[email protected]>
When changing the PKRU_AD_KEY macro to be used for PKS the name came into question.[1]
The intent of PKRU_AD_KEY is to set an initial value for the PKRU register but that is just a mask value.
Clarify this by changing the name to PKRU_AD_MASK().
NOTE the checkpatch errors are ignored for the init_pkru_value to align the values in the code.
[1] https://lore.kernel.org/lkml/[email protected]/
---
Changes for V9
New Patch
--- arch/x86/mm/pkeys.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/mm/pkeys.c +9 -6
@@ -110,7 +110,7 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkeyreturn vma_pkey (vma); } -#define PKRU_AD_KEY (pkey) (PKRU_AD_BIT << ((pkey) * PKRU_BITS_PER_PKEY)) +#define PKRU_AD_MASK (pkey) (PKRU_AD_BIT << ((pkey) * PKRU_BITS_PER_PKEY))/* * Make the default PKRU value (at execve() time) as restrictive
@@ -118,11 +118,14 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkey* in the process's lifetime will not accidentally get access * to data which is pkey-protected later on. */ -u32 init_pkru_value =PKRU_AD_KEY (1 ) |PKRU_AD_KEY (2 ) |PKRU_AD_KEY (3 ) | -PKRU_AD_KEY (4 ) |PKRU_AD_KEY (5 ) |PKRU_AD_KEY (6 ) | -PKRU_AD_KEY (7 ) |PKRU_AD_KEY (8 ) |PKRU_AD_KEY (9 ) | -PKRU_AD_KEY (10 ) |PKRU_AD_KEY (11 ) |PKRU_AD_KEY (12 ) | -PKRU_AD_KEY (13 ) |PKRU_AD_KEY (14 ) |PKRU_AD_KEY (15 ); +u32 init_pkru_value =PKRU_AD_MASK (1 ) |PKRU_AD_MASK (2 ) | +PKRU_AD_MASK (3 ) |PKRU_AD_MASK (4 ) | +PKRU_AD_MASK (5 ) |PKRU_AD_MASK (6 ) | +PKRU_AD_MASK (7 ) |PKRU_AD_MASK (8 ) | +PKRU_AD_MASK (9 ) |PKRU_AD_MASK (10 ) | +PKRU_AD_MASK (11 ) |PKRU_AD_MASK (12 ) | +PKRU_AD_MASK (13 ) |PKRU_AD_MASK (14 ) | +PKRU_AD_MASK (15 );static ssize_t init_pkru_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos)
Signature · 1 line
2.35.1
PATCH v9 4/45 x86/pkeys: Make PKRU macros generic
In reply to [email protected]From: Ira Weiny <[email protected]>
Protection Keys User (PKU) and Protection Keys Supervisor (PKS) work in similar fashions and can share common defines. Specifically PKS and PKU each have:
1. A single control register
2. The same number of keys
3. The same number of bits in the register per key
4. Access and Write disable in the same bit locations
Given the above, share all the macros that synthesize and manipulate register values between the two features. Share these defines by moving them into a new header, change their names to reflect the common use, and include the header where needed. This mostly takes the form of converting names from the PKU-specific "PKRU" to a user/supervisor agnostic "PKR".
Also while editing the code remove the use of 'we' from comments being touched.
NOTE the checkpatch errors are ignored for the init_pkru_value to align the values in the code.
---
Changes for V9:
From Dave Hansen
Add detail to commit message
Add Ack
s/PKR_AD_KEY/PKR_AD_MASK/
Changes from v7:
Rebased onto latest
--- arch/x86/include/asm/pkeys_common.h | 11 +++++++++++ arch/x86/include/asm/pkru.h | 20 ++++++++------------ arch/x86/kernel/fpu/xstate.c | 10 +++++----- arch/x86/mm/pkeys.c | 20 +++++++++----------- 4 files changed, 33 insertions(+), 28 deletions(-) create mode 100644 arch/x86/include/asm/pkeys_common.h
arch/x86/include/asm/pkeys_common.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_PKEYS_COMMON_H +#define _ASM_X86_PKEYS_COMMON_H + +#define PKR_AD_BIT 0x1u +#define PKR_WD_BIT 0x2u +#define PKR_BITS_PER_PKEY 2 + +#define PKR_AD_MASK (pkey) (PKR_AD_BIT << ((pkey) * PKR_BITS_PER_PKEY)) + +#endif /*_ASM_X86_PKEYS_COMMON_H */
arch/x86/include/asm/pkru.h +8 -12
@@ -3,10 +3,7 @@#define _ASM_X86_PKRU_H#include <asm/cpufeature.h> - -#define PKRU_AD_BIT 0x1u -#define PKRU_WD_BIT 0x2u -#define PKRU_BITS_PER_PKEY 2 +#include <asm/pkeys_common.h> #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS extern u32 init_pkru_value;
@@ -18,18 +15,17 @@ extern u32 init_pkru_value;static inline bool __pkru_allows_read (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKRU_BITS_PER_PKEY ; -return !(pkru & (PKRU_AD_BIT << pkru_pkey_bits)); +int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ; + +return !(pkru & (PKR_AD_BIT << pkru_pkey_bits)); }static inline bool __pkru_allows_write (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKRU_BITS_PER_PKEY ; -/* -* Access-disable disables writes too so we need to check -* both bits here. -*/ -return !(pkru & ((PKRU_AD_BIT |PKRU_WD_BIT ) << pkru_pkey_bits)); +int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ; + +/* Access-disable disables writes too so check both bits here. */ +return !(pkru & ((PKR_AD_BIT |PKR_WD_BIT ) << pkru_pkey_bits)); }static inline u32 read_pkru (void )
arch/x86/kernel/fpu/xstate.c +5 -5
@@ -1089,19 +1089,19 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,if (WARN_ON_ONCE (pkey >=arch_max_pkey ()))return -EINVAL ; -/* Set the bits we need in PKRU: */ +/* Set the bits needed in PKRU: */ if (init_val &PKEY_DISABLE_ACCESS ) - new_pkru_bits |=PKRU_AD_BIT ; + new_pkru_bits |=PKR_AD_BIT ;if (init_val &PKEY_DISABLE_WRITE ) - new_pkru_bits |=PKRU_WD_BIT ; + new_pkru_bits |=PKR_WD_BIT ;/* Shift the bits in to the correct place in PKRU for pkey: */ - pkey_shift = pkey *PKRU_BITS_PER_PKEY ; + pkey_shift = pkey *PKR_BITS_PER_PKEY ; new_pkru_bits <<= pkey_shift;/* Get old PKRU and mask off any old bits in place: */ old_pkru =read_pkru (); - old_pkru &= ~((PKRU_AD_BIT |PKRU_WD_BIT ) << pkey_shift); + old_pkru &= ~((PKR_AD_BIT |PKR_WD_BIT ) << pkey_shift);/* Write old part along with new part: */ write_pkru (old_pkru | new_pkru_bits);
arch/x86/mm/pkeys.c +9 -11
@@ -110,22 +110,20 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkeyreturn vma_pkey (vma); } -#define PKRU_AD_MASK (pkey) (PKRU_AD_BIT << ((pkey) * PKRU_BITS_PER_PKEY)) -/* * Make the default PKRU value (at execve() time) as restrictive * as possible. This ensures that any threads clone()'d early * in the process's lifetime will not accidentally get access * to data which is pkey-protected later on. */ -u32 init_pkru_value =PKRU_AD_MASK (1 ) |PKRU_AD_MASK (2 ) | -PKRU_AD_MASK (3 ) |PKRU_AD_MASK (4 ) | -PKRU_AD_MASK (5 ) |PKRU_AD_MASK (6 ) | -PKRU_AD_MASK (7 ) |PKRU_AD_MASK (8 ) | -PKRU_AD_MASK (9 ) |PKRU_AD_MASK (10 ) | -PKRU_AD_MASK (11 ) |PKRU_AD_MASK (12 ) | -PKRU_AD_MASK (13 ) |PKRU_AD_MASK (14 ) | -PKRU_AD_MASK (15 ); +u32 init_pkru_value =PKR_AD_MASK (1 ) |PKR_AD_MASK (2 ) | +PKR_AD_MASK (3 ) |PKR_AD_MASK (4 ) | +PKR_AD_MASK (5 ) |PKR_AD_MASK (6 ) | +PKR_AD_MASK (7 ) |PKR_AD_MASK (8 ) | +PKR_AD_MASK (9 ) |PKR_AD_MASK (10 ) | +PKR_AD_MASK (11 ) |PKR_AD_MASK (12 ) | +PKR_AD_MASK (13 ) |PKR_AD_MASK (14 ) | +PKR_AD_MASK (15 );static ssize_t init_pkru_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos)
@@ -158,7 +156,7 @@ static ssize_t init_pkru_write_file(struct file *file,* up immediately if someone attempts to disable access * or writes to pkey 0. */ -if (new_init_pkru & (PKRU_AD_BIT |PKRU_WD_BIT )) +if (new_init_pkru & (PKR_AD_BIT |PKR_WD_BIT ))return -EINVAL ;WRITE_ONCE (init_pkru_value, new_init_pkru);
Signature · 1 line
2.35.1
PATCH v9 5/45 x86/fpu: Refactor arch_set_user_pkey_access()
In reply to [email protected]From: Ira Weiny <[email protected]>
Both PKU and PKS update their register values in the same way. They can therefore share the update code.
Define a helper, pkey_update_pkval(), which will be used to support both Protection Key User (PKU) and the new Protection Key for Supervisor (PKS) in subsequent patches.
pkey_update_pkval() contributed by Thomas
---
Update for V8:
From Rick Edgecombe
Change pkey type to u8
Replace the code Peter provided in update_pkey_reg() for
Thomas' pkey_update_pkval()
-- https://lore.kernel.org/lkml/[email protected]/
--- arch/x86/include/asm/pkeys.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 22 ++++------------------ arch/x86/mm/pkeys.c | 16 ++++++++++++++++ 3 files changed, 22 insertions(+), 18 deletions(-)
arch/x86/include/asm/pkeys.h +2 -0
@@ -131,4 +131,6 @@ static inline int vma_pkey(struct vm_area_struct *vma) return (vma->vm_flags & vma_pkey_mask) >> VM_PKEY_SHIFT; } +u32 pkey_update_pkval (u32 pkval,u8 pkey,u32 accessbits); +#endif /*_ASM_X86_PKEYS_H */
arch/x86/kernel/fpu/xstate.c +4 -18
@@ -1071,8 +1071,7 @@ void *get_xsave_addr(struct xregs_state *xsave, int xfeature_nr)int arch_set_user_pkey_access (struct task_struct *tsk,int pkey,unsigned long init_val) { -u32 old_pkru, new_pkru_bits =0 ; -int pkey_shift; +u32 pkru;/* * This check implies XSAVE support. OSPKE only gets
@@ -1089,22 +1088,9 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,if (WARN_ON_ONCE (pkey >=arch_max_pkey ()))return -EINVAL ; -/* Set the bits needed in PKRU: */ -if (init_val &PKEY_DISABLE_ACCESS ) - new_pkru_bits |=PKR_AD_BIT ; -if (init_val &PKEY_DISABLE_WRITE ) - new_pkru_bits |=PKR_WD_BIT ; - -/* Shift the bits in to the correct place in PKRU for pkey: */ - pkey_shift = pkey *PKR_BITS_PER_PKEY ; - new_pkru_bits <<= pkey_shift; - -/* Get old PKRU and mask off any old bits in place: */ - old_pkru =read_pkru (); - old_pkru &= ~((PKR_AD_BIT |PKR_WD_BIT ) << pkey_shift); - -/* Write old part along with new part: */ -write_pkru (old_pkru | new_pkru_bits); + pkru =read_pkru (); + pkru =pkey_update_pkval (pkru, pkey, init_val); +write_pkru (pkru);return 0 ; }
arch/x86/mm/pkeys.c +16 -0
@@ -193,3 +193,19 @@ static __init int setup_init_pkru(char *opt)return 1 ; }__setup ("init_pkru=" , setup_init_pkru); + +/* +* Kernel users use the same flags as user space: +* PKEY_DISABLE_ACCESS +* PKEY_DISABLE_WRITE +*/ +u32 pkey_update_pkval (u32 pkval,u8 pkey,u32 accessbits) +{ +int shift = pkey *PKR_BITS_PER_PKEY ; + +if (WARN_ON_ONCE (accessbits & ~PKEY_ACCESS_MASK )) + accessbits &=PKEY_ACCESS_MASK ; + + pkval &= ~(PKEY_ACCESS_MASK << shift); +return pkval | accessbits << shift; +}
Signature · 1 line
2.35.1
PATCH v9 6/45 mm/pkeys: Add Kconfig options for PKS
In reply to [email protected]From: Ira Weiny <[email protected]>
Consumers wishing to implement additional protections on memory pages can use PKS. However, PKS is only available on some architectures.
For this reason PKS code, both in the core and in the consumers, is dead code without PKS being both available and used.
Add Kconfig options to allow for the elimination of unneeded code by detecting architecture PKS support (ARCH_HAS_SUPERVISOR_PKEYS) and requiring an indication of consumer need (ARCH_ENABLE_SUPERVISOR_PKEYS).
In this patch ARCH_ENABLE_SUPERVISOR_PKEYS remains off until the first kernel consumer sets it.
---
Changes for V9
Dave Hansen
Don't exclude AMD, cpu supported bits will properly turn the feature off.
Clarify commit message
Depend on CPU_SUP_INTEL
Changes for V8
Split this out to a single change patch
--- arch/x86/Kconfig | 1 + mm/Kconfig | 4 ++++ 2 files changed, 5 insertions(+)
arch/x86/Kconfig +1 -0
@@ -1868,6 +1868,7 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS depends on X86_64 && (CPU_SUP_INTEL || CPU_SUP_AMD) select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS + select ARCH_HAS_SUPERVISOR_PKEYS help Memory Protection Keys provides a mechanism for enforcing page-based protections, but without requiring modification of the
mm/Kconfig +4 -0
@@ -804,6 +804,10 @@ config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS bool +config ARCH_HAS_SUPERVISOR_PKEYS + bool +config ARCH_ENABLE_SUPERVISOR_PKEYS + bool config PERCPU_STATS bool "Collect percpu memory statistics"
Signature · 1 line
2.35.1
PATCH v9 7/45 x86/pkeys: Add PKS CPU feature bit
In reply to [email protected]From: Ira Weiny <[email protected]>
Memory Protection Keys (pkeys) provides a mechanism for enforcing page-based protections, but without requiring modification of the page tables when an application changes protection domains.
The supervisor support for memory protection keys is referred to as PKS (Protection Keys Supervisor).
Add the defines for the CPU support bit and the boilerplate disable infrastructure predicated on the new ARCH_ENABLE_SUPERVISOR_PKEYS Kconfig option.
---
Changes for V9
Dave Hansen
New commit message
Changes for V8
Split this out into it's own patch
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -370,6 +370,7 @@#define X86_FEATURE_MOVDIR64B (16*32+28)/* MOVDIR64B instruction */ #define X86_FEATURE_ENQCMD (16*32+29)/* ENQCMD and ENQCMDS instructions */ #define X86_FEATURE_SGX_LC (16*32+30)/* Software Guard Extensions Launch Control */ +#define X86_FEATURE_PKS (16*32+31)/* Protection Keys for Supervisor pages */ /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */ #define X86_FEATURE_OVERFLOW_RECOV (17*32+ 0)/* MCA overflow recovery support */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -44,6 +44,12 @@# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +# define DISABLE_PKS 0 +#else +# define DISABLE_PKS (1<<(X86_FEATURE_PKS & 31)) +#endif +#ifdef CONFIG_X86_5LEVEL # define DISABLE_LA57 0#else
@@ -85,7 +91,7 @@#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_PKS)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK19 0
Signature · 1 line
2.35.1
PATCH v9 8/45 x86/fault: Adjust WARN_ON for pkey fault
In reply to [email protected]From: Ira Weiny <[email protected]>
Previously if a protection key fault occurred on a kernel address it indicated something wrong because user page mappings are not supposed to be in the kernel address space.
With the addition of PKS, pkey faults may now happen on kernel mappings.
If PKS is enabled, avoid the warning in the fault path. Simplify the comment.
---
Changes for V9
From Dave Hansen
Clarify the comment and commit message
--- arch/x86/mm/fault.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/mm/fault.c +4 -4
@@ -1148,11 +1148,11 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) {/* -* Protection keys exceptions only happen on user pages. We -* have no user pages in the kernel portion of the address -* space, so do not expect them here. +* PF_PF faults should only occur on kernel +* addresses when supervisor pkeys are enabled. */ -WARN_ON_ONCE (hw_error_code &X86_PF_PK ); +WARN_ON_ONCE (!cpu_feature_enabled (X86_FEATURE_PKS ) && + (hw_error_code &X86_PF_PK ));#ifdef CONFIG_X86_32 /*
Signature · 1 line
2.35.1
PATCH v9 9/45 Documentation/pkeys: Add initial PKS documentation
In reply to [email protected]From: Ira Weiny <[email protected]>
Add initial overview and configuration information about PKS.
---
Changes for V9
Feedback from Dave Hansen
Remove overview and move relevant text to the main pkey overview which covers both user ans kernel keys. Add an example of using Kconfig
Move MSR details to later patches
--- Documentation/core-api/protection-keys.rst | 43 ++++++++++++++++++++-- 1 file changed, 39 insertions(+), 4 deletions(-)
Documentation/core-api/protection-keys.rst +39 -4
@@ -13,6 +13,11 @@ Pkeys Userspace (PKU) is a feature which can be found on: * Intel client CPUs, Tiger Lake (11th Gen Core) and later * Future AMD CPUs +Protection Keys Supervisor (PKS) is a feature which can be found on: + * Sapphire Rapids (and later) "Scalable Processor" Server CPUs + * Future non-server Intel parts. + * qemu: https://www.qemu.org/2021/04/30/qemu-6-0-0/ + Pkeys work by dedicating 4 previously Reserved bits in each page table entry to a "protection key", giving 16 possible keys.
@@ -23,13 +28,20 @@ and Write Disable) for each of 16 keys. Being a CPU register, PKRU is inherently thread-local, potentially giving each thread a different set of protections from every other thread. -There are two instructions (RDPKRU/WRPKRU) for reading and writing to the -register. The feature is only available in 64-bit mode, even though there is +For Userspace (PKU), there are two instructions (RDPKRU/WRPKRU) for reading and +writing to the register. + +For Supervisor (PKS), the register (MSR_IA32_PKRS) is accessible only to the +kernel through rdmsr and wrmsr. + +The feature is only available in 64-bit mode, even though there is theoretically space in the PAE PTEs. These permissions are enforced on data access only and have no effect on instruction fetches. -Syscalls -======== + + +Syscalls for user space keys +============================ There are 3 system calls which directly interact with pkeys::
@@ -96,3 +108,26 @@ with a read():: The kernel will send a SIGSEGV in both cases, but si_code will be set to SEGV_PKERR when violating protection keys versus SEGV_ACCERR when the plain mprotect() permissions are violated. + + +Kernel API for PKS support +========================== + +Kconfig +------- + +Kernel users intending to use PKS support should depend on +ARCH_HAS_SUPERVISOR_PKEYS, and select ARCH_ENABLE_SUPERVISOR_PKEYS to turn on +this support within the core. For example: + +.. code-block:: c + + config MY_NEW_FEATURE + depends on ARCH_HAS_SUPERVISOR_PKEYS + select ARCH_ENABLE_SUPERVISOR_PKEYS + +This will make "MY_NEW_FEATURE" unavailable unless the architecture sets +ARCH_HAS_SUPERVISOR_PKEYS. It also makes it possible for multiple independent +features to "select ARCH_ENABLE_SUPERVISOR_PKEYS". If no features enable PKS +by selecting ARCH_ENABLE_SUPERVISOR_PKEYS, PKS support will not be compiled +into the kernel.
Signature · 1 line
2.35.1
PATCH v9 10/45 mm/pkeys: Provide for PKS key allocation
In reply to [email protected]From: Ira Weiny <[email protected]>
Kernel consumers of PKS need a way to allocate a PKS pkey and assign the initial permissions for that key. It is desirable to not allocate keys for consumers which are not configured.
Introduce a macro to allocate keys sequentially based on which consumers are configured. In addition define a macro to set the proper permission bits based on the actual pkey value allocated.
pks-keys.h is added as a new header with minimal header dependencies. This allows the use of PKS_INIT_VALUE within other headers where the additional includes from other pkey headers caused major conflicts. The main conflict was using PKS_INIT_VALUE for INIT_TRHEAD in asm/processor.h
Add documentation.
---
Changes for V9
Reword the commit message
Move this patch ahead of the enable patch so that the enable
patch can use PKS_INIT_VALUE
From Dan Williams
Use Dan's macro magic
enhanced it to account for the max number of keys
Update documentation for the change
From Dave Hansen
use pkey
s/PKR_RW_KEY/PKR_RW_MASK
Changes for V8
Create pks-keys.h to solve header conflicts in subsequent
patches.
Remove create_initial_pkrs_value() which did not work
Replace it with PKS_INIT_VALUE
Fix up documentation to match
s/PKR_RW_BIT/PKR_RW_KEY()/ s/PKRS_INIT_VALUE/PKS_INIT_VALUE
Split this off of the previous patch
Update documentation and embed it in the code to help ensure it is kept up to date.
Changes for V7
Create a dynamic pkrs_initial_value in early init code. Clean up comments
Add comment to macro guard
--- Documentation/core-api/protection-keys.rst | 5 ++ arch/x86/include/asm/pkeys_common.h | 9 ++- include/linux/pks-keys.h | 78 ++++++++++++++++++++++ 3 files changed, 91 insertions(+), 1 deletion(-) create mode 100644 include/linux/pks-keys.h
Documentation/core-api/protection-keys.rst +5 -0
@@ -131,3 +131,8 @@ ARCH_HAS_SUPERVISOR_PKEYS. It also makes it possible for multiple independent features to "select ARCH_ENABLE_SUPERVISOR_PKEYS". If no features enable PKS by selecting ARCH_ENABLE_SUPERVISOR_PKEYS, PKS support will not be compiled into the kernel. + +PKS Key Allocation +------------------ +.. kernel-doc:: include/linux/pks-keys.h + :doc: PKS_KEY_ALLOCATION
arch/x86/include/asm/pkeys_common.h +8 -1
@@ -2,10 +2,17 @@#ifndef _ASM_X86_PKEYS_COMMON_H#define _ASM_X86_PKEYS_COMMON_H +#define PKS_NUM_PKEYS 16 +#define PKS_ALL_AD (0x55555555UL) +#define PKR_AD_BIT 0x1u#define PKR_WD_BIT 0x2u#define PKR_BITS_PER_PKEY 2 -#define PKR_AD_MASK (pkey) (PKR_AD_BIT << ((pkey) * PKR_BITS_PER_PKEY)) +#define PKR_PKEY_SHIFT (pkey) (pkey * PKR_BITS_PER_PKEY) + +#define PKR_RW_MASK (pkey) (0 << PKR_PKEY_SHIFT(pkey)) +#define PKR_AD_MASK (pkey) (PKR_AD_BIT << PKR_PKEY_SHIFT(pkey)) +#define PKR_WD_MASK (pkey) (PKR_WD_BIT << PKR_PKEY_SHIFT(pkey))#endif /*_ASM_X86_PKEYS_COMMON_H */
include/linux/pks-keys.h new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v9 11/45 x86/pkeys: Enable PKS on cpus which support it
In reply to [email protected]From: Ira Weiny <[email protected]>
Protection Keys for Supervisor pages (PKS) enables fast, hardware thread specific, manipulation of permission restrictions on supervisor page mappings. It uses a supervisor specific MSR to assign permissions to the pkeys.
When PKS is configured and the cpu supports PKS, initialize the MSR, and enable the hardware.
Add asm/pks.h to store new internal functions and structures such as pks_setup().
---
Changes for V9
Reword commit message
Move this after the patch defining PKS_INIT_VALUE
Changes for V8
Move setup_pks() into this patch with a default of all access
for all pkeys.
From Thomas
s/setup_pks/pks_setup/
Update Change log to better reflect exactly what this patch does.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/pks.h | 15 +++++++++++++++ arch/x86/include/uapi/asm/processor-flags.h | 2 ++ arch/x86/kernel/cpu/common.c | 2 ++ arch/x86/mm/pkeys.c | 17 +++++++++++++++++ 5 files changed, 37 insertions(+) create mode 100644 arch/x86/include/asm/pks.h
arch/x86/include/asm/msr-index.h +1 -0
@@ -787,6 +787,7 @@#define MSR_IA32_TSC_DEADLINE 0x000006E0 +#define MSR_IA32_PKRS 0x000006E1#define MSR_TSX_FORCE_ABORT 0x0000010F
arch/x86/include/asm/pks.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_PKS_H +#define _ASM_X86_PKS_H + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +void pks_setup (void ); + +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +static inline void pks_setup (void ) { } + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +#endif /* _ASM_X86_PKS_H */
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -130,6 +130,8 @@#define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT)#define X86_CR4_PKE_BIT 22/* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_PKS_BIT 24/* enable Protection Keys for Supervisor */ +#define X86_CR4_PKS _BITUL(X86_CR4_PKS_BIT)/* * x86-64 Task Priority Register, CR8
arch/x86/kernel/cpu/common.c +2 -0
@@ -59,6 +59,7 @@#include <asm/cpu_device_id.h> #include <asm/uv/uv.h> #include <asm/sigframe.h> +#include <asm/pks.h> #include "cpu.h"
@@ -1632,6 +1633,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)x86_init_rdrand (c);setup_pku (c); +pks_setup ();/* * Clear/Set all flags overridden by options, need do it
arch/x86/mm/pkeys.c +17 -0
@@ -6,6 +6,7 @@#include <linux/debugfs.h> /* debugfs_create_u32() */ #include <linux/mm_types.h> /* mm_struct, vma, etc... */ #include <linux/pkeys.h> /* PKEY_* */ +#include <linux/pks-keys.h> #include <uapi/asm-generic/mman-common.h> #include <asm/cpufeature.h> /* boot_cpu_has, ... */
@@ -209,3 +210,19 @@ u32 pkey_update_pkval(u32 pkval, u8 pkey, u32 accessbits) pkval &= ~(PKEY_ACCESS_MASK << shift);return pkval | accessbits << shift; } + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +/* +* PKS is independent of PKU and either or both may be supported on a CPU. +*/ +void pks_setup (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +wrmsrl (MSR_IA32_PKRS ,PKS_INIT_VALUE ); +cr4_set_bits (X86_CR4_PKS ); +} + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.35.1
PATCH v9 12/45 mm/pkeys: Define PKS page table macros
In reply to [email protected]From: Fenghua Yu <[email protected]>
Kernel PKS consumers will need a way to assign their pkey to pages.
Define _PAGE_PKEY() and PAGE_KERNEL_PKEY() to allow users to set a pkey on a PTE.
Add documentation.
---
Changes for V9
From Dave Hansen
s/PKey/pkey
Changes for V8
Split out from the 'Add PKS kernel API' patch Include documentation in this patch
--- Documentation/core-api/protection-keys.rst | 6 ++++++ arch/x86/include/asm/pgtable_types.h | 22 ++++++++++++++++++++++ include/linux/pgtable.h | 4 ++++ 3 files changed, 32 insertions(+)
Documentation/core-api/protection-keys.rst +6 -0
@@ -136,3 +136,9 @@ PKS Key Allocation ------------------ .. kernel-doc:: include/linux/pks-keys.h :doc: PKS_KEY_ALLOCATION + +Adding pages to a pkey protected domain +--------------------------------------- + +.. kernel-doc:: arch/x86/include/asm/pgtable_types.h + :doc: PKS_KEY_ASSIGNMENT
arch/x86/include/asm/pgtable_types.h +22 -0
@@ -71,6 +71,22 @@ _PAGE_PKEY_BIT2 | \ _PAGE_PKEY_BIT3) +/** +* DOC: PKS_KEY_ASSIGNMENT +* +* The following macros are used to set a pkey value in a supervisor PTE. +* +* .. code-block:: c +* +* #define _PAGE_KEY(pkey) +* #define PAGE_KERNEL_PKEY(pkey) +*/ +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +#define _PAGE_PKEY (pkey) (_AT(pteval_t, pkey) << _PAGE_BIT_PKEY_BIT0) +#else +#define _PAGE_PKEY (pkey) (_AT(pteval_t, 0)) +#endif +#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED)#else
@@ -226,6 +242,12 @@ enum page_cache_mode {#define PAGE_KERNEL_IO __pgprot_mask(__PAGE_KERNEL_IO)#define PAGE_KERNEL_IO_NOCACHE __pgprot_mask(__PAGE_KERNEL_IO_NOCACHE) +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +#define PAGE_KERNEL_PKEY (pkey) __pgprot_mask(__PAGE_KERNEL | _PAGE_PKEY(pkey)) +#else +#define PAGE_KERNEL_PKEY (pkey) PAGE_KERNEL +#endif +#endif /* __ASSEMBLY__ */ /* xwr */
include/linux/pgtable.h +4 -0
@@ -1511,6 +1511,10 @@ static inline bool arch_has_pfn_modify_check(void)# define PAGE_KERNEL_EXEC PAGE_KERNEL#endif +#ifndef PAGE_KERNEL_PKEY +#define PAGE_KERNEL_PKEY (pkey) PAGE_KERNEL +#endif +/* * Page Table Modification bits for pgtbl_mod_mask. *
Signature · 1 line
2.35.1
PATCH v9 13/45 mm/pkeys: PKS testing, add initial test code
In reply to [email protected]From: Ira Weiny <[email protected]>
Define a PKS consumer for testing.
Two initial tests are created. One to check that the default values have been properly assigned and a second which purposely causes a fault.
Add documentation.
---
Changes for V9
Simplify the commit message
Simplify documentation in favor of using test_pks Complete re-arch of test code...
Return -ENOENT for unknown tests
Adjust the key allocation
Reduce the globals used during fault detection Introduce a session structure to track information as long as the
debugfs file remains open.
Use pr_debug() for internal debug output.
Document how to run tests from debugfs with trace_printk()
output.
Feedback from Rick Edgecombe
Change pkey type to u8
remove pks_test_exit
set file data within the crash test to be cleaned up on
file close
Resolve when memory barriers are needed
From Dave Hansen
Place a lock around the execution of tests so that only
a single thread execute at a time.
Changes for V8
Ensure that unknown tests are flagged as failures. Split out the various tests into their own patches which test
the functionality as the series goes.
Move this basic test forward in the series
Changes for V7
Add testing for pks_abandon_protections()
Adjust pkrs_init_value
Adjust for new defines
Clean up comments
Adjust test for static allocation of pkeys
Use lookup_address() instead of follow_pte()
follow_pte only works on IO and raw PFN mappings, use lookup_address() instead. lookup_address() is constrained to architectures which support it.
--- Documentation/core-api/protection-keys.rst | 6 + include/linux/pks-keys.h | 8 +- lib/Kconfig.debug | 12 + lib/Makefile | 3 + lib/pks/Makefile | 3 + lib/pks/pks_test.c | 301 +++++++++++++++++++++ 6 files changed, 331 insertions(+), 2 deletions(-) create mode 100644 lib/pks/Makefile create mode 100644 lib/pks/pks_test.c
Documentation/core-api/protection-keys.rst +6 -0
@@ -142,3 +142,9 @@ Adding pages to a pkey protected domain .. kernel-doc:: arch/x86/include/asm/pgtable_types.h :doc: PKS_KEY_ASSIGNMENT + +Testing +------- + +.. kernel-doc:: lib/pks/pks_test.c + :doc: PKS_TEST
include/linux/pks-keys.h +6 -2
@@ -60,17 +60,21 @@/* PKS_KEY_DEFAULT must be 0 */ #define PKS_KEY_DEFAULT 0 -#define PKS_KEY_MAX PKS_NEW_KEY(PKS_KEY_DEFAULT, 1) +#define PKS_KEY_TEST PKS_NEW_KEY(PKS_KEY_DEFAULT, CONFIG_PKS_TEST) +#define PKS_KEY_MAX PKS_NEW_KEY(PKS_KEY_TEST, 1)/* PKS_KEY_DEFAULT_INIT must be RW */ #define PKS_KEY_DEFAULT_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_DEFAULT, RW, 1) +#define PKS_KEY_TEST_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_TEST, AD, \ + CONFIG_PKS_TEST)#define PKS_ALL_AD_MASK \ GENMASK(PKS_NUM_PKEYS * PKR_BITS_PER_PKEY, \ PKS_KEY_MAX * PKR_BITS_PER_PKEY)#define PKS_INIT_VALUE ((PKS_ALL_AD & PKS_ALL_AD_MASK) | \ - PKS_KEY_DEFAULT_INIT \ + PKS_KEY_DEFAULT_INIT | \ + PKS_KEY_TEST_INIT \ )#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
lib/Kconfig.debug +12 -0
@@ -2685,6 +2685,18 @@ config HYPERV_TESTING help Select this option to enable Hyper-V vmbus testing. +config PKS_TEST + bool "PKey (S)upervisor testing" + depends on ARCH_HAS_SUPERVISOR_PKEYS + select ARCH_ENABLE_SUPERVISOR_PKEYS + help + Select this option to enable testing of PKS core software and + hardware. + + Answer N if you don't know what supervisor keys are. + + If unsure, say N. + endmenu # "Kernel Testing and Coverage" source "Documentation/Kconfig"
lib/Makefile +3 -0
@@ -398,3 +398,6 @@ $(obj)/$(TEST_FORTIFY_LOG): $(addprefix $(obj)/, $(TEST_FORTIFY_LOGS)) FORCE ifeq ($(CONFIG_FORTIFY_SOURCE),y) $(obj)/string.o: $(obj)/$(TEST_FORTIFY_LOG) endif + +# PKS test +obj-y += pks/
lib/pks/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_PKS_TEST) += pks_test.o
lib/pks/pks_test.c new file +301 -0
301 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v9 14/45 x86/selftests: Add test_pks
In reply to [email protected]From: Ira Weiny <[email protected]>
The PKS kernel tests are clumsy to run using debugfs directly. It is much nicer to have a user space application trigger the execution of those tests.
Create test_pks as a selftest.
Output is as follows.
$ ./test_pks_64 -h Usage: ./test_pks_64 [-h,-d] [test] --help,-h This help --debug,-d Output kernel debug via dynamic debug if available
Run all PKS tests or the [test] specified.
[test] can be one of:
'check_defaults'
'create_fault' (Not included in run all)
$ ./test_pks_64 [RUN] check_defaults [OK]
---
Changes for V9:
New Patch
--- Documentation/core-api/protection-keys.rst | 3 + tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/test_pks.c | 353 +++++++++++++++++++++ 3 files changed, 357 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/test_pks.c
Documentation/core-api/protection-keys.rst +3 -0
@@ -148,3 +148,6 @@ Testing .. kernel-doc:: lib/pks/pks_test.c :doc: PKS_TEST + +.. kernel-doc:: tools/testing/selftests/x86/test_pks.c + :doc: PKS_TEST_USER
tools/testing/selftests/x86/Makefile +1 -1
@@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh $(CC) trivial_program.c -no-pie) TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \ check_initial_reg_state sigreturn iopl ioperm \ test_vsyscall mov_ss_trap \ - syscall_arg_fault fsgsbase_restore sigaltstack + syscall_arg_fault fsgsbase_restore sigaltstack test_pks TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer
tools/testing/selftests/x86/test_pks.c new file +353 -0
353 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v9 15/45 x86/pkeys: Introduce pks_write_pkrs()
In reply to [email protected]From: Ira Weiny <[email protected]>
Writing to MSR's is inefficient. Even though the underlying PKS register, MSR_IA32_PKRS, is not serializing; writing to the MSR should be avoided if possible. Especially when updates are made in critical paths such as the scheduler or the entry code.
Introduce pks_write_pkrs(). pks_write_pkrs() avoids writing MSR_IA32_PKRS if the pkrs value has not changed for the current CPU. Most of the callers are in a non-preemptable code path. Therefore, avoid calling preempt_{disable,enable}() to protect the per-cpu cache and instead rely on outer calls for this protection. Do the same with checks to X86_FEATURE_PKS.
On startup, while unlikely, the PKS_INIT_VALUE may be 0. This would prevent pks_write_pkrs() from updating the MSR because of the initial value of the per-cpu cache. Therefore, keep the MSR write in pks_setup() to ensure the MSR is initialized at least one time.
---
Changes for V9
From Dave Hansen
Update commit message with a bit more detail about why
this optimization is needed
Update the code comments as well.
Changes for V8
From Thomas
Remove get/put_cpu_ptr() and make this a 'lower level call. This makes it preemption unsafe but it is called mostly where preemption is already disabled. Add this as a predicate of the call and those calls which need to can disable preemption.
Add lockdep assert for preemption
Ensure MSR gets written even if the PKS_INIT_VALUE is 0. Completely re-write the commit message. s/write_pkrs/pks_write_pkrs/
Split this off into a singular patch
Changes for V7
Create a dynamic pkrs_initial_value in early init code. Clean up comments
Add comment to macro guard
--- arch/x86/mm/pkeys.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+)
arch/x86/mm/pkeys.c +41 -0
@@ -213,15 +213,56 @@ u32 pkey_update_pkval(u32 pkval, u8 pkey, u32 accessbits)#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +static DEFINE_PER_CPU(u32, pkrs_cache); + +/* +* pks_write_pkrs() - Write the pkrs of the current CPU +* @new_pkrs: New value to write to the current CPU register +* +* Optimizes the MSR writes by maintaining a per cpu cache. +* +* Context: must be called with preemption disabled +* Context: must only be called if PKS is enabled +* +* It should also be noted that the underlying WRMSR(MSR_IA32_PKRS) is not +* serializing but still maintains ordering properties similar to WRPKRU. +* The current SDM section on PKRS needs updating but should be the same as +* that of WRPKRU. Quote from the WRPKRU text: +* +* WRPKRU will never execute transiently. Memory accesses +* affected by PKRU register will not execute (even transiently) +* until all prior executions of WRPKRU have completed execution +* and updated the PKRU register. +*/ +static inline void pks_write_pkrs (u32 new_pkrs) +{ +u32 pkrs =__this_cpu_read (pkrs_cache); + +lockdep_assert_preemption_disabled (); + +if (pkrs != new_pkrs) { +__this_cpu_write (pkrs_cache, new_pkrs); +wrmsrl (MSR_IA32_PKRS , new_pkrs); + } +} +/* * PKS is independent of PKU and either or both may be supported on a CPU. +* +* Context: must be called with preemption disabled */ void pks_setup (void ) {if (!cpu_feature_enabled (X86_FEATURE_PKS ))return ; +/* +* If the PKS_INIT_VALUE is 0 then pks_write_pkrs() will fail to +* initialize the MSR. Do a single write here to ensure the MSR is +* written at least one time. +*/ wrmsrl (MSR_IA32_PKRS ,PKS_INIT_VALUE ); +pks_write_pkrs (PKS_INIT_VALUE );cr4_set_bits (X86_CR4_PKS ); }
Signature · 1 line
2.35.1
PATCH v9 16/45 x86/pkeys: Preserve the PKS MSR on context switch
In reply to [email protected]From: Ira Weiny <[email protected]>
The PKS MSR (PKRS) is a per-logical-processor register. Unfortunately, the MSR is not managed by XSAVE. Therefore, software must save/restore the MSR value on context switch.
Allocate space in thread_struct to hold the saved MSR value. Ensure all tasks, including the init_task are properly initialized. Set the CPU PKRS value when a task is scheduled.
---
Changes for V9
From Dave Hansen
Clarify the commit message s/pks_saved_pkrs/pkrs/ s/pks_write_current/x86_pkrs_load/
Change x86_pkrs_load to take the next thread instead of
'current'
Changes for V8
From Thomas
Ensure pkrs_write_current() does not suffer the overhead of preempt disable.
Fix setting of initial value
Remove flawed and broken create_initial_pkrs_value() in
favor of a much simpler and robust macro default
Update function names to be consistent.
s/pkrs_write_current/pks_write_current
This is a more consistent name
s/saved_pkrs/pks_saved_pkrs s/pkrs_init_value/PKS_INIT_VALUE
Remove pks_init_task()
This function was added mainly to avoid the header file issue. Adding pks-keys.h solved that and saves the complexity.
Changes for V7
Move definitions from asm/processor.h to asm/pks.h s/INIT_PKRS_VALUE/pkrs_init_value
Change pks_init_task()/pks_sched_in() to functions s/pks_sched_in/pks_write_current to be used more generically later in the series
--- arch/x86/include/asm/pks.h | 2 ++ arch/x86/include/asm/processor.h | 15 ++++++++++++++- arch/x86/kernel/process_64.c | 2 ++ arch/x86/mm/pkeys.c | 9 +++++++++ 4 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/pks.h +2 -0
@@ -5,10 +5,12 @@#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS void pks_setup (void ); +void x86_pkrs_load (struct thread_struct *thread);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void pks_setup (void ) { } +static inline void x86_pkrs_load (struct thread_struct *thread) { }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/include/asm/processor.h +14 -1
@@ -2,6 +2,8 @@#ifndef _ASM_X86_PROCESSOR_H#define _ASM_X86_PROCESSOR_H +#include <linux/pks-keys.h> +#include <asm/processor-flags.h> /* Forward declaration, a strange C thing */
@@ -527,6 +529,10 @@ struct thread_struct {* PKRU is the hardware itself. */ u32 pkru; +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* Saved Protection key register for supervisor mappings */ +u32 pkrs; +#endif /* Floating point and extended processor state */ struct fpu fpu;
@@ -769,7 +775,14 @@ static inline void spin_lock_prefetch(const void *x)#define KSTK_ESP (task) (task_pt_regs(task)->sp)#else -#define INIT_THREAD { } + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +#define INIT_THREAD { \ + .pkrs = PKS_INIT_VALUE, \ +} +#else +#define INIT_THREAD { } +#endif extern unsigned long KSTK_ESP (struct task_struct *task);
arch/x86/kernel/process_64.c +2 -0
@@ -59,6 +59,7 @@/* Not included via unistd.h */ #include <asm/unistd_32_ia32.h> #endif +#include <asm/pks.h> #include "process.h"
@@ -612,6 +613,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)x86_fsgsbase_load (prev, next);x86_pkru_load (prev, next); +x86_pkrs_load (next);/* * Switch the PDA and FPU contexts.
arch/x86/mm/pkeys.c +9 -0
@@ -246,6 +246,15 @@ static inline void pks_write_pkrs(u32 new_pkrs) } } +/* x86_pkrs_load() - Update CPU with the incoming thread pkrs value */ +void x86_pkrs_load (struct thread_struct *thread) +{ +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +pks_write_pkrs (thread->pkrs); +} +/* * PKS is independent of PKU and either or both may be supported on a CPU. *
Signature · 1 line
2.35.1
PATCH v9 17/45 mm/pkeys: Introduce pks_set_readwrite()
In reply to [email protected]From: Ira Weiny <[email protected]>
When kernel code needs access to a PKS protected page they will need to change the protections for the pkey to Read/Write.
Define pks_set_readwrite() to update the specified pkey. Define pks_update_protection() as a helper to do the heavy lifting and allow for subsequent pks_set_*() calls.
Define PKEY_READ_WRITE rather than use a magic value of '0' in pks_update_protection().
Finally, ensure preemption is disabled for pks_write_pkrs() because the context of this call can not generally be predicted.
pks.h is created to avoid conflicts and header dependencies with the user space pkey code.
Add documentation.
---
changes for v9
Move MSR documentation note to this patch move declarations to incline/linux/pks.h from rick edgecombe
change pkey type to u8
validate pkey range in pks_update_protection from 0day
fix documentation link
from dave hansen
s/pks_mk_*/pks_set_*/
use pkey s/pks_saved_pkrs/pkrs/
changes for v8
define pkey_read_write
make the call inline
clean up the names
use pks_write_pkrs() with preemption disabled split this out from 'add pks kernel api' include documentation in this patch
--- Documentation/core-api/protection-keys.rst | 15 +++++++++++ arch/x86/mm/pkeys.c | 31 ++++++++++++++++++++++ include/linux/pks.h | 31 ++++++++++++++++++++++ include/uapi/asm-generic/mman-common.h | 1 + 4 files changed, 78 insertions(+) create mode 100644 include/linux/pks.h
Documentation/core-api/protection-keys.rst +15 -0
@@ -143,6 +143,21 @@ Adding pages to a pkey protected domain .. kernel-doc:: arch/x86/include/asm/pgtable_types.h :doc: PKS_KEY_ASSIGNMENT +Changing permissions of individual keys +--------------------------------------- + +.. kernel-doc:: include/linux/pks.h + :identifiers: pks_set_readwrite + +MSR details +~~~~~~~~~~~ + +WRMSR is typically an architecturally serializing instruction. However, +WRMSR(MSR_IA32_PKRS) is an exception. It is not a serializing instruction and +instead maintains ordering properties similar to WRPKRU. Thus it is safe to +immediately use a mapping when the pks_set*() functions returns. Check the +latest SDM for details. + Testing -------
arch/x86/mm/pkeys.c +31 -0
@@ -6,6 +6,7 @@#include <linux/debugfs.h> /* debugfs_create_u32() */ #include <linux/mm_types.h> /* mm_struct, vma, etc... */ #include <linux/pkeys.h> /* PKEY_* */ +#include <linux/pks.h> #include <linux/pks-keys.h> #include <uapi/asm-generic/mman-common.h>
@@ -275,4 +276,34 @@ void pks_setup(void) cr4_set_bits(X86_CR4_PKS); } +/* +* Do not call this directly, see pks_set*(). +* +* @pkey: Key for the domain to change +* @protection: protection bits to be used +* +* Protection utilizes the same protection bits specified for User pkeys +* PKEY_DISABLE_ACCESS +* PKEY_DISABLE_WRITE +* +*/ +void pks_update_protection (u8 pkey,u8 protection) +{ +u32 pkrs; + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +if (WARN_ON_ONCE (pkey >=PKS_KEY_MAX )) +return ; + + pkrs = current->thread.pkrs; + current->thread.pkrs =pkey_update_pkval (pkrs, pkey, + protection); +preempt_disable (); +pks_write_pkrs (current->thread.pkrs); +preempt_enable (); +} +EXPORT_SYMBOL_GPL (pks_update_protection); +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
include/linux/pks.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PKS_H +#define _LINUX_PKS_H + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +#include <linux/types.h> + +#include <uapi/asm-generic/mman-common.h> + +void pks_update_protection (u8 pkey,u8 protection); + +/** +* pks_set_readwrite() - Make the domain Read/Write +* @pkey: the pkey for which the access should change. +* +* Allow all access, read and write, to the domain specified by pkey. This is +* not a global update and only affects the current running thread. +*/ +static inline void pks_set_readwrite (u8 pkey) +{ +pks_update_protection (pkey,PKEY_READ_WRITE ); +} + +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +static inline void pks_set_readwrite (u8 pkey) {} + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +#endif /* _LINUX_PKS_H */
include/uapi/asm-generic/mman-common.h +1 -0
@@ -78,6 +78,7 @@/* compatibility flags */ #define MAP_FILE 0 +#define PKEY_READ_WRITE 0x0#define PKEY_DISABLE_ACCESS 0x1#define PKEY_DISABLE_WRITE 0x2#define PKEY_ACCESS_MASK (PKEY_DISABLE_ACCESS |\
Signature · 1 line
2.35.1
PATCH v9 18/45 mm/pkeys: Introduce pks_set_noaccess()
In reply to [email protected]From: Ira Weiny <[email protected]>
After a valid access consumers will want to change PKS protections back to No Access for their pkey.
Define pks_set_noaccess() to update the specified pkey.
Add documentation.
---
Changes for V9
Move to pks.h
Change pkey type to u8
From 0day
Fix documentation link
From Dave Hansen
use pkey
s/pks_mk*/pks_set*/
Changes for V8
Make the call inline
Split this patch out from 'Add PKS kernel API' Include documentation in this patch
--- Documentation/core-api/protection-keys.rst | 2 +- include/linux/pks.h | 13 +++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-)
Documentation/core-api/protection-keys.rst +1 -1
@@ -147,7 +147,7 @@ Changing permissions of individual keys --------------------------------------- .. kernel-doc:: include/linux/pks.h - :identifiers: pks_set_readwrite + :identifiers: pks_set_readwrite pks_set_noaccess MSR details ~~~~~~~~~~~
include/linux/pks.h +13 -0
@@ -10,6 +10,18 @@void pks_update_protection (u8 pkey,u8 protection); +/** +* pks_set_noaccess() - Disable all access to the domain +* @pkey: the pkey for which the access should change. +* +* Disable all access to the domain specified by pkey. This is not a global +* update and only affects the current running thread. +*/ +static inline void pks_set_noaccess (u8 pkey) +{ +pks_update_protection (pkey,PKEY_DISABLE_ACCESS ); +} +/** * pks_set_readwrite() - Make the domain Read/Write * @pkey: the pkey for which the access should change.
@@ -24,6 +36,7 @@ static inline void pks_set_readwrite(u8 pkey)#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ +static inline void pks_set_noaccess (u8 pkey) {}static inline void pks_set_readwrite (u8 pkey) {}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.35.1
PATCH v9 19/45 mm/pkeys: Introduce PKS fault callbacks
In reply to [email protected]From: Rick Edgecombe <[email protected]>
Some PKS consumers will want special handling on violations of pkey permissions. Such a consumer is PMEM which will want to have a mode that logs the access violation, disables protection, and continues rather than oops'ing the machine.
Provide an API to assign callbacks for individual pkeys.
Since PKS faults do not provide the key that faulted, this information needs to be recovered by walking the page tables and extracting it from the leaf entry. The key can then be used to call the proper callback.
Add documentation.
---
Changes for V9:
Rework commit message
Adjust for the use of linux/pks.h
From the new key allocation: s/PKS_NR_CONSUMERS/PKS_KEY_MAX From Dave Hansen
use pkey
Fix conflicts with other users in the test code by
moving this forward in the series
Changes for V8:
Add pt_regs to the callback signature so that
pks_update_exception() can be called if needed.
Update commit message
Determine if page is large prior to not present Update commit message with more clarity as to why this was kept
separate from pks_abandon_protections() and pks_test_callback()
Embed documentation in c file.
Move handle_pks_key_fault() to pkeys.c
s/handle_pks_key_fault/pks_handle_key_fault/ This consolidates the PKS code nicely
Add feature check to pks_handle_key_fault()
From Rick Edgecombe
Fix key value check
From kernel test robot
Add static to handle_pks_key_fault
Changes for V7:
New patch
--- Documentation/core-api/protection-keys.rst | 6 ++ arch/x86/include/asm/pks.h | 10 +++ arch/x86/mm/fault.c | 17 +++-- arch/x86/mm/pkeys.c | 86 ++++++++++++++++++++++ include/linux/pks.h | 3 + 5 files changed, 116 insertions(+), 6 deletions(-)
Documentation/core-api/protection-keys.rst +6 -0
@@ -149,6 +149,12 @@ Changing permissions of individual keys .. kernel-doc:: include/linux/pks.h :identifiers: pks_set_readwrite pks_set_noaccess +Overriding Default Fault Behavior +--------------------------------- + +.. kernel-doc:: arch/x86/mm/pkeys.c + :doc: DEFINE_PKS_FAULT_CALLBACK + MSR details ~~~~~~~~~~~
arch/x86/include/asm/pks.h +10 -0
@@ -7,11 +7,21 @@void pks_setup (void );void x86_pkrs_load (struct thread_struct *thread); +bool pks_handle_key_fault (struct pt_regs *regs,unsigned long hw_error_code, +unsigned long address); +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void pks_setup (void ) { }static inline void x86_pkrs_load (struct thread_struct *thread) { } +static inline bool pks_handle_key_fault (struct pt_regs *regs, +unsigned long hw_error_code, +unsigned long address) +{ +return false; +} +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ #endif /* _ASM_X86_PKS_H */
arch/x86/mm/fault.c +11 -6
@@ -33,6 +33,7 @@#include <asm/kvm_para.h> /* kvm_handle_async_pf */ #include <asm/vdso.h> /* fixup_vdso_exception() */ #include <asm/irq_stack.h> +#include <asm/pks.h> /* pks_handle_key_fault() */ #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h>
@@ -1147,12 +1148,16 @@ static void do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) { -/* -* PF_PF faults should only occur on kernel -* addresses when supervisor pkeys are enabled. -*/ - WARN_ON_ONCE(!cpu_feature_enabled(X86_FEATURE_PKS) && - (hw_error_code & X86_PF_PK)); + if (hw_error_code & X86_PF_PK) { +/* +* PF_PF faults should only occur on kernel +* addresses when supervisor pkeys are enabled. +*/ + WARN_ON_ONCE(!cpu_feature_enabled(X86_FEATURE_PKS)); + +if (pks_handle_key_fault (regs, hw_error_code, address)) +return ; + }#ifdef CONFIG_X86_32 /*
arch/x86/mm/pkeys.c +86 -0
98 lines not on this page · raw message
include/linux/pks.h +3 -0
@@ -34,6 +34,9 @@ static inline void pks_set_readwrite(u8 pkey) pks_update_protection(pkey, PKEY_READ_WRITE); } +typedef bool (*pks_key_callback )(struct pt_regs *regs,unsigned long address, +bool write); +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void pks_set_noaccess (u8 pkey) {}
Signature · 1 line
2.35.1
PATCH v9 20/45 mm/pkeys: PKS testing, add a fault call back
In reply to [email protected]From: Ira Weiny <[email protected]>
PKS testing will need to know when a fault occurs due to it's actions so that it can properly determine functionality.
Install a PKS fault handler for the PKS test pkey.
---
Changes for V9
New Patch
--- arch/x86/mm/pkeys.c | 6 +++++- include/linux/pks.h | 7 +++++++ lib/pks/pks_test.c | 6 ++++++ 3 files changed, 18 insertions(+), 1 deletion(-)
arch/x86/mm/pkeys.c +5 -1
@@ -243,7 +243,11 @@ static DEFINE_PER_CPU(u32, pkrs_cache);* #endif * }; */ -static const pks_key_callback pks_key_callbacks[PKS_KEY_MAX ] = {0 }; +static const pks_key_callback pks_key_callbacks[PKS_KEY_MAX ] = { +#ifdef CONFIG_PKS_TEST + [PKS_KEY_TEST ] = pks_test_fault_callback, +#endif +};static bool pks_call_fault_callback (struct pt_regs *regs,unsigned long address,bool write,u16 key)
include/linux/pks.h +7 -0
@@ -44,4 +44,11 @@ static inline void pks_set_readwrite(u8 pkey) {}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ +#ifdef CONFIG_PKS_TEST + +bool pks_test_fault_callback (struct pt_regs *regs,unsigned long address, +bool write); + +#endif /* CONFIG_PKS_TEST */ +#endif /* _LINUX_PKS_H */
lib/pks/pks_test.c +6 -0
@@ -85,6 +85,12 @@ static void debug_result(const char *label, int test_num, sd->last_test_pass ? "PASS" : "FAIL"); } +bool pks_test_fault_callback (struct pt_regs *regs,unsigned long address, +bool write) +{ +return false; +} +static void *alloc_test_page (u8 pkey) { return __vmalloc_node_range(PKS_TEST_MEM_SIZE, 1, VMALLOC_START,
Signature · 1 line
2.35.1
PATCH v9 21/45 mm/pkeys: PKS testing, add pks_set_*() tests
In reply to [email protected]From: Ira Weiny <[email protected]>
Test that the pks_set_*() functions operate as intended.
First, verify that the pkey was properly set in the PTE.
Second, use the fault callback mechanism to detect if a fault occurred when expected and if so clear the fault.
The test iterates each of the following test cases.
PKS_TEST_NO_ACCESS, WRITE, FAULT_EXPECTED
PKS_TEST_NO_ACCESS, READ, FAULT_EXPECTED
PKS_TEST_RDWR, WRITE, NO_FAULT_EXPECTED
PKS_TEST_RDWR, READ, NO_FAULT_EXPECTED
Add documentation.
---
Changes for V9
Update commit message
Clarify use of global state for faults to be used by all tests Add test to test_pks user app
Remove an incorrect comment in the kdoc
Change pkey type to u8
From Dave Hansen
s/pks_mk*/pks_set*/
From Rick Edgecombe
Use standard fault callback instead of the custom PKS test one
Changes for V8
Remove readonly test, as that patch is not needed for PMEM Split this off into a patch which follows the pks_mk_*()
patches. Thus allowing for a better view of how the test works compared to the functionality added with those patches.
Remove unneeded prints
--- lib/pks/pks_test.c | 161 ++++++++++++++++++++++++- tools/testing/selftests/x86/test_pks.c | 5 +- 2 files changed, 162 insertions(+), 4 deletions(-)
lib/pks/pks_test.c +158 -3
@@ -33,11 +33,14 @@#include <linux/module.h> #include <linux/slab.h> #include <linux/vmalloc.h> +#include <linux/pgtable.h> +#include <linux/pks.h> #include <linux/pks-keys.h> #define PKS_TEST_MEM_SIZE (PAGE_SIZE)#define CHECK_DEFAULTS 0 +#define RUN_SINGLE 1#define RUN_CRASH_TEST 9static struct dentry *pks_test_dentry;
@@ -48,6 +51,7 @@ struct pks_test_ctx {u8 pkey;char data[64 ];void *test_page; +bool fault_seen; };static void debug_context (const char *label,struct pks_test_ctx *ctx)
@@ -85,10 +89,103 @@ static void debug_result(const char *label, int test_num, sd->last_test_pass ? "PASS" : "FAIL"); } +/* Global data protected by test_run_lock */ +struct pks_test_ctx *g_ctx_under_test; + +/* +* Call set_context_for_fault() after the context has been set up and prior to +* the expected fault. +*/ +static void set_context_for_fault (struct pks_test_ctx *ctx) +{ + g_ctx_under_test = ctx; +/* Ensure the state of the global context is correct prior to a fault */ +barrier (); +} +bool pks_test_fault_callback (struct pt_regs *regs,unsigned long address,bool write) { -return false; +pr_debug ("PKS Fault callback: ctx %p\n" , g_ctx_under_test); + +if (!g_ctx_under_test) +return false; + +pks_set_readwrite (g_ctx_under_test->pkey); + g_ctx_under_test->fault_seen = true; +return true; +} + +enum pks_access_mode { +PKS_TEST_NO_ACCESS , +PKS_TEST_RDWR , +}; + +#define PKS_WRITE true +#define PKS_READ false +#define PKS_FAULT_EXPECTED true +#define PKS_NO_FAULT_EXPECTED false + +static char *get_mode_str (enum pks_access_mode mode) +{ +switch (mode) { +case PKS_TEST_NO_ACCESS : +return "No Access" ; +case PKS_TEST_RDWR : +return "Read Write" ; + } + +return "" ; +} + +struct pks_access_test { +enum pks_access_mode mode; +bool write; +bool fault; +}; + +static struct pks_access_test pkey_test_ary[] = { + {PKS_TEST_NO_ACCESS ,PKS_WRITE ,PKS_FAULT_EXPECTED }, + {PKS_TEST_NO_ACCESS ,PKS_READ ,PKS_FAULT_EXPECTED }, + + {PKS_TEST_RDWR ,PKS_WRITE ,PKS_NO_FAULT_EXPECTED }, + {PKS_TEST_RDWR ,PKS_READ ,PKS_NO_FAULT_EXPECTED }, +}; + +static bool run_access_test (struct pks_test_ctx *ctx, +struct pks_access_test *test, +void *ptr) +{ +switch (test->mode) { +case PKS_TEST_NO_ACCESS : +pks_set_noaccess (ctx->pkey); +break ; +case PKS_TEST_RDWR : +pks_set_readwrite (ctx->pkey); +break ; +default : +pr_debug ("BUG in test, invalid mode\n" ); +return false; + } + + ctx->fault_seen = false; +set_context_for_fault (ctx); + +if (test->write) +memcpy (ptr, ctx->data,8 ); +else +memcpy (ctx->data, ptr,8 ); + +if (test->fault != ctx->fault_seen) { +pr_err ("pkey test FAILED: mode %s; write %s; fault %s != %s\n" , +get_mode_str (test->mode), + test->write ?"TRUE" :"FALSE" , + test->fault ?"YES" :"NO" , + ctx->fault_seen ?"YES" :"NO" ); +return false; + } + +return true; }static void *alloc_test_page (u8 pkey)
@@ -108,6 +205,37 @@ static void free_ctx(struct pks_test_ctx *ctx)kfree (ctx); } +static bool test_ctx (struct pks_test_ctx *ctx) +{ +bool rc = true; +int i; +u8 pkey; +void *ptr = ctx->test_page; +pte_t *ptep =NULL ; +unsigned int level; + + ptep =lookup_address ((unsigned long )ptr, &level); +if (!ptep) { +pr_err ("Failed to lookup address???\n" ); +return false; + } + + pkey =pte_flags_pkey (ptep->pte); +if (pkey != ctx->pkey) { +pr_err ("invalid pkey found: %u, test_pkey: %u\n" , + pkey, ctx->pkey); +return false; + } + +for (i =0 ; i <ARRAY_SIZE (pkey_test_ary); i++) { +/* sticky fail */ +if (!run_access_test (ctx, &pkey_test_ary[i], ptr)) + rc = false; + } + +return rc; +} +static struct pks_test_ctx *alloc_ctx (u8 pkey) {struct pks_test_ctx *ctx =kzalloc (sizeof (*ctx),GFP_KERNEL );
@@ -139,6 +267,23 @@ static void set_ctx_data(struct pks_session_data *sd, struct pks_test_ctx *ctx) sd->ctx = ctx; } +static bool run_single (struct pks_session_data *sd) +{ +struct pks_test_ctx *ctx; +bool rc; + + ctx =alloc_ctx (PKS_KEY_TEST ); +if (IS_ERR (ctx)) +return false; + +set_ctx_data (sd, ctx); + + rc =test_ctx (ctx); +pks_set_noaccess (ctx->pkey); + +return rc; +} +static void crash_it (struct pks_session_data *sd) {struct pks_test_ctx *ctx;
@@ -203,6 +348,12 @@ static ssize_t pks_read_file(struct file *file, char __user *user_buf,return simple_read_from_buffer (user_buf, count, ppos, buf, len); } +static void cleanup_test (void ) +{ + g_ctx_under_test =NULL ; +mutex_unlock (&test_run_lock); +} +static ssize_t pks_write_file (struct file *file,const char __user *user_buf,size_t count,loff_t *ppos) {
@@ -235,6 +386,10 @@ static ssize_t pks_write_file(struct file *file, const char __user *user_buf,pr_debug ("check defaults test: 0x%lx\n" ,PKS_INIT_VALUE );on_each_cpu (check_pkey_settings, file->private_data,1 );break ; +case RUN_SINGLE : +pr_debug ("Single key\n" ); + sd->last_test_pass =run_single (file->private_data); +break ;default :pr_debug ("Unknown test\n" ); sd->last_test_pass = false;
@@ -251,7 +406,7 @@ static ssize_t pks_write_file(struct file *file, const char __user *user_buf,* Normal exit; clear up the locking flag */ sd->need_unlock = false; -mutex_unlock (&test_run_lock); +cleanup_test ();debug_result ("Test complete" , test_num, sd);return count; }
@@ -282,7 +437,7 @@ static int pks_release_file(struct inode *inode, struct file *file)* not exit normally. */ if (sd->need_unlock) -mutex_unlock (&test_run_lock); +cleanup_test ();free_ctx (sd->ctx);kfree (sd);return 0 ;
tools/testing/selftests/x86/test_pks.c +4 -1
@@ -31,6 +31,7 @@/* Values from the kernel */ #define CHECK_DEFAULTS "0" +#define RUN_SINGLE "1"#define RUN_CRASH_TEST "9"time_t g_start_time;
@@ -53,6 +54,7 @@ static int do_simple_test(const char *debugfs_str);*/ enum {TEST_DEFAULTS =0 , +TEST_SINGLE ,MAX_TESTS , } tests;
@@ -64,7 +66,8 @@ struct test_item {const char *debugfs_str;int (*test_fn)(const char *debugfs_str); } test_list[] = { - { "check_defaults", CHECK_DEFAULTS, do_simple_test } + { "check_defaults", CHECK_DEFAULTS, do_simple_test }, + { "single", RUN_SINGLE, do_simple_test } };static char *get_test_name (int test_num)
Signature · 1 line
2.35.1
PATCH v9 22/45 mm/pkeys: PKS testing, test context switching
In reply to [email protected]From: Ira Weiny <[email protected]>
PKS software must maintain the PKRS value during a context switch. Test this by running two processes simultaneously on the same CPU while using different permissions for the same pkey.
Leverage test_pks to create two threads scheduled on the same cpu.
On the kernel side create two commands. One to set up the pkey prior to the context switch (arm context switch) and a second to check the pkey after the context switch (check context switch).
---
Changes for V9
From Rick Edgecombe
Ensure the parent/child threads don't cause each other
to hang if one experiences a failure
Adjust for the new test_pks user space component Adjust the debug output for '-d' option s/pks_mk_*/pks_set_*/
Use new set_file_data() call
Changes for V8
Split this off from the main testing patch Remove unneeded prints
--- lib/pks/pks_test.c | 54 +++++++++ tools/testing/selftests/x86/test_pks.c | 157 ++++++++++++++++++++++++- 2 files changed, 207 insertions(+), 4 deletions(-)
lib/pks/pks_test.c +54 -0
@@ -37,10 +37,14 @@#include <linux/pks.h> #include <linux/pks-keys.h> +#include <uapi/asm-generic/mman-common.h> +#define PKS_TEST_MEM_SIZE (PAGE_SIZE)#define CHECK_DEFAULTS 0#define RUN_SINGLE 1 +#define ARM_CTX_SWITCH 2 +#define CHECK_CTX_SWITCH 3#define RUN_CRASH_TEST 9static struct dentry *pks_test_dentry;
@@ -336,6 +340,48 @@ static void arm_or_run_crash_test(struct pks_session_data *sd)crash_it (sd); } +static void arm_ctx_switch (struct pks_session_data *sd) +{ +struct pks_test_ctx *ctx; + + ctx =alloc_ctx (PKS_KEY_TEST ); +if (IS_ERR (ctx)) { +pr_err ("Failed to allocate a context\n" ); + sd->last_test_pass = false; +return ; + } + +set_ctx_data (sd, ctx); + +/* Ensure a known state to test context switch */ +pks_set_readwrite (ctx->pkey); +} + +static void check_ctx_switch (struct pks_session_data *sd) +{ +struct pks_test_ctx *ctx = sd->ctx; +unsigned long reg_pkrs; +int access; + + sd->last_test_pass = true; + +if (!ctx) { +pr_err ("No Context switch configured\n" ); + sd->last_test_pass = false; +return ; + } + +rdmsrl (MSR_IA32_PKRS , reg_pkrs); + + access = (reg_pkrs >>PKR_PKEY_SHIFT (ctx->pkey)) & +PKEY_ACCESS_MASK ; +if (access !=0 ) { +pr_err ("Context switch check failed: pkey %u: 0x%x reg: 0x%lx\n" , + ctx->pkey, access, reg_pkrs); + sd->last_test_pass = false; + } +} +static ssize_t pks_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos) {
@@ -390,6 +436,14 @@ static ssize_t pks_write_file(struct file *file, const char __user *user_buf,pr_debug ("Single key\n" ); sd->last_test_pass =run_single (file->private_data);break ; +case ARM_CTX_SWITCH : +pr_debug ("Arming Context switch test\n" ); +arm_ctx_switch (file->private_data); +break ; +case CHECK_CTX_SWITCH : +pr_debug ("Checking Context switch test\n" ); +check_ctx_switch (file->private_data); +break ;default :pr_debug ("Unknown test\n" ); sd->last_test_pass = false;
tools/testing/selftests/x86/test_pks.c +153 -4
215 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v9 23/45 x86/entry: Add auxiliary pt_regs space
In reply to [email protected]From: Ira Weiny <[email protected]>
The PKRS MSR is not managed by XSAVE. In order for the MSR to be saved during an exception the current CPU MSR value needs to be saved somewhere during the exception and restored when returning to the previous context.
Two possible places for preserving this state were considered, irqentry_state_t or pt_regs.[1] pt_regs was much more complicated and was potentially fraught with unintended consequences.[2] However, Andy Lutomirski came up with a way to hide additional values on the stack which could be accessed as "extended_pt_regs".[3] This method allows any function with current access to pt_regs to obtain access to the extra information without expanding the use of irqentry_state_t and leaving pt_regs intact for compatibility with outside tools like BPF.
Prepare the assembly code to add a hidden auxiliary pt_regs space. To simplify, the assembly code only adds space on the stack as defined by the C code which needs it. The use of this space is left to the C code which is required to select ARCH_HAS_PTREGS_AUXILIARY to enable this support.
Each nested exception gets another copy of this auxiliary space allowing for any number of levels of exception handling.
Initially the space is left empty and results in no code changes because ARCH_HAS_PTREGS_AUXILIARY is not set. Subsequent patches adding data to pt_regs_auxiliary must set ARCH_HAS_PTREGS_AUXILIARY or a build failure will occur. The use of ARCH_HAS_PTREGS_AUXILIARY also avoids the introduction of 2 instructions (addq/subq) on every entry call when the extra space is not needed.
32bit is specifically excluded as the current consumer of this, PKS, will not support 32bit either.
Peter, Thomas, Andy, Dave, and Dan all suggested parts of the patch or aided in the development of the patch..
[1] https://lore.kernel.org/lkml/CALCETrVe1i5JdyzD_BcctxQJn+ZE3T38EFPgjxN1F577M36g+w@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/#t
[3] https://lore.kernel.org/lkml/CALCETrUHwZPic89oExMMe-WyDY8-O3W68NcZvse3=PGW+iW5=w@mail.gmail.com/
---
Changes for V9:
Update commit message
Changes for V8:
Exclude 32bit
Introduce ARCH_HAS_PTREGS_AUXILIARY to optimize this away when
not needed.
From Thomas
s/EXTENDED_PT_REGS_SIZE/PT_REGS_AUX_SIZE
Fix up PTREGS_AUX_SIZE macro to be based on the
structures and used in assembly code via the nifty asm-offset macros
Bound calls into c code with [PUSH|POP]_RTREGS_AUXILIARY
instead of using a macro 'call'
Split this patch out and put the PKS specific stuff in a
separate patch
Changes for V7:
Rebased to 5.14 entry code
declare write_pkrs() in pks.h s/INIT_PKRS_VALUE/pkrs_init_value
Remove unnecessary INIT_PKRS_VALUE def s/pkrs_save_set_irq/pkrs_save_irq/
The inital value for exceptions is best managed completely within the pkey code.
--- arch/x86/Kconfig | 4 ++++ arch/x86/entry/calling.h | 20 ++++++++++++++++++++ arch/x86/entry/entry_64.S | 22 ++++++++++++++++++++++ arch/x86/entry/entry_64_compat.S | 6 ++++++ arch/x86/include/asm/ptrace.h | 18 ++++++++++++++++++ arch/x86/kernel/asm-offsets_64.c | 15 +++++++++++++++ arch/x86/kernel/head_64.S | 6 ++++++ 7 files changed, 91 insertions(+)
arch/x86/Kconfig +4 -0
@@ -1878,6 +1878,10 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. +config ARCH_HAS_PTREGS_AUXILIARY + depends on X86_64 + bool + choice prompt "TSX enable mode" depends on CPU_SUP_INTEL
arch/x86/entry/calling.h +20 -0
@@ -63,6 +63,26 @@ For 32-bit we have the following conventions - kernel is built with* for assembly code: */ + +#ifdef CONFIG_ARCH_HAS_PTREGS_AUXILIARY + +.macro PUSH_PTREGS_AUXILIARY + /* add space for pt_regs_auxiliary */ + subq $PTREGS_AUX_SIZE, %rsp +.endm + +.macro POP_PTREGS_AUXILIARY + /* remove space for pt_regs_auxiliary */ + addq $PTREGS_AUX_SIZE, %rsp +.endm + +#else + +#define PUSH_PTREGS_AUXILIARY +#define POP_PTREGS_AUXILIARY + +#endif + .macro PUSH_REGS rdx=%rdx rax=%rax save_ret=0 .if \save_ret pushq %rsi /* pt_regs->si */
arch/x86/entry/entry_64.S +22 -0
@@ -332,7 +332,9 @@ SYM_CODE_END(ret_from_fork) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif + PUSH_PTREGS_AUXILIARY call \cfunc + POP_PTREGS_AUXILIARY jmp error_return .endm
@@ -435,7 +437,9 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ + PUSH_PTREGS_AUXILIARY call \cfunc + POP_PTREGS_AUXILIARY jmp paranoid_exit
@@ -496,7 +500,9 @@ SYM_CODE_START(\asmsym) * stack. */ movq %rsp, %rdi /* pt_regs pointer */ + PUSH_PTREGS_AUXILIARY call vc_switch_off_ist + POP_PTREGS_AUXILIARY movq %rax, %rsp /* Switch to new stack */ UNWIND_HINT_REGS
@@ -507,7 +513,9 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ + PUSH_PTREGS_AUXILIARY call kernel_\cfunc + POP_PTREGS_AUXILIARY /* * No need to switch back to the IST stack. The current stack is either
@@ -542,7 +550,9 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer into first argument */ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ + PUSH_PTREGS_AUXILIARY call \cfunc + POP_PTREGS_AUXILIARY jmp paranoid_exit
@@ -784,7 +794,9 @@ SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS + PUSH_PTREGS_AUXILIARY call xen_pv_evtchn_do_upcall + POP_PTREGS_AUXILIARY jmp error_return SYM_CODE_END(exc_xen_hypervisor_callback)
@@ -984,7 +996,9 @@ SYM_CODE_START_LOCAL(error_entry) /* Put us onto the real thread stack. */ popq %r12 /* save return addr in %12 */ movq %rsp, %rdi /* arg0 = pt_regs pointer */ + PUSH_PTREGS_AUXILIARY call sync_regs + POP_PTREGS_AUXILIARY movq %rax, %rsp /* switch stack */ ENCODE_FRAME_POINTER pushq %r12
@@ -1040,7 +1054,9 @@ SYM_CODE_START_LOCAL(error_entry) * as if we faulted immediately after IRET. */ mov %rsp, %rdi + PUSH_PTREGS_AUXILIARY call fixup_bad_iret + POP_PTREGS_AUXILIARY mov %rax, %rsp jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
@@ -1146,7 +1162,9 @@ SYM_CODE_START(asm_exc_nmi) movq %rsp, %rdi movq $-1, %rsi + PUSH_PTREGS_AUXILIARY call exc_nmi + POP_PTREGS_AUXILIARY /* * Return back to user mode. We must *not* do the normal exit
@@ -1182,6 +1200,8 @@ SYM_CODE_START(asm_exc_nmi) * +---------------------------------------------------------+ * | pt_regs | * +---------------------------------------------------------+ + * | (Optionally) pt_regs_extended | + * +---------------------------------------------------------+ * * The "original" frame is used by hardware. Before re-enabling * NMIs, we need to be done with it, and we need to leave enough
@@ -1358,7 +1378,9 @@ end_repeat_nmi: movq %rsp, %rdi movq $-1, %rsi + PUSH_PTREGS_AUXILIARY call exc_nmi + POP_PTREGS_AUXILIARY /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14
arch/x86/entry/entry_64_compat.S +6 -0
@@ -136,7 +136,9 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) .Lsysenter_flags_fixed: movq %rsp, %rdi + PUSH_PTREGS_AUXILIARY call do_SYSENTER_32 + POP_PTREGS_AUXILIARY /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
@@ -253,7 +255,9 @@ SYM_INNER_LABEL(entry_SYSCALL_compat_after_hwframe, SYM_L_GLOBAL) UNWIND_HINT_REGS movq %rsp, %rdi + PUSH_PTREGS_AUXILIARY call do_fast_syscall_32 + POP_PTREGS_AUXILIARY /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
@@ -410,6 +414,8 @@ SYM_CODE_START(entry_INT80_compat) cld movq %rsp, %rdi + PUSH_PTREGS_AUXILIARY call do_int80_syscall_32 + POP_PTREGS_AUXILIARY jmp swapgs_restore_regs_and_return_to_usermode SYM_CODE_END(entry_INT80_compat)
arch/x86/include/asm/ptrace.h +18 -0
@@ -2,6 +2,7 @@#ifndef _ASM_X86_PTRACE_H#define _ASM_X86_PTRACE_H +#include <linux/container_of.h> #include <asm/segment.h> #include <asm/page_types.h> #include <uapi/asm/ptrace.h>
@@ -91,6 +92,23 @@ struct pt_regs {/* top of stack page */ }; +/* +* NOTE: Features which add data to pt_regs_auxiliary must select +* ARCH_HAS_PTREGS_AUXILIARY. Failure to do so will result in a build failure. +*/ +struct pt_regs_auxiliary { +}; + +struct pt_regs_extended { +struct pt_regs_auxiliary aux; +struct pt_regs pt_regs__aligned(8) ; +}; + +static inline struct pt_regs_extended *to_extended_pt_regs (struct pt_regs *regs) +{ + return container_of(regs, struct pt_regs_extended, pt_regs); +} +#endif /* !__i386__ */ #ifdef CONFIG_PARAVIRT
arch/x86/kernel/asm-offsets_64.c +15 -0
@@ -4,6 +4,7 @@#endif #include <asm/ia32.h> +#include <asm/ptrace.h> #if defined(CONFIG_KVM_GUEST ) && defined(CONFIG_PARAVIRT_SPINLOCKS )#include <asm/kvm_para.h>
@@ -60,5 +61,19 @@ int main(void)DEFINE (stack_canary_offset, offsetof(struct fixed_percpu_data , stack_canary));BLANK ();#endif + +#ifdef CONFIG_ARCH_HAS_PTREGS_AUXILIARY +/* Size of Auxiliary pt_regs data */ +DEFINE (PTREGS_AUX_SIZE ,sizeof (struct pt_regs_extended ) - +sizeof (struct pt_regs )); +#else +/* +* Adding data to struct pt_regs_auxiliary requires setting +* ARCH_HAS_PTREGS_AUXILIARY +*/ +BUILD_BUG_ON ((sizeof (struct pt_regs_extended ) - +sizeof (struct pt_regs )) !=0 ); +#endif +return 0 ; }
arch/x86/kernel/head_64.S +6 -0
@@ -336,8 +336,10 @@ SYM_CODE_START_NOALIGN(vc_boot_ghcb) movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi movq initial_vc_handler(%rip), %rax + PUSH_PTREGS_AUXILIARY ANNOTATE_RETPOLINE_SAFE call *%rax + POP_PTREGS_AUXILIARY /* Unwind pt_regs */ POP_REGS
@@ -414,7 +416,9 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) UNWIND_HINT_REGS movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */ + PUSH_PTREGS_AUXILIARY call do_early_exception + POP_PTREGS_AUXILIARY decl early_recursion_flag(%rip) jmp restore_regs_and_return_to_kernel
@@ -438,7 +442,9 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Call C handler */ movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi + PUSH_PTREGS_AUXILIARY call do_vc_no_ghcb + POP_PTREGS_AUXILIARY /* Unwind pt_regs */ POP_REGS
Signature · 1 line
2.35.1
PATCH v9 24/45 entry: Split up irqentry_exit_cond_resched()
In reply to [email protected]From: Ira Weiny <[email protected]>
Auxiliary pt_regs space needs to be manipulated by the generic entry/exit code.
Normally irqentry_exit() would take care of handling any auxiliary pt_regs on exit. Unfortunately, the call to irqentry_exit_cond_resched() from xen_pv_evtchn_do_upcall() bypasses the normal irqentry_exit() call. Because of this bypass irqentry_exit_cond_resched() will be required to handle any auxiliary pt_regs exit handling. However, this prevents irqentry_exit() from being able to call irqentry_exit_cond_resched() and while maintaining control of the auxiliary pt_regs.
Separate out the common functionality of irqentry_exit_cond_resched() so that functionality can be used by irqentry_exit(). Add a pt_regs parameter in anticipation of having irqentry_exit_cond_resched() handle the auxiliary pt_regs separately from irqentry_exit().
---
Changes for V9
Update commit message
Changes for V8
New Patch
--- arch/x86/entry/common.c | 2 +- include/linux/entry-common.h | 3 ++- kernel/entry/common.c | 9 +++++++-- 3 files changed, 10 insertions(+), 4 deletions(-)
arch/x86/entry/common.c +1 -1
@@ -309,7 +309,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { -irqentry_exit_cond_resched (); +irqentry_exit_cond_resched (regs);instrumentation_end ();restore_inhcall (inhcall); }else {
include/linux/entry-common.h +2 -1
@@ -451,10 +451,11 @@ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs);/** * irqentry_exit_cond_resched - Conditionally reschedule on return from interrupt +* @regs: Pointer to pt_regs of interrupted context * * Conditional reschedule with additional sanity checks. */ -void irqentry_exit_cond_resched (void ); +void irqentry_exit_cond_resched (struct pt_regs *regs);void __irqentry_exit_cond_resched (void );#ifdef CONFIG_PREEMPT_DYNAMIC
kernel/entry/common.c +7 -2
@@ -395,7 +395,7 @@ void __irqentry_exit_cond_resched(void)DEFINE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif -void irqentry_exit_cond_resched (void ) +static void exit_cond_resched (void ) {if (IS_ENABLED (CONFIG_PREEMPTION )) {#ifdef CONFIG_PREEMPT_DYNAMIC
@@ -406,6 +406,11 @@ void irqentry_exit_cond_resched(void) } } +void irqentry_exit_cond_resched (struct pt_regs *regs) +{ +exit_cond_resched (); +} + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) {lockdep_assert_irqs_disabled ();
@@ -431,7 +436,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) }instrumentation_begin (); -irqentry_exit_cond_resched (); +exit_cond_resched ();/* Covers both tracing and lockdep */ trace_hardirqs_on ();instrumentation_end ();
Signature · 1 line
2.35.1
Re: [PATCH V9 24/45] entry: Split up irqentry_exit_cond_resched()
Quoted 4 lines · Ira
From: Ira Weiny <[email protected]>
Auxiliary
pt_regsspace needs to be manipulated by the generic entry/exit code.
Because of fix to the irqentry_exit_cond_resched() code[1] this patch needed rework upon rebasing to 5.18-rc1
This basic design of this patch remains but the code is different. The irqentry_exit_cond_resched() still needs to have pt_regs passed into it.
However, this could be safely ignored for this review cycle as well.
As soon as I have a series based on 5.18 I'll resend the full series.
Thanks for understanding,
Ira
[1] 4624a14f4daa ("sched/preempt: Simplify irqentry_exit_cond_resched() callers")
Quoted 96 lines
Normally
irqentry_exit()would take care of handling any auxiliarypt_regson exit. Unfortunately, the call toirqentry_exit_cond_resched()fromxen_pv_evtchn_do_upcall()bypasses the normalirqentry_exit()call. Because of this bypassirqentry_exit_cond_resched()will be required to handle any auxiliarypt_regsexit handling. However, this preventsirqentry_exit()from being able to callirqentry_exit_cond_resched()and while maintaining control of the auxiliarypt_regs.Separate out the common functionality of
irqentry_exit_cond_resched()so that functionality can be used byirqentry_exit(). Add apt_regsparameter in anticipation of havingirqentry_exit_cond_resched()handle the auxiliarypt_regsseparately fromirqentry_exit().Signed-off-by Ira Weiny <[email protected]>---
Changes for V9
Update commit messageChanges for V8
New Patch--- arch/x86/entry/common.c | 2 +- include/linux/entry-common.h | 3 ++- kernel/entry/common.c | 9 +++++++-- 3 files changed, 10 insertions(+), 4 deletions(-)arch/x86/entry/common.c +1
-1@@ -309,7 +309,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { -irqentry_exit_cond_resched (); +irqentry_exit_cond_resched (regs);instrumentation_end ();restore_inhcall (inhcall); }else {include/linux/entry-common.h +2
-1@@ -451,10 +451,11 @@ irqentry_state_t noinstr irqentry_enter(struct pt_regs *regs);/** * irqentry_exit_cond_resched - Conditionally reschedule on return from interrupt +* @regs: Pointer to pt_regs of interrupted context * * Conditional reschedule with additional sanity checks. */ -void irqentry_exit_cond_resched (void ); +void irqentry_exit_cond_resched (struct pt_regs *regs);void __irqentry_exit_cond_resched (void );#ifdef CONFIG_PREEMPT_DYNAMIC kernel/entry/common.c +7
-2@@ -395,7 +395,7 @@ void __irqentry_exit_cond_resched(void)DEFINE_STATIC_CALL (__irqentry_exit_cond_resched, __irqentry_exit_cond_resched);#endif -void irqentry_exit_cond_resched (void ) +static void exit_cond_resched (void ) {if (IS_ENABLED (CONFIG_PREEMPTION )) {#ifdef CONFIG_PREEMPT_DYNAMIC @@ -406,6 +406,11 @@ void irqentry_exit_cond_resched(void) } } +void irqentry_exit_cond_resched (struct pt_regs *regs) +{ +exit_cond_resched (); +} + noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) {lockdep_assert_irqs_disabled ();@@ -431,7 +436,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) }instrumentation_begin (); -irqentry_exit_cond_resched (); +exit_cond_resched ();/* Covers both tracing and lockdep */ trace_hardirqs_on ();instrumentation_end ();Signature · 1 line
2.35.1
PATCH v9 25/45 entry: Add calls for save/restore auxiliary pt_regs
In reply to [email protected]From: Ira Weiny <[email protected]>
Some architectures have auxiliary pt_regs space which is available to store extra information on the stack. For ease of implementation the common C code was left to fill in the data when needed.
Add calls to the architecture save and restore auxiliary pt_regs functions. Define empty calls for any architecture which does not have auxiliary pt_regs.
NOTE: Due to the split nature of the Xen exit code irqentry_exit_cond_resched() requires an unbalanced call to arch_restore_aux_pt_regs() regardless of the nature of the preemption configuration.
---
Changes for V9
Update commit message
Changes for V8
New patch which introduces a generic auxiliary pt_register save
restore.
--- include/linux/entry-common.h | 7 +++++++ kernel/entry/common.c | 16 ++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/entry-common.h +7 -0
@@ -99,6 +99,13 @@ static inline __must_check int arch_syscall_enter_tracehook(struct pt_regs *regs }#endif +#ifndef CONFIG_ARCH_HAS_PTREGS_AUXILIARY + +static inline void arch_save_aux_pt_regs (struct pt_regs *regs) { } +static inline void arch_restore_aux_pt_regs (struct pt_regs *regs) { } + +#endif +/** * enter_from_user_mode - Establish state when coming from user mode *
kernel/entry/common.c +14 -2
@@ -323,7 +323,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)if (user_mode (regs)) {irqentry_enter_from_user_mode (regs); -return ret; + goto aux_save; }/*
@@ -362,7 +362,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_end (); ret.exit_rcu = true; -return ret; + goto aux_save; }/*
@@ -377,6 +377,11 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)trace_hardirqs_off_finish ();instrumentation_end (); +aux_save: +instrumentation_begin (); +arch_save_aux_pt_regs (regs); +instrumentation_end (); +return ret; }
@@ -408,6 +413,7 @@ static void exit_cond_resched(void)void irqentry_exit_cond_resched (struct pt_regs *regs) { +arch_restore_aux_pt_regs (regs);exit_cond_resched (); }
@@ -415,6 +421,10 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) {lockdep_assert_irqs_disabled (); +instrumentation_begin (); +arch_restore_aux_pt_regs (regs); +instrumentation_end (); +/* Check whether this returns to user mode */ if (user_mode (regs)) {irqentry_exit_to_user_mode (regs);
@@ -464,6 +474,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)instrumentation_begin ();trace_hardirqs_off_finish ();ftrace_nmi_enter (); +arch_save_aux_pt_regs (regs);instrumentation_end ();return irq_state;
@@ -472,6 +483,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) {instrumentation_begin (); +arch_restore_aux_pt_regs (regs);ftrace_nmi_exit ();if (irq_state.lockdep) {trace_hardirqs_on_prepare ();
Signature · 1 line
2.35.1
PATCH v9 26/45 x86/entry: Define arch_{save|restore}_auxiliary_pt_regs()
In reply to [email protected]From: Ira Weiny <[email protected]>
The x86 architecture supports the new auxiliary pt_regs space if ARCH_HAS_PTREGS_AUXILIARY is enabled.
Define the callbacks within the x86 code required by the core entry code when this support is enabled.
---
Changes for V8
New patch
--- arch/x86/include/asm/entry-common.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/x86/include/asm/entry-common.h +12 -0
@@ -95,4 +95,16 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +#ifdef CONFIG_ARCH_HAS_PTREGS_AUXILIARY + +static inline void arch_save_aux_pt_regs (struct pt_regs *regs) +{ +} + +static inline void arch_restore_aux_pt_regs (struct pt_regs *regs) +{ +} + +#endif +#endif
Signature · 1 line
2.35.1
PATCH v9 27/45 x86/pkeys: Preserve PKRS MSR across exceptions
In reply to [email protected]From: Ira Weiny <[email protected]>
PKRS is a per-logical-processor MSR which overlays additional protection for pages which have been mapped with a protection key. It is desired to protect PKS pages while executing exception code while also allowing exception code to access PKS pages with the proper pks_set_*() calls.
To do this the current thread value must be saved, the CPU MSR value set to the default value during the exception, and the saved thread value restored upon completion. This can be done with the new auxiliary pt_regs space.
When PKS is configured, configure auxiliary pt_regs, add space to pt_regs_auxiliary, and define save/restore functions.
Update the PKS test code to maintain functionality by clearing the saved PKRS value before returning.
Peter, Thomas, Andy, Dave, and Dan all suggested parts of the patch or aided in the development of the patch.
[1] https://lore.kernel.org/lkml/CALCETrVe1i5JdyzD_BcctxQJn+ZE3T38EFPgjxN1F577M36g+w@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/#t
[3] https://lore.kernel.org/lkml/CALCETrUHwZPic89oExMMe-WyDY8-O3W68NcZvse3=PGW+iW5=w@mail.gmail.com/
---
Changes for V9:
Update commit message
s/pks_thread_pkrs/pkrs/
From Dave Hansen
s/pks_saved_pkrs/pkrs/
Changes for V8:
Tie this into the new generic auxiliary pt_regs support. Build this on the new irqentry_*() refactoring patches Split this patch off from the PKS portion of the auxiliary
pt_regs functionality.
From Thomas
Fix noinstr mess s/write_pkrs/pks_write_pkrs s/pkrs_init_value/PKRS_INIT_VALUE
Simplify the number and location of the save/restore calls.
Cover entry from user space as well.
Changes for V7:
Rebased to 5.14 entry code
declare write_pkrs() in pks.h s/INIT_PKRS_VALUE/pkrs_init_value
Remove unnecessary INIT_PKRS_VALUE def s/pkrs_save_set_irq/pkrs_save_irq/
The inital value for exceptions is best managed completely within the pkey code.
--- arch/x86/Kconfig | 3 ++- arch/x86/include/asm/entry-common.h | 3 +++ arch/x86/include/asm/pks.h | 4 ++++ arch/x86/include/asm/ptrace.h | 3 +++ arch/x86/mm/pkeys.c | 32 +++++++++++++++++++++++++++++ lib/pks/pks_test.c | 9 +++++++- 6 files changed, 52 insertions(+), 2 deletions(-)
arch/x86/Kconfig +2 -1
@@ -1879,8 +1879,9 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS If unsure, say y. config ARCH_HAS_PTREGS_AUXILIARY + def_bool y depends on X86_64 - bool + depends on ARCH_ENABLE_SUPERVISOR_PKEYS choice prompt "TSX enable mode"
arch/x86/include/asm/entry-common.h +3 -0
@@ -8,6 +8,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/fpu/api.h> +#include <asm/pks.h> /* Check that the stack and regs on entry from user mode are sane. */ static __always_inline void arch_check_user_regs(struct pt_regs *regs)
@@ -99,10 +100,12 @@ static __always_inline void arch_exit_to_user_mode(void)static inline void arch_save_aux_pt_regs (struct pt_regs *regs) { +pks_save_pt_regs (regs); }static inline void arch_restore_aux_pt_regs (struct pt_regs *regs) { +pks_restore_pt_regs (regs); }#endif
arch/x86/include/asm/pks.h +4 -0
@@ -6,6 +6,8 @@void pks_setup (void );void x86_pkrs_load (struct thread_struct *thread); +void pks_save_pt_regs (struct pt_regs *regs); +void pks_restore_pt_regs (struct pt_regs *regs);bool pks_handle_key_fault (struct pt_regs *regs,unsigned long hw_error_code,unsigned long address);
@@ -14,6 +16,8 @@ bool pks_handle_key_fault(struct pt_regs *regs, unsigned long hw_error_code,static inline void pks_setup (void ) { }static inline void x86_pkrs_load (struct thread_struct *thread) { } +static inline void pks_save_pt_regs (struct pt_regs *regs) { } +static inline void pks_restore_pt_regs (struct pt_regs *regs) { } static inline bool pks_handle_key_fault(struct pt_regs *regs, unsigned long hw_error_code,
arch/x86/include/asm/ptrace.h +3 -0
@@ -97,6 +97,9 @@ struct pt_regs {* ARCH_HAS_PTREGS_AUXILIARY. Failure to do so will result in a build failure. */ struct pt_regs_auxiliary { +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +u32 pkrs; +#endif };struct pt_regs_extended {
arch/x86/mm/pkeys.c +32 -0
@@ -346,6 +346,38 @@ void x86_pkrs_load(struct thread_struct *thread)pks_write_pkrs (thread->pkrs); } +/* +* PKRS is a per-logical-processor MSR which overlays additional protection for +* pages which have been mapped with a protection key. +* +* To protect against exceptions having potentially privileged access to memory +* of an interrupted thread, save the current thread value and set the PKRS +* value to be used during the exception. +*/ +void pks_save_pt_regs (struct pt_regs *regs) +{ +struct pt_regs_auxiliary *aux_pt_regs; + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + + aux_pt_regs = &to_extended_pt_regs (regs)->aux; + aux_pt_regs->pkrs = current->thread.pkrs; +pks_write_pkrs (PKS_INIT_VALUE ); +} + +void pks_restore_pt_regs (struct pt_regs *regs) +{ +struct pt_regs_auxiliary *aux_pt_regs; + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + + aux_pt_regs = &to_extended_pt_regs (regs)->aux; + current->thread.pkrs = aux_pt_regs->pkrs; +pks_write_pkrs (current->thread.pkrs); +} +/* * PKS is independent of PKU and either or both may be supported on a CPU. *
lib/pks/pks_test.c +8 -1
@@ -34,11 +34,14 @@#include <linux/slab.h> #include <linux/vmalloc.h> #include <linux/pgtable.h> +#include <linux/pkeys.h> #include <linux/pks.h> #include <linux/pks-keys.h> #include <uapi/asm-generic/mman-common.h> +#include <asm/ptrace.h> +#define PKS_TEST_MEM_SIZE (PAGE_SIZE)#define CHECK_DEFAULTS 0
@@ -110,12 +113,16 @@ static void set_context_for_fault(struct pks_test_ctx *ctx)bool pks_test_fault_callback (struct pt_regs *regs,unsigned long address,bool write) { +struct pt_regs_extended *ept_regs =to_extended_pt_regs (regs); +struct pt_regs_auxiliary *aux_pt_regs = &ept_regs->aux; +u32 pkrs = aux_pt_regs->pkrs; +pr_debug ("PKS Fault callback: ctx %p\n" , g_ctx_under_test);if (!g_ctx_under_test)return false; -pks_set_readwrite (g_ctx_under_test->pkey); + aux_pt_regs->pkrs =pkey_update_pkval (pkrs, g_ctx_under_test->pkey,0 ); g_ctx_under_test->fault_seen = true;return true; }
Signature · 1 line
2.35.1
PATCH v9 28/45 x86/fault: Print PKS MSR on fault
In reply to [email protected]From: Ira Weiny <[email protected]>
If a PKS fault occurs it will be easier to debug if the PKS MSR value at the time of the fault is known.
Add pks_show_regs() to __show_regs() to show the PKRS MSR on fault if enabled.
An 'executive summary' of the pt_regs are saved in __die_header() which ensures that the first registers are saved in the event of multiple faults. Teach this code about the extended pt_registers such that the PKS code can get to the original pkrs value as well.
---
Changes for V9
From Dave Hansen
Move this output to __show_regs() next to the PKRU
register dump
Changes for V8
Split this into it's own patch.
--- arch/x86/include/asm/pks.h | 3 +++ arch/x86/kernel/dumpstack.c | 32 ++++++++++++++++++++++++++++++-- arch/x86/kernel/process_64.c | 1 + arch/x86/mm/pkeys.c | 11 +++++++++++ 4 files changed, 45 insertions(+), 2 deletions(-)
arch/x86/include/asm/pks.h +3 -0
@@ -8,6 +8,7 @@ void pks_setup(void);void x86_pkrs_load (struct thread_struct *thread);void pks_save_pt_regs (struct pt_regs *regs);void pks_restore_pt_regs (struct pt_regs *regs); +void pks_show_regs (struct pt_regs *regs,const char *log_lvl);bool pks_handle_key_fault (struct pt_regs *regs,unsigned long hw_error_code,unsigned long address);
@@ -18,6 +19,8 @@ static inline void pks_setup(void) { }static inline void x86_pkrs_load (struct thread_struct *thread) { }static inline void pks_save_pt_regs (struct pt_regs *regs) { }static inline void pks_restore_pt_regs (struct pt_regs *regs) { } +static inline void pks_show_regs (struct pt_regs *regs, +const char *log_lvl) { } static inline bool pks_handle_key_fault(struct pt_regs *regs, unsigned long hw_error_code,
arch/x86/kernel/dumpstack.c +30 -2
@@ -27,8 +27,36 @@ int panic_on_unrecovered_nmi;int panic_on_io_nmi;static int die_counter; +#ifdef CONFIG_ARCH_HAS_PTREGS_AUXILIARY + +static struct pt_regs_extended exec_summary_regs; + +static void save_exec_summary (struct pt_regs *regs) +{ + exec_summary_regs = *(to_extended_pt_regs (regs)); +} + +static struct pt_regs *retrieve_exec_summary (void ) +{ +return &exec_summary_regs.pt_regs; +} + +#else /* !CONFIG_ARCH_HAS_PTREGS_AUXILIARY */ +static struct pt_regs exec_summary_regs; +static void save_exec_summary (struct pt_regs *regs) +{ + exec_summary_regs = *regs; +} + +static struct pt_regs *retrieve_exec_summary (void ) +{ +return &exec_summary_regs; +} + +#endif /* CONFIG_ARCH_HAS_PTREGS_AUXILIARY */ + bool noinstr in_task_stack(unsigned long *stack, struct task_struct *task,struct stack_info *info) {
@@ -369,7 +397,7 @@ void oops_end(unsigned long flags, struct pt_regs *regs, int signr)oops_exit ();/* Executive summary in case the oops scrolled away */ -__show_regs (&exec_summary_regs,SHOW_REGS_ALL ,KERN_DEFAULT ); +__show_regs (retrieve_exec_summary (),SHOW_REGS_ALL ,KERN_DEFAULT );if (!signr)return ;
@@ -396,7 +424,7 @@ static void __die_header(const char *str, struct pt_regs *regs, long err)/* Save the regs of the first oops for the executive summary later. */ if (!die_counter) - exec_summary_regs = *regs; +save_exec_summary (regs);if (IS_ENABLED (CONFIG_PREEMPTION )) pr =IS_ENABLED (CONFIG_PREEMPT_RT ) ?" PREEMPT_RT" :" PREEMPT" ;
arch/x86/kernel/process_64.c +1 -0
@@ -140,6 +140,7 @@ void __show_regs(struct pt_regs *regs, enum show_regs_mode mode,if (cpu_feature_enabled (X86_FEATURE_OSPKE ))printk ("%sPKRU: %08x\n" , log_lvl,read_pkru ()); +pks_show_regs (regs, log_lvl); }void release_thread (struct task_struct *dead_task)
arch/x86/mm/pkeys.c +11 -0
@@ -378,6 +378,17 @@ void pks_restore_pt_regs(struct pt_regs *regs)pks_write_pkrs (current->thread.pkrs); } +void pks_show_regs (struct pt_regs *regs,const char *log_lvl) +{ +struct pt_regs_auxiliary *aux_pt_regs; + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + + aux_pt_regs = &to_extended_pt_regs (regs)->aux; +printk ("%sPKRS: 0x%x\n" , log_lvl, aux_pt_regs->pkrs); +} +/* * PKS is independent of PKU and either or both may be supported on a CPU. *
Signature · 1 line
2.35.1
PATCH v9 29/45 mm/pkeys: PKS testing, Add exception test
In reply to [email protected]From: Ira Weiny <[email protected]>
During an exception the interrupted threads PKRS value is preserved and the exception receives the default value for that pkey. Upon return from exception the threads PKRS value is restored.
Add a PKS test which forces a fault to check that this works as intended. Check that both the thread as well as the exception PKRS state is correct at the beginning, during, and after the exception.
Add the test to the test_pks app.
---
Change for V9
Add test to test_pks
Clean up the globals shared with the fault handler Use the PKS Test specific fault callback s/pks_mk*/pks_set*/
Change pkey type to u8
From Dave Hansen
use pkey
Change for V8
Split this test off from the testing patch and place it after the exception saving code.
--- arch/x86/mm/pkeys.c | 2 +- include/linux/pks.h | 6 ++ lib/pks/pks_test.c | 133 +++++++++++++++++++++++++ tools/testing/selftests/x86/test_pks.c | 5 +- 4 files changed, 144 insertions(+), 2 deletions(-)
arch/x86/mm/pkeys.c +1 -1
@@ -215,7 +215,7 @@ u32 pkey_update_pkval(u32 pkval, u8 pkey, u32 accessbits)#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS -static DEFINE_PER_CPU(u32, pkrs_cache); +__static_or_pks_test DEFINE_PER_CPU (u32 ,pkrs_cache );/** * DOC: DEFINE_PKS_FAULT_CALLBACK
include/linux/pks.h +6 -0
@@ -46,9 +46,15 @@ static inline void pks_set_readwrite(u8 pkey) {}#ifdef CONFIG_PKS_TEST +#define __static_or_pks_test +bool pks_test_fault_callback (struct pt_regs *regs,unsigned long address,bool write); +#else /* !CONFIG_PKS_TEST */ + +#define __static_or_pks_test static +#endif /* CONFIG_PKS_TEST */ #endif /* _LINUX_PKS_H */
lib/pks/pks_test.c +133 -0
182 lines not on this page · raw message
tools/testing/selftests/x86/test_pks.c +4 -1
@@ -35,6 +35,7 @@#define RUN_SINGLE "1"#define ARM_CTX_SWITCH "2"#define CHECK_CTX_SWITCH "3" +#define RUN_EXCEPTION "4"#define RUN_CRASH_TEST "9"time_t g_start_time;
@@ -61,6 +62,7 @@ enum {TEST_DEFAULTS =0 ,TEST_SINGLE ,TEST_CTX_SWITCH , +TEST_EXCEPTION ,MAX_TESTS , } tests;
@@ -74,7 +76,8 @@ struct test_item { } test_list[] = { { "check_defaults", CHECK_DEFAULTS, do_simple_test }, { "single", RUN_SINGLE, do_simple_test }, - { "context_switch", ARM_CTX_SWITCH, do_context_switch } + { "context_switch", ARM_CTX_SWITCH, do_context_switch }, + { "exception", RUN_EXCEPTION, do_simple_test } }; static char *get_test_name(int test_num)
Signature · 1 line
2.35.1
PATCH v9 30/45 mm/pkeys: Introduce pks_update_exception()
In reply to [email protected]From: Ira Weiny <[email protected]>
Some PKS use cases will want to catch permissions violations with the fault callback mechanism and optionally allow the access.
The pks_set_*() calls update the protection of the current running context. They will not work to change the protections of a thread which has been interrupted. Therefore updating a thread from within an exception requires a different method.
Introduce pks_update_exception() which updates the faulted threads protections in addition to the current context.
Add documentation
---
Changes for V9
Add preemption disable around pkrs per cpu cache Update commit message
Change pkey type to u8 s/pks_saved_pkrs/pkrs
Changes for V8
Remove the concept of abandoning a pkey in favor of using the
custom fault handler via this new pks_update_exception() call
Without an abandon call there is no need for an abandon mask on
sched in, new thread creation, or within exceptions...
This now lets all invalid access' fault
Ensure that all entry points into the pks has feature checks... Place abandon fault check before the test callback to ensure
testing does not detect the double fault of the abandon code and flag it incorrectly as a fault.
Change return type of pks_handle_abandoned_pkeys() to bool
--- Documentation/core-api/protection-keys.rst | 3 ++ arch/x86/mm/pkeys.c | 58 +++++++++++++++++++--- include/linux/pks.h | 5 ++ 3 files changed, 58 insertions(+), 8 deletions(-)
Documentation/core-api/protection-keys.rst +3 -0
@@ -149,6 +149,9 @@ Changing permissions of individual keys .. kernel-doc:: include/linux/pks.h :identifiers: pks_set_readwrite pks_set_noaccess +.. kernel-doc:: arch/x86/mm/pkeys.c + :identifiers: pks_update_exception + Overriding Default Fault Behavior ---------------------------------
arch/x86/mm/pkeys.c +50 -8
77 lines not on this page · raw message
include/linux/pks.h +5 -0
@@ -9,6 +9,7 @@#include <uapi/asm-generic/mman-common.h> void pks_update_protection (u8 pkey,u8 protection); +void pks_update_exception (struct pt_regs *regs,u8 pkey,u8 protection);/** * pks_set_noaccess() - Disable all access to the domain
@@ -41,6 +42,10 @@ typedef bool (*pks_key_callback)(struct pt_regs *regs, unsigned long address,static inline void pks_set_noaccess (u8 pkey) {}static inline void pks_set_readwrite (u8 pkey) {} +static inline void pks_update_exception (struct pt_regs *regs, +u8 pkey, +u8 protection) +{ }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.35.1
PATCH v9 31/45 mm/pkeys: PKS testing, test pks_update_exception()
In reply to [email protected]From: Ira Weiny <[email protected]>
A common use case for the custom fault callbacks will be for the callback to warn of the violation and relax the permissions rather than crash the kernel. pks_update_exception() was added for this purpose.
Add a test which uses pks_update_exception() to clear the pkey permissions. Verify that the permissions are changed in the interrupted thread.
---
Changes for V9
Update the commit message
Clean up test name
Add test_pks support
s/pks_mk_*/pks_set_*/
Simplify the use of globals for the faults
From Rick Edgecombe
Use WRITE_ONCE to protect against races with the fault handler s/RUN_FAULT_ABANDON/RUN_FAULT_CALLBACK
Changes for V8
New test developed just to double check for regressions while reworking the code.
--- lib/pks/pks_test.c | 60 ++++++++++++++++++++++++++ tools/testing/selftests/x86/test_pks.c | 5 ++- 2 files changed, 64 insertions(+), 1 deletion(-)
lib/pks/pks_test.c +60 -0
@@ -49,6 +49,7 @@#define ARM_CTX_SWITCH 2#define CHECK_CTX_SWITCH 3#define RUN_EXCEPTION 4 +#define RUN_EXCEPTION_UPDATE 5#define RUN_CRASH_TEST 9DECLARE_PER_CPU (u32, pkrs_cache);
@@ -64,6 +65,7 @@ struct pks_test_ctx {void *test_page;bool fault_seen;bool validate_exp_handling; +bool validate_update_exp; };static bool check_pkey_val (u32 pk_reg,u8 pkey,u32 expected)
@@ -164,6 +166,16 @@ static void validate_exception(struct pks_test_ctx *ctx, u32 thread_pkrs) } } +static bool handle_update_exception (struct pt_regs *regs,struct pks_test_ctx *ctx) +{ +pr_debug ("Updating pkey %d during exception\n" , ctx->pkey); + + ctx->fault_seen = true; +pks_update_exception (regs, ctx->pkey,0 ); + +return true; +} +/* Global data protected by test_run_lock */ struct pks_test_ctx *g_ctx_under_test;
@@ -190,6 +202,9 @@ bool pks_test_fault_callback(struct pt_regs *regs, unsigned long address,if (!g_ctx_under_test)return false; +if (g_ctx_under_test->validate_update_exp) +return handle_update_exception (regs, g_ctx_under_test); +if (g_ctx_under_test->validate_exp_handling) {validate_exception (g_ctx_under_test, pkrs);/*
@@ -518,6 +533,47 @@ static void check_ctx_switch(struct pks_session_data *sd) } } +static bool run_exception_update (struct pks_session_data *sd) +{ +struct pks_test_ctx *ctx; + + ctx =alloc_ctx (PKS_KEY_TEST ); +if (IS_ERR (ctx)) +return false; + +set_ctx_data (sd, ctx); + + ctx->fault_seen = false; + ctx->validate_update_exp = true; +pks_set_noaccess (ctx->pkey); + +set_context_for_fault (ctx); + +/* fault */ +memcpy (ctx->test_page, ctx->data,8 ); + +if (!ctx->fault_seen) { +pr_err ("Failed to see the callback\n" ); +return false; + } + + ctx->fault_seen = false; + ctx->validate_update_exp = false; + +set_context_for_fault (ctx); + +/* no fault */ +memcpy (ctx->test_page, ctx->data,8 ); + +if (ctx->fault_seen) { +pr_err ("Pkey %d failed to be set RD/WR in the callback\n" , + ctx->pkey); +return false; + } + +return true; +} +static ssize_t pks_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos) {
@@ -584,6 +640,10 @@ static ssize_t pks_write_file(struct file *file, const char __user *user_buf,pr_debug ("Exception checking\n" ); sd->last_test_pass =run_exception_test (file->private_data);break ; +case RUN_EXCEPTION_UPDATE : +pr_debug ("Fault clear test\n" ); + sd->last_test_pass =run_exception_update (file->private_data); +break ;default :pr_debug ("Unknown test\n" ); sd->last_test_pass = false;
tools/testing/selftests/x86/test_pks.c +4 -1
@@ -36,6 +36,7 @@#define ARM_CTX_SWITCH "2"#define CHECK_CTX_SWITCH "3"#define RUN_EXCEPTION "4" +#define RUN_EXCEPTION_UPDATE "5"#define RUN_CRASH_TEST "9"time_t g_start_time;
@@ -63,6 +64,7 @@ enum {TEST_SINGLE ,TEST_CTX_SWITCH ,TEST_EXCEPTION , +TEST_FAULT_CALLBACK ,MAX_TESTS , } tests;
@@ -77,7 +79,8 @@ struct test_item { { "check_defaults", CHECK_DEFAULTS, do_simple_test }, { "single", RUN_SINGLE, do_simple_test }, { "context_switch", ARM_CTX_SWITCH, do_context_switch }, - { "exception", RUN_EXCEPTION, do_simple_test } + { "exception", RUN_EXCEPTION, do_simple_test }, + { "exception_update", RUN_EXCEPTION_UPDATE, do_simple_test } };static char *get_test_name (int test_num)
Signature · 1 line
2.35.1
PATCH v9 32/45 mm/pkeys: PKS testing, add test for all keys
In reply to [email protected]From: Ira Weiny <[email protected]>
To help test hardware and qemu it is necessary to be able to run through all the available pkeys and run the access checks. However, running this test will conflict with normal PKS consumers.
Make a test, which is mutually exclusive from all other PKS consumers, that loops through all the pkeys and tests the various access modes.
Update the documentation.
---
Changes for V9
Update commit message
Create ENABLE_PKS_CONSUMER Kconfig to make this test mutually
exclusive with any other pks consumer
Changes for V8
Split this off from the large testing patch Remove debugging version
--- Documentation/core-api/protection-keys.rst | 12 +++---- arch/x86/mm/pkeys.c | 10 ++++++ include/linux/pks-keys.h | 5 +++ lib/Kconfig.debug | 21 +++++++++++ lib/pks/pks_test.c | 41 +++++++++++++++++++++- mm/Kconfig | 9 +++++ tools/testing/selftests/x86/test_pks.c | 5 ++- 7 files changed, 95 insertions(+), 8 deletions(-)
Documentation/core-api/protection-keys.rst +6 -6
@@ -117,20 +117,20 @@ Kconfig ------- Kernel users intending to use PKS support should depend on -ARCH_HAS_SUPERVISOR_PKEYS, and select ARCH_ENABLE_SUPERVISOR_PKEYS to turn on -this support within the core. For example: +ARCH_HAS_SUPERVISOR_PKEYS, and select ARCH_ENABLE_PKS_CONSUMER to turn on this +support within the core. For example: .. code-block:: c config MY_NEW_FEATURE depends on ARCH_HAS_SUPERVISOR_PKEYS - select ARCH_ENABLE_SUPERVISOR_PKEYS + select ARCH_ENABLE_PKS_CONSUMER This will make "MY_NEW_FEATURE" unavailable unless the architecture sets ARCH_HAS_SUPERVISOR_PKEYS. It also makes it possible for multiple independent -features to "select ARCH_ENABLE_SUPERVISOR_PKEYS". If no features enable PKS -by selecting ARCH_ENABLE_SUPERVISOR_PKEYS, PKS support will not be compiled -into the kernel. +features to "select ARCH_ENABLE_PKS_CONSUMER". If no features enable PKS by +selecting ARCH_ENABLE_PKS_CONSUMER, PKS support will not be compiled into the +kernel. PKS Key Allocation ------------------
arch/x86/mm/pkeys.c +10 -0
@@ -243,12 +243,22 @@ __static_or_pks_test DEFINE_PER_CPU(u32, pkrs_cache);* #endif * }; */ +#ifndef CONFIG_PKS_TEST_ALL_KEYS +static const pks_key_callback pks_key_callbacks[PKS_KEY_MAX ] = { #ifdef CONFIG_PKS_TEST [PKS_KEY_TEST ] = pks_test_fault_callback, #endif }; +#else /* CONFIG_PKS_TEST_ALL_KEYS */ + +static const pks_key_callback pks_key_callbacks[PKS_KEY_MAX ] = { + [1 ... (PKS_KEY_MAX -1 )] = pks_test_fault_callback, +}; + +#endif +static bool pks_call_fault_callback (struct pt_regs *regs,unsigned long address,bool write,u16 key) {
include/linux/pks-keys.h +5 -0
@@ -63,6 +63,11 @@#define PKS_KEY_TEST PKS_NEW_KEY(PKS_KEY_DEFAULT, CONFIG_PKS_TEST)#define PKS_KEY_MAX PKS_NEW_KEY(PKS_KEY_TEST, 1) +#ifdef CONFIG_PKS_TEST_ALL_KEYS +#undef PKS_KEY_MAX +#define PKS_KEY_MAX PKS_NUM_PKEYS +#endif +/* PKS_KEY_DEFAULT_INIT must be RW */ #define PKS_KEY_DEFAULT_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_DEFAULT, RW, 1)#define PKS_KEY_TEST_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_TEST, AD, \
lib/Kconfig.debug +21 -0
@@ -2685,6 +2685,12 @@ config HYPERV_TESTING help Select this option to enable Hyper-V vmbus testing. +# PKS_TEST is a special PKS consumer and therefore sets +# ARCH_ENABLE_SUPERVISOR_PKEYS directly rather than through +# ARCH_ENABLE_PKS_CONSUMER +# +# This allows PKS_TEST_ALL_KEYS to remain mutially exclusive to any real PKS +# consumer config PKS_TEST bool "PKey (S)upervisor testing" depends on ARCH_HAS_SUPERVISOR_PKEYS
@@ -2697,6 +2703,21 @@ config PKS_TEST If unsure, say N. +config PKS_TEST_ALL_KEYS + bool "PKS test all keys" + depends on (PKS_TEST && !ARCH_ENABLE_PKS_CONSUMER) + help + Select this option to enable testing of all the PKS keys available in + the architecture. This option is mutually exclusive with PKS + consumers other than PKS_TEST. This is because it will consume all + PKS keys for testing purposes. + + Answer N if you don't know what supervisor keys are or want to have + supervisor keys available for other consumers. + + If unsure, say N. + + endmenu # "Kernel Testing and Coverage" source "Documentation/Kconfig"
lib/pks/pks_test.c +40 -1
64 lines not on this page · raw message
mm/Kconfig +9 -0
@@ -806,6 +806,15 @@ config ARCH_HAS_PKEYS bool config ARCH_HAS_SUPERVISOR_PKEYS bool + +config ARCH_ENABLE_PKS_CONSUMER + select ARCH_ENABLE_SUPERVISOR_PKEYS + bool + +# WARNING Do not set ARCH_ENABLE_SUPERVISOR_PKEYS directly use +# ARCH_ENABLE_PKS_CONSUMER instead. +# +# See the PKey (S)upervisor testing (PKS_TEST) config option for details. config ARCH_ENABLE_SUPERVISOR_PKEYS bool
tools/testing/selftests/x86/test_pks.c +4 -1
@@ -37,6 +37,7 @@#define CHECK_CTX_SWITCH "3"#define RUN_EXCEPTION "4"#define RUN_EXCEPTION_UPDATE "5" +#define RUN_ALL_KEYS "6"#define RUN_CRASH_TEST "9"time_t g_start_time;
@@ -65,6 +66,7 @@ enum {TEST_CTX_SWITCH ,TEST_EXCEPTION ,TEST_FAULT_CALLBACK , +TEST_ALL ,MAX_TESTS , } tests;
@@ -80,7 +82,8 @@ struct test_item { { "single", RUN_SINGLE, do_simple_test }, { "context_switch", ARM_CTX_SWITCH, do_context_switch }, { "exception", RUN_EXCEPTION, do_simple_test }, - { "exception_update", RUN_EXCEPTION_UPDATE, do_simple_test } + { "exception_update", RUN_EXCEPTION_UPDATE, do_simple_test }, + { "run_all", RUN_ALL_KEYS, do_simple_test } };static char *get_test_name (int test_num)
Signature · 1 line
2.35.1
PATCH v9 33/45 mm/pkeys: Add pks_available()
In reply to [email protected]From: Ira Weiny <[email protected]>
If PKS is configured within the kernel but the CPU does not support PKS, the PKS calls remain safe to execute even without protection. However, adding the overhead of these calls on CPUs which don't support PKS is inefficient and best avoided.
Define pks_available() to allow users to check if PKS is enabled on the current system.
The implementation of pks_available() is placed in the asm headers while being directly exported via linux/pks.h to allow for the inline calling of cpu_feature_enabled() by consumers outside of the architecture.
---
Changes for V9
Driven by a request by Dan Williams to make this static inline
Place this in pks.h to avoid header conflicts while allowing for an optimized call to cpu_feature_enabled()
Changes for V8
s/pks_enabled/pks_available
--- Documentation/core-api/protection-keys.rst | 3 +++ arch/x86/include/asm/pks.h | 12 ++++++++++++ include/linux/pks.h | 8 ++++++++ 3 files changed, 23 insertions(+)
Documentation/core-api/protection-keys.rst +3 -0
@@ -152,6 +152,9 @@ Changing permissions of individual keys .. kernel-doc:: arch/x86/mm/pkeys.c :identifiers: pks_update_exception +.. kernel-doc:: arch/x86/include/asm/pks.h + :identifiers: pks_available + Overriding Default Fault Behavior ---------------------------------
arch/x86/include/asm/pks.h +12 -0
@@ -2,8 +2,20 @@#ifndef _ASM_X86_PKS_H#define _ASM_X86_PKS_H +#include <asm/cpufeature.h> +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/** +* pks_available() - Is PKS available on this system +* +* Return if PKS is currently supported and enabled on this system. +*/ +static inline bool pks_available (void ) +{ +return cpu_feature_enabled (X86_FEATURE_PKS ); +} +void pks_setup (void );void x86_pkrs_load (struct thread_struct *thread);void pks_save_pt_regs (struct pt_regs *regs);
include/linux/pks.h +8 -0
@@ -8,6 +8,9 @@#include <uapi/asm-generic/mman-common.h> +#include <asm/pks.h> + +bool pks_available (void );void pks_update_protection (u8 pkey,u8 protection);void pks_update_exception (struct pt_regs *regs,u8 pkey,u8 protection);
@@ -40,6 +43,11 @@ typedef bool (*pks_key_callback)(struct pt_regs *regs, unsigned long address,#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ +static inline bool pks_available (void ) +{ +return false; +} +static inline void pks_set_noaccess (u8 pkey) {}static inline void pks_set_readwrite (u8 pkey) {} static inline void pks_update_exception(struct pt_regs *regs,
Signature · 1 line
2.35.1
PATCH v9 34/45 memremap_pages: Add Kconfig for DEVMAP_ACCESS_PROTECTION
In reply to [email protected]From: Ira Weiny <[email protected]>
The persistent memory (PMEM) driver uses the memremap_pages facility to provide 'struct page' metadata (vmemmap) for PMEM. Given that PMEM capacity may be orders of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Unlike stray writes to System RAM, which may result in a crash or other undesirable behavior, stray writes to PMEM additionally are more likely to result in permanent data loss. Reboot is not a remediation for PMEM corruption like it is for System RAM.
Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, and direct-I/O to a DAX page), it is amenable to supervisor pkey protection.
Add a Kconfig option to configure additional devmap protections using PKS.
Only PMEM which is advertised to the memory subsystem needs this protection. Therefore, the feature depends on NVDIMM_PFN.
---
Changes for V9
Change this to enable arch pks consumer for mutual exclusion
with testing all pkeys
From Dan Williams
Default to no
Clean up commit message
Changes for V8
Split this out from
[PATCH V7 13/18] memremap_pages: Add access protection via supervisor Protection Keys (PKS)
--- mm/Kconfig | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
mm/Kconfig +18 -0
@@ -776,6 +776,24 @@ config ZONE_DEVICE If FS_DAX is enabled, then say Y. +config DEVMAP_ACCESS_PROTECTION + bool "Access protection for memremap_pages()" + depends on NVDIMM_PFN + depends on ARCH_HAS_SUPERVISOR_PKEYS + select ARCH_ENABLE_PKS_CONSUMER + default n + + help + Enable extra protections on device memory. This protects against + unintended access to devices such as a stray writes. This feature is + particularly useful to protect against corruption of persistent + memory. + + This depends on architecture support of supervisor PKeys and has no + overhead if the architecture does not support them. + + If you have persistent memory say 'Y'. + config DEV_PAGEMAP_OPS bool
Signature · 1 line
2.35.1
PATCH v9 35/45 memremap_pages: Introduce pgmap_protection_available()
In reply to [email protected]From: Ira Weiny <[email protected]>
PMEM will flag additional dev_pagemap protection through (struct dev_pagemap)->flags. However, it is more efficient to know if that protection is available prior to requesting it and failing the mapping.
Define pgmap_protection_available() to check if protection is available prior to being requested. The name of pgmap_protection_available() was specifically chosen to isolate the implementation of the protection from higher level users.
---
Changes for V9
Clean up commit message
From Dan Williams
make call stack static inline throughout this call and
pks_available() such that callers calls cpu_feature_enabled() directly
Changes for V8
Split this out to it's own patch. s/pgmap_protection_enabled/pgmap_protection_available
--- include/linux/mm.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
include/linux/mm.h +17 -0
@@ -31,6 +31,7 @@#include <linux/sizes.h> #include <linux/sched.h> #include <linux/pgtable.h> +#include <linux/pks.h> #include <linux/kasan.h> struct mempolicy ;
@@ -1143,6 +1144,22 @@ static inline bool is_pci_p2pdma_page(const struct page *page) page->pgmap->type ==MEMORY_DEVICE_PCI_P2PDMA ; } +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + +static inline bool pgmap_protection_available (void ) +{ +return pks_available (); +} + +#else + +static inline bool pgmap_protection_available (void ) +{ +return false; +} + +#endif /* CONFIG_DEVMAP_ACCESS_PROTECTION */ +/* 127: arbitrary random number, small enough to assemble well */ #define folio_ref_zero_or_close_to_overflow (folio) \ ((unsigned int) folio_ref_count(folio) + 127u <= 127u)
Signature · 1 line
2.35.1
PATCH v9 36/45 memremap_pages: Introduce a PGMAP_PROTECTION flag
In reply to [email protected]From: Ira Weiny <[email protected]>
The persistent memory (PMEM) driver uses the memremap_pages facility to provide 'struct page' metadata (vmemmap) for PMEM. Given that PMEM capacity maybe orders of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Unlike stray writes to System RAM, which may result in a crash or other undesirable behavior, stray writes to PMEM additionally are more likely to result in permanent data loss. Reboot is not a remediation for PMEM corruption like it is for System RAM.
Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, and direct-I/O to a DAX page), it is amenable to supervisor pkey protection.
Some systems which have configured DEVMAP_ACCESS_PROTECTION may not have PMEM installed. Or the PMEM may not be mapped into the direct map. In addition, some callers of memremap_pages() will not want the mapped pages protected.
Define a new PGMAP flag to distinguish page maps which are protected. Use this flag to enable runtime protection support. A static key is used to optimize the runtime support.
Specifying this flag on a system which can't support protections will fail. Callers are expected to check if protections are supported via pgmap_protection_available(). It was considered to have callers specify the flag and check if the dev_pagemap object returned was protected or not. But this was considered less efficient than a direct check beforehand.
---
Changes for V9
Clean up commit message
Changes for V8
Split this out into it's own patch
--- include/linux/memremap.h | 1 + mm/memremap.c | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+)
include/linux/memremap.h +1 -0
@@ -80,6 +80,7 @@ struct dev_pagemap_ops { };#define PGMAP_ALTMAP_VALID (1 << 0) +#define PGMAP_PROTECTION (1 << 1)/** * struct dev_pagemap - metadata for ZONE_DEVICE mappings
mm/memremap.c +40 -0
@@ -63,6 +63,37 @@ static void devmap_managed_enable_put(struct dev_pagemap *pgmap) }#endif /* CONFIG_DEV_PAGEMAP_OPS */ +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + +/* +* Note; all devices which have asked for protections share the same key. The +* key may, or may not, have been provided by the core. If not, protection +* will be disabled. The key acquisition is attempted when the first ZONE +* DEVICE requests it and freed when all zones have been unmapped. +* +* Also this must be EXPORT_SYMBOL rather than EXPORT_SYMBOL_GPL because it is +* intended to be used in the kmap API. +*/ +DEFINE_STATIC_KEY_FALSE (dev_pgmap_protection_static_key); +EXPORT_SYMBOL (dev_pgmap_protection_static_key); + +static void devmap_protection_enable (void ) +{ +static_branch_inc (&dev_pgmap_protection_static_key); +} + +static void devmap_protection_disable (void ) +{ +static_branch_dec (&dev_pgmap_protection_static_key); +} + +#else /* !CONFIG_DEVMAP_ACCESS_PROTECTION */ + +static void devmap_protection_enable (void ) { } +static void devmap_protection_disable (void ) { } + +#endif /* CONFIG_DEVMAP_ACCESS_PROTECTION */ +static void pgmap_array_delete (struct range *range) { xa_store_range(&pgmap_array, PHYS_PFN(range->start), PHYS_PFN(range->end),
@@ -162,6 +193,9 @@ void memunmap_pages(struct dev_pagemap *pgmap)WARN_ONCE (pgmap->altmap.alloc,"failed to free all reserved pages\n" );devmap_managed_enable_put (pgmap); + +if (pgmap->flags &PGMAP_PROTECTION ) +devmap_protection_disable (); }EXPORT_SYMBOL_GPL (memunmap_pages);
@@ -308,6 +342,12 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)if (WARN_ONCE (!nr_range,"nr_range must be specified\n" ))return ERR_PTR (-EINVAL ); +if (pgmap->flags &PGMAP_PROTECTION ) { +if (!pgmap_protection_available ()) +return ERR_PTR (-EINVAL ); +devmap_protection_enable (); + } +switch (pgmap->type) {case MEMORY_DEVICE_PRIVATE :if (!IS_ENABLED (CONFIG_DEVICE_PRIVATE )) {
Signature · 1 line
2.35.1
PATCH v9 37/45 memremap_pages: Introduce devmap_protected()
In reply to [email protected]From: Ira Weiny <[email protected]>
Consumers of protected dev_pagemaps can check the PGMAP_PROTECTION flag to see if the devmap is protected. However, most contexts will have a struct page not the pagemap structure directly.
Define devmap_protected() to determine if a page is part of a dev_pagemap mapping and if the page is protected by additional protections.
--- include/linux/mm.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
include/linux/mm.h +17 -0
@@ -1151,6 +1151,23 @@ static inline bool pgmap_protection_available(void) return pks_available(); } +DECLARE_STATIC_KEY_FALSE (dev_pgmap_protection_static_key); + +/* +* devmap_protected() requires a reference on the page to ensure there is no +* races with dev_pagemap tear down. +*/ +static inline bool devmap_protected (struct page *page) +{ +if (!static_branch_unlikely (&dev_pgmap_protection_static_key)) +return false; +if (!is_zone_device_page (page)) +return false; +if (page->pgmap->flags &PGMAP_PROTECTION ) +return true; +return false; +} +#else static inline bool pgmap_protection_available (void )
Signature · 1 line
2.35.1
PATCH v9 38/45 memremap_pages: Reserve a PKS pkey for eventual use by PMEM
In reply to [email protected]From: Ira Weiny <[email protected]>
Reserve a pkey for use by the memmap facility and set the default protections to Access Disabled.
---
Changes for V9
Adjust for new key allocation
From Dave Hansen
use pkey
--- include/linux/pks-keys.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/pks-keys.h +7 -2
@@ -61,7 +61,9 @@/* PKS_KEY_DEFAULT must be 0 */ #define PKS_KEY_DEFAULT 0#define PKS_KEY_TEST PKS_NEW_KEY(PKS_KEY_DEFAULT, CONFIG_PKS_TEST) -#define PKS_KEY_MAX PKS_NEW_KEY(PKS_KEY_TEST, 1) +#define PKS_KEY_PGMAP_PROTECTION \ + PKS_NEW_KEY(PKS_KEY_TEST, CONFIG_DEVMAP_ACCESS_PROTECTION) +#define PKS_KEY_MAX PKS_NEW_KEY(PKS_KEY_PGMAP_PROTECTION, 1)#ifdef CONFIG_PKS_TEST_ALL_KEYS #undef PKS_KEY_MAX
@@ -72,6 +74,8 @@#define PKS_KEY_DEFAULT_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_DEFAULT, RW, 1)#define PKS_KEY_TEST_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_TEST, AD, \ CONFIG_PKS_TEST) +#define PKS_KEY_PGMAP_INIT PKS_DECLARE_INIT_VALUE(PKS_KEY_PGMAP_PROTECTION, \ + AD, CONFIG_DEVMAP_ACCESS_PROTECTION)#define PKS_ALL_AD_MASK \ GENMASK(PKS_NUM_PKEYS * PKR_BITS_PER_PKEY, \
@@ -79,7 +83,8 @@#define PKS_INIT_VALUE ((PKS_ALL_AD & PKS_ALL_AD_MASK) | \ PKS_KEY_DEFAULT_INIT | \ - PKS_KEY_TEST_INIT \ + PKS_KEY_TEST_INIT | \ + PKS_KEY_PGMAP_INIT \ )#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.35.1
PATCH v9 39/45 memremap_pages: Set PKS pkey in PTEs if requested
In reply to [email protected]From: Ira Weiny <[email protected]>
When a devmap caller requests protections, the dev_pagemap PTE's need to have a PKEY set.
When PGMAP_PROTECTIONS is requested add the pkey to the page protections.
---
Changes for V9
From Dave Hansen
use pkey
--- mm/memremap.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/memremap.c +13 -0
@@ -82,6 +82,14 @@ static void devmap_protection_enable(void)static_branch_inc (&dev_pgmap_protection_static_key); } +static pgprot_t devmap_protection_adjust_pgprot (pgprot_t prot) +{ +pgprotval_t val; + + val =pgprot_val (prot); +return __pgprot (val |_PAGE_PKEY (PKS_KEY_PGMAP_PROTECTION )); +} +static void devmap_protection_disable (void ) {static_branch_dec (&dev_pgmap_protection_static_key);
@@ -92,6 +100,10 @@ static void devmap_protection_disable(void)static void devmap_protection_enable (void ) { }static void devmap_protection_disable (void ) { } +static pgprot_t devmap_protection_adjust_pgprot (pgprot_t prot) +{ +return prot; +}#endif /* CONFIG_DEVMAP_ACCESS_PROTECTION */ static void pgmap_array_delete (struct range *range)
@@ -346,6 +358,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid)if (!pgmap_protection_available ())return ERR_PTR (-EINVAL );devmap_protection_enable (); + params.pgprot =devmap_protection_adjust_pgprot (params.pgprot); }switch (pgmap->type) {
Signature · 1 line
2.35.1
PATCH v9 40/45 memremap_pages: Define pgmap_set_{readwrite|noaccess}() calls
In reply to [email protected]From: Ira Weiny <[email protected]>
A thread that wants to access memory protected by PGMAP protections must first enable access, and then disable access when it is done.
Introduce pgmap_set_{readwrite|noaccess}() for this purpose. The two calls are destined to be used by the kmap API and take a struct page for convenience. They determine if the page is protected and, if so, perform the requested operation.
Toggling between Read/Write and No Access was chosen as it fits well with the accessibility of a kmap'ed page. Discussions did occur regarding making a finer grained mapping for Read Only but that is something which can be added at a later date.
In addition, two lower level functions are exported. They take the dev_pagemap object directly for internal consumers who have knowledge of the of the dev_pagemap.
All changes in the protections must be through the above calls. They abstract the protection implementation (currently the PKS api) from upper layer consumers.
The calls are made nestable by the use of a per task reference count. This ensures that the first call to re-enable protection does not 'break' the last access of the device memory. Expansion of the task struct is unavoidable due to the desire to maintain kmap_local_page() as non-atomic and migratable. The only other idea to track a reference count was in a per-cpu variable. However, doing so would make kmap_local_page() equivalent to kmap_atomic() which is undesirable.
Access to device memory during exceptions (#PF) is expected only from user faults. Therefore there is no need to maintain the reference count during exceptions.
NOTE: It is not anticipated that any code path will directly nest these calls. For this reason multiple reviewers, including Dan and Thomas, asked why this reference counting was needed at this level rather than in a higher level call such as kmap_local_page(). The reason is that pgmap_set_readwrite() can nest with kmap_{atomic,local_page}(). Therefore this reference counting is pushed to the lower level to ensure that any combination of calls is nestable.
---
Changes for V9
From Dan Williams
Update the commit message with details on why the thread
struct needs to be expanded.
Following on Dave Hansens suggestion for pks_mk
s/pgmap_mk_*/pgmap_set_*/
Changes for V8
Split these functions into their own patch.
This helps to clarify the commit message and usage.
--- include/linux/mm.h | 35 +++++++++++++++++++++++++++++++++++ include/linux/sched.h | 7 +++++++ init/init_task.c | 3 +++ mm/memremap.c | 14 ++++++++++++++ 4 files changed, 59 insertions(+)
include/linux/mm.h +35 -0
@@ -1168,8 +1168,43 @@ static inline bool devmap_protected(struct page *page)return false; } +void __pgmap_set_readwrite (struct dev_pagemap *pgmap); +void __pgmap_set_noaccess (struct dev_pagemap *pgmap); + +static inline bool pgmap_check_pgmap_prot (struct page *page) +{ +if (!devmap_protected (page)) +return false; + +/* +* There is no known use case to change permissions in an irq for pgmap +* pages +*/ +lockdep_assert_in_irq (); +return true; +} + +static inline void pgmap_set_readwrite (struct page *page) +{ +if (!pgmap_check_pgmap_prot (page)) +return ; +__pgmap_set_readwrite (page->pgmap); +} + +static inline void pgmap_set_noaccess (struct page *page) +{ +if (!pgmap_check_pgmap_prot (page)) +return ; +__pgmap_set_noaccess (page->pgmap); +} +#else +static inline void __pgmap_set_readwrite (struct dev_pagemap *pgmap) { } +static inline void __pgmap_set_noaccess (struct dev_pagemap *pgmap) { } +static inline void pgmap_set_readwrite (struct page *page) { } +static inline void pgmap_set_noaccess (struct page *page) { } +static inline bool pgmap_protection_available (void ) {return false;
include/linux/sched.h +7 -0
@@ -1492,6 +1492,13 @@ struct task_struct {struct callback_head l1d_flush_kill;#endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION +/* +* NOTE: pgmap_prot_count is modified within a single thread of +* execution. So it does not need to be atomic_t. +*/ +u32 pgmap_prot_count; +#endif /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct.
init/init_task.c +3 -0
@@ -209,6 +209,9 @@ struct task_struct init_task#ifdef CONFIG_SECCOMP_FILTER .seccomp = { .filter_count = ATOMIC_INIT(0) },#endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + .pgmap_prot_count = 0, +#endif };EXPORT_SYMBOL (init_task);
mm/memremap.c +14 -0
@@ -95,6 +95,20 @@ static void devmap_protection_disable(void) static_branch_dec(&dev_pgmap_protection_static_key); } +void __pgmap_set_readwrite (struct dev_pagemap *pgmap) +{ +if (!current->pgmap_prot_count++) +pks_set_readwrite (PKS_KEY_PGMAP_PROTECTION ); +} +EXPORT_SYMBOL_GPL (__pgmap_set_readwrite); + +void __pgmap_set_noaccess (struct dev_pagemap *pgmap) +{ +if (!--current->pgmap_prot_count) +pks_set_noaccess (PKS_KEY_PGMAP_PROTECTION ); +} +EXPORT_SYMBOL_GPL (__pgmap_set_noaccess); +#else /* !CONFIG_DEVMAP_ACCESS_PROTECTION */ static void devmap_protection_enable (void ) { }
Signature · 1 line
2.35.1
PATCH v9 41/45 memremap_pages: Add memremap.pks_fault_mode
In reply to [email protected]From: Ira Weiny <[email protected]>
When PKS protections for PMEM are enabled the kernel may capture stray writes, or it may capture false positive access violations. An example of a false positive access violation is a code path that neglects to call kmap_{atomic,local_page}, but is otherwise a valid access. In the false positive scenario there is no actual risk to data integrity, but the kernel still needs to make a decision as to whether to report the access violation and continue, or treat the violation as fatal. That policy decision is captured in a new pks_fault_mode kernel parameter.
2 modes are available:
'relaxed' (default) -- WARN_ONCE, removed the protections, and continuing to operate.
'strict' -- Stop kernel execution via fault. This is the most protective of the PMEM memory but may be undesirable in some configurations.
NOTE: There was some debate about if a 3rd mode called 'silent' should be available. 'silent' would be the same as 'relaxed' but not print any output. While 'silent' is nice for admins to reduce console/log output it would result in less motivation to fix invalid access to the protected pmem pages. Therefore, 'silent' is left out.
NOTE: The __param_check macro requires a type to correctly verify the values passed as the module parameter. Therefore a typedef is made of the pks_fault_modes and the checkpatch warning regarding new typedefs is ignored.
---
Changes for V9
From Dan Williams
Clarify commit message
Remove code comment regarding checkpatch
From Rick Edgecombe
Remove unnecessary initialization
Changes for V8
Use pks_update_exception() instead of abandoning the pkey. Split out pgmap_protection_flag_invalid() into a separate patch
for clarity.
From Rick Edgecombe
Fix sysfs_streq() checks
From Randy Dunlap
Fix Documentation closing parans
Changes for V7
Leverage Rick Edgecombe's fault callback infrastructure to relax invalid
uses and prevent crashes
From Dan Williams
Use sysfs_* calls for parameter
Make pgmap_disable_protection inline
Remove pfn from warn output
Remove silent parameter option
--- .../admin-guide/kernel-parameters.txt | 12 ++++ arch/x86/mm/pkeys.c | 4 ++ include/linux/mm.h | 3 + mm/memremap.c | 65 +++++++++++++++++++ 4 files changed, 84 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +12 -0
@@ -4158,6 +4158,18 @@ pirq= [SMP,APIC] Manual mp-table setup See Documentation/x86/i386/IO-APIC.rst. + memremap.pks_fault_mode= [X86] Control the behavior of page map + protection violations. + (depends on CONFIG_DEVMAP_ACCESS_PROTECTION) + + Format: { relaxed | strict } + + relaxed - Print a warning, disable the protection and + continue execution. + strict - Stop kernel execution via fault + + default: relaxed + plip= [PPT,NET] Parallel port network link Format: { parport<nr> | timid | 0 } See also Documentation/admin-guide/parport.rst.
arch/x86/mm/pkeys.c +4 -0
@@ -8,6 +8,7 @@#include <linux/pkeys.h> /* PKEY_* */ #include <linux/pks.h> #include <linux/pks-keys.h> +#include <linux/mm.h> /* fault callback */ #include <uapi/asm-generic/mman-common.h> #include <asm/cpufeature.h> /* boot_cpu_has, ... */
@@ -249,6 +250,9 @@ static const pks_key_callback pks_key_callbacks[PKS_KEY_MAX] = { #ifdef CONFIG_PKS_TEST [PKS_KEY_TEST ] = pks_test_fault_callback, #endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + [PKS_KEY_PGMAP_PROTECTION] = pgmap_pks_fault_callback, +#endif };#else /* CONFIG_PKS_TEST_ALL_KEYS */
include/linux/mm.h +3 -0
@@ -1198,6 +1198,9 @@ static inline void pgmap_set_noaccess(struct page *page) __pgmap_set_noaccess(page->pgmap); } +bool pgmap_pks_fault_callback (struct pt_regs *regs,unsigned long address, +bool write); +#else static inline void __pgmap_set_readwrite (struct dev_pagemap *pgmap) { }
mm/memremap.c +65 -0
77 lines not on this page · raw message
Signature · 1 line
2.35.1
PATCH v9 42/45 kmap: Make kmap work for devmap protected pages
In reply to [email protected]From: Ira Weiny <[email protected]>
Today, kmap_{local_page,atomic}() handle granting access to HIGHMEM pages without the caller needing to know if the page is HIGHMEM, or not. Use that existing infrastructure to grant access to PGMAP (PKS) protected pages.
kmap_{local_page,atomic}() are both thread local mappings so they work well with the thread specific protections available within PKS.
On the other hand, the kmap() call is not changed. kmap() allows for a mapping to be shared with other threads, while PKS protections operate on a thread local basis. For this reason, and the desire to move away from mappings like this, kmap() is left unsupported.
This behavior is safe because neither of the 2 current DAX-capable filesystems (ext4 and xfs) perform such global mappings. And known device drivers that would handle devmap pages are not using kmap(). Any future filesystems that gain DAX support, or device drivers wanting to support devmap protected pages will need to use kmap_local_page().
Note: HIGHMEM support is mutually exclusive with PGMAP protection. The rationale is mainly to reduce complexity, but also because direct-map exposure is already mitigated by default on HIGHMEM systems because by definition HIGHMEM systems do not have large capacities of memory in the direct map.
---
Changes for V9
From Dan Williams
Update commit message
Clarify why kmap is not 'compatible' with PKS Explain the HIGHMEM system exclusion more
Remove pgmap_protection_flag_invalid() from kmap s/pks_mk*/pks_set*/
Changes for V8
Reword commit message
--- include/linux/highmem-internal.h | 4 ++++ mm/Kconfig | 1 + 2 files changed, 5 insertions(+)
include/linux/highmem-internal.h +4 -0
@@ -174,6 +174,7 @@ static inline void kunmap(struct page *page)static inline void *kmap_local_page (struct page *page) { +pgmap_set_readwrite (page);return page_address (page); }
@@ -197,6 +198,7 @@ static inline void __kunmap_local(void *addr)#ifdef ARCH_HAS_FLUSH_ON_KUNMAP kunmap_flush_on_unmap (addr);#endif +pgmap_set_noaccess (kmap_to_page (addr)); }static inline void *kmap_atomic (struct page *page)
@@ -206,6 +208,7 @@ static inline void *kmap_atomic(struct page *page)else preempt_disable ();pagefault_disable (); +pgmap_set_readwrite (page);return page_address (page); }
@@ -224,6 +227,7 @@ static inline void __kunmap_atomic(void *addr)#ifdef ARCH_HAS_FLUSH_ON_KUNMAP kunmap_flush_on_unmap (addr);#endif +pgmap_set_noaccess (kmap_to_page (addr));pagefault_enable ();if (IS_ENABLED (CONFIG_PREEMPT_RT ))migrate_enable ();
mm/Kconfig +1 -0
@@ -779,6 +779,7 @@ config ZONE_DEVICE config DEVMAP_ACCESS_PROTECTION bool "Access protection for memremap_pages()" depends on NVDIMM_PFN + depends on !HIGHMEM depends on ARCH_HAS_SUPERVISOR_PKEYS select ARCH_ENABLE_PKS_CONSUMER default n
Signature · 1 line
2.35.1
PATCH v9 43/45 dax: Stray access protection for dax_direct_access()
In reply to [email protected]From: Ira Weiny <[email protected]>
dax_direct_access() provides a way to obtain the direct map address of PMEM memory. With the new devmap protections the use of this address needs to be bracketed by calls to enable and disable protection of those pages. These calls only need to be used to guard actual access to the memory. Other uses of dax_direct_access() do not need to use these guards.
Introduce 2 new calls dax_set_readwrite() and dax_set_noaccess(). Bracket all uses of the address returned by dax_direct_access() with those calls.
For consumers who require a permanent address to the dax device, such as the DM write cache, dax_map_protected() is used to query for additional protections.
Update the DM write cache code to create a permanent mapping if dax_map_protected() is true.
---
Changes for V9
Do not add a new dax operation. Instead teach struct dax_device
about the dev_pagemap PGMAP_PROTECTION flag and call the ops directly if needed.
s/dax_mk_*/dax_set_*/
Changes for V8
Rebase changes on 5.17-rc1
Clean up the cover letter
dax_read_lock() is not required s/dax_protected()/dax_map_protected()/
Testing revealed a dax_flush() which was not properly protected.
Changes for V7
Rework cover letter.
Do not include a FS_DAX_LIMITED restriction for dcss. It will
simply not implement the protection and there is no need to special case this.
Clean up commit message because I did not originally understand the nuance of the s390 device.
Introduce dax_{protected,mk_readwrite,mk_noaccess}() From Dan Williams
Remove old clean up cruft from previous versions Remove map_protected
Remove 'global' parameters all calls
--- drivers/dax/super.c | 59 ++++++++++++++++++++++++++++++++++++++ drivers/md/dm-writecache.c | 8 +++++- fs/dax.c | 8 ++++++ fs/fuse/virtio_fs.c | 2 ++ include/linux/dax.h | 5 ++++ 5 files changed, 81 insertions(+), 1 deletion(-)
drivers/dax/super.c +59 -0
@@ -28,6 +28,7 @@ struct dax_device {void *private;unsigned long flags;const struct dax_operations *ops; +struct dev_pagemap *pgmap; };static dev_t dax_devt;
@@ -117,6 +118,8 @@ enum dax_device_flags {* @pgoff: offset in pages from the start of the device to translate * @nr_pages: number of consecutive pages caller can handle relative to @pfn * @kaddr: output parameter that returns a virtual address mapping of pfn +* Direct access through this pointer must be guarded by calls to +* dax_set_{readwrite,noaccess}() * @pfn: output parameter that returns an absolute pfn translation of @pgoff * * Return: negative errno if an error occurs, otherwise the number of
@@ -209,6 +212,56 @@ void dax_flush(struct dax_device *dax_dev, void *addr, size_t size)#endif EXPORT_SYMBOL_GPL (dax_flush); +bool dax_map_protected (struct dax_device *dax_dev) +{ +struct dev_pagemap *pgmap = dax_dev->pgmap; + +if (!dax_alive (dax_dev)) +return false; + +return pgmap && (pgmap->flags &PGMAP_PROTECTION ); +} +EXPORT_SYMBOL_GPL (dax_map_protected); + +/** +* dax_set_readwrite() - make protected dax devices read/write +* @dax_dev: the dax device representing the memory to access +* +* Any access of the kaddr memory returned from dax_direct_access() must be +* guarded by dax_set_readwrite() and dax_set_noaccess(). This ensures that any +* dax devices which have additional protections are allowed to relax those +* protections for the thread using this memory. +* +* NOTE these calls must be contained within a single thread of execution and +* both must be guarded by dax_read_lock() Which is also a requirement for +* dax_direct_access() anyway. +*/ +void dax_set_readwrite (struct dax_device *dax_dev) +{ +if (!dax_map_protected (dax_dev)) +return ; + +__pgmap_set_readwrite (dax_dev->pgmap); +} +EXPORT_SYMBOL_GPL (dax_set_readwrite); + +/** +* dax_set_noaccess() - restore protection to dax devices if needed +* @dax_dev: the dax device representing the memory to access +* +* See dax_direct_access() and dax_set_readwrite() +* +* NOTE Must be called prior to dax_read_unlock() +*/ +void dax_set_noaccess (struct dax_device *dax_dev) +{ +if (!dax_map_protected (dax_dev)) +return ; + +__pgmap_set_noaccess (dax_dev->pgmap); +} +EXPORT_SYMBOL_GPL (dax_set_noaccess); +void dax_write_cache (struct dax_device *dax_dev,bool wc) {if (wc)
@@ -248,6 +301,12 @@ void set_dax_nomc(struct dax_device *dax_dev) }EXPORT_SYMBOL_GPL (set_dax_nomc); +void set_dax_pgmap (struct dax_device *dax_dev,struct dev_pagemap *pgmap) +{ + dax_dev->pgmap = pgmap; +} +EXPORT_SYMBOL_GPL (set_dax_pgmap); +bool dax_alive (struct dax_device *dax_dev) {lockdep_assert_held (&dax_srcu);
drivers/md/dm-writecache.c +7 -1
@@ -297,7 +297,13 @@ static int persistent_memory_claim(struct dm_writecache *wc) r = -EOPNOTSUPP ; goto err2; } -if (da != p) { + +/* +* Force the write cache to map the pages directly if the dax device +* mapping is protected or if the number of pages returned was not what +* was requested. +*/ +if (dax_map_protected (wc->ssd_dev->dax_dev) || da != p) {long i; wc->memory_map =NULL ; pages =kvmalloc_array (p,sizeof (struct page *),GFP_KERNEL );
fs/dax.c +8 -0
@@ -728,7 +728,9 @@ static int copy_cow_page_dax(struct vm_fault *vmf, const struct iomap_iter *iterreturn rc; } vto =kmap_atomic (vmf->cow_page); +dax_set_readwrite (iter->iomap.dax_dev);copy_user_page (vto, kaddr, vmf->address, vmf->cow_page); +dax_set_noaccess (iter->iomap.dax_dev);kunmap_atomic (vto);dax_read_unlock (id);return 0 ;
@@ -937,8 +939,10 @@ static int dax_writeback_one(struct xa_state *xas, struct dax_device *dax_dev, count =1UL <<dax_entry_order (entry); index = xas->xa_index & ~(count -1 ); +dax_set_readwrite (dax_dev);dax_entry_mkclean (mapping, index, pfn);dax_flush (dax_dev,page_address (pfn_to_page (pfn)), count *PAGE_SIZE ); +dax_set_noaccess (dax_dev);/* * After we have flushed the cache, we can clear the dirty tag. There * cannot be new dirty data in the pfn after the flush has completed as
@@ -1125,8 +1129,10 @@ static int dax_memzero(struct dax_device *dax_dev, pgoff_t pgoff, ret =dax_direct_access (dax_dev, pgoff,1 , &kaddr,NULL );if (ret >0 ) { +dax_set_readwrite (dax_dev);memset (kaddr + offset,0 , size);dax_flush (dax_dev, kaddr + offset, size); +dax_set_noaccess (dax_dev); }return ret; }
@@ -1260,12 +1266,14 @@ static loff_t dax_iomap_iter(const struct iomap_iter *iomi,if (map_len > end - pos) map_len = end - pos; +dax_set_readwrite (dax_dev);if (iov_iter_rw (iter) ==WRITE ) xfer =dax_copy_from_iter (dax_dev, pgoff, kaddr, map_len, iter);else xfer =dax_copy_to_iter (dax_dev, pgoff, kaddr, map_len, iter); +dax_set_noaccess (dax_dev); pos += xfer; length -= xfer;
fs/fuse/virtio_fs.c +2 -0
@@ -774,8 +774,10 @@ static int virtio_fs_zero_page_range(struct dax_device *dax_dev, rc =dax_direct_access (dax_dev, pgoff, nr_pages, &kaddr,NULL );if (rc <0 )return rc; +dax_set_readwrite (dax_dev);memset (kaddr,0 , nr_pages <<PAGE_SHIFT );dax_flush (dax_dev, kaddr, nr_pages <<PAGE_SHIFT ); +dax_set_noaccess (dax_dev);return 0 ; }
include/linux/dax.h +5 -0
@@ -91,6 +91,7 @@ static inline bool daxdev_mapping_supported(struct vm_area_struct *vma,void set_dax_nocache (struct dax_device *dax_dev);void set_dax_nomc (struct dax_device *dax_dev); +void set_dax_pgmap (struct dax_device *dax_dev,struct dev_pagemap *pgmap);struct writeback_control ;#if defined(CONFIG_BLOCK ) && defined(CONFIG_FS_DAX )
@@ -187,6 +188,10 @@ int dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff, size_t nr_pages);void dax_flush (struct dax_device *dax_dev,void *addr,size_t size); +bool dax_map_protected (struct dax_device *dax_dev); +void dax_set_readwrite (struct dax_device *dax_dev); +void dax_set_noaccess (struct dax_device *dax_dev); +ssize_t dax_iomap_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops); vm_fault_t dax_iomap_fault(struct vm_fault *vmf, enum page_entry_size pe_size,
Signature · 1 line
2.35.1
PATCH v9 44/45 nvdimm/pmem: Enable stray access protection
In reply to [email protected]From: Ira Weiny <[email protected]>
The persistent memory (PMEM) driver uses the memremap_pages facility to provide 'struct page' metadata (vmemmap) for PMEM. Given that PMEM capacity maybe orders of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Unlike stray writes to System RAM, which may result in a crash or other undesirable behavior, stray writes to PMEM additionally are more likely to result in permanent data loss. Reboot is not a remediation for PMEM corruption like it is for System RAM.
Now that all valid kernel access' to PMEM have been annotated with {__}pgmap_set_{readwrite,noaccess}() PGMAP_PROTECTION is safe to enable in the pmem layer.
Set PGMAP_PROTECTION if pgmap protections are available and set the pgmap property of the dax device for it's use.
Internally, the pmem driver uses a cached virtual address, pmem->virt_addr (pmem_addr). Call __pgmap_set_{readwrite,noaccess}() directly when PGMAP_PROTECTION is active on those mappings.
---
Changes for V9
Remove the dax operations and pass the pgmap to the dax_device for its use. s/pgmap_mk_*/pgmap_set_*/ s/pmem_mk_*/pmem_set_*/
Changes for V8
Rebase to 5.17-rc1
Remove global param
Add internal structure which uses the pmem device and pgmap
device directly in the *_mk_*() calls.
Add pmem dax ops callbacks
Use pgmap_protection_available() s/PGMAP_PKEY_PROTECT/PGMAP_PROTECTION
--- drivers/nvdimm/pmem.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+)
drivers/nvdimm/pmem.c +26 -0
@@ -138,6 +138,18 @@ static blk_status_t read_pmem(struct page *page, unsigned int off,return BLK_STS_OK ; } +static void pmem_set_readwrite (struct pmem_device *pmem) +{ +if (pmem->pgmap.flags &PGMAP_PROTECTION ) +__pgmap_set_readwrite (&pmem->pgmap); +} + +static void pmem_set_noaccess (struct pmem_device *pmem) +{ +if (pmem->pgmap.flags &PGMAP_PROTECTION ) +__pgmap_set_noaccess (&pmem->pgmap); +} +static blk_status_t pmem_do_read (struct pmem_device *pmem,struct page *page,unsigned int page_off,sector_t sector,unsigned int len)
@@ -149,7 +161,11 @@ static blk_status_t pmem_do_read(struct pmem_device *pmem,if (unlikely (is_bad_pmem (&pmem->bb, sector, len)))return BLK_STS_IOERR ; +/* Enable direct use of pmem->virt_addr */ +pmem_set_readwrite (pmem); rc =read_pmem (page, page_off, pmem_addr, len); +pmem_set_noaccess (pmem); +flush_dcache_page (page);return rc; }
@@ -181,11 +197,15 @@ static blk_status_t pmem_do_write(struct pmem_device *pmem,* after clear poison. */ flush_dcache_page (page); + +/* Enable direct use of pmem->virt_addr */ +pmem_set_readwrite (pmem);write_pmem (pmem_addr, page, page_off, len);if (unlikely (bad_pmem)) { rc =pmem_clear_poison (pmem, pmem_off, len);write_pmem (pmem_addr, page, page_off, len); } +pmem_set_noaccess (pmem);return rc; }
@@ -427,6 +447,8 @@ static int pmem_attach_disk(struct device *dev, pmem->pfn_flags =PFN_DEV ;if (is_nd_pfn (dev)) { pmem->pgmap.type =MEMORY_DEVICE_FS_DAX ; +if (pgmap_protection_available ()) + pmem->pgmap.flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, &pmem->pgmap); pfn_sb = nd_pfn->pfn_sb; pmem->data_offset =le64_to_cpu (pfn_sb->dataoff);
@@ -440,6 +462,8 @@ static int pmem_attach_disk(struct device *dev, pmem->pgmap.range.end = res->end; pmem->pgmap.nr_range =1 ; pmem->pgmap.type =MEMORY_DEVICE_FS_DAX ; +if (pgmap_protection_available ()) + pmem->pgmap.flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, &pmem->pgmap); pmem->pfn_flags |=PFN_MAP ; bb_range = pmem->pgmap.range;
@@ -481,6 +505,8 @@ static int pmem_attach_disk(struct device *dev, }set_dax_nocache (dax_dev);set_dax_nomc (dax_dev); +if (pmem->pgmap.flags &PGMAP_PROTECTION ) +set_dax_pgmap (dax_dev, &pmem->pgmap);if (is_nvdimm_sync (nd_region))set_dax_synchronous (dax_dev); rc =dax_add_host (dax_dev, disk);
Signature · 1 line
2.35.1
PATCH v9 45/45 devdax: Enable stray access protection
In reply to [email protected]From: Ira Weiny <[email protected]>
Device dax is primarily accessed through user space and kernel access is controlled through the kmap interfaces.
Now that all valid kernel initiated access to dax devices have been accounted for, turn on PGMAP_PKEYS_PROTECT for device dax.
---
Changes for V9
Add Review tag
Changes for V8
Rebase to 5.17-rc1 Use pgmap_protection_available() s/PGMAP_PKEYS_PROTECT/PGMAP_PROTECTION/ --- drivers/dax/device.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/dax/device.c +2 -0
@@ -452,6 +452,8 @@ int dev_dax_probe(struct dev_dax *dev_dax)if (dev_dax->align >PAGE_SIZE ) pgmap->vmemmap_shift =order_base_2 (dev_dax->align >>PAGE_SHIFT ); +if (pgmap_protection_available ()) + pgmap->flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, pgmap);if (IS_ERR (addr))return PTR_ERR (addr);
Signature · 1 line
2.35.1
Quoted 6 lines · Ira
From: Ira Weiny <[email protected]>
I'm looking for Intel acks on the series prior to submitting to maintainers. Most of the changes from V8 to V9 was in getting the tests straightened out. But there are some improvements in the actual code.
Is there any feedback on this?
Ira
Quoted 355 lines
Changes for V9
Review and update all commit messages.
Update cover letter belowPKS Core
Separate user and supervisor pkey code in the headers
createlinux/pks.hfor supervisor calls
This facilitated making the pmem code more efficient
Completely rearchitect the test code
[After Dave Hansen and Rick Edgecombe found issues in the test
code it was easier to rearchitect the code completely rather than attempt to fix it.]
Removepks_test_callbackin favor of using fault hooks
Fault hooks also isolate the fault callbacks from being false positives if non-test consumers are running
Make additionalPKS_TEST_RUN_ALLKconfigoption which is
mutually exclusive to any non-test PKS consumerPKS_TEST_RUN_ALLtakes over all pkey callbacks
Ensure that each test runs within it's own context and is
mutually exclusive from running while any other test is running.
Ensure test session and context memory is cleaned up on file
close
Usepr_debug()and dynamic debug for in kernel debug messages Enhancetest_pksselftest
Add the ability to run all tests not just the context
switch test
Standardize output [PASS][FAIL][SKIP]
Add '-d' option enables dynamic debug to see the kernel
debug messagesIncorporate feedback from Rick Edgecombe
Update all pkey types to u8
Fix up test code barriers
Move patch declaringPKS_INIT_VALUEahead of the patch which enables
PKS so thatPKS_INIT_VALUEcan be used whenpks_setup()is first created
From Dan Williams
Use macros instead of an enum for a pkey allocation scheme
which is predicated on the config options of consumers This almost worked perfectly. It required a bit of tweeking to be able to allocate all of the keys.From Dave Hansen
Reposition some code to be near/similar to user pkeys
s/pks_write_current/x86_pkrs_load s/pks_saved_pkrs/pkrs
Update Documentation s/PKR_{RW,AD,WD}_KEY/PKR_{RW,AD,WD}_MASK Consistently use lower case for pkey Update commit messages
Add AcksPMEM Stray Write
Building on the change to the pks_mk_*() function rename
s/pgmap_mk_*/pgmap_set_*/
s/dax_mk_*/dax_set_*/
From Dan Williams
Avoid adding new dax operations by teachingdax_deviceabout pgmap Removepgmap_protection_flag_invalid()patch (Just let
kmap'ings fail)PKS/PMEM Stray write protection
===============================This series is broken into 2 parts.
1) Introduce Protection Key Supervisor (PKS), testing, and documentation
2) Use PKS to protect PMEM from stray writesIntroduce Protection Key Supervisor (PKS)
-----------------------------------------PKS enables protections on 'domains' of supervisor pages to limit supervisor mode access to pages beyond the normal paging protections. PKS works in a similar fashion to user space pkeys, PKU. As with PKU, supervisor pkeys are checked in addition to normal paging protections. And page mappings are assigned to a domain by setting a 4 bit pkey in the PTE of that mapping.
Unlike PKU, permissions are changed via a MSR update. This update avoids TLB flushes making this an efficient way to alter protections vs PTE updates.
Also, unlike PTE updates PKS permission changes apply only to the current processor. Therefore changing permissions apply only to that thread and not any other cpu/process. This allows protections to remain in place on other cpus for additional protection and isolation.
Even though PKS updates are thread local, XSAVE is not supported for the PKRS MSR. Therefore this implementation saves and restores the MSR across context switches and during exceptions within software. Nested exceptions are supported by each exception getting a new PKS state.
For consistent behavior with current paging protections, pkey 0 is reserved and configured to allow full access via the pkey mechanism, thus preserving the default paging protections because PTEs naturally have a pkey value of 0.
Other keys, (1-15) are statically allocated by kernel consumers when configured. This is done by adding the appropriate
PKS_NEW_KEYandPKS_DECLARE_INIT_VALUEmacros topks-keys.h.Two PKS consumers,
PKS_TESTand PMEM stray write protection, are included in this series. When the number of users grows larger the sharing of keys will need to be resolved depending on the needs of the users at that time. Many methods have been contemplated but the number of kernel users and use cases envisioned is still quite small, much less than the 15 available keys.To summarize, the following are key attributes of PKS.
1) Fast switching of permissions
1a) Prevents access without page table manipulations 1b) No TLB flushes required
2) Works on a per thread basis, thus allowing protections to be preserved on threads which are not actively accessing data through the mapping.PKS is available with 4 and 5 level paging. For this and simplicity of implementation, the feature is restricted to
x86_64.Use PKS to protect PMEM from stray writes
-----------------------------------------DAX leverages the direct-map to enable 'struct page' services for PMEM. Given that PMEM capacity may be an order of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Such a stray write becomes a silent data corruption bug.
Stray pointers to System RAM may result in a crash or other undesirable behavior which, while unfortunate, are usually recoverable with a reboot. Stray writes to PMEM are permanent in nature and thus are more likely to result in permanent user data loss. Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, direct-I/O, and any properly kmap'ed page), it is amenable to PKS protection.
Set up an infrastructure for extra device access protection. Then implement the protection using the new Protection Keys Supervisor (PKS) on architectures which support it.
Because PMEM pages are all associated with a struct
dev_pagemapand flags in struct page are valuable the flag of protecting memory can be stored in structdev_pagemap. All PMEM is protected by the same pkey. So a single flag is all that is needed in eachdev_pagemapto indicate protection.General access in the kernel is supported by modifying the kmap infrastructure which can detect if a page is pks protected and enable access until the corresponding unmap is called.
Because PKS is a thread local mechanism and because kmap was never really intended to create a long term mapping, this implementation does not support the kmap()/kunmap() calls. Calling
kmap()on a PMEM protected page is allowed but accessing that mapping will cause a fault.Originally this series modified many of the kmap call sites to indicate they were thread local.[1] And an attempt to support kmap()[2] was made. But now that
kmap_local_page()has been developed[3] and in more wide spread use,kmap()can safely be left unsupported.How the fault is handled is configurable via a new module parameter memremap.pks_fault_mode. Two modes are supported.
'relaxed' (default) --
WARN_ONCE, disable the protection and allow
access'strict' -- prevent any unguarded access to a protected
dev_pagemap
rangeThis 'safety valve' feature has already been useful in the development of this feature.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/----------------------------------------------------------------------------
Changes for V8Feedback from Thomas
* clean up noinstr mess
* Fix static PKEY allocation mess
* Ensure all functions are consistently named.
* Split up patches to do 1 thing per patch
*pkey_update_pkval()implementation
* Streamline the use ofpks_write_pkrs()by not disabling preemption
- Leave this to the callers who require it.
- Use documentation and lockdep to prevent errors
* Clean up commit messages to explain in detail _why_ each patch is there.Feedback from Dave H.
* Leave outpks_mk_readonly()as it is not used by the PMEM use caseFeedback from Peter Anvin
* Replacepks_abandon_pkey()withpks_update_exception()This is an even greater simplification in that it no longer attempts to shield users from faults. As the main use case for abandoning a key was to allow a system to continue running even with an error. This should be a rare event so the performance should not be an issue.* Simplify
ARCH_ENABLE_SUPERVISOR_PKEYS* Update PKS Test code
- Add default value test
- Split up the test code into patches which follow each feature addition
- simplify test code processing
- ensure consistent reporting of errors.* Ensure all entry points to the PKS code are protected by
cpu_feature_enabled(X86_FEATURE_PKS)
- At the same time make sure non-entry points or sub-functions to the PKS code are not _unnecessarily_ protected by the feature check* Update documentation
- Use kernel docs to place the docs with the code for easier internal developer use* Adjust the PMEM use cases for the core changes
* Split the PMEM patches up to be 1 change per patch and help clarify review
* Review all header files and remove those no longer needed
* Review/update/clarify all commit messages
Fenghua Yu (1):
mm/pkeys: Define PKS page table macrosIra Weiny (43):
entry: Create an internalirqentry_exit_cond_resched()callDocumentation/protection-keys: Clean up documentation for User Space pkeys
x86/pkeys: ClarifyPKRU_AD_KEYmacro
x86/pkeys: Make PKRU macros generic
x86/fpu: Refactorarch_set_user_pkey_access()
mm/pkeys: AddKconfigoptions for PKS
x86/pkeys: Add PKS CPU feature bit
x86/fault: AdjustWARN_ONfor pkey faultDocumentation/pkeys: Add initial PKS documentation mm/pkeys: Provide for PKS key allocation
x86/pkeys: Enable PKS on cpus which support it
mm/pkeys: PKS testing, add initial test code x86/selftests: Addtest_pks
x86/pkeys: Introducepks_write_pkrs()
x86/pkeys: Preserve the PKS MSR on context switch mm/pkeys: Introducepks_set_readwrite()
mm/pkeys: Introducepks_set_noaccess()
mm/pkeys: PKS testing, add a fault call back
mm/pkeys: PKS testing, add pks_set_*() tests
mm/pkeys: PKS testing, test context switching
x86/entry: Add auxiliarypt_regsspace
entry: Split upirqentry_exit_cond_resched()
entry: Add calls for save/restore auxiliarypt_regsx86/entry: Define arch_{save|restore}_auxiliary_pt_regs() x86/pkeys: Preserve PKRS MSR across exceptions
x86/fault: Print PKS MSR on fault
mm/pkeys: PKS testing, Add exception test
mm/pkeys: Introducepks_update_exception()
mm/pkeys: PKS testing, testpks_update_exception()mm/pkeys: PKS testing, add test for all keys
mm/pkeys: Addpks_available()memremap_pages: AddKconfigforDEVMAP_ACCESS_PROTECTIONmemremap_pages: Introducepgmap_protection_available()memremap_pages: Introduce aPGMAP_PROTECTIONflagmemremap_pages: Introducedevmap_protected()memremap_pages: Reserve a PKS pkey for eventual use by PMEMmemremap_pages: Set PKS pkey in PTEs if requestedmemremap_pages: Define pgmap_set_{readwrite|noaccess}() callsmemremap_pages: Add memremap.pks_fault_mode
kmap: Make kmap work for devmap protected pages
dax: Stray access protection fordax_direct_access()nvdimm/pmem: Enable stray access protection
devdax: Enable stray access protectionRick Edgecombe (1):
mm/pkeys: Introduce PKS fault callbacks
.../admin-guide/kernel-parameters.txt| 12 +Documentation/core-api/protection-keys.rst| 130 ++-arch/x86/Kconfig| 6 +arch/x86/entry/calling.h| 20 +arch/x86/entry/common.c| 2 +-arch/x86/entry/entry_64.S| 22 +arch/x86/entry/entry_64_compat.S| 6 +arch/x86/include/asm/cpufeatures.h| 1 +arch/x86/include/asm/disabled-features.h| 8 +-arch/x86/include/asm/entry-common.h| 15 +arch/x86/include/asm/msr-index.h| 1 +arch/x86/include/asm/pgtable_types.h| 22 +arch/x86/include/asm/pkeys.h| 2 +arch/x86/include/asm/pkeys_common.h| 18 +arch/x86/include/asm/pkru.h| 20 +-arch/x86/include/asm/pks.h| 46 ++arch/x86/include/asm/processor.h| 15 +-arch/x86/include/asm/ptrace.h| 21 +arch/x86/include/uapi/asm/processor-flags.h| 2 +arch/x86/kernel/asm-offsets_64.c| 15 +arch/x86/kernel/cpu/common.c| 2 +arch/x86/kernel/dumpstack.c| 32 +-arch/x86/kernel/fpu/xstate.c| 22 +-arch/x86/kernel/head_64.S| 6 +arch/x86/kernel/process_64.c| 3 +arch/x86/mm/fault.c| 17 +-arch/x86/mm/pkeys.c| 320 +++++++-drivers/dax/device.c| 2 +drivers/dax/super.c| 59 ++drivers/md/dm-writecache.c| 8 +-drivers/nvdimm/pmem.c| 26 +fs/dax.c| 8 +fs/fuse/virtio_fs.c| 2 +include/linux/dax.h| 5 +include/linux/entry-common.h| 15 +-include/linux/highmem-internal.h| 4 +include/linux/memremap.h| 1 +include/linux/mm.h| 72 ++include/linux/pgtable.h| 4 +include/linux/pks-keys.h| 92 +++include/linux/pks.h| 73 ++include/linux/sched.h| 7 +include/uapi/asm-generic/mman-common.h| 1 +init/init_task.c| 3 +kernel/entry/common.c| 44 +-kernel/sched/core.c| 40 +-lib/Kconfig.debug| 33 +
lib/Makefile | 3 +lib/pks/Makefile| 3 +lib/pks/pks_test.c| 755 ++++++++++++++++++ mm/Kconfig | 32 +mm/memremap.c| 132 +++tools/testing/selftests/x86/Makefile| 2 +-tools/testing/selftests/x86/test_pks.c| 514 ++++++++++++54 files changed, 2617 insertions(+), 109 deletions(-)create mode 100644
arch/x86/include/asm/pkeys_common.hcreate mode 100644arch/x86/include/asm/pks.hcreate mode 100644include/linux/pks-keys.hcreate mode 100644include/linux/pks.h
create mode 100644lib/pks/Makefile
create mode 100644lib/pks/pks_test.c
create mode 100644tools/testing/selftests/x86/test_pks.c--
2.35.1