PKS/PMEM: Add Stray Write Protection
Subsystems (20)
v7 cover PKS/PMEM: Add Stray Write Protection
From: Ira Weiny <[email protected]>
NOTE: x86 maintainers, I'm submitting this for ack/review by Dave Hansen and Dan Williams. Feel free to ignore it but we have had a lot of internal debate on a number of design decisions so we would like to have the remaining reviews public such that everyone can see the remaining debate/decisions.
Furthermore, this gives a public reference for Rick to build other PKS use cases on.
PKS/PMEM Stray write protection
===============================
This series is broken into 2 parts.
1) Introduce Protection Key Supervisor (PKS)
2) Use PKS to protect PMEM from stray writes
Introduce Protection Key Supervisor (PKS)
-----------------------------------------
PKS enables protections on 'domains' of supervisor pages to limit supervisor mode access to pages beyond the normal paging protections. PKS works in a similar fashion to user space pkeys, PKU. As with PKU, supervisor pkeys are checked in addition to normal paging protections and Access or Writes can be disabled via a MSR update without TLB flushes when permissions change.
Also like PKU, a page mapping is assigned to a domain by setting pkey bits in the page table entry for that mapping.
Access is controlled through a PKRS register which is updated via WRMSR/RDMSR.
XSAVE is not supported for the PKRS MSR. Therefore the implementation saves/restores the MSR across context switches and during exceptions. Nested exceptions are supported by each exception getting a new PKS state.
For consistent behavior with current paging protections, pkey 0 is reserved and configured to allow full access via the pkey mechanism, thus preserving the default paging protections.
Other keys, (1-15) are statically allocated by kernel users adding an entry to 'enum pks_pkey_consumers' and adding a corresponding default value in consumer_defaults in create_initial_pkrs_value(). This patch series allocates a single key for use by persistent memory stray write protection. When the number of users grows larger the sharing of keys will need to be resolved depending on the needs of the users at that time.
More usage details can be found in the documentation.
The following are key attributes of PKS.
1) Fast switching of permissions
1a) Prevents access without page table manipulations 1b) No TLB flushes required
2) Works on a per thread basis
PKS is available with 4 and 5 level paging. Like PKRU it consumes 4 bits from the PTE to store the pkey within the entry.
Use PKS to protect PMEM from stray writes
-----------------------------------------
DAX leverages the direct-map to enable 'struct page' services for PMEM. Given that PMEM capacity may be an order of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Such a stray write becomes a silent data corruption bug.
Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, and direct-I/O), it is amenable to PKS protection. Set up an infrastructure for extra device access protection. Then implement the protection using the new Protection Keys Supervisor (PKS) on architectures which support it.
Because PMEM pages are all associated with a struct dev_pagemap the flag of protecting memory can be stored there. All PMEM is protected by the same pkey. So a single flag is all that is needed to indicate protection.
General access in the kernel is supported by modifying the kmap infrastructure which can detect if a page is PMEM and pks protected. If so kmap_local_page() and kmap_atomic() can enable access until their unmap's are called.
Because PKS is a thread local mechanism and because kmap was never really intended to create a long term mapping,
This implementation avoids supporting the kmap()/kunmap() for a number of reasons. First, kmap was never really intended to create long term mappings. Second, no known kernel users of pmem use kmap. Third, PKS is a thread local mechanism.
Originally this series modified many of the kmap call sites to indicate they were thread local.[1] And an attempt to support kmap()[2] was made. But now that kmap_local_page() has been developed[3] and in more wide spread use, kmap() should be safe to leave unsupported and is considered an invalid access.
Handling invalid access to these pages is configurable via a new module parameter memremap.pks_fault_mode. 2 modes are suported.
'relaxed' (default) -- WARN_ONCE, disable the protection and allow
access
'strict' -- prevent any unguarded access to a protected dev_pagemap
range
The fault handler detects the PMEM fault and applies the above configuration to the faulting thread. The kmap call is a special case. It is considered an invalid access but uses the configuration early before any access such that the kmap code path can be better evaluated and fixed.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/ https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/[email protected]/
Fenghua Yu (1):
x86/pks: Add PKS kernel API
Ira Weiny (16):
x86/pkeys: Create pkeys_common.h
x86/fpu: Refactor arch_set_user_pkey_access()
x86/pks: Add additional PKEY helper macros
x86/pks: Add PKS defines and Kconfig options
x86/pks: Add PKS setup code
x86/fault: Adjust WARN_ON for PKey fault
x86/pks: Preserve the PKRS MSR on context switch x86/entry: Preserve PKRS MSR across exceptions
x86/pks: Introduce pks_abandon_protections()
x86/pks: Add PKS Test code
memremap_pages: Add access protection via supervisor Protection Keys
(PKS)
memremap_pages: Add memremap.pks_fault_mode
kmap: Add stray access protection for devmap pages
dax: Stray access protection for dax_direct_access() nvdimm/pmem: Enable stray access protection
devdax: Enable stray access protection
Rick Edgecombe (1):
x86/pks: Add PKS fault callbacks
.../admin-guide/kernel-parameters.txt | 14 + Documentation/core-api/protection-keys.rst | 153 +++- arch/x86/Kconfig | 1 + arch/x86/entry/calling.h | 26 + arch/x86/entry/common.c | 56 ++ arch/x86/entry/entry_64.S | 22 +- arch/x86/entry/entry_64_compat.S | 6 +- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/pgtable_types.h | 12 + arch/x86/include/asm/pkeys.h | 2 + arch/x86/include/asm/pkeys_common.h | 19 + arch/x86/include/asm/pkru.h | 16 +- arch/x86/include/asm/pks.h | 67 ++ arch/x86/include/asm/processor-flags.h | 2 + arch/x86/include/asm/processor.h | 19 +- arch/x86/include/uapi/asm/processor-flags.h | 2 + arch/x86/kernel/cpu/common.c | 2 + arch/x86/kernel/fpu/xstate.c | 22 +- arch/x86/kernel/head_64.S | 7 +- arch/x86/kernel/process.c | 3 + arch/x86/kernel/process_64.c | 3 + arch/x86/mm/fault.c | 82 +- arch/x86/mm/pkeys.c | 277 +++++- drivers/dax/device.c | 2 + drivers/dax/super.c | 54 ++ drivers/md/dm-writecache.c | 8 +- drivers/nvdimm/pmem.c | 55 +- fs/dax.c | 8 + fs/fuse/virtio_fs.c | 2 + include/linux/dax.h | 8 + include/linux/highmem-internal.h | 5 + include/linux/memremap.h | 1 + include/linux/mm.h | 88 ++ include/linux/pgtable.h | 4 + include/linux/pkeys.h | 36 + include/linux/sched.h | 7 + init/init_task.c | 3 + kernel/entry/common.c | 14 +- kernel/fork.c | 3 + lib/Kconfig.debug | 13 + lib/Makefile | 3 + lib/pks/Makefile | 3 + lib/pks/pks_test.c | 864 ++++++++++++++++++ mm/Kconfig | 26 + mm/memremap.c | 158 ++++ tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/test_pks.c | 157 ++++ 49 files changed, 2261 insertions(+), 86 deletions(-) create mode 100644 arch/x86/include/asm/pkeys_common.h create mode 100644 arch/x86/include/asm/pks.h create mode 100644 lib/pks/Makefile create mode 100644 lib/pks/pks_test.c create mode 100644 tools/testing/selftests/x86/test_pks.c
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 1/18 x86/pkeys: Create pkeys_common.h
From: Ira Weiny <[email protected]>
Protection Keys User (PKU) and Protection Keys Supervisor (PKS) work in similar fashions and can share common defines. Specifically PKS and PKU each have:
1. A single control register
2. The same number of keys
3. The same number of bits in the register per key
4. Access and Write disable in the same bit locations
Given the above, share all the macros that synthesize and manipulate register values between the two features. Share these defines by moving them into a new header, change their names to reflect the common use, and include the header where needed.
Also while editing the code remove the use of 'we' from comments being touched.
--- arch/x86/include/asm/pkeys_common.h | 11 +++++++++++ arch/x86/include/asm/pkru.h | 18 ++++++------------ arch/x86/kernel/fpu/xstate.c | 8 ++++---- arch/x86/mm/pkeys.c | 14 ++++++-------- 4 files changed, 27 insertions(+), 24 deletions(-) create mode 100644 arch/x86/include/asm/pkeys_common.h
arch/x86/include/asm/pkeys_common.h new file +11 -0
@@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_PKEYS_COMMON_H +#define _ASM_X86_PKEYS_COMMON_H + +#define PKR_AD_BIT 0x1 +#define PKR_WD_BIT 0x2 +#define PKR_BITS_PER_PKEY 2 + +#define PKR_AD_KEY (pkey) (PKR_AD_BIT << ((pkey) * PKR_BITS_PER_PKEY)) + +#endif /*_ASM_X86_PKEYS_COMMON_H */
arch/x86/include/asm/pkru.h +6 -12
@@ -3,10 +3,7 @@#define _ASM_X86_PKRU_H#include <asm/fpu/xstate.h> - -#define PKRU_AD_BIT 0x1 -#define PKRU_WD_BIT 0x2 -#define PKRU_BITS_PER_PKEY 2 +#include <asm/pkeys_common.h> #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS extern u32 init_pkru_value;
@@ -18,18 +15,15 @@ extern u32 init_pkru_value;static inline bool __pkru_allows_read (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKRU_BITS_PER_PKEY ; -return !(pkru & (PKRU_AD_BIT << pkru_pkey_bits)); +int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ; +return !(pkru & (PKR_AD_BIT << pkru_pkey_bits)); }static inline bool __pkru_allows_write (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKRU_BITS_PER_PKEY ; -/* -* Access-disable disables writes too so we need to check -* both bits here. -*/ -return !(pkru & ((PKRU_AD_BIT |PKRU_WD_BIT ) << pkru_pkey_bits)); +int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ; +/* Access-disable disables writes too so check both bits here. */ +return !(pkru & ((PKR_AD_BIT |PKR_WD_BIT ) << pkru_pkey_bits)); }static inline u32 read_pkru (void )
arch/x86/kernel/fpu/xstate.c +4 -4
@@ -933,11 +933,11 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,if (WARN_ON_ONCE (pkey >=arch_max_pkey ()))return -EINVAL ; -/* Set the bits we need in PKRU: */ +/* Set the bits needed in PKRU: */ if (init_val &PKEY_DISABLE_ACCESS ) - new_pkru_bits |=PKRU_AD_BIT ; + new_pkru_bits |=PKR_AD_BIT ;if (init_val &PKEY_DISABLE_WRITE ) - new_pkru_bits |=PKRU_WD_BIT ; + new_pkru_bits |=PKR_WD_BIT ;/* Shift the bits in to the correct place in PKRU for pkey: */ pkey_shift = pkey *PKRU_BITS_PER_PKEY ;
@@ -945,7 +945,7 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,/* Get old PKRU and mask off any old bits in place: */ old_pkru =read_pkru (); - old_pkru &= ~((PKRU_AD_BIT |PKRU_WD_BIT ) << pkey_shift); + old_pkru &= ~((PKR_AD_BIT |PKR_WD_BIT ) << pkey_shift);/* Write old part along with new part: */ write_pkru (old_pkru | new_pkru_bits);
arch/x86/mm/pkeys.c +6 -8
@@ -110,19 +110,17 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkeyreturn vma_pkey (vma); } -#define PKRU_AD_KEY (pkey) (PKRU_AD_BIT << ((pkey) * PKRU_BITS_PER_PKEY)) -/* * Make the default PKRU value (at execve() time) as restrictive * as possible. This ensures that any threads clone()'d early * in the process's lifetime will not accidentally get access * to data which is pkey-protected later on. */ -u32 init_pkru_value =PKRU_AD_KEY (1 ) |PKRU_AD_KEY (2 ) |PKRU_AD_KEY (3 ) | -PKRU_AD_KEY (4 ) |PKRU_AD_KEY (5 ) |PKRU_AD_KEY (6 ) | -PKRU_AD_KEY (7 ) |PKRU_AD_KEY (8 ) |PKRU_AD_KEY (9 ) | -PKRU_AD_KEY (10 ) |PKRU_AD_KEY (11 ) |PKRU_AD_KEY (12 ) | -PKRU_AD_KEY (13 ) |PKRU_AD_KEY (14 ) |PKRU_AD_KEY (15 ); +u32 init_pkru_value =PKR_AD_KEY (1 ) |PKR_AD_KEY (2 ) |PKR_AD_KEY (3 ) | +PKR_AD_KEY (4 ) |PKR_AD_KEY (5 ) |PKR_AD_KEY (6 ) | +PKR_AD_KEY (7 ) |PKR_AD_KEY (8 ) |PKR_AD_KEY (9 ) | +PKR_AD_KEY (10 ) |PKR_AD_KEY (11 ) |PKR_AD_KEY (12 ) | +PKR_AD_KEY (13 ) |PKR_AD_KEY (14 ) |PKR_AD_KEY (15 );static ssize_t init_pkru_read_file (struct file *file,char __user *user_buf,size_t count,loff_t *ppos)
@@ -155,7 +153,7 @@ static ssize_t init_pkru_write_file(struct file *file,* up immediately if someone attempts to disable access * or writes to pkey 0. */ -if (new_init_pkru & (PKRU_AD_BIT |PKRU_WD_BIT )) +if (new_init_pkru & (PKR_AD_BIT |PKR_WD_BIT ))return -EINVAL ;WRITE_ONCE (init_pkru_value, new_init_pkru);
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 2/18 x86/fpu: Refactor arch_set_user_pkey_access()
In reply to [email protected]From: Ira Weiny <[email protected]>
Both PKU and PKS update their register values in the same way. They can therefore share the update code.
Define a helper, update_pkey_val(), which will be used to support both Protection Key User (PKU) and the new Protection Key for Supervisor (PKS) in subsequent patches.
Use that helper in arch_set_user_pkey_access().
--- arch/x86/include/asm/pkeys.h | 2 ++ arch/x86/kernel/fpu/xstate.c | 22 ++++------------------ arch/x86/mm/pkeys.c | 23 +++++++++++++++++++++++ 3 files changed, 29 insertions(+), 18 deletions(-)
arch/x86/include/asm/pkeys.h +2 -0
@@ -133,4 +133,6 @@ static inline int vma_pkey(struct vm_area_struct *vma) return (vma->vm_flags & vma_pkey_mask) >> VM_PKEY_SHIFT; } +u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags); +#endif /*_ASM_X86_PKEYS_H */
arch/x86/kernel/fpu/xstate.c +4 -18
@@ -915,8 +915,7 @@ EXPORT_SYMBOL_GPL(get_xsave_addr);int arch_set_user_pkey_access (struct task_struct *tsk,int pkey,unsigned long init_val) { -u32 old_pkru, new_pkru_bits =0 ; -int pkey_shift; +u32 pkru;/* * This check implies XSAVE support. OSPKE only gets
@@ -933,22 +932,9 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,if (WARN_ON_ONCE (pkey >=arch_max_pkey ()))return -EINVAL ; -/* Set the bits needed in PKRU: */ -if (init_val &PKEY_DISABLE_ACCESS ) - new_pkru_bits |=PKR_AD_BIT ; -if (init_val &PKEY_DISABLE_WRITE ) - new_pkru_bits |=PKR_WD_BIT ; - -/* Shift the bits in to the correct place in PKRU for pkey: */ - pkey_shift = pkey *PKRU_BITS_PER_PKEY ; - new_pkru_bits <<= pkey_shift; - -/* Get old PKRU and mask off any old bits in place: */ - old_pkru =read_pkru (); - old_pkru &= ~((PKR_AD_BIT |PKR_WD_BIT ) << pkey_shift); - -/* Write old part along with new part: */ -write_pkru (old_pkru | new_pkru_bits); + pkru =read_pkru (); + pkru =update_pkey_val (pkru, pkey, init_val); +write_pkru (pkru);return 0 ; }
arch/x86/mm/pkeys.c +23 -0
@@ -190,3 +190,26 @@ static __init int setup_init_pkru(char *opt)return 1 ; }__setup ("init_pkru=" , setup_init_pkru); + +/* +* Replace disable bits for @pkey with values from @flags +* +* Kernel users use the same flags as user space: +* PKEY_DISABLE_ACCESS +* PKEY_DISABLE_WRITE +*/ +u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags) +{ +int pkey_shift = pkey *PKR_BITS_PER_PKEY ; + +/* Mask out old bit values */ + pk_reg &= ~(((1 <<PKR_BITS_PER_PKEY ) -1 ) << pkey_shift); + +/* Or in new values */ +if (flags &PKEY_DISABLE_ACCESS ) + pk_reg |=PKR_AD_BIT << pkey_shift; +if (flags &PKEY_DISABLE_WRITE ) + pk_reg |=PKR_WD_BIT << pkey_shift; + +return pk_reg; +}
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 02/18] x86/fpu: Refactor arch_set_user_pkey_access()
arch/x86/mm/pkeys.c static __init int setup_init_pkru(char *opt) · line 201
Quoted 8 lines · ira weiny
arch/x86/mm/pkeys.c +8
-0+/* +* Replace disable bits for @pkey with values from @flags +* +* Kernel users use the same flags as user space: +* PKEY_DISABLE_ACCESS +* PKEY_DISABLE_WRITE +*/ +u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags)
pkey_.... please.
arch/x86/mm/pkeys.c static __init int setup_init_pkru(char *opt) · line 214
Quoted 13 lines
arch/x86/mm/pkeys.c +13
-0+{ +int pkey_shift = pkey *PKR_BITS_PER_PKEY ; + +/* Mask out old bit values */ + pk_reg &= ~(((1 <<PKR_BITS_PER_PKEY ) -1 ) << pkey_shift); + +/* Or in new values */ +if (flags &PKEY_DISABLE_ACCESS ) + pk_reg |=PKR_AD_BIT << pkey_shift; +if (flags &PKEY_DISABLE_WRITE ) + pk_reg |=PKR_WD_BIT << pkey_shift; + +return pk_reg;
Also this code is silly.
#define PKEY_ACCESS_MASK (PKEY_DISABLE_ACCESS | PKEY_DISABLE_WRITE)
u32 pkey_update_pkval(u32 pkval, int pkey, u32 accessbits)
{
int shift = pkey * PKR_BITS_PER_PKEY;
if (WARN_ON_ONCE(accessbits & ~PKEY_ACCESS_MASK))
accessbits &= PKEY_ACCESS_MASK;
pkval &= ~(PKEY_ACCESS_MASK << shift);
return pkval | accessbit << pkey_shift;
}See? It does not even need comments because it's self explaining and uses sensible argument names matching the function name.
PATCH v7 3/18 x86/pks: Add additional PKEY helper macros
In reply to [email protected]From: Ira Weiny <[email protected]>
Avoid open coding shift and mask operations by defining and using helper macros for PKey operations.
--- arch/x86/include/asm/pkeys_common.h | 6 +++++- arch/x86/include/asm/pkru.h | 6 ++---- arch/x86/mm/pkeys.c | 8 +++----- 3 files changed, 10 insertions(+), 10 deletions(-)
arch/x86/include/asm/pkeys_common.h +5 -1
@@ -6,6 +6,10 @@#define PKR_WD_BIT 0x2#define PKR_BITS_PER_PKEY 2 -#define PKR_AD_KEY (pkey) (PKR_AD_BIT << ((pkey) * PKR_BITS_PER_PKEY)) +#define PKR_PKEY_SHIFT (pkey) (pkey * PKR_BITS_PER_PKEY) +#define PKR_PKEY_MASK (pkey) (((1 << PKR_BITS_PER_PKEY) - 1) << PKR_PKEY_SHIFT(pkey)) + +#define PKR_AD_KEY (pkey) (PKR_AD_BIT << PKR_PKEY_SHIFT(pkey)) +#define PKR_WD_KEY (pkey) (PKR_WD_BIT << PKR_PKEY_SHIFT(pkey))#endif /*_ASM_X86_PKEYS_COMMON_H */
arch/x86/include/asm/pkru.h +2 -4
@@ -15,15 +15,13 @@ extern u32 init_pkru_value;static inline bool __pkru_allows_read (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ; -return !(pkru & (PKR_AD_BIT << pkru_pkey_bits)); +return !(pkru &PKR_AD_KEY (pkey)); }static inline bool __pkru_allows_write (u32 pkru,u16 pkey) { -int pkru_pkey_bits = pkey *PKR_BITS_PER_PKEY ;/* Access-disable disables writes too so check both bits here. */ -return !(pkru & ((PKR_AD_BIT |PKR_WD_BIT ) << pkru_pkey_bits)); +return !(pkru & (PKR_AD_KEY (pkey) |PKR_WD_KEY (pkey))); }static inline u32 read_pkru (void )
arch/x86/mm/pkeys.c +3 -5
@@ -200,16 +200,14 @@ __setup("init_pkru=", setup_init_pkru);*/ u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags) { -int pkey_shift = pkey *PKR_BITS_PER_PKEY ; -/* Mask out old bit values */ - pk_reg &= ~(((1 <<PKR_BITS_PER_PKEY ) -1 ) << pkey_shift); + pk_reg &= ~PKR_PKEY_MASK (pkey);/* Or in new values */ if (flags &PKEY_DISABLE_ACCESS ) - pk_reg |=PKR_AD_BIT << pkey_shift; + pk_reg |=PKR_AD_KEY (pkey);if (flags &PKEY_DISABLE_WRITE ) - pk_reg |=PKR_WD_BIT << pkey_shift; + pk_reg |=PKR_WD_KEY (pkey);return pk_reg; }
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 03/18] x86/pks: Add additional PKEY helper macros
Quoted 7 lines · ira weiny
arch/x86/mm/pkeys.c +0
-2@@ -200,16 +200,14 @@ __setup("init_pkru=", setup_init_pkru);*/ u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags) { -int pkey_shift = pkey *PKR_BITS_PER_PKEY ; -/* Mask out old bit values */
arch/x86/mm/pkeys.c __setup("init_pkru=", setup_init_pkru); · line 210
Quoted 10 lines
arch/x86/mm/pkeys.c +3
-3- pk_reg &= ~(((1 <<PKR_BITS_PER_PKEY ) -1 ) << pkey_shift); + pk_reg &= ~PKR_PKEY_MASK (pkey);/* Or in new values */ if (flags &PKEY_DISABLE_ACCESS ) - pk_reg |=PKR_AD_BIT << pkey_shift; + pk_reg |=PKR_AD_KEY (pkey);if (flags &PKEY_DISABLE_WRITE ) - pk_reg |=PKR_WD_BIT << pkey_shift; + pk_reg |=PKR_WD_KEY (pkey);
I'm not seeing how this is improving that code. Quite the contrary.
Thanks,
tglx
Re: [PATCH V7 03/18] x86/pks: Add additional PKEY helper macros
Quoted 11 lines · Thomas Gleixner
Quoted 10 lines · ira weiny
arch/x86/mm/pkeys.c +1
-3@@ -200,16 +200,14 @@ __setup("init_pkru=", setup_init_pkru);*/ u32 update_pkey_val (u32 pk_reg,int pkey,unsigned int flags) { -int pkey_shift = pkey *PKR_BITS_PER_PKEY ; -/* Mask out old bit values */ - pk_reg &= ~(((1 <<PKR_BITS_PER_PKEY ) -1 ) << pkey_shift); + pk_reg &= ~PKR_PKEY_MASK (pkey);
Quoted 9 lines
arch/x86/mm/pkeys.c __setup("init_pkru=", setup_init_pkru); · line 210
Quoted 7 lines
arch/x86/mm/pkeys.c +2
-2/* Or in new values */ if (flags &PKEY_DISABLE_ACCESS ) - pk_reg |=PKR_AD_BIT << pkey_shift; + pk_reg |=PKR_AD_KEY (pkey);if (flags &PKEY_DISABLE_WRITE ) - pk_reg |=PKR_WD_BIT << pkey_shift; + pk_reg |=PKR_WD_KEY (pkey);I'm not seeing how this is improving that code. Quite the contrary.
Aside of that why are you ordering it the wrong way around, i.e.
1) implement turd
2) polish turd
instead of implementing the required helpers first if they are really providing value.
Thanks,
tglx
Re: [PATCH V7 03/18] x86/pks: Add additional PKEY helper macros
In reply to Thomas GleixnerQuoted 11 lines · Thomas Gleixner
Quoted 10 lines · ira weiny
(quoted diff) +1
-3@@ -200,16 +200,14 @@ __setup("init_pkru=", setup_init_pkru); */ u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) { - int pkey_shift = pkey * PKR_BITS_PER_PKEY; - /* Mask out old bit values */ - pk_reg &= ~(((1 << PKR_BITS_PER_PKEY) - 1) << pkey_shift); + pk_reg &= ~PKR_PKEY_MASK(pkey);
Quoted 9 lines
Review __setup("init_pkru=", setup_init_pkru); · line 210
Quoted 7 lines
(quoted diff) +2
-2/* Or in new values */ if (flags & PKEY_DISABLE_ACCESS) - pk_reg |= PKR_AD_BIT << pkey_shift; + pk_reg |= PKR_AD_KEY(pkey); if (flags & PKEY_DISABLE_WRITE) - pk_reg |= PKR_WD_BIT << pkey_shift; + pk_reg |= PKR_WD_KEY(pkey);I'm not seeing how this is improving that code. Quite the contrary.
Fair enough. Even more so when using the code you suggested for pkey_update_pkval().
In that case it boils down to:
arch/x86/mm/pkeys.c +1 -1
@@ -198,7 +198,7 @@ __setup("init_pkru=", setup_init_pkru);*/ u32 pkey_update_pkval (u32 pkval,int pkey,u32 accessbits) { -int shift = pkey *PKR_BITS_PER_PKEY ; +int shift =PKR_PKEY_SHIFT (pkey);if (WARN_ON_ONCE (accessbits & ~PKEY_ACCESS_MASK )) accessbits &=PKEY_ACCESS_MASK ;
Better?
As to the reason of why to put this patch after the other one. Why would I improve the old pre-refactoring code only to throw it away when moving it to pkey_update_pkval()? This reasoning is even stronger when pkey_update_pkval() is implemented.
I agree with Dan regarding the macros though. I think they make it easier to see what is going on without dealing with masks and shifts directly. But I can remove this patch if you feel that strongly about it.
Ira
Quoted 4 lines
Thanks,
tglx
Re: [PATCH V7 03/18] x86/pks: Add additional PKEY helper macros
Ira,
Quoted 11 lines · Ira Weiny
On Thu, Nov 25, 2021 at 03:25:09PM +0100, Thomas Gleixner wrote:
u32
pkey_update_pkval(u32 pkval, int pkey, u32 accessbits){arch/x86/mm/pkeys.c +1
-1-int shift = pkey *PKR_BITS_PER_PKEY ; +int shift =PKR_PKEY_SHIFT (pkey);if (WARN_ON_ONCE(accessbits & ~PKEY_ACCESS_MASK)) accessbits &= PKEY_ACCESS_MASK;Better?
Let me postpone this question.
Quoted 4 lines
As to the reason of why to put this patch after the other one. Why would I improve the old pre-refactoring code only to throw it away when moving it to
pkey_update_pkval()? This reasoning is even stronger whenpkey_update_pkval()is implemented.
Which refactoring? We seem to have fundamentally definitions of that term. Let me illustrate why.
The original version of this was:
u32 get_new_pkr(u32 old_pkr, int pkey, unsigned long init_val)
{
int pkey_shift = (pkey * PKR_BITS_PER_PKEY);
u32 new_pkr_bits = 0;
/* Set the bits we need in the register: */
if (init_val & PKEY_DISABLE_ACCESS)
new_pkr_bits |= PKR_AD_BIT;
if (init_val & PKEY_DISABLE_WRITE)
new_pkr_bits |= PKR_WD_BIT; /* Shift the bits in to the correct place: */ new_pkr_bits <<= pkey_shift;
/* Mask off any old bits in place: */ old_pkr &= ~((PKR_AD_BIT | PKR_WD_BIT) << pkey_shift);
/* Return the old part along with the new part: */ return old_pkr | new_pkr_bits;
}
IOW, mechanical Cut & Paste.
Then PeterZ came along and suggested to improve it this way:
u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags)
{
int pkey_shift = pkey * PKR_BITS_PER_PKEY;
/* Mask out old bit values */
pk_reg &= ~(((1 << PKR_BITS_PER_PKEY) - 1) << pkey_shift);
/* Or in new values */
if (flags & PKEY_DISABLE_ACCESS)
pk_reg |= PKR_AD_BIT << pkey_shift;
if (flags & PKEY_DISABLE_WRITE)
pk_reg |= PKR_WD_BIT << pkey_shift;
return pk_reg;}
which is already better. So you changed your approach from Cut & Paste to Copy & Paste.
But neither Cut & Paste nor Copy & Paste match what refactoring is really about. Just throwing the term refactoring at it does not make it so.
Refactoring is about improving the code in design and implementation. The keyword is: improving.
There are obviously cases where you can take the code as is and split it out into a new helper function.
You really have to look at it and answer the question whether it's good code or not, whether it could be written in better ways and with improved functionality.
I could have given you this minimalistic one:
u32 pkey_update_pkval(u32 pkval, int pkey, u32 accessbits)
{
int shift = pkey * PKR_BITS_PER_PKEY;
pkval &= ~(PKEY_ACCESS_MASK << shift);
return pkval | (accessbit & PKEY_ACCESS_MASK) << shift;}
But I gave you this:
u32 pkey_update_pkval(u32 pkval, int pkey, u32 accessbits)
{
int shift = pkey * PKR_BITS_PER_PKEY;
if (WARN_ON_ONCE(accessbits & ~PKEY_ACCESS_MASK))
accessbits &= PKEY_ACCESS_MASK;
pkval &= ~(PKEY_ACCESS_MASK << shift);
return pkval | accessbit << shift;}
This is what refactoring is about. See?
Quoted 3 lines
I agree with Dan regarding the macros though. I think they make it easier to see what is going on without dealing with masks and shifts directly. But I can remove this patch if you feel that strongly about it.
I'm not against macros per se, but not everything is automatically better when it is hidden behind a macro.
What I'm arguing against is the claim that macros are an improvement by definition. Especially when they are just blindly thrown into code which should not exist in the first place.
Also versus ordering. What's wrong with doing it this way:
1) Define the macros first without changing the code
2) Implement pkey_update_pkval() in a sensible way and use the macros where appropriate. Thereby replacing the existing version in the other function.
Which would end up in the obviously even simpler code:
u32 pkey_update_pkval(u32 pkval, int pkey, u32 accessbits)
{
if (WARN_ON_ONCE(accessbits & ~PKEY_ACCESS_MASK))
accessbits &= PKEY_ACCESS_MASK;
pkval &= ~PKR_PKEY_VALUE(pkey, PKEY_ACCESS_MASK);
return pkval | PKR_PKEY_VALUE(pkey, accessbits);}
That fits the goal of that macro exercise to make it easy to read and obvious what's going on, no?
Instead of:
Quoted 5 lines
u32
pkey_update_pkval(u32 pkval, int pkey, u32 accessbits){arch/x86/mm/pkeys.c +1
-1-int shift = pkey *PKR_BITS_PER_PKEY ; +int shift =PKR_PKEY_SHIFT (pkey);
Quoted 8 lines
if (WARN_ON_ONCE(accessbits & ~PKEY_ACCESS_MASK)) accessbits &= PKEY_ACCESS_MASK; pkval &= ~(PKEY_ACCESS_MASK << shift); return pkval | accessbit << shift;}
Better?
You surely can answer this question yourself, no?
"By continuously improving the design of code, we make it easier and
easier to work with. This is in sharp contrast to what typically
happens: little refactoring and a great deal of attention paid to expediently adding new features. If you get into the hygienic habit of refactoring continuously, you'll find that it is easier to extend and maintain code." -- Joshua Kerievsky
If you study that quote carefully, you surely can find our diverging approach to refactoring in it, no?
Thanks,
tglx
PATCH v7 4/18 x86/pks: Add PKS defines and Kconfig options
In reply to [email protected]From: Ira Weiny <[email protected]>
Protection Keys for Supervisor pages (PKS) enables fast, hardware thread specific, manipulation of permission restrictions on supervisor page mappings. It uses the same mechanism of Protection Keys as those on User mappings but applies that mechanism to supervisor mappings using a supervisor specific MSR.
Define the PKS CPU feature bits.
Add the Kconfig ARCH_HAS_SUPERVISOR_PKEYS to indicate to kernel consumers that an architecture supports pkeys.
Introduce ARCH_ENABLE_SUPERVISOR_PKEYS to allow architectures to avoid PKS code unless a kernel consumers is configured.
ARCH_ENABLE_SUPERVISOR_PKEYS remains off until the first kernel use case sets it.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/uapi/asm/processor-flags.h | 2 ++ mm/Kconfig | 4 ++++ 5 files changed, 15 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -1837,6 +1837,7 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS depends on X86_64 && (CPU_SUP_INTEL || CPU_SUP_AMD) select ARCH_USES_HIGH_VMA_FLAGS select ARCH_HAS_PKEYS + select ARCH_HAS_SUPERVISOR_PKEYS help Memory Protection Keys provides a mechanism for enforcing page-based protections, but without requiring modification of the
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -365,6 +365,7 @@#define X86_FEATURE_MOVDIR64B (16*32+28)/* MOVDIR64B instruction */ #define X86_FEATURE_ENQCMD (16*32+29)/* ENQCMD and ENQCMDS instructions */ #define X86_FEATURE_SGX_LC (16*32+30)/* Software Guard Extensions Launch Control */ +#define X86_FEATURE_PKS (16*32+31)/* Protection Keys for Supervisor pages */ /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */ #define X86_FEATURE_OVERFLOW_RECOV (17*32+ 0)/* MCA overflow recovery support */
arch/x86/include/asm/disabled-features.h +7 -1
@@ -44,6 +44,12 @@# define DISABLE_OSPKE (1<<(X86_FEATURE_OSPKE & 31))#endif /* CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS */ +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +# define DISABLE_PKS 0 +#else +# define DISABLE_PKS (1<<(X86_FEATURE_PKS & 31)) +#endif +#ifdef CONFIG_X86_5LEVEL # define DISABLE_LA57 0#else
@@ -85,7 +91,7 @@#define DISABLED_MASK14 0#define DISABLED_MASK15 0#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_PKS)#define DISABLED_MASK17 0#define DISABLED_MASK18 0#define DISABLED_MASK19 0
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -130,6 +130,8 @@#define X86_CR4_SMAP _BITUL(X86_CR4_SMAP_BIT)#define X86_CR4_PKE_BIT 22/* enable Protection Keys support */ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) +#define X86_CR4_PKS_BIT 24/* enable Protection Keys for Supervisor */ +#define X86_CR4_PKS _BITUL(X86_CR4_PKS_BIT)/* * x86-64 Task Priority Register, CR8
mm/Kconfig +4 -0
@@ -818,6 +818,10 @@ config ARCH_USES_HIGH_VMA_FLAGS bool config ARCH_HAS_PKEYS bool +config ARCH_HAS_SUPERVISOR_PKEYS + bool +config ARCH_ENABLE_SUPERVISOR_PKEYS + bool config PERCPU_STATS bool "Collect percpu memory statistics"
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 5/18 x86/pks: Add PKS setup code
In reply to [email protected]From: Ira Weiny <[email protected]>
Protection Keys for Supervisor pages (PKS) enables fast, hardware thread specific, manipulation of permission restrictions on supervisor page mappings. It uses the same mechanism of Protection Keys as those on User mappings but applies that mechanism to supervisor mappings using a supervisor specific MSR.
Add setup code and the lowest level of PKS MSR write support. Pkeys values are allocated statically via the pks_pkey_consumers enumeration. create_initial_pkrs_value() builds the initial protection values for each pkey. Users who need a default value other than Access Disabled should update consumer_defaults[].
The PKRS value is cached per-cpu to avoid the overhead of the MSR write if the value has not changed.
That said, it should be noted that the underlying WRMSR(MSR_IA32_PKRS) is not serializing but still maintains ordering properties similar to WRPKRU. The current SDM section on PKRS needs updating but should be the same as that of WRPKRU. So to quote from the WRPKRU text:
WRPKRU will never execute transiently. Memory accesses affected by PKRU register will not execute (even transiently) until all prior executions of WRPKRU have completed execution and updated the PKRU register.
write_pkrs() contributed by Peter Zijlstra. create_initial_pkrs_value() contributed by Dave Hansen
setup_pks() is an internal x86 function call. Introduce asm/pks.h to declare functions and internal structures such as this.
---
Changes for V7
Create a dynamic pkrs_initial_value in early init code. Clean up comments
Add comment to macro guard
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/pkeys_common.h | 4 ++ arch/x86/include/asm/pks.h | 15 ++++++ arch/x86/kernel/cpu/common.c | 2 + arch/x86/mm/pkeys.c | 75 +++++++++++++++++++++++++++++ include/linux/pkeys.h | 8 +++ 6 files changed, 105 insertions(+) create mode 100644 arch/x86/include/asm/pks.h
arch/x86/include/asm/msr-index.h +1 -0
@@ -767,6 +767,7 @@#define MSR_IA32_TSC_DEADLINE 0x000006E0 +#define MSR_IA32_PKRS 0x000006E1#define MSR_TSX_FORCE_ABORT 0x0000010F
arch/x86/include/asm/pkeys_common.h +4 -0
@@ -2,14 +2,18 @@#ifndef _ASM_X86_PKEYS_COMMON_H#define _ASM_X86_PKEYS_COMMON_H +#define PKR_RW_BIT 0x0#define PKR_AD_BIT 0x1#define PKR_WD_BIT 0x2#define PKR_BITS_PER_PKEY 2 +#define PKS_NUM_PKEYS 16 +#define PKR_PKEY_SHIFT (pkey) (pkey * PKR_BITS_PER_PKEY)#define PKR_PKEY_MASK (pkey) (((1 << PKR_BITS_PER_PKEY) - 1) << PKR_PKEY_SHIFT(pkey))#define PKR_AD_KEY (pkey) (PKR_AD_BIT << PKR_PKEY_SHIFT(pkey))#define PKR_WD_KEY (pkey) (PKR_WD_BIT << PKR_PKEY_SHIFT(pkey)) +#define PKR_VALUE (pkey, val) (val << PKR_PKEY_SHIFT(pkey))#endif /*_ASM_X86_PKEYS_COMMON_H */
arch/x86/include/asm/pks.h new file +15 -0
@@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_PKS_H +#define _ASM_X86_PKS_H + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +void setup_pks (void ); + +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +static inline void setup_pks (void ) { } + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +#endif /* _ASM_X86_PKS_H */
arch/x86/kernel/cpu/common.c +2 -0
@@ -59,6 +59,7 @@#include <asm/cpu_device_id.h> #include <asm/uv/uv.h> #include <asm/sigframe.h> +#include <asm/pks.h> #include "cpu.h"
@@ -1590,6 +1591,7 @@ static void identify_cpu(struct cpuinfo_x86 *c)x86_init_rdrand (c);setup_pku (c); +setup_pks ();/* * Clear/Set all flags overridden by options, need do it
arch/x86/mm/pkeys.c +75 -0
78 lines not on this page · raw message
include/linux/pkeys.h +8 -0
@@ -46,4 +46,12 @@ static inline bool arch_pkeys_enabled(void)#endif /* ! CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +enum pks_pkey_consumers { +PKS_KEY_DEFAULT =0 ,/* Must be 0 for default PTE values */ +PKS_KEY_NR_CONSUMERS +}; +extern u32 pkrs_init_value; +#endif +#endif /* _LINUX_PKEYS_H */
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 05/18] x86/pks: Add PKS setup code
arch/x86/include/asm/pks.h line 7
Quoted 3 lines · ira weiny
arch/x86/include/asm/pks.h new file +3
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +void setup_pks (void );
pks_setup()
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +static DEFINE_PER_CPU(u32, pkrs_cache); +u32 __read_mostly pkrs_init_value; + +/* +* write_pkrs() optimizes MSR writes by maintaining a per cpu cache which can +* be checked quickly. +* +* It should also be noted that the underlying WRMSR(MSR_IA32_PKRS) is not
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 234
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+* serializing but still maintains ordering properties similar to WRPKRU. +* The current SDM section on PKRS needs updating but should be the same as +* that of WRPKRU. So to quote from the WRPKRU text: +* +* WRPKRU will never execute transiently. Memory accesses +* affected by PKRU register will not execute (even transiently) +* until all prior executions of WRPKRU have completed execution +* and updated the PKRU register. +*/ +void write_pkrs (u32 new_pkrs)
pkrs_write()
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 239
Quoted 5 lines
arch/x86/mm/pkeys.c +5
-0+{ +u32 *pkrs; + +if (!static_cpu_has (X86_FEATURE_PKS )) +return ;
cpu_feature_enabled() if at all. Why is this function even invoked when PKS is off?
Review
Quoted 2 lines
(quoted diff) +2
-0+ + pkrs = get_cpu_ptr(&pkrs_cache);
As far as I've seen this is mostly called from non-preemptible regions. So that get/put pair is pointless. Stick a lockdep assert into the code and disable preemption at the maybe one callsite which needs it.
Quoted 14 lines
arch/x86/mm/pkeys.c +14
-0+if (*pkrs != new_pkrs) { + *pkrs = new_pkrs; +wrmsrl (MSR_IA32_PKRS , new_pkrs); + } +put_cpu_ptr (pkrs); +} + +/* +* Build a default PKRS value from the array specified by consumers +*/ +static int __init create_initial_pkrs_value (void ) +{ +/* All users get Access Disabled unless changed below */ +u8 consumer_defaults[PKS_NUM_PKEYS ] = {
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 265
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+ [0 ... PKS_NUM_PKEYS-1] = PKR_AD_BIT + }; +int i; + + consumer_defaults[PKS_KEY_DEFAULT ] =PKR_RW_BIT ; + +/* Ensure the number of consumers is less than the number of keys */ +BUILD_BUG_ON (PKS_KEY_NR_CONSUMERS >PKS_NUM_PKEYS ); + + pkrs_init_value =0 ;
This needs to be cleared because the BSS might be non zero?
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 269
Quoted 3 lines
arch/x86/mm/pkeys.c +3
-0+/* Fill the defaults for the consumers */ +for (i =0 ; i <PKS_NUM_PKEYS ; i++) + pkrs_init_value |=PKR_VALUE (i, consumer_defaults[i]);
Also PKR_RW_BIT is a horrible invention:
Quoted 4 lines
+#define
PKR_RW_BIT0x0
#definePKR_AD_BIT0x1
#definePKR_WD_BIT0x2
#definePKR_BITS_PER_PKEY2
This makes my brain spin. How do you fit 3 bits into 2 bits per key? That's really non-intuitive.
PKR_RW_ENABLE 0x0PKR_ACCESS_DISABLE 0x1PKR_WRITE_DISABLE 0x2
makes it obvious what this is about, no?
Aside of that, the function which set's up the init value is really bogus. As you explained in the cover letter a kernel user has to:
1) Claim an index in the enum
2) Add a default value to the array in that function
Seriously? How is that any better than doing:
#define PKS_KEY0_DEFAULT PKR_RW_ENABLE
#define PKS_KEY1_DEFAULT PKR_ACCESS_DISABLE
...
#define PKS_KEY15_DEFAULT PKR_ACCESS_DISABLE
and let the compiler construct pkrs_init_value?
TBH, it's not and this function has to be ripped out in case that you need a dynamic allocation of keys some day. So what is this buying us aside of horrible to read and utterly pointless code?
Quoted 4 lines
arch/x86/mm/pkeys.c +4
-0+return 0 ; +} +early_initcall (create_initial_pkrs_value); +
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 284
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+/* +* PKS is independent of PKU and either or both may be supported on a CPU. +* Configure PKS if the CPU supports the feature. +*/ +void setup_pks (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +write_pkrs (pkrs_init_value);
Is the init value set up _before_ this function is invoked for the first time?
Thanks,
tglx
Re: [PATCH V7 05/18] x86/pks: Add PKS setup code
Quoted 131 lines · Thomas Gleixner
arch/x86/include/asm/pks.h line 7
Quoted 3 lines · ira weiny
arch/x86/include/asm/pks.h new file +3
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +void setup_pks (void );
pks_setup()Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +static DEFINE_PER_CPU(u32, pkrs_cache); +u32 __read_mostly pkrs_init_value; + +/* +* write_pkrs() optimizes MSR writes by maintaining a per cpu cache which can +* be checked quickly. +* +* It should also be noted that the underlying WRMSR(MSR_IA32_PKRS) is not arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 234
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+* serializing but still maintains ordering properties similar to WRPKRU. +* The current SDM section on PKRS needs updating but should be the same as +* that of WRPKRU. So to quote from the WRPKRU text: +* +* WRPKRU will never execute transiently. Memory accesses +* affected by PKRU register will not execute (even transiently) +* until all prior executions of WRPKRU have completed execution +* and updated the PKRU register. +*/ +void write_pkrs (u32 new_pkrs)
pkrs_write()arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 239
Quoted 5 lines
arch/x86/mm/pkeys.c +5
-0+{ +u32 *pkrs; + +if (!static_cpu_has (X86_FEATURE_PKS )) +return ;
cpu_feature_enabled()if at all. Why is this function even invoked when PKS is off?Review
Quoted 2 lines
(quoted diff) +2
-0+ + pkrs = get_cpu_ptr(&pkrs_cache);As far as I've seen this is mostly called from non-preemptible regions. So that get/put pair is pointless. Stick a lockdep assert into the code and disable preemption at the maybe one callsite which needs it.
Quoted 14 lines
arch/x86/mm/pkeys.c +14
-0+if (*pkrs != new_pkrs) { + *pkrs = new_pkrs; +wrmsrl (MSR_IA32_PKRS , new_pkrs); + } +put_cpu_ptr (pkrs); +} + +/* +* Build a default PKRS value from the array specified by consumers +*/ +static int __init create_initial_pkrs_value (void ) +{ +/* All users get Access Disabled unless changed below */ +u8 consumer_defaults[PKS_NUM_PKEYS ] = {arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 265
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+ [0 ... PKS_NUM_PKEYS-1] = PKR_AD_BIT + }; +int i; + + consumer_defaults[PKS_KEY_DEFAULT ] =PKR_RW_BIT ; + +/* Ensure the number of consumers is less than the number of keys */ +BUILD_BUG_ON (PKS_KEY_NR_CONSUMERS >PKS_NUM_PKEYS ); + + pkrs_init_value =0 ;This needs to be cleared because the BSS might be non zero?
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 269
Quoted 3 lines
arch/x86/mm/pkeys.c +3
-0+/* Fill the defaults for the consumers */ +for (i =0 ; i <PKS_NUM_PKEYS ; i++) + pkrs_init_value |=PKR_VALUE (i, consumer_defaults[i]);Also
PKR_RW_BITis a horrible invention:Quoted 4 lines
+#define
PKR_RW_BIT0x0
#definePKR_AD_BIT0x1
#definePKR_WD_BIT0x2
#definePKR_BITS_PER_PKEY2This makes my brain spin. How do you fit 3 bits into 2 bits per key? That's really non-intuitive.
PKR_RW_ENABLE0x0PKR_ACCESS_DISABLE0x1PKR_WRITE_DISABLE0x2makes it obvious what this is about, no?
Aside of that, the function which set's up the init value is really bogus. As you explained in the cover letter a kernel user has to:
1) Claim an index in the enum
2) Add a default value to the array in that functionSeriously? How is that any better than doing:
#define
PKS_KEY0_DEFAULTPKR_RW_ENABLE
#definePKS_KEY1_DEFAULTPKR_ACCESS_DISABLE
...
#definePKS_KEY15_DEFAULTPKR_ACCESS_DISABLEand let the compiler construct
pkrs_init_value?TBH, it's not and this function has to be ripped out in case that you need a dynamic allocation of keys some day. So what is this buying us aside of horrible to read and utterly pointless code?
Quoted 4 lines
arch/x86/mm/pkeys.c +4
-0+return 0 ; +} +early_initcall (create_initial_pkrs_value); +arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 284
Quoted 10 lines
arch/x86/mm/pkeys.c +10
-0+/* +* PKS is independent of PKU and either or both may be supported on a CPU. +* Configure PKS if the CPU supports the feature. +*/ +void setup_pks (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +write_pkrs (pkrs_init_value);Is the init value set up _before_ this function is invoked for the first time?
Quoted 4 lines
Thanks,
tglx
Setting up for cpu0 is before create_initial_pkrs_value. therefore pkrs value of cpu0 won't be set correctly.
[root@AliYun ~]# rdmsr -a 0x000006E1
0
55555554
55555554
55555554
55555554
55555554
55555554
55555554
55555554
55555554
Here are my test results after applying the patches
Thanks.
Re: [PATCH V7 05/18] x86/pks: Add PKS setup code
Quoted 26 lines · taoyi ty
Quoted 10 lines · Thomas Gleixner
arch/x86/mm/pkeys.c u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags) · line 284
Quoted 6 lines
arch/x86/mm/pkeys.c +6
-0+void setup_pks (void ) +{ +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + +write_pkrs (pkrs_init_value);Is the init value set up _before_ this function is invoked for the first time?
Setting up for cpu0 is before
create_initial_pkrs_value. therefore pkrs value of cpu0 won't be set correctly.[root@AliYun ~]# rdmsr -a
0x000006E1
0
55555554
55555554
55555554
55555554
55555554
55555554
55555554
55555554
55555554
Quoted 1 line
Here are my test results after applying the patches
Thanks for confirming what I assumed from looking at the patches!
tglx
Re: [PATCH V7 05/18] x86/pks: Add PKS setup code
In reply to Thomas GleixnerQuoted 14 lines · Thomas Gleixner
On Tue, Aug 03 2021 at 21:32, ira weiny wrote:
Aside of that, the function which set's up the init value is really bogus. As you explained in the cover letter a kernel user has to:1) Claim an index in the enum
2) Add a default value to the array in that functionSeriously? How is that any better than doing:
#define
PKS_KEY0_DEFAULTPKR_RW_ENABLE
#definePKS_KEY1_DEFAULTPKR_ACCESS_DISABLE
...
#definePKS_KEY15_DEFAULTPKR_ACCESS_DISABLE
Quoted 5 lines
and let the compiler construct
pkrs_init_value?TBH, it's not and this function has to be ripped out in case that you need a dynamic allocation of keys some day. So what is this buying us aside of horrible to read and utterly pointless code?
And as Taoyi confirmed its broken.
It surely takes a reviewer to spot that and an external engineer to run rdmsr -a to validate that this is not working as expected, right?
Sigh...
PATCH v7 6/18 x86/fault: Adjust WARN_ON for PKey fault
In reply to [email protected]From: Ira Weiny <[email protected]>
Previously if a Protection key fault occurred it indicated something very wrong because user page mappings are not supposed to be in the kernel address space.
Now PKey faults may happen on kernel mappings if the feature is enabled.
Remove the warning in the fault path and allow the oops to occur without extra debugging if PKS is enabled.
--- arch/x86/mm/fault.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/mm/fault.c +8 -4
@@ -1141,11 +1141,15 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) {/* -* Protection keys exceptions only happen on user pages. We -* have no user pages in the kernel portion of the address -* space, so do not expect them here. +* X86_PF_PK (Protection key exceptions) may occur on kernel addresses +* when PKS (PKeys Supervisor) is enabled. +* +* However, if PKS is not enabled WARN if this exception is seen +* because there are no user pages in the kernel portion of the address +* space. */ -WARN_ON_ONCE (hw_error_code &X86_PF_PK ); +WARN_ON_ONCE (!cpu_feature_enabled (X86_FEATURE_PKS ) && + (hw_error_code &X86_PF_PK ));#ifdef CONFIG_X86_32 /*
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 7/18 x86/pks: Preserve the PKRS MSR on context switch
In reply to [email protected]From: Ira Weiny <[email protected]>
The PKRS MSR is defined as a per-logical-processor register. This isolates memory access by logical CPU. Unfortunately, the MSR is not managed by XSAVE. Therefore, tasks must save/restore the MSR value on context switch.
Define a saved PKRS value in the task struct. Initialize all tasks with the INIT_PKRS_VALUE and call pkrs_write_current() to set the MSR to the saved task value on schedule in.
---
Changes for V7
Move definitions from asm/processor.h to asm/pks.h s/INIT_PKRS_VALUE/pkrs_init_value
Change pks_init_task()/pks_sched_in() to functions s/pks_sched_in/pks_write_current to be used more generically later in the series
--- arch/x86/include/asm/pks.h | 4 ++++ arch/x86/include/asm/processor.h | 19 ++++++++++++++++++- arch/x86/kernel/process.c | 3 +++ arch/x86/kernel/process_64.c | 3 +++ arch/x86/mm/pkeys.c | 16 ++++++++++++++++ 5 files changed, 44 insertions(+), 1 deletion(-)
arch/x86/include/asm/pks.h +4 -0
@@ -5,10 +5,14 @@#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS void setup_pks (void ); +void pkrs_write_current (void ); +void pks_init_task (struct task_struct *task);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void setup_pks (void ) { } +static inline void pkrs_write_current (void ) { } +static inline void pks_init_task (struct task_struct *task) { }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/include/asm/processor.h +18 -1
@@ -502,6 +502,12 @@ struct thread_struct {unsigned long cr2;unsigned long trap_nr;unsigned long error_code; + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* Saved Protection key register for supervisor mappings */ +u32 saved_pkrs; +#endif +#ifdef CONFIG_VM86 /* Virtual 86 mode info */ struct vm86 *vm86;
@@ -768,7 +774,18 @@ static inline void spin_lock_prefetch(const void *x)#define KSTK_ESP (task) (task_pt_regs(task)->sp)#else -#define INIT_THREAD { } + +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* +* Early task gets full permissions, the restrictive value is set in +* pks_init_task() +*/ +#define INIT_THREAD { \ + .saved_pkrs = 0, \ +} +#else +#define INIT_THREAD { } +#endif extern unsigned long KSTK_ESP (struct task_struct *task);
arch/x86/kernel/process.c +3 -0
@@ -43,6 +43,7 @@#include <asm/io_bitmap.h> #include <asm/proto.h> #include <asm/frame.h> +#include <asm/pks.h> #include "process.h"
@@ -223,6 +224,8 @@ void flush_thread(void)fpu_flush_thread ();pkru_flush_thread (); + +pks_init_task (tsk); }void disable_TSC (void )
arch/x86/kernel/process_64.c +3 -0
@@ -59,6 +59,7 @@/* Not included via unistd.h */ #include <asm/unistd_32_ia32.h> #endif +#include <asm/pks.h> #include "process.h"
@@ -658,6 +659,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)/* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in (); +pkrs_write_current (); +return prev_p; }
arch/x86/mm/pkeys.c +16 -0
@@ -284,5 +284,21 @@ void setup_pks(void)write_pkrs (pkrs_init_value); cr4_set_bits(X86_CR4_PKS); } +; + +/* +* PKRS is only temporarily changed during specific code paths. Only a +* preemption during these windows away from the default value would +* require updating the MSR. write_pkrs() handles this optimization. +*/ +void pkrs_write_current (void ) +{ +write_pkrs (current->thread.saved_pkrs); +} + +void pks_init_task (struct task_struct *task) +{ + task->thread.saved_pkrs = pkrs_init_value; +}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 07/18] x86/pks: Preserve the PKRS MSR on context switch
arch/x86/kernel/process_64.c __switch_to(struct task_struct *prev_p, struct task_struct *next_p) · line 662
Quoted 5 lines · ira weiny
arch/x86/kernel/process_64.c +1
-0@@ -658,6 +659,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)/* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in (); +pkrs_write_current ();
This is invoked from switch_to() and does this extra get/put_cpu_ptr() dance in the write function for no reason. Can you please stop sticking overhead into the hotpath just because?
Thanks,
tglx
PATCH v7 8/18 x86/entry: Preserve PKRS MSR across exceptions
In reply to [email protected]From: Ira Weiny <[email protected]>
The PKRS MSR is not managed by XSAVE. It is preserved through a context switch but this support leaves exception handling code open to memory accesses during exceptions.
2 possible places for preserving this state were considered, irqentry_state_t or pt_regs.[1] pt_regs was much more complicated and was potentially fraught with unintended consequences.[2] However, Andy came up with a way to hide additional values on the stack which could be accessed as "extended_pt_regs".[3] This method allows for; any place which has struct pt_regs can get access to the extra information; no extra information is added to irq_state; and pt_regs is left intact for compatibility with outside tools like BPF.
To simplify, the assembly code only adds space on the stack. The setting or use of any needed values are left to the C code. While some entry points may not use this space it is still added where ever pt_regs is passed to the C code for consistency.
Each nested exception gets another copy of this extended space allowing for any number of levels of exception handling.
In the assembly, a macro is defined to allow a central place to add space for other uses should the need arise.
Finally export pkrs_{save|restore}_irq to the common code to allow it to preserve the current task's PKRS in the new extended pt_regs if enabled.
Peter, Thomas, Andy, Dave, and Dan all suggested parts of the patch or aided in the development of the patch..
[1] https://lore.kernel.org/lkml/CALCETrVe1i5JdyzD_BcctxQJn+ZE3T38EFPgjxN1F577M36g+w@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/#t
[3] https://lore.kernel.org/lkml/CALCETrUHwZPic89oExMMe-WyDY8-O3W68NcZvse3=PGW+iW5=w@mail.gmail.com/
---
Changes for V7:
Rebased to 5.14 entry code
declare write_pkrs() in pks.h s/INIT_PKRS_VALUE/pkrs_init_value
Remove unnecessary INIT_PKRS_VALUE def s/pkrs_save_set_irq/pkrs_save_irq/
The inital value for exceptions is best managed completely within the pkey code.
--- arch/x86/entry/calling.h | 26 +++++++++++++ arch/x86/entry/common.c | 54 ++++++++++++++++++++++++++ arch/x86/entry/entry_64.S | 22 ++++++----- arch/x86/entry/entry_64_compat.S | 6 +-- arch/x86/include/asm/pks.h | 18 +++++++++ arch/x86/include/asm/processor-flags.h | 2 + arch/x86/kernel/head_64.S | 7 ++-- arch/x86/mm/fault.c | 3 ++ include/linux/pkeys.h | 11 +++++- kernel/entry/common.c | 14 ++++++- 10 files changed, 143 insertions(+), 20 deletions(-)
arch/x86/entry/calling.h +26 -0
@@ -63,6 +63,32 @@ For 32-bit we have the following conventions - kernel is built with* for assembly code: */ +/* +* __call_ext_ptregs - Helper macro to call into C with extended pt_regs +* @cfunc: C function to be called +* +* This will ensure that extended_ptregs is added and removed as needed during +* a call into C code. +*/ +.macro __call_ext_ptregs cfunc annotate_retpoline_safe:req +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rsp +#endif + .if \annotate_retpoline_safe == 1 + ANNOTATE_RETPOLINE_SAFE + .endif + call \cfunc +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* remove space for extended_pt_regs */ + addq $EXTENDED_PT_REGS_SIZE, %rsp +#endif +.endm + +.macro call_ext_ptregs cfunc + __call_ext_ptregs \cfunc, annotate_retpoline_safe=0 +.endm + .macro PUSH_REGS rdx=%rdx rax=%rax save_ret=0 .if \save_ret pushq %rsi /* pt_regs->si */
arch/x86/entry/common.c +54 -0
78 lines not on this page · raw message
arch/x86/entry/entry_64.S +12 -10
@@ -332,7 +332,7 @@ SYM_CODE_END(ret_from_fork) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif - call \cfunc + call_ext_ptregs \cfunc jmp error_return .endm
@@ -435,7 +435,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ - call \cfunc + call_ext_ptregs \cfunc jmp paranoid_exit
@@ -496,7 +496,7 @@ SYM_CODE_START(\asmsym) * stack. */ movq %rsp, %rdi /* pt_regs pointer */ - call vc_switch_off_ist + call_ext_ptregs vc_switch_off_ist movq %rax, %rsp /* Switch to new stack */ UNWIND_HINT_REGS
@@ -507,7 +507,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ - call kernel_\cfunc + call_ext_ptregs kernel_\cfunc /* * No need to switch back to the IST stack. The current stack is either
@@ -542,7 +542,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer into first argument */ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ - call \cfunc + call_ext_ptregs \cfunc jmp paranoid_exit
@@ -781,7 +781,7 @@ SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - call xen_pv_evtchn_do_upcall + call_ext_ptregs xen_pv_evtchn_do_upcall jmp error_return SYM_CODE_END(exc_xen_hypervisor_callback)
@@ -987,7 +987,7 @@ SYM_CODE_START_LOCAL(error_entry) /* Put us onto the real thread stack. */ popq %r12 /* save return addr in %12 */ movq %rsp, %rdi /* arg0 = pt_regs pointer */ - call sync_regs + call_ext_ptregs sync_regs movq %rax, %rsp /* switch stack */ ENCODE_FRAME_POINTER pushq %r12
@@ -1042,7 +1042,7 @@ SYM_CODE_START_LOCAL(error_entry) * as if we faulted immediately after IRET. */ mov %rsp, %rdi - call fixup_bad_iret + call_ext_ptregs fixup_bad_iret mov %rax, %rsp jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)
@@ -1148,7 +1148,7 @@ SYM_CODE_START(asm_exc_nmi) movq %rsp, %rdi movq $-1, %rsi - call exc_nmi + call_ext_ptregs exc_nmi /* * Return back to user mode. We must *not* do the normal exit
@@ -1184,6 +1184,8 @@ SYM_CODE_START(asm_exc_nmi) * +---------------------------------------------------------+ * | pt_regs | * +---------------------------------------------------------+ + * | (Optionally) extended_pt_regs | + * +---------------------------------------------------------+ * * The "original" frame is used by hardware. Before re-enabling * NMIs, we need to be done with it, and we need to leave enough
@@ -1360,7 +1362,7 @@ end_repeat_nmi: movq %rsp, %rdi movq $-1, %rsi - call exc_nmi + call_ext_ptregs exc_nmi /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14
arch/x86/entry/entry_64_compat.S +3 -3
@@ -136,7 +136,7 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) .Lsysenter_flags_fixed: movq %rsp, %rdi - call do_SYSENTER_32 + call_ext_ptregs do_SYSENTER_32 /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
@@ -253,7 +253,7 @@ SYM_INNER_LABEL(entry_SYSCALL_compat_after_hwframe, SYM_L_GLOBAL) UNWIND_HINT_REGS movq %rsp, %rdi - call do_fast_syscall_32 + call_ext_ptregs do_fast_syscall_32 /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV
@@ -410,6 +410,6 @@ SYM_CODE_START(entry_INT80_compat) cld movq %rsp, %rdi - call do_int80_syscall_32 + call_ext_ptregs do_int80_syscall_32 jmp swapgs_restore_regs_and_return_to_usermode SYM_CODE_END(entry_INT80_compat)
arch/x86/include/asm/pks.h +18 -0
@@ -4,15 +4,33 @@#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +struct extended_pt_regs { +u32 thread_pkrs; +/* Keep stack 8 byte aligned */ +u32 pad; +struct pt_regs pt_regs; +}; +void setup_pks (void );void pkrs_write_current (void );void pks_init_task (struct task_struct *task); +void write_pkrs (u32 new_pkrs); + +static inline struct extended_pt_regs *extended_pt_regs (struct pt_regs *regs) +{ + return container_of(regs, struct extended_pt_regs, pt_regs); +} + +void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void setup_pks (void ) { }static inline void pkrs_write_current (void ) { }static inline void pks_init_task (struct task_struct *task) { } +static inline void write_pkrs (u32 new_pkrs) { } +static inline void show_extended_regs_oops (struct pt_regs *regs, +unsigned long error_code) { }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/include/asm/processor-flags.h +2 -0
@@ -53,4 +53,6 @@# define X86_CR3_PTI_PCID_USER_BIT 11#endif +#define EXTENDED_PT_REGS_SIZE 8 +#endif /* _ASM_X86_PROCESSOR_FLAGS_H */
arch/x86/kernel/head_64.S +3 -4
@@ -319,8 +319,7 @@ SYM_CODE_START_NOALIGN(vc_boot_ghcb) movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi movq initial_vc_handler(%rip), %rax - ANNOTATE_RETPOLINE_SAFE - call *%rax + __call_ext_ptregs *%rax, annotate_retpoline_safe=1 /* Unwind pt_regs */ POP_REGS
@@ -397,7 +396,7 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) UNWIND_HINT_REGS movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */ - call do_early_exception + call_ext_ptregs do_early_exception decl early_recursion_flag(%rip) jmp restore_regs_and_return_to_kernel
@@ -421,7 +420,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Call C handler */ movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi - call do_vc_no_ghcb + call_ext_ptregs do_vc_no_ghcb /* Unwind pt_regs */ POP_REGS
arch/x86/mm/fault.c +3 -0
@@ -32,6 +32,7 @@#include <asm/pgtable_areas.h> /* VMALLOC_START, ... */ #include <asm/kvm_para.h> /* kvm_handle_async_pf */ #include <asm/vdso.h> /* fixup_vdso_exception() */ +#include <asm/pks.h> #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h>
@@ -547,6 +548,8 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad (error_code &X86_PF_PK ) ?"protection keys violation" : "permissions violation"); +show_extended_regs_oops (regs, error_code); +if (!(error_code &X86_PF_USER ) &&user_mode (regs)) {struct desc_ptr idt, gdt;u16 ldtr, tr;
include/linux/pkeys.h +10 -1
@@ -52,6 +52,15 @@ enum pks_pkey_consumers {PKS_KEY_NR_CONSUMERS };extern u32 pkrs_init_value; -#endif + +void pkrs_save_irq (struct pt_regs *regs); +void pkrs_restore_irq (struct pt_regs *regs); + +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +static inline void pkrs_save_irq (struct pt_regs *regs) { } +static inline void pkrs_restore_irq (struct pt_regs *regs) { } + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ #endif /* _LINUX_PKEYS_H */
kernel/entry/common.c +12 -2
@@ -6,6 +6,7 @@#include <linux/livepatch.h> #include <linux/audit.h> #include <linux/tick.h> +#include <linux/pkeys.h> #include "common.h"
@@ -364,7 +365,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_end (); ret.exit_rcu = true; -return ret; + goto done; }/*
@@ -379,6 +380,8 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)trace_hardirqs_off_finish ();instrumentation_end (); +done: +pkrs_save_irq (regs);return ret; }
@@ -404,7 +407,12 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Check whether this returns to user mode */ if (user_mode (regs)) {irqentry_exit_to_user_mode (regs); - }else if (!regs_irqs_disabled (regs)) { +return ; + } + +pkrs_restore_irq (regs); + +if (!regs_irqs_disabled (regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing
@@ -458,11 +466,13 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)ftrace_nmi_enter ();instrumentation_end (); +pkrs_save_irq (regs);return irq_state; } void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) { +pkrs_restore_irq (regs);instrumentation_begin ();ftrace_nmi_exit ();if (irq_state.lockdep) {
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
Quoted 11 lines · Ira Weiny
From: Ira Weiny <[email protected]>
The PKRS MSR is not managed by XSAVE. It is preserved through a context switch but this support leaves exception handling code open to memory accesses during exceptions.
2 possible places for preserving this state were considered,
irqentry_state_tor pt_regs.[1]pt_regswas much more complicated and was potentially fraught with unintended consequences.[2] However, Andy came up with a way to hide additional values on the stack which could be accessed as "extended_pt_regs".[3]
Andy,
I'm preparing to send V8 of this PKS work. But I have not seen any feed back since I originally implemented this in V4[1].
Does this meets your expectations? Are there any issues you can see with this code?
I would appreciate your feedback.
Thanks,
Ira
[1] https://lore.kernel.org/lkml/[email protected]/
Quoted 498 lines
This method allows for; any place
which has structpt_regscan get access to the extra information; no extra information is added toirq_state; andpt_regsis left intact for compatibility with outside tools like BPF.To simplify, the assembly code only adds space on the stack. The setting or use of any needed values are left to the C code. While some entry points may not use this space it is still added where ever
pt_regsis passed to the C code for consistency.Each nested exception gets another copy of this extended space allowing for any number of levels of exception handling.
In the assembly, a macro is defined to allow a central place to add space for other uses should the need arise.
Finally export pkrs_{save|restore}_irq to the common code to allow it to preserve the current task's PKRS in the new extended
pt_regsif enabled.Peter, Thomas, Andy, Dave, and Dan all suggested parts of the patch or aided in the development of the patch..
[1] https://lore.kernel.org/lkml/CALCETrVe1i5JdyzD_BcctxQJn+ZE3T38EFPgjxN1F577M36g+w@mail.gmail.com/
[2] https://lore.kernel.org/lkml/[email protected]/#t
[3] https://lore.kernel.org/lkml/CALCETrUHwZPic89oExMMe-WyDY8-O3W68NcZvse3=PGW+iW5=w@mail.gmail.com/Cc Dave Hansen <[email protected]>Cc Dan Williams <[email protected]>Suggested-by Dave Hansen <[email protected]>Suggested-by Dan Williams <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Ira Weiny <[email protected]>---
Changes for V7:
Rebased to 5.14 entry code
declarewrite_pkrs()inpks.hs/INIT_PKRS_VALUE/pkrs_init_value
Remove unnecessaryINIT_PKRS_VALUEdef s/pkrs_save_set_irq/pkrs_save_irq/
The inital value for exceptions is best managed completely within the pkey code.--- arch/x86/entry/calling.h | 26 +++++++++++++ arch/x86/entry/common.c | 54 ++++++++++++++++++++++++++ arch/x86/entry/entry_64.S | 22 ++++++----- arch/x86/entry/entry_64_compat.S | 6 +-- arch/x86/include/asm/pks.h | 18 +++++++++ arch/x86/include/asm/processor-flags.h | 2 + arch/x86/kernel/head_64.S | 7 ++-- arch/x86/mm/fault.c | 3 ++ include/linux/pkeys.h | 11 +++++- kernel/entry/common.c | 14 ++++++- 10 files changed, 143 insertions(+), 20 deletions(-)arch/x86/entry/calling.h +26
-0@@ -63,6 +63,32 @@ For 32-bit we have the following conventions - kernel is built with* for assembly code: */ +/* +* __call_ext_ptregs - Helper macro to call into C with extended pt_regs +* @cfunc: C function to be called +* +* This will ensure that extended_ptregs is added and removed as needed during +* a call into C code. +*/ +.macro __call_ext_ptregs cfunc annotate_retpoline_safe:req +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rsp +#endif + .if \annotate_retpoline_safe == 1 + ANNOTATE_RETPOLINE_SAFE + .endif + call \cfunc +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* remove space for extended_pt_regs */ + addq $EXTENDED_PT_REGS_SIZE, %rsp +#endif +.endm + +.macro call_ext_ptregs cfunc + __call_ext_ptregs \cfunc, annotate_retpoline_safe=0 +.endm + .macro PUSH_REGS rdx=%rdx rax=%rax save_ret=0 .if \save_ret pushq %rsi /* pt_regs->si */arch/x86/entry/common.c +54
-0@@ -19,6 +19,7 @@#include <linux/nospec.h> #include <linux/syscalls.h> #include <linux/uaccess.h> +#include <linux/pkeys.h> #ifdef CONFIG_XEN_PV #include <xen/xen-ops.h> @@ -34,6 +35,7 @@#include <asm/io_bitmap.h> #include <asm/syscall.h> #include <asm/irq_stack.h> +#include <asm/pks.h> #ifdef CONFIG_X86_64 @@ -252,6 +254,56 @@ SYSCALL_DEFINE0(ni_syscall)return -ENOSYS ; } +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + +void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code) +{ +struct extended_pt_regs *ept_regs =extended_pt_regs (regs); + +if (cpu_feature_enabled (X86_FEATURE_PKS ) && (error_code &X86_PF_PK )) +pr_alert ("PKRS: 0x%x\n" , ept_regs->thread_pkrs); +} + +/* +* PKRS is a per-logical-processor MSR which overlays additional protection for +* pages which have been mapped with a protection key. +* +* Context switches save the MSR in the task struct thus taking that value to +* other processors if necessary. +* +* To protect against exceptions having access to this memory save the current +* thread value and set the PKRS value to be used during the exception. +*/ +void pkrs_save_irq (struct pt_regs *regs) +{ +struct extended_pt_regs *ept_regs; + +BUILD_BUG_ON (sizeof (struct extended_pt_regs ) + !=EXTENDED_PT_REGS_SIZE + +sizeof (struct pt_regs )); + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + + ept_regs =extended_pt_regs (regs); + ept_regs->thread_pkrs = current->thread.saved_pkrs; +write_pkrs (pkrs_init_value); +} + +void pkrs_restore_irq (struct pt_regs *regs) +{ +struct extended_pt_regs *ept_regs; + +if (!cpu_feature_enabled (X86_FEATURE_PKS )) +return ; + + ept_regs =extended_pt_regs (regs); +write_pkrs (ept_regs->thread_pkrs); + current->thread.saved_pkrs = ept_regs->thread_pkrs; +} + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ +#ifdef CONFIG_XEN_PV #ifndef CONFIG_PREEMPTION /* @@ -309,6 +361,8 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { +/* Normally called by irqentry_exit, restore pkrs here */ +pkrs_restore_irq (regs);irqentry_exit_cond_resched ();instrumentation_end ();restore_inhcall (inhcall);arch/x86/entry/entry_64.S +12
-10@@ -332,7 +332,7 @@ SYM_CODE_END(ret_from_fork) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif - call \cfunc + call_ext_ptregs \cfunc jmp error_return .endm@@ -435,7 +435,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ - call \cfunc + call_ext_ptregs \cfunc jmp paranoid_exit@@ -496,7 +496,7 @@ SYM_CODE_START(\asmsym) * stack. */ movq %rsp, %rdi /* pt_regs pointer */ - call vc_switch_off_ist + call_ext_ptregs vc_switch_off_ist movq %rax, %rsp /* Switch to new stack */ UNWIND_HINT_REGS@@ -507,7 +507,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer */ - call kernel_\cfunc + call_ext_ptregs kernel_\cfunc /* * No need to switch back to the IST stack. The current stack is either@@ -542,7 +542,7 @@ SYM_CODE_START(\asmsym) movq %rsp, %rdi /* pt_regs pointer into first argument */ movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ - call \cfunc + call_ext_ptregs \cfunc jmp paranoid_exit@@ -781,7 +781,7 @@ SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - call xen_pv_evtchn_do_upcall + call_ext_ptregs xen_pv_evtchn_do_upcall jmp error_return SYM_CODE_END(exc_xen_hypervisor_callback)@@ -987,7 +987,7 @@ SYM_CODE_START_LOCAL(error_entry) /* Put us onto the real thread stack. */ popq %r12 /* save return addr in %12 */ movq %rsp, %rdi /* arg0 = pt_regs pointer */ - call sync_regs + call_ext_ptregs sync_regs movq %rax, %rsp /* switch stack */ ENCODE_FRAME_POINTER pushq %r12@@ -1042,7 +1042,7 @@ SYM_CODE_START_LOCAL(error_entry) * as if we faulted immediately after IRET. */ mov %rsp, %rdi - call fixup_bad_iret + call_ext_ptregs fixup_bad_iret mov %rax, %rsp jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry)@@ -1148,7 +1148,7 @@ SYM_CODE_START(asm_exc_nmi) movq %rsp, %rdi movq $-1, %rsi - call exc_nmi + call_ext_ptregs exc_nmi /* * Return back to user mode. We must *not* do the normal exit@@ -1184,6 +1184,8 @@ SYM_CODE_START(asm_exc_nmi) * +---------------------------------------------------------+ * | pt_regs | * +---------------------------------------------------------+ + * | (Optionally) extended_pt_regs | + * +---------------------------------------------------------+ * * The "original" frame is used by hardware. Before re-enabling * NMIs, we need to be done with it, and we need to leave enough@@ -1360,7 +1362,7 @@ end_repeat_nmi: movq %rsp, %rdi movq $-1, %rsi - call exc_nmi + call_ext_ptregs exc_nmi /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14arch/x86/entry/entry_64_compat.S +3
-3@@ -136,7 +136,7 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) .Lsysenter_flags_fixed: movq %rsp, %rdi - call do_SYSENTER_32 + call_ext_ptregs do_SYSENTER_32 /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV@@ -253,7 +253,7 @@ SYM_INNER_LABEL(entry_SYSCALL_compat_after_hwframe, SYM_L_GLOBAL) UNWIND_HINT_REGS movq %rsp, %rdi - call do_fast_syscall_32 + call_ext_ptregs do_fast_syscall_32 /* XEN PV guests always use IRET path */ ALTERNATIVE "testl %eax, %eax; jz swapgs_restore_regs_and_return_to_usermode", \ "jmp swapgs_restore_regs_and_return_to_usermode", X86_FEATURE_XENPV@@ -410,6 +410,6 @@ SYM_CODE_START(entry_INT80_compat) cld movq %rsp, %rdi - call do_int80_syscall_32 + call_ext_ptregs do_int80_syscall_32 jmp swapgs_restore_regs_and_return_to_usermode SYM_CODE_END(entry_INT80_compat)arch/x86/include/asm/pks.h +18
-0@@ -4,15 +4,33 @@#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +struct extended_pt_regs { +u32 thread_pkrs; +/* Keep stack 8 byte aligned */ +u32 pad; +struct pt_regs pt_regs; +}; +void setup_pks (void );void pkrs_write_current (void );void pks_init_task (struct task_struct *task); +void write_pkrs (u32 new_pkrs); + +static inline struct extended_pt_regs *extended_pt_regs (struct pt_regs *regs) +{ + return container_of(regs, struct extended_pt_regs, pt_regs); +} + +void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void setup_pks (void ) { }static inline void pkrs_write_current (void ) { }static inline void pks_init_task (struct task_struct *task) { } +static inline void write_pkrs (u32 new_pkrs) { } +static inline void show_extended_regs_oops (struct pt_regs *regs, +unsigned long error_code) { }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ arch/x86/include/asm/processor-flags.h +2
-0@@ -53,4 +53,6 @@# define X86_CR3_PTI_PCID_USER_BIT 11#endif +#define EXTENDED_PT_REGS_SIZE 8 +#endif /* _ASM_X86_PROCESSOR_FLAGS_H */ arch/x86/kernel/head_64.S +3
-4@@ -319,8 +319,7 @@ SYM_CODE_START_NOALIGN(vc_boot_ghcb) movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi movq initial_vc_handler(%rip), %rax - ANNOTATE_RETPOLINE_SAFE - call *%rax + __call_ext_ptregs *%rax, annotate_retpoline_safe=1 /* Unwind pt_regs */ POP_REGS@@ -397,7 +396,7 @@ SYM_CODE_START_LOCAL(early_idt_handler_common) UNWIND_HINT_REGS movq %rsp,%rdi /* RDI = pt_regs; RSI is already trapnr */ - call do_early_exception + call_ext_ptregs do_early_exception decl early_recursion_flag(%rip) jmp restore_regs_and_return_to_kernel@@ -421,7 +420,7 @@ SYM_CODE_START_NOALIGN(vc_no_ghcb) /* Call C handler */ movq %rsp, %rdi movq ORIG_RAX(%rsp), %rsi - call do_vc_no_ghcb + call_ext_ptregs do_vc_no_ghcb /* Unwind pt_regs */ POP_REGSarch/x86/mm/fault.c +3
-0@@ -32,6 +32,7 @@#include <asm/pgtable_areas.h> /* VMALLOC_START, ... */ #include <asm/kvm_para.h> /* kvm_handle_async_pf */ #include <asm/vdso.h> /* fixup_vdso_exception() */ +#include <asm/pks.h> #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h> @@ -547,6 +548,8 @@ show_fault_oops(struct pt_regs *regs, unsigned long error_code, unsigned long ad (error_code &X86_PF_PK ) ?"protection keys violation" : "permissions violation"); +show_extended_regs_oops (regs, error_code); +if (!(error_code &X86_PF_USER ) &&user_mode (regs)) {struct desc_ptr idt, gdt;u16 ldtr, tr;include/linux/pkeys.h +10
-1@@ -52,6 +52,15 @@ enum pks_pkey_consumers {PKS_KEY_NR_CONSUMERS };extern u32 pkrs_init_value; -#endif + +void pkrs_save_irq (struct pt_regs *regs); +void pkrs_restore_irq (struct pt_regs *regs); + +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +static inline void pkrs_save_irq (struct pt_regs *regs) { } +static inline void pkrs_restore_irq (struct pt_regs *regs) { } + +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ #endif /* _LINUX_PKEYS_H */ kernel/entry/common.c +12
-2@@ -6,6 +6,7 @@#include <linux/livepatch.h> #include <linux/audit.h> #include <linux/tick.h> +#include <linux/pkeys.h> #include "common.h" @@ -364,7 +365,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_end (); ret.exit_rcu = true; -return ret; + goto done; }/* @@ -379,6 +380,8 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)trace_hardirqs_off_finish ();instrumentation_end (); +done: +pkrs_save_irq (regs);return ret; }@@ -404,7 +407,12 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Check whether this returns to user mode */ if (user_mode (regs)) {irqentry_exit_to_user_mode (regs); - }else if (!regs_irqs_disabled (regs)) { +return ; + } + +pkrs_restore_irq (regs); + +if (!regs_irqs_disabled (regs)) {/* * If RCU was not watching on entry this needs to be done * carefully and needs the same ordering of lockdep/tracing @@ -458,11 +466,13 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)ftrace_nmi_enter ();instrumentation_end (); +pkrs_save_irq (regs);return irq_state; } void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) { +pkrs_restore_irq (regs);instrumentation_begin ();ftrace_nmi_exit ();if (irq_state.lockdep) {Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
Quoted 15 lines · Ira Weiny
Quoted 11 lines · Ira Weiny
From: Ira Weiny <[email protected]>
The PKRS MSR is not managed by XSAVE. It is preserved through a context switch but this support leaves exception handling code open to memory accesses during exceptions.
2 possible places for preserving this state were considered,
irqentry_state_tor pt_regs.[1]pt_regswas much more complicated and was potentially fraught with unintended consequences.[2] However, Andy came up with a way to hide additional values on the stack which could be accessed as "extended_pt_regs".[3]Andy,
Quoted 7 lines
I'm preparing to send V8 of this PKS work. But I have not seen any feed back since I originally implemented this in V4[1].
Does this meets your expectations? Are there any issues you can see with this code?
I would appreciate your feedback.
Not Andy here, but I'll respond to the patch in a minute.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
In reply to Ira WeinyQuoted 13 lines · Ira Weiny
Quoted 11 lines · Ira Weiny
From: Ira Weiny <[email protected]>
The PKRS MSR is not managed by XSAVE. It is preserved through a context switch but this support leaves exception handling code open to memory accesses during exceptions.
2 possible places for preserving this state were considered,
irqentry_state_tor pt_regs.[1]pt_regswas much more complicated and was potentially fraught with unintended consequences.[2] However, Andy came up with a way to hide additional values on the stack which could be accessed as "extended_pt_regs".[3]
Quoted 7 lines
Andy,
I'm preparing to send V8 of this PKS work. But I have not seen any feed back since I originally implemented this in V4[1].
Does this meets your expectations? Are there any issues you can see with this code?
I think I'm generally okay with the approach to allocating space. All of Thomas' comments still apply, though. (Sorry, I'm horribly behind.)
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
In reply to [email protected]Ira,
Quoted 5 lines · ira weiny
arch/x86/entry/calling.h +5
-0+/* +* __call_ext_ptregs - Helper macro to call into C with extended pt_regs +* @cfunc: C function to be called +* +* This will ensure that extended_ptregs is added and removed as needed during
arch/x86/entry/calling.h For 32-bit we have the following conventions - kernel is built with · line 80
Quoted 10 lines
arch/x86/entry/calling.h +10
-0+* a call into C code. +*/ +.macro __call_ext_ptregs cfunc annotate_retpoline_safe:req +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rsp +#endif + .if \annotate_retpoline_safe == 1 + ANNOTATE_RETPOLINE_SAFE + .endif
This annotation is new and nowhere mentioned why it is part of this patch.
Can you please do _ONE_ functional change per patch and not a unreviewable pile of changes in one go? Same applies for the ASM and the C code changes. The ASM change has to go first and then the C code can build upon it.
arch/x86/entry/calling.h For 32-bit we have the following conventions - kernel is built with · line 85
Quoted 5 lines
arch/x86/entry/calling.h +5
-0+ call \cfunc +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* remove space for extended_pt_regs */ + addq $EXTENDED_PT_REGS_SIZE, %rsp +#endif
I really have to ask the question whether this #ifdeffery has any value at all. 8 bytes extra stack usage is not going to be the end of the world and distro kernels will enable that config anyway.
If we really want to save the space then certainly not by sprinkling CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS all over the place and hiding the extra sized ptregs in the pkrs header.
You are changing generic architecture code so you better think about making such a change generic and extensible. Can folks please start thinking beyond the brim of their teacup and not pretend that the feature they are working on is the unicorn which requires unique magic brandnamed after the unicorn of the day.
If the next feature comes around which needs to save something in that extended area then we are going to change the world again, right? Certainly not.
This wants to go into asm/ptrace.h:
struct pt_regs_aux {
u32 pkrs;
};struct pt_regs_extended {
struct pt_regs_aux aux;
struct pt_regs regs __attribute__((aligned(8)));
};and then have in asm-offset:
DEFINE(PT_REGS_AUX_SIZE, sizeof(struct pt_regs_extended) - sizeof(struct pt_regs));
which does the right thing whatever the size of pt_regs_aux is. So for the above it will have:
#define PT_REGS_AUX_SIZE 8 /* sizeof(struct pt_regs_extended) - sizeof(struct pt_regs) */
Even, if you do
struct pt_regs_aux {
#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS u32 pkrs;
#endif
};
and the config switch is disabled. It's still correct:
#define PT_REGS_AUX_SIZE 0 /* sizeof(struct pt_regs_extended) - sizeof(struct pt_regs) */
See? No magic hardcoded constant, no build time error checking for that constant. Nothing, it just works.
That's one part, but let me come back to this:
arch/x86/entry/calling.h For 32-bit we have the following conventions - kernel is built with · line 76
Quoted 3 lines
arch/x86/entry/calling.h +3
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rsp
What guarantees that RSP points to pt_regs at this point? Nothing at all. It's just pure luck and a question of time until this explodes in hard to diagnose ways.
Because between
movq %rsp, %rdi
and
call ....can legitimately be other code which causes the stack pointer to change. It's not the case today, but nothing prevents this in the future.
The correct thing to do is:
movq %rsp, %rdi
RSP_MAKE_PT_REGS_AUX_SPACE
call ...
RSP_REMOVE_PT_REGS_AUX_SPACEThe few extra macro lines in the actual code are way better as they make it completely obvious what's going on and any misuse can be spotted easily.
arch/x86/entry/common.c SYSCALL_DEFINE0(ni_syscall) · line 277
Quoted 12 lines
arch/x86/entry/common.c +12
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* +* PKRS is a per-logical-processor MSR which overlays additional protection for +* pages which have been mapped with a protection key. +* +* Context switches save the MSR in the task struct thus taking that value to +* other processors if necessary. +* +* To protect against exceptions having access to this memory save the current +* thread value and set the PKRS value to be used during the exception. +*/ +void pkrs_save_irq (struct pt_regs *regs)
That's a misnomer as this is invoked for _any_ exception not just interrupts.
arch/x86/entry/common.c __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) · line 366
Quoted 10 lines
#ifdef
CONFIG_XEN_PV
#ifndefCONFIG_PREEMPTION
/*arch/x86/entry/common.c +2
-0@@ -309,6 +361,8 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { +/* Normally called by irqentry_exit, restore pkrs here */ +pkrs_restore_irq (regs);irqentry_exit_cond_resched ();
Sigh. Consistency is overrated....
arch/x86/include/asm/pks.h line 17
Quoted 5 lines
arch/x86/include/asm/pks.h +2
-0+void setup_pks (void );void pkrs_write_current (void );void pks_init_task (struct task_struct *task); +void write_pkrs (u32 new_pkrs);
So we have pkrs_write_current() and write_pkrs() now. Can you please stick to a common prefix, i.e. pkrs_ ?
Quoted 22 lines
kernel/entry/common.c +2
-1@@ -6,6 +6,7 @@#include <linux/livepatch.h> #include <linux/audit.h> #include <linux/tick.h> +#include <linux/pkeys.h> #include "common.h" @@ -364,7 +365,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_end (); ret.exit_rcu = true; -return ret; + goto done; }/*
kernel/entry/common.c noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) · line 384
Quoted 5 lines
kernel/entry/common.c +2
-0@@ -379,6 +380,8 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)trace_hardirqs_off_finish ();instrumentation_end (); +done: +pkrs_save_irq (regs);
This still calls out into instrumentable code. I explained you before why this is wrong. Also objtool emits warnings to that effect if you do a proper verified build.
kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 413
Quoted 12 lines
kernel/entry/common.c +4
-1return ret; }@@ -404,7 +407,12 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Check whether this returns to user mode */ if (user_mode (regs)) {irqentry_exit_to_user_mode (regs); - }else if (!regs_irqs_disabled (regs)) { +return ; + } + +pkrs_restore_irq (regs);
At least you are now putting it consistently at the wrong place vs. noinstr.
Though, if you look at the xen_pv_evtchn_do_upcall() part where you added this extra invocation you might figure out that adding pkrs_restore_irq() to irqentry_exit_cond_resched() and explicitely to the 'else' path in irqentry_exit() makes it magically consistent for both use cases.
Thanks,
tglx
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
Thomas,
Thanks for the review. Sorry for being so late to respond I was sick all last week and so it took me longer to figure out some of this stuff.
Quoted 13 lines · Thomas Gleixner
Ira,
Quoted 10 lines · ira weiny
(quoted diff) +10
-0+/* + * __call_ext_ptregs - Helper macro to call into C with extended pt_regs + * @cfunc: C function to be called + * + * This will ensure that extended_ptregs is added and removed as needed during + * a call into C code. + */ +.macro __call_ext_ptregs cfunc annotate_retpoline_safe:req +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* add space for extended_pt_regs */
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ subq $EXTENDED_PT_REGS_SIZE, %rsp +#endif + .if \annotate_retpoline_safe == 1 + ANNOTATE_RETPOLINE_SAFE + .endifThis annotation is new and nowhere mentioned why it is part of this patch.
I don't understand. ANNOTATE_RETPOLINE_SAFE has been around since:
9e0e3c5130e9 x86/speculation, objtool: Annotate indirect calls/jumps for objtool
Quoted 5 lines
Can you please do _ONE_ functional change per patch and not a unreviewable pile of changes in one go? Same applies for the ASM and the C code changes. The ASM change has to go first and then the C code can build upon it.
I'm sorry for having the ASM and C code together but this all seemed like 1 change to me.
I can split it if you prefer. How about a patch with just the x86 extended pt_regs stuff but that would leave a zero size for the extended stuff? Then followed by the pks bits?
Quoted 10 lines
Review
Quoted 6 lines
(quoted diff) +5
-0+ call \cfunc +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* remove space for extended_pt_regs */ + addq $EXTENDED_PT_REGS_SIZE, %rsp +#endifI really have to ask the question whether this #ifdeffery has any value at all. 8 bytes extra stack usage is not going to be the end of the world and distro kernels will enable that config anyway.
My goal with this has always been 0 overhead if turned off. So this seemed like a logical addition. Furthermore, ARCH_ENABLE_SUPERVISOR_PKEYS is predicated on ARCH_HAS_SUPERVISOR_PKEYS which is only available with x86_64. This removes the space for x86 when not needed.
All the config stuff was introduced in patch 04/18.[0]
Quoted 7 lines
If we really want to save the space then certainly not by sprinkling
CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYSall over the place and hiding the extra sized ptregs in the pkrs header.You are changing generic architecture code so you better think about making such a change generic and extensible.
I agree. And I tried to do so. The generic entry code is modified only by the addition of pkrs_[save|restore]_irq(). These are only defined if the arch defines ARCH_HAS_SUPERVISOR_PKEYS and furthermore, if something triggers enabling ARCH_ENABLE_SUPERVISOR_PKEYS.
ARCH_HAS_SUPERVISOR_PKEYS is restricted to x86_64 at the moment. All other arch's including x86 should not see any changes in the generic code.
I thought we had agreed that it was ok for me to restrict the addition of the extended pt_regs to what was required for PKS when these changes were discussed. Because at the time I was concerned about my lack of knowledge of all the other architectures.[1]
Quoted 8 lines
Can folks please start
thinking beyond the brim of their teacup and not pretend that the feature they are working on is the unicorn which requires unique magic brandnamed after the unicorn of the day.If the next feature comes around which needs to save something in that extended area then we are going to change the world again, right?
I'm not sure what you mean by 'change the world'. I would anticipate the entry code to be modified with something similar to pks_[save|restore]_irq() and let the arch deal with the specifics.
Also in [1] I thought Peter and Andy agreed that placing additional generic state in the extended pt_regs was not needed and does not buy us anything. I specifically asked if that was something we wanted to do in.[2]
Quoted 4 lines
Certainly not.
This wants to go into
asm/ptrace.h:
Quoted 8 lines
struct pt_regs_aux { u32 pkrs; };struct pt_regs_extended { struct pt_regs_aux aux; struct pt_regs regs __attribute__((aligned(8))); };
Ok the aligned attribute does what I was doing much more gracefully. This is a good idea yes, thank you.
Quoted 18 lines
and then have in asm-offset:
DEFINE(PT_REGS_AUX_SIZE, sizeof(struct pt_regs_extended) - sizeof(struct pt_regs));which does the right thing whatever the size of
pt_regs_auxis. So for the above it will have:#define
PT_REGS_AUX_SIZE8 /*sizeof(struct pt_regs_extended)-sizeof(struct pt_regs)*/Even, if you do
struct
pt_regs_aux{#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS u32 pkrs;#endif
};
Quoted 6 lines
and the config switch is disabled. It's still correct:
#define
PT_REGS_AUX_SIZE0 /*sizeof(struct pt_regs_extended)-sizeof(struct pt_regs)*/See? No magic hardcoded constant, no build time error checking for that constant. Nothing, it just works.
Yes agreed definitely an improvement.
Quoted 10 lines
That's one part, but let me come back to this:
Review
Quoted 3 lines
(quoted diff) +3
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rspWhat guarantees that RSP points to
pt_regsat this point? Nothing at all. It's just pure luck and a question of time until this explodes in hard to diagnose ways.
It took me a bit to wrap my head around what I think you mean. My initial response was that rsp should be the stack pointer for __call_ext_ptregs() just like it was for call. But I think I see that it is better to open code this since others may want to play the same trick without using this code and therefore we may not be getting the extended pt_regs structure on the stack like we think. For example if someone did...
movq %rsp, %rdi RSP_ADD_OTHER_STACK_STUFF __call_ext_ptregs ... RSP_REMOVE_OTHER_STACK_STUFF
... it would be broken.
My assumption was that would be illegal after this patch. But indeed there is no way to easily see that in the future.
Quoted 13 lines
Because between
movq %rsp, %rdi and call ....can legitimately be other code which causes the stack pointer to change. It's not the case today, but nothing prevents this in the future.
The correct thing to do is:
Quoted 8 lines
movq %rsp, %rdi RSP_MAKE_PT_REGS_AUX_SPACE call ... RSP_REMOVE_PT_REGS_AUX_SPACEThe few extra macro lines in the actual code are way better as they make it completely obvious what's going on and any misuse can be spotted easily.
Sure FWIW this is what I had originally but thought it would be cleaner to wrap the 'call'. I will convert it back. Also this removes the annotate_retpoline_safe stuff above.
Quoted 16 lines
Review
Quoted 13 lines
(quoted diff) +12
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* + * PKRS is a per-logical-processor MSR which overlays additional protection for + * pages which have been mapped with a protection key. + * + * Context switches save the MSR in the task struct thus taking that value to + * other processors if necessary. + * + * To protect against exceptions having access to this memory save the current + * thread value and set the PKRS value to be used during the exception. + */ +void pkrs_save_irq(struct pt_regs *regs)That's a misnomer as this is invoked for _any_ exception not just interrupts.
I'm confused by the naming in kernel/entry/common.c then. I'm more than willing to change the name. But I only see irq* for almost everything in that file. And I was trying to follow that convention.
Quoted 6 lines
Quoted 6 lines
#ifdef
CONFIG_XEN_PV
#ifndefCONFIG_PREEMPTION
/*(quoted diff) +0
-0@@ -309,6 +361,8 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
Quoted 7 lines
Review __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) · line 366
Quoted 5 lines
(quoted diff) +2
-0inhcall = get_and_clear_inhcall(); if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) { + /* Normally called by irqentry_exit, restore pkrs here */ + pkrs_restore_irq(regs); irqentry_exit_cond_resched();Sigh. Consistency is overrated....
I'm not that familiar with the xen code so perhaps I missed something?
Quoted 9 lines
Review __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
Quoted 6 lines
(quoted diff) +2
-0+ void setup_pks(void); void pkrs_write_current(void); void pks_init_task(struct task_struct *task); +void write_pkrs(u32 new_pkrs);So we have
pkrs_write_current()andwrite_pkrs()now. Can you please stick to a common prefix, i.e.pkrs_?
Sorry, yes.
Quoted 26 lines
Quoted 26 lines
kernel/entry/common.c +2
-1@@ -6,6 +6,7 @@#include <linux/livepatch.h> #include <linux/audit.h> #include <linux/tick.h> +#include <linux/pkeys.h> #include "common.h" @@ -364,7 +365,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_end (); ret.exit_rcu = true; -return ret; + goto done; }/* @@ -379,6 +380,8 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)trace_hardirqs_off_finish ();instrumentation_end ();
Quoted 6 lines
kernel/entry/common.c noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) · line 384
Quoted 2 lines
kernel/entry/common.c +2
-0+done: +pkrs_save_irq (regs);This still calls out into instrumentable code. I explained you before why this is wrong. Also objtool emits warnings to that effect if you do a proper verified build.
I was not sure what a 'proper verified build' was and objtool was not throwing any warnings for me even if I ran it directly.
10:49:27 > ./tools/objtool/objtool check -n vmlinux.o
vmlinux.o: warning: objtool: ftrace_caller()+0x94: call without frame pointer save/setup
vmlinux.o: warning: objtool: ftrace_regs_caller()+0xde: call without frame pointer save/setup vmlinux.o: warning: objtool: return_to_handler()+0x10: call without frame pointer save/setup vmlinux.o: warning: objtool: copy_mc_fragile() falls through to next function copy_mc_fragile_handle_tail() vmlinux.o: warning: objtool: copy_user_enhanced_fast_string() falls through to next function copy_user_generic_unrolled() vmlinux.o: warning: objtool: __memset() falls through to next function memset_erms()
vmlinux.o: warning: objtool: __memcpy() falls through to next function memcpy_erms()
vmlinux.o: warning: objtool: file already has .static_call_sites section, skipping
After asking around and digging quite a bit I found CONFIG_DEBUG_ENTRY which enabled the check and the error. [But only during a build and not with the above command??? Shouldn't the above command work too?]
What other config options should we be running with to verify the build?
Regardless, reading more about noinstr and looking at the code more carefully I realize I _completely_ misunderstood what you meant before in [3]. I should have asked for clarification.
Yes this was originally marked noinstr because it was called from a noinstr function. I see now, or at least I think I see, that you were taking exception to my blindly marking pkrs_save_irq() noinstr without a good reason.
When you said 'there is absolutely no reason to have this marked noinstr.' I thought that meant we could simply remove it from noinstr. But what I think you meant is that there is no reason to have it _be_ noinstr _and_ I should also make it called from the instrumentable sections of the irqentry_*() calls.
So something like this patch on top of this series? [With an equivalent change for pkrs_restore_irq().]
11:03:18 > git di
kernel/entry/common.c +4 -4
@@ -360,10 +360,11 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)rcu_irq_enter ();instrumentation_begin ();trace_hardirqs_off_finish (); +pkrs_save_irq (regs);instrumentation_end (); ret.exit_rcu = true; - goto done; +return ret; }/*
@@ -376,10 +377,9 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)instrumentation_begin ();rcu_irq_enter_check_tick ();trace_hardirqs_off_finish (); +pkrs_save_irq (regs);instrumentation_end (); -done: -pkrs_save_irq (regs);return ret; }
@@ -462,9 +462,9 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)instrumentation_begin ();trace_hardirqs_off_finish ();ftrace_nmi_enter (); +pkrs_save_irq (regs);instrumentation_end (); -pkrs_save_irq (regs);return irq_state; }
Quoted 4 lines
Quoted 4 lines
kernel/entry/common.c +0
-0return ret; }
Quoted 12 lines
kernel/entry/common.c noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) · line 413
Quoted 9 lines
kernel/entry/common.c +4
-1@@ -404,7 +407,12 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Check whether this returns to user mode */ if (user_mode (regs)) {irqentry_exit_to_user_mode (regs); - }else if (!regs_irqs_disabled (regs)) { +return ; + } + +pkrs_restore_irq (regs);At least you are now putting it consistently at the wrong place vs. noinstr.
Indeed. Sorry about not understanding noinstr fully.
Quoted 7 lines
Though, if you look at the
xen_pv_evtchn_do_upcall()part where you added this extra invocation you might figure out that addingpkrs_restore_irq()toirqentry_exit_cond_resched()and explicitely to the 'else' path inirqentry_exit()makes it magically consistent for both use cases.
Thank you, yes good catch. However, I think I need at least 1 more call in the !regs_irqs_disabled() && state.exit_rcu case right?
11:29:48 > git di
arch/x86/entry/common.c +0 -2
@@ -363,8 +363,6 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { -/* Normally called by irqentry_exit, restore pkrs here */ -pkrs_restore_irq (regs);irqentry_exit_cond_resched ();instrumentation_end ();restore_inhcall (inhcall);
kernel/entry/common.c +7 -3
@@ -385,6 +385,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)void irqentry_exit_cond_resched (void ) { +pkrs_restore_irq (regs);if (!preempt_count ()) {/* Sanity check RCU and thread stack */ rcu_irq_exit_check_preempt ();
@@ -408,8 +409,6 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)return ; } -pkrs_restore_irq (regs); -if (!regs_irqs_disabled (regs)) {/* * If RCU was not watching on entry this needs to be done
@@ -421,6 +420,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +pkrs_restore_irq (regs);instrumentation_end ();rcu_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 );
@@ -439,6 +439,10 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)trace_hardirqs_on ();instrumentation_end (); }else { +instrumentation_begin (); +pkrs_restore_irq (regs); +instrumentation_end (); +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry.
@@ -470,8 +474,8 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) { -pkrs_restore_irq (regs);instrumentation_begin (); +pkrs_restore_irq (regs);ftrace_nmi_exit ();if (irq_state.lockdep) {trace_hardirqs_on_prepare ();
Thank you again for the review,
Ira
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
Quoted 3 lines
Thanks,
tglx
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
On Mon, Dec 06, 2021 at 05:54:23PM -0800, 'Ira Weiny' wrote:
[snip]
Quoted 30 lines
Quoted 7 lines
Though, if you look at the
xen_pv_evtchn_do_upcall()part where you added this extra invocation you might figure out that addingpkrs_restore_irq()toirqentry_exit_cond_resched()and explicitely to the 'else' path inirqentry_exit()makes it magically consistent for both use cases.Thank you, yes good catch. However, I think I need at least 1 more call in the !regs_irqs_disabled() && state.exit_rcu case right?
11:29:48 > git di
arch/x86/entry/common.c +0
-2@@ -363,8 +363,6 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { -/* Normally called by irqentry_exit, restore pkrs here */ -pkrs_restore_irq (regs);irqentry_exit_cond_resched ();instrumentation_end ();restore_inhcall (inhcall);kernel/entry/common.c +0
-0
kernel/entry/common.c noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) · line 386
Quoted 2 lines
kernel/entry/common.c +0
-0@@ -385,6 +385,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)void irqentry_exit_cond_resched (void )
Opps... Of course regs will need to be passed in here now...
Ira
Quoted 57 lines
kernel/entry/common.c +7
-3{ +pkrs_restore_irq (regs);if (!preempt_count ()) {/* Sanity check RCU and thread stack */ rcu_irq_exit_check_preempt ();@@ -408,8 +409,6 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)return ; } -pkrs_restore_irq (regs); -if (!regs_irqs_disabled (regs)) {/* * If RCU was not watching on entry this needs to be done @@ -421,6 +420,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)/* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +pkrs_restore_irq (regs);instrumentation_end ();rcu_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 );@@ -439,6 +439,10 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)trace_hardirqs_on ();instrumentation_end (); }else { +instrumentation_begin (); +pkrs_restore_irq (regs); +instrumentation_end (); +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. @@ -470,8 +474,8 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs) void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) { -pkrs_restore_irq (regs);instrumentation_begin (); +pkrs_restore_irq (regs);ftrace_nmi_exit ();if (irq_state.lockdep) {trace_hardirqs_on_prepare ();Thank you again for the review,
Ira[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/Quoted 3 lines
Thanks,
tglx
Re: [PATCH V7 08/18] x86/entry: Preserve PKRS MSR across exceptions
In reply to Ira WeinyIra,
Quoted 13 lines · Ira Weiny
Quoted 11 lines · Thomas Gleixner
Review
Quoted 8 lines
+.macro
__call_ext_ptregscfunc annotate_retpoline_safe:req(quoted diff) +7
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rsp +#endif + .if \annotate_retpoline_safe == 1 + ANNOTATE_RETPOLINE_SAFE + .endifThis annotation is new and nowhere mentioned why it is part of this patch.
Quoted 4 lines
I don't understand.
ANNOTATE_RETPOLINE_SAFEhas been around since:
9e0e3c5130e9x86/speculation, objtool: Annotate indirect calls/jumps for objtool
Sorry, I misread that macro maze. It's conditional obviously.
Quoted 3 lines
I can split it if you prefer. How about a patch with just the x86 extended
pt_regsstuff but that would leave a zero size for the extended stuff? Then followed by the pks bits?
Whatever makes sense and does one thing per patch.
Quoted 8 lines
Quoted 3 lines
I really have to ask the question whether this #ifdeffery has any value at all. 8 bytes extra stack usage is not going to be the end of the world and distro kernels will enable that config anyway.
My goal with this has always been 0 overhead if turned off. So this seemed like a logical addition. Furthermore,
ARCH_ENABLE_SUPERVISOR_PKEYSis predicated onARCH_HAS_SUPERVISOR_PKEYSwhich is only available withx86_64. This removes the space for x86 when not needed.
The question is not about 64 vs. 32bit. The question is whether the conditional makes sense for 64bit in the first place. Whether this matters for 32bit has to be determined. It makes some sense, but less #ifdeffery and less obfuscation makes sense too.
Quoted 11 lines
Quoted 6 lines
If we really want to save the space then certainly not by sprinkling
CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYSall over the place and hiding the extra sized ptregs in the pkrs header.You are changing generic architecture code so you better think about making such a change generic and extensible.
I agree. And I tried to do so. The generic entry code is modified only by the addition of pkrs_[save|restore]_irq(). These are only defined if the arch defines
ARCH_HAS_SUPERVISOR_PKEYSand furthermore, if something triggers enablingARCH_ENABLE_SUPERVISOR_PKEYS.
I'm talking about generic _architecture_ code, i.e. the code in arch/x86/ which affects all vendors and systems.
Quoted 2 lines
ARCH_HAS_SUPERVISOR_PKEYSis restricted tox86_64at the moment. All other arch's including x86 should not see any changes in the generic code.
That was not the question and I'm well aware of that.
Quoted 6 lines
Quoted 2 lines
If the next feature comes around which needs to save something in that extended area then we are going to change the world again, right?
I'm not sure what you mean by 'change the world'. I would anticipate the entry code to be modified with something similar to pks_[save|restore]_irq() and let the arch deal with the specifics.
If on X86 the next X86 specific feature comes around which needs extra reg space then someone has to change world in arch/x86 again, replace all the ARCH_ENABLE_SUPERVISOR_PKEYS #ifdefs with something else, right?
Instead of adding a new field to pt_regs_aux and be done with it.
Quoted 3 lines
Also in [1] I thought Peter and Andy agreed that placing additional generic state in the extended
pt_regswas not needed and does not buy us anything. I specifically asked if that was something we wanted to do in.[2]
This was about a generic representation which affects the common entry code in kernel/entry/... Can you spot the difference?
What I suggested is _solely_ x86 specific and does not trickle into anything outside of arch/x86.
Quoted 4 lines
Quoted 2 lines
See? No magic hardcoded constant, no build time error checking for that constant. Nothing, it just works.
Yes agreed definitely an improvement.
It's the right thing to do.
Quoted 22 lines
Quoted 9 lines
That's one part, but let me come back to this:
Review
Quoted 3 lines
(quoted diff) +3
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS + /* add space for extended_pt_regs */ + subq $EXTENDED_PT_REGS_SIZE, %rspWhat guarantees that RSP points to
pt_regsat this point? Nothing at all. It's just pure luck and a question of time until this explodes in hard to diagnose ways.It took me a bit to wrap my head around what I think you mean. My initial response was that rsp should be the stack pointer for
__call_ext_ptregs()just like it was for call. But I think I see that it is better to open code this since others may want to play the same trick without using this code and therefore we may not be getting the extendedpt_regsstructure on the stack like we think. For example if someone did...movq %rsp, %rdi RSP_ADD_OTHER_STACK_STUFF __call_ext_ptregs ... RSP_REMOVE_OTHER_STACK_STUFF
Quoted 4 lines
... it would be broken.
My assumption was that would be illegal after this patch. But indeed there is no way to easily see that in the future.
There is no law which forbids to put code there. Aside of that software developmemnt is strictly not based on assumptions by definition.
Quoted 17 lines
Quoted 17 lines
Because between
movq %rsp, %rdi and call ....can legitimately be other code which causes the stack pointer to change. It's not the case today, but nothing prevents this in the future.
The correct thing to do is:
movq %rsp, %rdi RSP_MAKE_PT_REGS_AUX_SPACE call ... RSP_REMOVE_PT_REGS_AUX_SPACE
Quoted 7 lines
Quoted 3 lines
The few extra macro lines in the actual code are way better as they make it completely obvious what's going on and any misuse can be spotted easily.
Sure FWIW this is what I had originally but thought it would be cleaner to wrap the 'call'. I will convert it back. Also this removes the
annotate_retpoline_safestuff above.
It makes the whole ifdeffery more palatable. Hiding everything in hideous macros in not an improvement at all.
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
(quoted diff) +7
-0+#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +/* + * PKRS is a per-logical-processor MSR which overlays additional protection for + * pages which have been mapped with a protection key. + * + * Context switches save the MSR in the task struct thus taking that value to + * other processors if necessary.Quoted 8 lines
Quoted 5 lines
+ *
+ * To protect against exceptions having access to this memory save the current
+ * thread value and set the PKRS value to be used during the exception.
+ */
+voidpkrs_save_irq(struct pt_regs *regs)That's a misnomer as this is invoked for _any_ exception not just interrupts.
Quoted 3 lines
I'm confused by the naming in
kernel/entry/common.cthen. I'm more than willing to change the name. But I only see irq* for almost everything in that file. And I was trying to follow that convention.
Do you see anything named irq* in the NMI parts?
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
#ifdef
CONFIG_XEN_PV
#ifndefCONFIG_PREEMPTION
/*(quoted diff) +0
-0@@ -309,6 +361,8 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)Quoted 7 lines
arch/x86/entry/common.c __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) · line 366
Quoted 5 lines
arch/x86/entry/common.c +2
-0inhcall =get_and_clear_inhcall ();if (inhcall && !WARN_ON_ONCE (state.exit_rcu)) { +/* Normally called by irqentry_exit, restore pkrs here */ +pkrs_restore_irq (regs);irqentry_exit_cond_resched ();Sigh. Consistency is overrated....
Quoted 1 line
I'm not that familiar with the xen code so perhaps I missed something?
Yes, taste. And that has nothing to do with being familiar with XEN code.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
(quoted diff) +2
-0+ /* Normally called by irqentry_exit, restore pkrs here */ + pkrs_restore_irq(regs);irqentry_exit_cond_resched();
Your comment says: Normally called by irqentry_exit
Alone writing this comment should have made you look into the invoked function below:
irqentry_exit_cond_resched();
And because the generic entry code is pretty consequent about naming conventions it's not a surprise that you can find an invocation of irqentry_exit_cond_resched() in irqentry_exit(), right?
So instead of writing 'Normally' which is completely confusing you could have done a proper analysis and figured out what I suggested:
Quoted 5 lines
Quoted 5 lines
Though, if you look at the
xen_pv_evtchn_do_upcall()part where you added this extra invocation you might figure out that addingpkrs_restore_irq()toirqentry_exit_cond_resched()and explicitely to the 'else' path inirqentry_exit()makes it magically consistent for both use cases.
But because your preference seems to be have random naming conventions, i.e. chosing the prefix of the day, I'm not that surprised.
Quoted 2 lines
Thank you, yes good catch. However, I think I need at least 1 more call in the !regs_irqs_disabled() && state.exit_rcu case right?
I take this as a rethorical question.
Quoted 9 lines
Quoted 6 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+done: + pkrs_save_irq(regs);This still calls out into instrumentable code. I explained you before why this is wrong. Also objtool emits warnings to that effect if you do a proper verified build.
I was not sure what a 'proper verified build' was and objtool was not throwing any warnings for me even if I ran it directly.
...
Quoted 2 lines
After asking around and digging quite a bit I found
CONFIG_DEBUG_ENTRYwhich enabled the check and the error.
May I ask the obvious question why you did not ask around when I told you the same thing several month ago?
Seriously, you are changing code which has very obviously placed 'noinstr' annotations on functions and a boat load of very obvious instrumentation_begin()/end() pairs in it along with a gazillion of comments and you just go and add your stuff as you can see fit?
Even _after_ I told you that this is wrong?
The word "engineering" has a meaning. Engineering is based on math and science. Both mandate structured and diligent problem analyis.
Can you pretty please explain me how ignoring these annotations in the first place and then ignoring the related review comments is related to that?
Quoted 2 lines
[But only during a build and not with the above command??? Shouldn't the above command work too?]
Did you even try to figure out what CONFIG_DEBUG_ENTRY does?
# git grep -l CONFIG_DEBUG_ENTRY
and looking at the resulting output which has one very obvious named file in it:
include/linux/instrumentation.h
might have told you the answer. Also
# git log ... # git blame ... # your_favourite_browser https://duckduckgo.com/?q=objtool+noinstr # your_favourite_browser https://duckduckgo.com/?q=objtool+CONFIG_DEBUG_ENTRY
aside of asking colleagues or replying to my previous review comment with a sensible question would have solved that, right?
Asking does not make you look stupid. Not asking and making uninformed assumptions will make you look stupid for sure.
But asking just to avoid homework is not in the book either. The community does not have the capacity to deal with that.
Quoted 3 lines
Regardless, reading more about noinstr and looking at the code more carefully I realize I _completely_ misunderstood what you meant before in [3]. I should have asked for clarification.
Bingo!
Quoted 11 lines
Yes this was originally marked noinstr because it was called from a noinstr function. I see now, or at least I think I see, that you were taking exception to my blindly marking
pkrs_save_irq()noinstr without a good reason.When you said 'there is absolutely no reason to have this marked noinstr.' I thought that meant we could simply remove it from noinstr. But what I think you meant is that there is no reason to have it _be_ noinstr _and_ I should also make it called from the instrumentable sections of the irqentry_*() calls.
So something like this patch on top of this series? [With an equivalent change for
pkrs_restore_irq().]
No comment. Why?
1) This is not an basic enginering course
2) Because I refuse to comment on hastily cobbled together crap which still gets it wrong. Hint:
I did not even look at the result of this patch applied. I just did a mental inventory based on the patch hunks you provided. They simply do not sum up.
Don't dare to ask me why.
Thanks,
tglx
PATCH v7 9/18 x86/pks: Add PKS kernel API
In reply to [email protected]From: Fenghua Yu <[email protected]>
PKS allows kernel users to define domains of page mappings which have additional protections beyond the paging protections. Violating those protections creates a fault which by default will oops.
Each kernel user defines a PKS_KEY_* key value which identifies a PKS domain to be used exclusively by that kernel user. This API is then used to control which pages are part of that domain and the current threads protection of those pages.
4 new functions are added pks_enabled(), pks_mk_noaccess(), pks_mk_readonly(), and pks_mk_readwrite(). 2 new macros are added PAGE_KERNEL_PKEY(key) and _PAGE_PKEY(pkey).
Update the protection key documentation to cover pkeys on supervisor pages. This includes how to reserve a key and set the default permissions on that key.
---
Change for V7
Add pks_enabled() to allow users more dynamic choice on PKS use. Update documentation for key allocation
Remove dynamic key allocation, keys will be allocated statically now.
Add expected CPU generation support to documentation
--- Documentation/core-api/protection-keys.rst | 121 ++++++++++++++++++--- arch/x86/include/asm/pgtable_types.h | 12 ++ arch/x86/mm/pkeys.c | 66 +++++++++++ include/linux/pgtable.h | 4 + include/linux/pkeys.h | 14 +++ 5 files changed, 199 insertions(+), 18 deletions(-)
Documentation/core-api/protection-keys.rst +103 -18
@@ -4,25 +4,30 @@ Memory Protection Keys ====================== -Memory Protection Keys for Userspace (PKU aka PKEYs) is a feature -which is found on Intel's Skylake (and later) "Scalable Processor" -Server CPUs. It will be available in future non-server Intel parts -and future AMD processors. +Memory Protection Keys provide a mechanism for enforcing page-based +protections, but without requiring modification of the page tables +when an application changes protection domains. -For anyone wishing to test or use this feature, it is available in -Amazon's EC2 C5 instances and is known to work there using an Ubuntu -17.04 image. +PKeys Userspace (PKU) is a feature which is found on Intel's Skylake "Scalable +Processor" Server CPUs and later. And it will be available in future +non-server Intel parts and future AMD processors. -Memory Protection Keys provides a mechanism for enforcing page-based -protections, but without requiring modification of the page tables -when an application changes protection domains. It works by -dedicating 4 previously ignored bits in each page table entry to a -"protection key", giving 16 possible keys. +Protection Keys for Supervisor pages (PKS) is available in the SDM since May +2020. + +pkeys work by dedicating 4 previously Reserved bits in each page table entry to +a "protection key", giving 16 possible keys. User and Supervisor pages are +treated separately. + +Protections for each page are controlled with per-CPU registers for each type +of page User and Supervisor. Each of these 32-bit register stores two separate +bits (Access Disable and Write Disable) for each key. -There is also a new user-accessible register (PKRU) with two separate -bits (Access Disable and Write Disable) for each key. Being a CPU -register, PKRU is inherently thread-local, potentially giving each -thread a different set of protections from every other thread. +For Userspace the register is user-accessible (rdpkru/wrpkru). For +Supervisor, the register (MSR_IA32_PKRS) is accessible only to the kernel. + +Being a CPU register, pkeys are inherently thread-local, potentially giving +each thread an independent set of protections from every other thread. There are two new instructions (RDPKRU/WRPKRU) for reading and writing to the new register. The feature is only available in 64-bit mode,
@@ -30,8 +35,11 @@ even though there is theoretically space in the PAE PTEs. These permissions are enforced on data access only and have no effect on instruction fetches. -Syscalls -======== +For kernel space rdmsr/wrmsr are used to access the kernel MSRs. + + +Syscalls for user space keys +============================ There are 3 system calls which directly interact with pkeys::
@@ -98,3 +106,80 @@ with a read():: The kernel will send a SIGSEGV in both cases, but si_code will be set to SEGV_PKERR when violating protection keys versus SEGV_ACCERR when the plain mprotect() permissions are violated. + + +Kernel API for PKS support +========================== + +Similar to user space pkeys, supervisor pkeys allow additional protections to +be defined for a supervisor mappings. Unlike user space pkeys, violations of +these protections result in a a kernel oops. + +Supervisor Memory Protection Keys (PKS) is a feature which is found on Intel's +Sapphire Rapids (and later) "Scalable Processor" Server CPUs. It will also be +available in future non-server Intel parts. + +Also qemu has some support as well: https://www.qemu.org/2021/04/30/qemu-6-0-0/ + +Kernel users intending to use PKS support should depend on +ARCH_HAS_SUPERVISOR_PKEYS, and add their config to ARCH_ENABLE_SUPERVISOR_PKEYS +to turn on this support within the core. + +Users reserve a key value by adding an entry to the enum pks_pkey_consumers and +defining the initial protections in the consumer_defaults[] array. + +For example to configure a key for 'MY_FEATURE' with a default of Write +Disabled. + +:: + + enum pks_pkey_consumers + { + PKS_KEY_DEFAULT, + PKS_KEY_MY_FEATURE, + PKS_KEY_NR_CONSUMERS + } + + ... + consumer_defaults[PKS_KEY_DEFAULT] = 0; + consumer_defaults[PKS_KEY_MY_FEATURE] = PKR_DISABLE_WRITE; + ... + +The following interface is used to manipulate the 'protection domain' defined +by a pkey within the kernel. Setting a pkey value in a supervisor PTE adds +this additional protection to the page. + +:: + + #define PAGE_KERNEL_PKEY(pkey) + #define _PAGE_KEY(pkey) + bool pks_enabled(void); + void pks_mk_noaccess(int pkey); + void pks_mk_readonly(int pkey); + void pks_mk_readwrite(int pkey); + +pks_enabled() allows users to know if PKS is configured and available on the +current running system. + +Kernel users must set the pkey in the page table entries for the mappings they +want to protect. This can be done with PAGE_KERNEL_PKEY() or _PAGE_KEY(). + +The pks_mk*() family of calls allow indinvidual threads to change the +protections for the domain identified by the pkey parameter. 3 states are +available: pks_mk_noaccess(), pks_mk_readonly(), and pks_mk_readwrite() which +set the access to none, read, and read/write respectively. + +The interface sets (Access Disabled (AD=1)) for all keys not in use. + +It should be noted that the underlying WRMSR(MSR_IA32_PKRS) is not serializing +but still maintains ordering properties similar to WRPKRU. Thus it is safe to +immediately use a mapping when the pks_mk*() functions return. + +Older versions of the SDM on PKRS may be wrong with regard to this +serialization. The text should be the same as that of WRPKRU. From the WRPKRU +text: + + WRPKRU will never execute transiently. Memory accesses + affected by PKRU register will not execute (even transiently) + until all prior executions of WRPKRU have completed execution + and updated the PKRU register.
arch/x86/include/asm/pgtable_types.h +12 -0
@@ -71,6 +71,12 @@ _PAGE_PKEY_BIT2 | \ _PAGE_PKEY_BIT3) +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +#define _PAGE_PKEY (pkey) (_AT(pteval_t, pkey) << _PAGE_BIT_PKEY_BIT0) +#else +#define _PAGE_PKEY (pkey) (_AT(pteval_t, 0)) +#endif +#if defined(CONFIG_X86_64 ) || defined(CONFIG_X86_PAE )#define _PAGE_KNL_ERRATUM_MASK (_PAGE_DIRTY | _PAGE_ACCESSED)#else
@@ -226,6 +232,12 @@ enum page_cache_mode {#define PAGE_KERNEL_IO __pgprot_mask(__PAGE_KERNEL_IO)#define PAGE_KERNEL_IO_NOCACHE __pgprot_mask(__PAGE_KERNEL_IO_NOCACHE) +#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS +#define PAGE_KERNEL_PKEY (pkey) __pgprot_mask(__PAGE_KERNEL | _PAGE_PKEY(pkey)) +#else +#define PAGE_KERNEL_PKEY (pkey) PAGE_KERNEL +#endif +#endif /* __ASSEMBLY__ */ /* xwr */
arch/x86/mm/pkeys.c +66 -0
82 lines not on this page · raw message
include/linux/pgtable.h +4 -0
@@ -1526,6 +1526,10 @@ static inline bool arch_has_pfn_modify_check(void)# define PAGE_KERNEL_EXEC PAGE_KERNEL#endif +#ifndef PAGE_KERNEL_PKEY +#define PAGE_KERNEL_PKEY (pkey) PAGE_KERNEL +#endif +/* * Page Table Modification bits for pgtbl_mod_mask. *
include/linux/pkeys.h +14 -0
@@ -56,11 +56,25 @@ extern u32 pkrs_init_value;void pkrs_save_irq (struct pt_regs *regs);void pkrs_restore_irq (struct pt_regs *regs); +bool pks_enabled (void ); +void pks_mk_noaccess (int pkey); +void pks_mk_readonly (int pkey); +void pks_mk_readwrite (int pkey); +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void pkrs_save_irq (struct pt_regs *regs) { }static inline void pkrs_restore_irq (struct pt_regs *regs) { } +static inline bool pks_enabled (void ) +{ +return false; +} + +static inline void pks_mk_noaccess (int pkey) {} +static inline void pks_mk_readonly (int pkey) {} +static inline void pks_mk_readwrite (int pkey) {} +#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ #endif /* _LINUX_PKEYS_H */
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 10/18 x86/pks: Introduce pks_abandon_protections()
In reply to [email protected]From: Ira Weiny <[email protected]>
Unanticipated access to PMEM by otherwise working kernel code would be very disruptive to otherwise working systems. Such access could be through valid uses such as kmap(). In this use case PMEM protections will require the ability to abandon all protections of a pkey on all threads system wide.
Introduce pks_abandon_protections() to allow a user to mask off protection values. This will filter through all the threads of the system as they are scheduled in and in the immediate case override the value should running threads PKS fault.
Update pkrs_write_current(), pks_init_task(), and pkrs_{save|restore}_irq() to account for pkrs_pkey_allowed_mask.
Add handle_abandoned_pks_value() to adjust any already running threads which may fault on an abandoned pkey.
---
Changes for V7
New patch
Significant internal review from Dan Williams and Rick Edgecombe
--- Documentation/core-api/protection-keys.rst | 7 +++- arch/x86/entry/common.c | 6 ++- arch/x86/include/asm/pks.h | 5 +++ arch/x86/mm/fault.c | 24 ++++++----- arch/x86/mm/pkeys.c | 49 ++++++++++++++++++++++ include/linux/pkeys.h | 2 + 6 files changed, 80 insertions(+), 13 deletions(-)
Documentation/core-api/protection-keys.rst +6 -1
@@ -157,6 +157,7 @@ this additional protection to the page. void pks_mk_noaccess(int pkey); void pks_mk_readonly(int pkey); void pks_mk_readwrite(int pkey); + void pks_abandon_protections(int pkey); pks_enabled() allows users to know if PKS is configured and available on the current running system.
@@ -169,7 +170,11 @@ protections for the domain identified by the pkey parameter. 3 states are available: pks_mk_noaccess(), pks_mk_readonly(), and pks_mk_readwrite() which set the access to none, read, and read/write respectively. -The interface sets (Access Disabled (AD=1)) for all keys not in use. +The interface sets Access Disabled for all keys not in use. The +pks_abandon_protections() call reduces the protections for the specified key to +be fully accessible thus abandoning the protections of the key. There is no +way to reverse this. As such pks_abandon_protections() is intended to provide +a 'relief valve' if the PKS protections should prove too restrictive. It should be noted that the underlying WRMSR(MSR_IA32_PKRS) is not serializing but still maintains ordering properties similar to WRPKRU. Thus it is safe to
arch/x86/entry/common.c +4 -2
@@ -37,6 +37,8 @@#include <asm/irq_stack.h> #include <asm/pks.h> +extern u32 pkrs_pkey_allowed_mask; +#ifdef CONFIG_X86_64 static __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)
@@ -287,7 +289,7 @@ void pkrs_save_irq(struct pt_regs *regs) ept_regs =extended_pt_regs (regs); ept_regs->thread_pkrs = current->thread.saved_pkrs; -write_pkrs (pkrs_init_value); +write_pkrs (pkrs_init_value & pkrs_pkey_allowed_mask); }void pkrs_restore_irq (struct pt_regs *regs)
@@ -298,8 +300,8 @@ void pkrs_restore_irq(struct pt_regs *regs)return ; ept_regs =extended_pt_regs (regs); -write_pkrs (ept_regs->thread_pkrs); current->thread.saved_pkrs = ept_regs->thread_pkrs; + pkrs_write_current(); }#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/include/asm/pks.h +5 -0
@@ -22,6 +22,7 @@ static inline struct extended_pt_regs *extended_pt_regs(struct pt_regs *regs) }void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code); +int handle_abandoned_pks_value (struct pt_regs *regs);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
@@ -31,6 +32,10 @@ static inline void pks_init_task(struct task_struct *task) { }static inline void write_pkrs (u32 new_pkrs) { }static inline void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code) { } +static inline int handle_abandoned_pks_value (struct pt_regs *regs) +{ +return 0 ; +}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/mm/fault.c +14 -10
@@ -1143,16 +1143,20 @@ static void do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) { -/* -* X86_PF_PK (Protection key exceptions) may occur on kernel addresses -* when PKS (PKeys Supervisor) is enabled. -* -* However, if PKS is not enabled WARN if this exception is seen -* because there are no user pages in the kernel portion of the address -* space. -*/ - WARN_ON_ONCE(!cpu_feature_enabled(X86_FEATURE_PKS) && - (hw_error_code & X86_PF_PK)); + if (hw_error_code & X86_PF_PK) { +/* +* X86_PF_PK (Protection key exceptions) may occur on kernel +* addresses when PKS (PKeys Supervisor) is enabled. +* +* However, if PKS is not enabled WARN if this exception is +* seen because there are no user pages in the kernel portion +* of the address space. +*/ + WARN_ON_ONCE(!cpu_feature_enabled(X86_FEATURE_PKS)); + +if (handle_abandoned_pks_value (regs)) +return ; + }#ifdef CONFIG_X86_32 /*
arch/x86/mm/pkeys.c +49 -0
77 lines not on this page · raw message
include/linux/pkeys.h +2 -0
@@ -60,6 +60,7 @@ bool pks_enabled(void);void pks_mk_noaccess (int pkey);void pks_mk_readonly (int pkey);void pks_mk_readwrite (int pkey); +void pks_abandon_protections (int pkey);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
@@ -74,6 +75,7 @@ static inline bool pks_enabled(void)static inline void pks_mk_noaccess (int pkey) {}static inline void pks_mk_readonly (int pkey) {}static inline void pks_mk_readwrite (int pkey) {} +static inline void pks_abandon_protections (int pkey) {}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 11/18 x86/pks: Add PKS Test code
In reply to [email protected]From: Ira Weiny <[email protected]>
The core PKS functionality provides an interface for kernel users to reserve a key and set up page tables with that key.
Define test code under CONFIG_PKS_TEST which exercises the core functionality of PKS via a debugfs entry. Basic checks can be triggered on boot with a kernel command line option while both basic and preemption checks can be triggered with separate debugfs values. [See the comment at the top of pks_test.c for details on the values which can be used and what tests they run.]
CONFIG_PKS_TEST enables ARCH_ENABLE_SUPERVISOR_PKEYS but can not co-exist with any GENERAL_PKS_USER. This is because the test code iterates through all the keys and is pretty much not useful in general kernel configs. General PKS users should select GENERAL_PKS_USER which will disable PKS_TEST as well as enable ARCH_ENABLE_SUPERVISOR_PKEYS.
A PKey is not reserved for this test and the test code defines its own PKS_KEY_PKS_TEST.
To test pks_abandon_protections() each test requires the thread to be re-run after resetting the abandoned mask value. Do this by allowing the test code access to the abandoned mask value.
---
Changes for V7
Add testing for pks_abandon_protections()
Adjust pkrs_init_value
Adjust for new defines
Clean up comments
Adjust test for static allocation of pkeys
Use lookup_address() instead of follow_pte()
follow_pte only works on IO and raw PFN mappings, use lookup_address() instead. lookup_address() is constrained to architectures which support it.
--- Documentation/core-api/protection-keys.rst | 6 +- arch/x86/include/asm/pks.h | 18 + arch/x86/mm/fault.c | 8 + arch/x86/mm/pkeys.c | 18 +- lib/Kconfig.debug | 13 + lib/Makefile | 3 + lib/pks/Makefile | 3 + lib/pks/pks_test.c | 864 +++++++++++++++++++++ mm/Kconfig | 5 +- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/test_pks.c | 157 ++++ 11 files changed, 1092 insertions(+), 5 deletions(-) create mode 100644 lib/pks/Makefile create mode 100644 lib/pks/pks_test.c create mode 100644 tools/testing/selftests/x86/test_pks.c
Documentation/core-api/protection-keys.rst +4 -2
@@ -122,8 +122,8 @@ available in future non-server Intel parts. Also qemu has some support as well: https://www.qemu.org/2021/04/30/qemu-6-0-0/ Kernel users intending to use PKS support should depend on -ARCH_HAS_SUPERVISOR_PKEYS, and add their config to ARCH_ENABLE_SUPERVISOR_PKEYS -to turn on this support within the core. +ARCH_HAS_SUPERVISOR_PKEYS, and add their config to GENERAL_PKS_USER to turn on +this support within the core. Users reserve a key value by adding an entry to the enum pks_pkey_consumers and defining the initial protections in the consumer_defaults[] array.
@@ -188,3 +188,5 @@ text: affected by PKRU register will not execute (even transiently) until all prior executions of WRPKRU have completed execution and updated the PKRU register. + +Example code can be found in lib/pks/pks_test.c
arch/x86/include/asm/pks.h +18 -0
@@ -39,4 +39,22 @@ static inline int handle_abandoned_pks_value(struct pt_regs *regs)#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ + +#ifdef CONFIG_PKS_TEST + +#define __static_or_pks_test + +bool pks_test_callback (struct pt_regs *regs); + +#else /* !CONFIG_PKS_TEST */ + +#define __static_or_pks_test static + +static inline bool pks_test_callback (struct pt_regs *regs) +{ +return false; +} + +#endif /* CONFIG_PKS_TEST */ +#endif /* _ASM_X86_PKS_H */
arch/x86/mm/fault.c +8 -0
@@ -1154,6 +1154,14 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned long hw_error_code,*/ WARN_ON_ONCE (!cpu_feature_enabled (X86_FEATURE_PKS )); +/* +* If a protection key exception occurs it could be because a PKS test +* is running. If so, pks_test_callback() will clear the protection +* mechanism and return true to indicate the fault was handled. +*/ +if (pks_test_callback (regs)) +return ; +if (handle_abandoned_pks_value (regs))return ; }
arch/x86/mm/pkeys.c +17 -1
@@ -218,7 +218,7 @@ u32 update_pkey_val(u32 pk_reg, int pkey, unsigned int flags)#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS -static DEFINE_PER_CPU(u32, pkrs_cache); +__static_or_pks_test DEFINE_PER_CPU (u32 ,pkrs_cache );u32 __read_mostly pkrs_init_value;/*
@@ -289,6 +289,22 @@ static int __init create_initial_pkrs_value(void) pkrs_init_value =0 ; pkrs_pkey_allowed_mask =PKRS_ALLOWED_MASK_DEFAULT ; +/* +* PKS_TEST is mutually exclusive to any real users of PKS so define a PKS_TEST +* appropriate value. +* +* NOTE: PKey 0 must still be fully permissive for normal kernel mappings to +* work correctly. +*/ +if (IS_ENABLED (CONFIG_PKS_TEST )) { + pkrs_init_value = (PKR_AD_KEY (1 ) |PKR_AD_KEY (2 ) |PKR_AD_KEY (3 ) | \ +PKR_AD_KEY (4 ) |PKR_AD_KEY (5 ) |PKR_AD_KEY (6 ) | \ +PKR_AD_KEY (7 ) |PKR_AD_KEY (8 ) |PKR_AD_KEY (9 ) | \ +PKR_AD_KEY (10 ) |PKR_AD_KEY (11 ) |PKR_AD_KEY (12 ) | \ +PKR_AD_KEY (13 ) |PKR_AD_KEY (14 ) |PKR_AD_KEY (15 )); +return 0 ; + } +/* Fill the defaults for the consumers */ for (i =0 ; i <PKS_NUM_PKEYS ; i++) pkrs_init_value |=PKR_VALUE (i, consumer_defaults[i]);
lib/Kconfig.debug +13 -0
@@ -2650,6 +2650,19 @@ config HYPERV_TESTING help Select this option to enable Hyper-V vmbus testing. +config PKS_TEST + bool "PKey (S)upervisor testing" + depends on ARCH_HAS_SUPERVISOR_PKEYS + depends on !GENERAL_PKS_USER + help + Select this option to enable testing of PKS core software and + hardware. The PKS core provides a mechanism to allocate keys as well + as maintain the protection settings across context switches. + + Answer N if you don't know what supervisor keys are. + + If unsure, say N. + endmenu # "Kernel Testing and Coverage" source "Documentation/Kconfig"
lib/Makefile +3 -0
@@ -360,3 +360,6 @@ obj-$(CONFIG_CMDLINE_KUNIT_TEST) += cmdline_kunit.o obj-$(CONFIG_SLUB_KUNIT_TEST) += slub_kunit.o obj-$(CONFIG_GENERIC_LIB_DEVMEM_IS_ALLOWED) += devmem_is_allowed.o + +# PKS test +obj-y += pks/
lib/pks/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_PKS_TEST) += pks_test.o
lib/pks/pks_test.c new file +864 -0
864 lines not on this page · raw message
mm/Kconfig +4 -1
@@ -820,8 +820,11 @@ config ARCH_HAS_PKEYS bool config ARCH_HAS_SUPERVISOR_PKEYS bool +config GENERAL_PKS_USER + def_bool n config ARCH_ENABLE_SUPERVISOR_PKEYS - bool + def_bool y + depends on PKS_TEST || GENERAL_PKS_USER config PERCPU_STATS bool "Collect percpu memory statistics"
tools/testing/selftests/x86/Makefile +1 -1
@@ -13,7 +13,7 @@ CAN_BUILD_WITH_NOPIE := $(shell ./check_cc.sh $(CC) trivial_program.c -no-pie) TARGETS_C_BOTHBITS := single_step_syscall sysret_ss_attrs syscall_nt test_mremap_vdso \ check_initial_reg_state sigreturn iopl ioperm \ test_vsyscall mov_ss_trap \ - syscall_arg_fault fsgsbase_restore sigaltstack + syscall_arg_fault fsgsbase_restore sigaltstack test_pks TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer
tools/testing/selftests/x86/test_pks.c new file +157 -0
157 lines not on this page · raw message
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 12/18 x86/pks: Add PKS fault callbacks
In reply to [email protected]From: Rick Edgecombe <[email protected]>
Some PKS keys will want special handling on accesses that violate their permissions. One of these is PMEM which will want to have a mode that logs the access violation, disables protection, and continues rather than oops the machine.
Since PKS faults do not provide the actual key that faulted, this information needs to be recovered by walking the page tables and extracting it from the leaf entry.
This infrastructure could be used to implement abandoned pkeys, but adds support in a separate call such that abandoned pkeys are handled more quickly by skipping the page table walk.
In pkeys.c, define a new api for setting callbacks for individual pkeys.
---
Changes for V7:
New patch
--- Documentation/core-api/protection-keys.rst | 27 +++++++++++- arch/x86/include/asm/pks.h | 7 +++ arch/x86/mm/fault.c | 51 ++++++++++++++++++++++ arch/x86/mm/pkeys.c | 13 ++++++ include/linux/pkeys.h | 2 + 5 files changed, 99 insertions(+), 1 deletion(-)
Documentation/core-api/protection-keys.rst +26 -1
@@ -113,7 +113,8 @@ Kernel API for PKS support Similar to user space pkeys, supervisor pkeys allow additional protections to be defined for a supervisor mappings. Unlike user space pkeys, violations of -these protections result in a a kernel oops. +these protections result in a a kernel oops unless a PKS fault handler is +provided which handles the fault. Supervisor Memory Protection Keys (PKS) is a feature which is found on Intel's Sapphire Rapids (and later) "Scalable Processor" Server CPUs. It will also be
@@ -145,6 +146,30 @@ Disabled. consumer_defaults[PKS_KEY_MY_FEATURE] = PKR_DISABLE_WRITE; ... + +Users may also provide a fault handler which can handle a fault differently +than an oops. Continuing our example from above if 'MY_FEATURE' wanted to +define a handler they can do so by adding the coresponding entry to the +pks_key_callbacks array. + +:: + + #ifdef CONFIG_MY_FEATURE + bool my_feature_pks_fault_callback(unsigned long address, bool write) + { + if (my_feature_fault_is_ok) + return true; + return false; + } + #endif + + static const pks_key_callback pks_key_callbacks[PKS_KEY_NR_CONSUMERS] = { + [PKS_KEY_DEFAULT] = NULL, + #ifdef CONFIG_MY_FEATURE + [PKS_KEY_PGMAP_PROTECTION] = my_feature_pks_fault_callback, + #endif + }; + The following interface is used to manipulate the 'protection domain' defined by a pkey within the kernel. Setting a pkey value in a supervisor PTE adds this additional protection to the page.
arch/x86/include/asm/pks.h +7 -0
@@ -23,6 +23,7 @@ static inline struct extended_pt_regs *extended_pt_regs(struct pt_regs *regs)void show_extended_regs_oops (struct pt_regs *regs,unsigned long error_code);int handle_abandoned_pks_value (struct pt_regs *regs); +bool handle_pks_key_callback (unsigned long address,bool write,u16 key);#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
@@ -36,6 +37,12 @@ static inline int handle_abandoned_pks_value(struct pt_regs *regs) {return 0 ; } +static inline bool handle_pks_key_fault (struct pt_regs *regs, +unsigned long hw_error_code, +unsigned long address) +{ +return false; +}#endif /* CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */
arch/x86/mm/fault.c +51 -0
63 lines not on this page · raw message
arch/x86/mm/pkeys.c +13 -0
@@ -241,6 +241,19 @@ int handle_abandoned_pks_value(struct pt_regs *regs)return (ept_regs->thread_pkrs != old); } +static const pks_key_callback pks_key_callbacks[PKS_KEY_NR_CONSUMERS ] = {0 }; + +bool handle_pks_key_callback (unsigned long address,bool write,u16 key) +{ +if (key >PKS_KEY_NR_CONSUMERS ) +return false; + +if (pks_key_callbacks[key]) +return pks_key_callbacks[key](address, write); + +return false; +} +/* * write_pkrs() optimizes MSR writes by maintaining a per cpu cache which can * be checked quickly.
include/linux/pkeys.h +2 -0
@@ -62,6 +62,8 @@ void pks_mk_readonly(int pkey);void pks_mk_readwrite (int pkey);void pks_abandon_protections (int pkey); +typedef bool (*pks_key_callback )(unsigned long address,bool write); +#else /* !CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS */ static inline void pkrs_save_irq (struct pt_regs *regs) { }
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Re: [PATCH V7 12/18] x86/pks: Add PKS fault callbacks
arch/x86/mm/pkeys.c int handle_abandoned_pks_value(struct pt_regs *regs) · line 249
Quoted 8 lines · On Tue, 2021-08-03 at 21:32 -0700, [email protected] wrote:
+static const
pks_key_callback
pks_key_callbacks[PKS_KEY_NR_CONSUMERS] = { 0 };(quoted diff) +2
-0+ +bool handle_pks_key_callback(unsigned long address, bool write, u16key)
arch/x86/mm/pkeys.c +3
-0+{ +if (key >PKS_KEY_NR_CONSUMERS ) +return false;
Good idea, should be >= though?
arch/x86/mm/pkeys.c int handle_abandoned_pks_value(struct pt_regs *regs) · line 256
Quoted 7 lines
arch/x86/mm/pkeys.c +7
-0+ +if (pks_key_callbacks[key]) +return pks_key_callbacks[key](address, write); + +return false; +} +
Otherwise, I've rebased on this series and didn't hit any problems. Thanks.
Re: [PATCH V7 12/18] x86/pks: Add PKS fault callbacks
Quoted 10 lines · Rick P
arch/x86/mm/pkeys.c int handle_abandoned_pks_value(struct pt_regs *regs) · line 249
Quoted 8 lines · On Tue, 2021-08-03 at 21:32 -0700, [email protected] wrote:
+static const
pks_key_callback
pks_key_callbacks[PKS_KEY_NR_CONSUMERS] = { 0 };(quoted diff) +2
-0+ +bool handle_pks_key_callback(unsigned long address, bool write, u16key)
arch/x86/mm/pkeys.c +3
-0+{ +if (key >PKS_KEY_NR_CONSUMERS ) +return false;Good idea, should be >= though?
Yep. Fixed thanks.
Quoted 11 lines
arch/x86/mm/pkeys.c int handle_abandoned_pks_value(struct pt_regs *regs) · line 256
Quoted 8 lines
arch/x86/mm/pkeys.c +7
-0+ +if (pks_key_callbacks[key]) +return pks_key_callbacks[key](address, write); + +return false; +} +Otherwise, I've rebased on this series and didn't hit any problems. Thanks.
Awesome! I still want Dave and Dan to weigh in prior to me respining with the changes so far.
Thanks,
Ira
PATCH v7 13/18 memremap_pages: Add access protection via supervisor Protection Keys (PKS)
In reply to [email protected]From: Ira Weiny <[email protected]>
The persistent memory (PMEM) driver uses the memremap_pages facility to provide 'struct page' metadata (vmemmap) for PMEM. Given that PMEM capacity maybe orders of magnitude higher capacity than System RAM it presents a large vulnerability surface to stray writes. Unlike stray writes to System RAM, which may result in a crash or other undesirable behavior, stray writes to PMEM additionally are more likely to result in permanent data loss. Reboot is not a remediation for PMEM corruption like it is for System RAM.
Given that PMEM access from the kernel is limited to a constrained set of locations (PMEM driver, Filesystem-DAX, and direct-I/O to a DAX page), it is amenable to supervisor pkey protection. Set up an infrastructure for thread local access protection. Then implement the protection using the new Protection Keys Supervisor (PKS) on architectures that support it.
To enable this extra protection memremap_pages users should check for protection support via pgmap_protection_enabled() and if enabled specify (PGMAP_PROTECTION) in (struct dev_pagemap)->flags to request that access protection.
NOTE: The name of pgmap_protection_enable() and PGMAP_PROTECTION were specifically chosen to isolate the implementation of the protection from higher level users.
Kernel code intending to access this memory can do so through 4 new calls. pgmap_mk_{readwrite,noaccess}() and __pgmap_mk_{readwrite,noaccess}() calls.
The pgmap_mk_*() take a page parameter and the __pgmap_mk_*() calls directly take the dev_pagemap objects. pgmap_mk_*() take care of checking if the page is a page map managed page and are safe to any user who has a reference on the page.
All changes in the protections must be through the above calls. They abstract the protection implementation (currently the PKS api) from the upper layer users.
Furthermore, the calls are nestable by the use of a per task reference count. This ensures that the first call to re-enable protection does not 'break' the last access of the device memory.
NOTE: There are no code paths which directly nest these calls. For this reason multiple reviewers, including Dan and Thomas, asked why this reference counting was needed at this level rather than in a higher level call such as kmap_{atomic,local_page}(). The reason is that pmgmap_mk_read_write() can nest with kmap_{atomic,local_page}(). Therefore push this reference counting to the lower level.
Access to device memory during exceptions (#PF) is expected only from user faults. Therefore there is no need to maintain the reference count when entering or exiting exceptions. However, reference counting will occur during the exception. Recall that protection is automatically enabled during exceptions by the PKS core.[1]
A default of (NVDIMM_PFN && ARCH_HAS_SUPERVISOR_PKEYS) was suggested but logically that is the same as saying default 'yes' because both NVDIMM_PFN and ARCH_HAS_SUPERVISOR_PKEYS are required. Therefore a default of 'yes' was used.
[1] https://lore.kernel.org/lkml/[email protected]/
---
Changes for V7
Add __pgmap_mk_*() calls to allow users who have a dev_pagemap
to call directly into that layer of the API
Add pgmap_protection_enabled() and fail memremap_pages() if
protection is requested and pgmap_protection_enabled() is false
s/PGMAP_PKEY_PROTECT/PGMAP_PROTECTION
This helps to isolate the implementation details of the protection from the higher layers.
s/dev_page_access_ref/pgmap_prot_count s/DEV_PAGEMAP_PROTECTION/DEVMAP_ACCESS_PROTECTION Adjust Kconfig dependency and default
Address feedback from Dan Williams
Add requirement comment to devmap_protected Make pgmap_mk_* static inline
Change to devmap_protected
Change config to DEV_PAGEMAP_PROTECTION
Remove dynamic key use from memremap
This greatly simplifies turning on PKS when requested by the remapping code
#define a static key for pmem use
--- arch/x86/mm/pkeys.c | 3 +- include/linux/memremap.h | 1 + include/linux/mm.h | 62 ++++++++++++++++++++++++++++++++++ include/linux/pkeys.h | 1 + include/linux/sched.h | 7 ++++ init/init_task.c | 3 ++ kernel/fork.c | 3 ++ mm/Kconfig | 18 ++++++++++ mm/memremap.c | 73 ++++++++++++++++++++++++++++++++++++++++ 9 files changed, 170 insertions(+), 1 deletion(-)
arch/x86/mm/pkeys.c +2 -1
@@ -294,7 +294,8 @@ static int __init create_initial_pkrs_value(void) };int i; - consumer_defaults[PKS_KEY_DEFAULT ] =PKR_RW_BIT ; + consumer_defaults[PKS_KEY_DEFAULT ] =PKR_RW_BIT ; + consumer_defaults[PKS_KEY_PGMAP_PROTECTION ] =PKR_AD_BIT ;/* Ensure the number of consumers is less than the number of keys */ BUILD_BUG_ON (PKS_KEY_NR_CONSUMERS >PKS_NUM_PKEYS );
include/linux/memremap.h +1 -0
@@ -90,6 +90,7 @@ struct dev_pagemap_ops { };#define PGMAP_ALTMAP_VALID (1 << 0) +#define PGMAP_PROTECTION (1 << 1)/** * struct dev_pagemap - metadata for ZONE_DEVICE mappings
include/linux/mm.h +62 -0
68 lines not on this page · raw message
include/linux/pkeys.h +1 -0
@@ -49,6 +49,7 @@ static inline bool arch_pkeys_enabled(void)#ifdef CONFIG_ARCH_ENABLE_SUPERVISOR_PKEYS enum pks_pkey_consumers {PKS_KEY_DEFAULT =0 ,/* Must be 0 for default PTE values */ +PKS_KEY_PGMAP_PROTECTION ,PKS_KEY_NR_CONSUMERS };extern u32 pkrs_init_value;
include/linux/sched.h +7 -0
@@ -1400,6 +1400,13 @@ struct task_struct {struct llist_head kretprobe_instances;#endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION +/* +* NOTE: pgmap_prot_count is modified within a single thread of +* execution. So it does not need to be atomic_t. +*/ +u32 pgmap_prot_count; +#endif /* * New fields for task_struct should be added above here, so that * they are included in the randomized portion of task_struct.
init/init_task.c +3 -0
@@ -213,6 +213,9 @@ struct task_struct init_task#ifdef CONFIG_SECCOMP_FILTER .seccomp = { .filter_count = ATOMIC_INIT(0) },#endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + .pgmap_prot_count = 0, +#endif };EXPORT_SYMBOL (init_task);
kernel/fork.c +3 -0
@@ -956,6 +956,9 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)#ifdef CONFIG_MEMCG tsk->active_memcg =NULL ; +#endif +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + tsk->pgmap_prot_count =0 ;#endif return tsk;
mm/Kconfig +18 -0
@@ -790,6 +790,24 @@ config ZONE_DEVICE If FS_DAX is enabled, then say Y. +config DEVMAP_ACCESS_PROTECTION + bool "Access protection for memremap_pages()" + depends on NVDIMM_PFN + depends on ARCH_HAS_SUPERVISOR_PKEYS + select GENERAL_PKS_USER + default y + + help + Enable extra protections on device memory. This protects against + unintended access to devices such as a stray writes. This feature is + particularly useful to protect against corruption of persistent + memory. + + This depends on architecture support of supervisor PKeys and has no + overhead if the architecture does not support them. + + If you have persistent memory say 'Y'. + config DEV_PAGEMAP_OPS bool
mm/memremap.c +73 -0
97 lines not on this page · raw message
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 14/18 memremap_pages: Add memremap.pks_fault_mode
In reply to [email protected]From: Ira Weiny <[email protected]>
Some systems may be using pmem in unanticipated ways. As such it is possible a code path may violation the restrictions of the PMEM PKS protections.
In order to provide a more seamless integration of the PMEM PKS feature provide a pks_fault_mode that allows for a relaxed mode should a previously working feature start to fault on PKS protected PMEM.
2 modes are available:
'relaxed' (default) -- WARN_ONCE, abandon the protections, and continuing to operate.
'strict' -- BUG_ON/or fault indicating the error. This is the most protective of the PMEM memory but may be undesirable in some configurations.
NOTE: There was some debate about if a 3rd mode called 'silent' should be available. 'silent' would be the same as 'relaxed' but not print any output. While 'silent' is nice for admins to reduce console/log output it would result in less motivation to fix invalid access to the protected pmem pages. Therefore, 'silent' is left out.
In addition, kmap() is known to not work with this protection. Provide a new call; pgmap_protection_flag_invalid(). This gives better debugging for missed kmap() users. This call also respects the pks_fault_mode settings.
---
Changes for V7
Leverage Rick Edgecombe's fault callback infrastructure to relax invalid
uses and prevent crashes
From Dan Williams
Use sysfs_* calls for parameter
Make pgmap_disable_protection inline
Remove pfn from warn output
Remove silent parameter option
--- .../admin-guide/kernel-parameters.txt | 14 +++ arch/x86/mm/pkeys.c | 8 +- include/linux/mm.h | 26 ++++++ mm/memremap.c | 85 +++++++++++++++++++ 4 files changed, 132 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +14 -0
@@ -4081,6 +4081,20 @@ pirq= [SMP,APIC] Manual mp-table setup See Documentation/x86/i386/IO-APIC.rst. + memremap.pks_fault_mode= [X86] Control the behavior of page map + protection violations. Violations may not be an actual + use of the memory but simply an attempt to map it in an + incompatible way. + (depends on CONFIG_DEVMAP_ACCESS_PROTECTION + + Format: { relaxed | strict } + + relaxed - Print a warning, disable the protection and + continue execution. + strict - Stop kernel execution via BUG_ON or fault + + default: relaxed + plip= [PPT,NET] Parallel port network link Format: { parport<nr> | timid | 0 } See also Documentation/admin-guide/parport.rst.
arch/x86/mm/pkeys.c +7 -1
@@ -9,6 +9,7 @@#include <linux/debugfs.h> /* debugfs_create_u32() */ #include <linux/mm_types.h> /* mm_struct, vma, etc... */ #include <linux/pkeys.h> /* PKEY_* */ +#include <linux/mm.h> /* fault callback */ #include <uapi/asm-generic/mman-common.h> #include <asm/cpufeature.h> /* boot_cpu_has, ... */
@@ -241,7 +242,12 @@ int handle_abandoned_pks_value(struct pt_regs *regs)return (ept_regs->thread_pkrs != old); } -static const pks_key_callback pks_key_callbacks[PKS_KEY_NR_CONSUMERS ] = {0 }; +static const pks_key_callback pks_key_callbacks[PKS_KEY_NR_CONSUMERS ] = { + [PKS_KEY_DEFAULT ] =NULL , +#ifdef CONFIG_DEVMAP_ACCESS_PROTECTION + [PKS_KEY_PGMAP_PROTECTION ] = pgmap_pks_fault_callback, +#endif +};bool handle_pks_key_callback (unsigned long address,bool write,u16 key) {
include/linux/mm.h +26 -0
@@ -1216,6 +1216,7 @@ static inline bool devmap_protected(struct page *page)return false; } +void __pgmap_protection_flag_invalid (struct dev_pagemap *pgmap);void __pgmap_mk_readwrite (struct dev_pagemap *pgmap);void __pgmap_mk_noaccess (struct dev_pagemap *pgmap);
@@ -1232,6 +1233,27 @@ static inline bool pgmap_check_pgmap_prot(struct page *page)return true; } +/* +* pgmap_protection_flag_invalid - Check and flag an invalid use of a pgmap +* protected page +* +* There are code paths which are known to not be compatible with pgmap +* protections. pgmap_protection_flag_invalid() is provided as a 'relief +* valve' to be used in those functions which are known to be incompatible. +* +* Thus an invalid code path can be flag more precisely what code contains the +* bug vs just flagging a fault. Like the fault handler code this abandons the +* use of the PKS key and optionally allows the calling code path to continue +* based on the configuration of the memremap.pks_fault_mode command line +* (and/or sysfs) option. +*/ +static inline void pgmap_protection_flag_invalid (struct page *page) +{ +if (!pgmap_check_pgmap_prot (page)) +return ; +__pgmap_protection_flag_invalid (page->pgmap); +} +static inline void pgmap_mk_readwrite (struct page *page) {if (!pgmap_check_pgmap_prot (page))
@@ -1247,10 +1269,14 @@ static inline void pgmap_mk_noaccess(struct page *page)bool pgmap_protection_enabled (void ); +bool pgmap_pks_fault_callback (unsigned long address,bool write); +#else static inline void __pgmap_mk_readwrite (struct dev_pagemap *pgmap) { }static inline void __pgmap_mk_noaccess (struct dev_pagemap *pgmap) { } + +static inline void pgmap_protection_flag_invalid (struct page *page) { }static inline void pgmap_mk_readwrite (struct page *page) { }static inline void pgmap_mk_noaccess (struct page *page) { }static inline bool pgmap_protection_enabled (void )
mm/memremap.c +85 -0
91 lines not on this page · raw message
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
Randy Dunlap
reviewerRe: [PATCH V7 14/18] memremap_pages: Add memremap.pks_fault_mode
Documentation/admin-guide/kernel-parameters.txt line 4088
Quoted 13 lines · On 8/3/21 9:32 PM, [email protected] wrote:
Documentation/admin-guide/kernel-parameters.txt +5
-0@@ -4081,6 +4081,20 @@ pirq= [SMP,APIC] Manual mp-table setup See Documentation/x86/i386/IO-APIC.rst. + memremap.pks_fault_mode= [X86] Control the behavior of page map + protection violations. Violations may not be an actual + use of the memory but simply an attempt to map it in an + incompatible way. + (depends on CONFIG_DEVMAP_ACCESS_PROTECTION
Missing closing ')' above.
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt +9
-0+ + Format: { relaxed | strict } + + relaxed - Print a warning, disable the protection and + continue execution. + strict - Stop kernel execution via BUG_ON or fault + + default: relaxed +
Signature · 1 line
~Randy
Re: [PATCH V7 14/18] memremap_pages: Add memremap.pks_fault_mode
Quoted 15 lines · Randy Dunlap
Documentation/admin-guide/kernel-parameters.txt line 4087
Quoted 12 lines · On 8/3/21 9:32 PM, [email protected] wrote:
Documentation/admin-guide/kernel-parameters.txt +5
-0@@ -4081,6 +4081,20 @@ pirq= [SMP,APIC] Manual mp-table setup See Documentation/x86/i386/IO-APIC.rst. + memremap.pks_fault_mode= [X86] Control the behavior of page map + protection violations. Violations may not be an actual + use of the memory but simply an attempt to map it in an + incompatible way. + (depends on CONFIG_DEVMAP_ACCESS_PROTECTIONMissing closing ')' above.
Fixed. Thank you!
Ira
Quoted 16 lines
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +9
-0+ + Format: { relaxed | strict } + + relaxed - Print a warning, disable the protection and + continue execution. + strict - Stop kernel execution via BUG_ON or fault + + default: relaxed +Signature · 1 line
~Randy
Re: [PATCH V7 14/18] memremap_pages: Add memremap.pks_fault_mode
In reply to [email protected]Quoted 6 lines · On Tue, 2021-08-03 at 21:32 -0700, [email protected] wrote:
+static int
param_set_pks_fault_mode(const char *val, const struct kernel_param *kp)mm/memremap.c +4
-0+{ +int ret = -EINVAL ; + +if (!sysfs_streq (val,"relaxed" )) {
mm/memremap.c static void devmap_protection_disable(void) · line 123
Quoted 10 lines
mm/memremap.c +10
-0+ pks_fault_mode =PKS_MODE_RELAXED ; + ret =0 ; + }else if (!sysfs_streq (val,"strict" )) { + pks_fault_mode =PKS_MODE_STRICT ; + ret =0 ; + } + +return ret; +} +
Looks like !sysfs_streq() should be just sysfs_streq().
Re: [PATCH V7 14/18] memremap_pages: Add memremap.pks_fault_mode
Quoted 11 lines · Rick P
Quoted 10 lines · On Tue, 2021-08-03 at 21:32 -0700, [email protected] wrote:
+static int
param_set_pks_fault_mode(const char *val, const struct kernel_param *kp)mm/memremap.c +8
-0+{ +int ret = -EINVAL ; + +if (!sysfs_streq (val,"relaxed" )) { + pks_fault_mode =PKS_MODE_RELAXED ; + ret =0 ; + }else if (!sysfs_streq (val,"strict" )) { + pks_fault_mode =PKS_MODE_STRICT ;
Quoted 8 lines
mm/memremap.c static void devmap_protection_disable(void) · line 123
Quoted 6 lines
mm/memremap.c +6
-0+ ret =0 ; + } + +return ret; +} +Looks like !sysfs_streq() should be just
sysfs_streq().
Indeed. Fixed.
Thanks!
Ira
PATCH v7 15/18 kmap: Add stray access protection for devmap pages
In reply to [email protected]From: Ira Weiny <[email protected]>
Enable PKS protection for devmap pages. The devmap protection facility wants to co-opt kmap_{local_page,atomic}() to mediate access to PKS protected pages.
kmap() allows for global mappings to be established, while the PKS facility depends on thread-local access. For this reason kmap() is not supported, but it leaves a policy decision for what to do when kmap() is attempted on a protected devmap page.
Neither of the 2 current DAX-capable filesystems (ext4 and xfs) perform such global mappings. The bulk of device drivers that would handle devmap pages are not using kmap(). Any future filesystems that gain DAX support, or device drivers wanting to support devmap protected pages will need to move to kmap_local_page(). In the meantime to handle these kmap() users call pgmap_protection_flag_invalid() to flag and invalid use of any potentially protected pages. This allows better debugging of invalided uses vs catching faults later on when the address is used.
Direct-map exposure is already mitigated by default on HIGHMEM systems because by definition HIGHMEM systems do not have large capacities of memory in the direct map. Therefore, to reduce complexity HIGHMEM systems are not supported.
--- include/linux/highmem-internal.h | 5 +++++ mm/Kconfig | 1 + 2 files changed, 6 insertions(+)
include/linux/highmem-internal.h +5 -0
@@ -142,6 +142,7 @@ static inline struct page *kmap_to_page(void *addr)static inline void *kmap (struct page *page) {might_sleep (); +pgmap_protection_flag_invalid (page);return page_address (page); }
@@ -157,6 +158,7 @@ static inline void kunmap(struct page *page)static inline void *kmap_local_page (struct page *page) { +pgmap_mk_readwrite (page);return page_address (page); }
@@ -175,12 +177,14 @@ static inline void __kunmap_local(void *addr)#ifdef ARCH_HAS_FLUSH_ON_KUNMAP kunmap_flush_on_unmap (addr);#endif +pgmap_mk_noaccess (kmap_to_page (addr)); }static inline void *kmap_atomic (struct page *page) {preempt_disable ();pagefault_disable (); +pgmap_mk_readwrite (page);return page_address (page); }
@@ -199,6 +203,7 @@ static inline void __kunmap_atomic(void *addr)#ifdef ARCH_HAS_FLUSH_ON_KUNMAP kunmap_flush_on_unmap (addr);#endif +pgmap_mk_noaccess (kmap_to_page (addr));pagefault_enable ();preempt_enable (); }
mm/Kconfig +1 -0
@@ -794,6 +794,7 @@ config DEVMAP_ACCESS_PROTECTION bool "Access protection for memremap_pages()" depends on NVDIMM_PFN depends on ARCH_HAS_SUPERVISOR_PKEYS + depends on !HIGHMEM select GENERAL_PKS_USER default y
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 16/18 dax: Stray access protection for dax_direct_access()
In reply to [email protected]From: Ira Weiny <[email protected]>
dax_direct_access() provides a way to obtain the direct map address of PMEM memory. Coordinate PKS protection with dax_direct_access() of protected devmap pages.
Introduce 3 new calls dax_{protected,mk_readwrite,mk_noaccess}() These 3 calls do not have to be implemented by the dax provider if no protection is implemented.
Single threads of execution can use dax_mk_{readwrite,noaccess}() to relax the protection of the dax device and allow direct use of the kaddr returned from dax_direct_access(). dax_mk_{readwrite,noaccess}() must be used within the dax_read_[un]lock() protected region. And they only need to be used to guard actual access to the memory pointed to. Other uses of dax_direct_access() do not need to use these guards.
For users who require a permanent address to the dax device such as the DM write cache. dax_protected() indicates that the dax device has additional protections. In this case the user choses to create it's own mapping of the memory.
---
Changes for V7
Rework cover letter.
Do not include a FS_DAX_LIMITED restriction for dcss. It will
simply not implement the protection and there is no need to special case this.
Clean up commit message because I did not originally understand the nuance of the s390 device.
Introduce dax_{protected,mk_readwrite,mk_noaccess}() From Dan Williams
Remove old clean up cruft from previous versions Remove map_protected
Remove 'global' parameters all calls
--- drivers/dax/super.c | 54 ++++++++++++++++++++++++++++++++++++++ drivers/md/dm-writecache.c | 8 +++++- fs/dax.c | 8 ++++++ fs/fuse/virtio_fs.c | 2 ++ include/linux/dax.h | 8 ++++++ 5 files changed, 79 insertions(+), 1 deletion(-)
drivers/dax/super.c +54 -0
@@ -296,6 +296,8 @@ EXPORT_SYMBOL_GPL(dax_attribute_group);* @pgoff: offset in pages from the start of the device to translate * @nr_pages: number of consecutive pages caller can handle relative to @pfn * @kaddr: output parameter that returns a virtual address mapping of pfn +* Direct access through this pointer must be guarded by calls to +* dax_mk_{readwrite,noaccess}() * @pfn: output parameter that returns an absolute pfn translation of @pgoff * * Return: negative errno if an error occurs, otherwise the number of
@@ -389,6 +391,58 @@ void dax_flush(struct dax_device *dax_dev, void *addr, size_t size)#endif EXPORT_SYMBOL_GPL (dax_flush); +bool dax_map_protected (struct dax_device *dax_dev) +{ +if (!dax_alive (dax_dev)) +return false; + +if (dax_dev->ops->map_protected) +return dax_dev->ops->map_protected (dax_dev); +return false; +} +EXPORT_SYMBOL_GPL (dax_map_protected); + +/** +* dax_mk_readwrite() - make protected dax devices read/write +* @dax_dev: the dax device representing the memory to access +* +* Any access of the kaddr memory returned from dax_direct_access() must be +* guarded by dax_mk_readwrite() and dax_mk_noaccess(). This ensures that any +* dax devices which have additional protections are allowed to relax those +* protections for the thread using this memory. +* +* NOTE these calls must be contained within a single thread of execution and +* both must be guarded by dax_read_lock() Which is also a requirement for +* dax_direct_access() anyway. +*/ +void dax_mk_readwrite (struct dax_device *dax_dev) +{ +if (!dax_alive (dax_dev)) +return ; + +if (dax_dev->ops->mk_readwrite) + dax_dev->ops->mk_readwrite (dax_dev); +} +EXPORT_SYMBOL_GPL (dax_mk_readwrite); + +/** +* dax_mk_noaccess() - restore protection to dax devices if needed +* @dax_dev: the dax device representing the memory to access +* +* See dax_direct_access() and dax_mk_readwrite() +* +* NOTE Must be called prior to dax_read_unlock() +*/ +void dax_mk_noaccess (struct dax_device *dax_dev) +{ +if (!dax_alive (dax_dev)) +return ; + +if (dax_dev->ops->mk_noaccess) + dax_dev->ops->mk_noaccess (dax_dev); +} +EXPORT_SYMBOL_GPL (dax_mk_noaccess); +void dax_write_cache (struct dax_device *dax_dev,bool wc) {if (wc)
drivers/md/dm-writecache.c +7 -1
@@ -284,7 +284,13 @@ static int persistent_memory_claim(struct dm_writecache *wc) r = -EOPNOTSUPP ; goto err2; } -if (da != p) { + +/* +* Force the write cache to map the pages directly if the dax device +* mapping is protected or if the number of pages returned was not what +* was requested. +*/ +if (dax_map_protected (wc->ssd_dev->dax_dev) || da != p) {long i; wc->memory_map =NULL ; pages =kvmalloc_array (p,sizeof (struct page *),GFP_KERNEL );
fs/dax.c +8 -0
@@ -728,7 +728,9 @@ static int copy_cow_page_dax(struct block_device *bdev, struct dax_device *dax_dreturn rc; } vto =kmap_atomic (to); +dax_mk_readwrite (dax_dev);copy_user_page (vto, (void __force *)kaddr, vaddr, to); +dax_mk_noaccess (dax_dev);kunmap_atomic (vto);dax_read_unlock (id);return 0 ;
@@ -1096,8 +1098,10 @@ s64 dax_iomap_zero(loff_t pos, u64 length, struct iomap *iomap) }if (!page_aligned) { +dax_mk_readwrite (iomap->dax_dev);memset (kaddr + offset,0 , size);dax_flush (iomap->dax_dev, kaddr + offset, size); +dax_mk_noaccess (iomap->dax_dev); }dax_read_unlock (id);return size;
@@ -1169,6 +1173,8 @@ dax_iomap_actor(struct inode *inode, loff_t pos, loff_t length, void *data,if (map_len > end - pos) map_len = end - pos; +dax_mk_readwrite (dax_dev); +/* * The userspace address for the memory copy has already been * validated via access_ok() in either vfs_read() or
@@ -1181,6 +1187,8 @@ dax_iomap_actor(struct inode *inode, loff_t pos, loff_t length, void *data, xfer =dax_copy_to_iter (dax_dev, pgoff, kaddr, map_len, iter); +dax_mk_noaccess (dax_dev); + pos += xfer; length -= xfer; done += xfer;
fs/fuse/virtio_fs.c +2 -0
@@ -776,8 +776,10 @@ static int virtio_fs_zero_page_range(struct dax_device *dax_dev, rc =dax_direct_access (dax_dev, pgoff, nr_pages, &kaddr,NULL );if (rc <0 )return rc; +dax_mk_readwrite (dax_dev);memset (kaddr,0 , nr_pages <<PAGE_SHIFT );dax_flush (dax_dev, kaddr, nr_pages <<PAGE_SHIFT ); +dax_mk_noaccess (dax_dev);return 0 ; }
include/linux/dax.h +8 -0
@@ -36,6 +36,10 @@ struct dax_operations { struct iov_iter *);/* zero_page_range: required operation. Zero page range */ int (*zero_page_range)(struct dax_device *,pgoff_t ,size_t ); + +bool (*map_protected)(struct dax_device *dax_dev); +void (*mk_readwrite)(struct dax_device *dax_dev); +void (*mk_noaccess)(struct dax_device *dax_dev); };extern struct attribute_group dax_attribute_group;
@@ -228,6 +232,10 @@ int dax_zero_page_range(struct dax_device *dax_dev, pgoff_t pgoff, size_t nr_pages);void dax_flush (struct dax_device *dax_dev,void *addr,size_t size); +bool dax_map_protected (struct dax_device *dax_dev); +void dax_mk_readwrite (struct dax_device *dax_dev); +void dax_mk_noaccess (struct dax_device *dax_dev); +ssize_t dax_iomap_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops); vm_fault_t dax_iomap_fault(struct vm_fault *vmf, enum page_entry_size pe_size,
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 17/18 nvdimm/pmem: Enable stray access protection
In reply to [email protected]From: Ira Weiny <[email protected]>
Now that all potential / valid kernel initiated access' to PMEM have been annotated with {__}pgmap_mk_{readwrite,noaccess}(), turn on PGMAP_PROTECTION.
Implement the dax_protected which communicates this memory has extra protection. Also implement pmem_mk_{readwrite,noaccess}() to relax those protections for valid users.
Internally, the pmem driver uses a cached virtual address, pmem->virt_addr (pmem_addr).
Call __pgmap_mk_{readwrite,noaccess}() directly when PGMAP_PROTECTION is active on the device.
---
Changes for V7
Remove global param
Add internal structure which uses the pmem device and pgmap
device directly in the *_mk_*() calls.
Add pmem dax ops callbacks
Use pgmap_protection_enabled() s/PGMAP_PKEY_PROTECT/PGMAP_PROTECTION
--- drivers/nvdimm/pmem.c | 55 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-)
drivers/nvdimm/pmem.c +54 -1
@@ -138,6 +138,18 @@ static blk_status_t read_pmem(struct page *page, unsigned int off,return BLK_STS_OK ; } +static void __pmem_mk_readwrite (struct pmem_device *pmem) +{ +if (pmem->pgmap.flags &PGMAP_PROTECTION ) +__pgmap_mk_readwrite (&pmem->pgmap); +} + +static void __pmem_mk_noaccess (struct pmem_device *pmem) +{ +if (pmem->pgmap.flags &PGMAP_PROTECTION ) +__pgmap_mk_noaccess (&pmem->pgmap); +} +static blk_status_t pmem_do_read (struct pmem_device *pmem,struct page *page,unsigned int page_off,sector_t sector,unsigned int len)
@@ -149,7 +161,10 @@ static blk_status_t pmem_do_read(struct pmem_device *pmem,if (unlikely (is_bad_pmem (&pmem->bb, sector, len)))return BLK_STS_IOERR ; +__pmem_mk_readwrite (pmem); rc =read_pmem (page, page_off, pmem_addr, len); +__pmem_mk_noaccess (pmem); +flush_dcache_page (page);return rc; }
@@ -181,11 +196,14 @@ static blk_status_t pmem_do_write(struct pmem_device *pmem,* after clear poison. */ flush_dcache_page (page); + +__pmem_mk_readwrite (pmem);write_pmem (pmem_addr, page, page_off, len);if (unlikely (bad_pmem)) { rc =pmem_clear_poison (pmem, pmem_off, len);write_pmem (pmem_addr, page, page_off, len); } +__pmem_mk_noaccess (pmem);return rc; }
@@ -320,6 +338,23 @@ static size_t pmem_copy_to_iter(struct dax_device *dax_dev, pgoff_t pgoff,return _copy_mc_to_iter (addr, bytes, i); } +static bool pmem_map_protected (struct dax_device *dax_dev) +{ +struct pmem_device *pmem =dax_get_private (dax_dev); + +return (pmem->pgmap.flags &PGMAP_PROTECTION ); +} + +static void pmem_mk_readwrite (struct dax_device *dax_dev) +{ +__pmem_mk_readwrite (dax_get_private (dax_dev)); +} + +static void pmem_mk_noaccess (struct dax_device *dax_dev) +{ +__pmem_mk_noaccess (dax_get_private (dax_dev)); +} +static const struct dax_operations pmem_dax_ops = { .direct_access = pmem_dax_direct_access, .dax_supported = generic_fsdax_supported,
@@ -328,6 +363,17 @@ static const struct dax_operations pmem_dax_ops = { .zero_page_range = pmem_dax_zero_page_range, }; +static const struct dax_operations pmem_protected_dax_ops = { + .direct_access = pmem_dax_direct_access, + .dax_supported = generic_fsdax_supported, + .copy_from_iter = pmem_copy_from_iter, + .copy_to_iter = pmem_copy_to_iter, + .zero_page_range = pmem_dax_zero_page_range, + .map_protected = pmem_map_protected, + .mk_readwrite = pmem_mk_readwrite, + .mk_noaccess = pmem_mk_noaccess, +}; +static const struct attribute_group *pmem_attribute_groups[] = { &dax_attribute_group,NULL ,
@@ -432,6 +478,8 @@ static int pmem_attach_disk(struct device *dev,if (is_nd_pfn (dev)) { pmem->pgmap.type =MEMORY_DEVICE_FS_DAX ; pmem->pgmap.ops = &fsdax_pagemap_ops; +if (pgmap_protection_enabled ()) + pmem->pgmap.flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, &pmem->pgmap); pfn_sb = nd_pfn->pfn_sb; pmem->data_offset =le64_to_cpu (pfn_sb->dataoff);
@@ -446,6 +494,8 @@ static int pmem_attach_disk(struct device *dev, pmem->pgmap.nr_range =1 ; pmem->pgmap.type =MEMORY_DEVICE_FS_DAX ; pmem->pgmap.ops = &fsdax_pagemap_ops; +if (pgmap_protection_enabled ()) + pmem->pgmap.flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, &pmem->pgmap); pmem->pfn_flags |=PFN_MAP ; bb_range = pmem->pgmap.range;
@@ -483,7 +533,10 @@ static int pmem_attach_disk(struct device *dev,if (is_nvdimm_sync (nd_region)) flags =DAXDEV_F_SYNC ; - dax_dev =alloc_dax (pmem, disk->disk_name, &pmem_dax_ops, flags); +if (pmem->pgmap.flags &PGMAP_PROTECTION ) + dax_dev =alloc_dax (pmem, disk->disk_name, &pmem_protected_dax_ops, flags); +else + dax_dev =alloc_dax (pmem, disk->disk_name, &pmem_dax_ops, flags);if (IS_ERR (dax_dev)) {return PTR_ERR (dax_dev); }
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9
PATCH v7 18/18 devdax: Enable stray access protection
In reply to [email protected]From: Ira Weiny <[email protected]>
Device dax is primarily accessed through user space. Kernel access is controlled through the kmap interfaces.
Now that all valid kernel initiated access to dax devices have been accounted for with pgmap_mk_{readwrite,noaccess}() through kmap, turn on PGMAP_PKEYS_PROTECT for device dax.
---
Changes for V7
Use pgmap_protetion_enabled()
s/PGMAP_PKEYS_PROTECT/PGMAP_PROTECTION/
--- drivers/dax/device.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/dax/device.c +2 -0
@@ -426,6 +426,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type =MEMORY_DEVICE_GENERIC ; +if (pgmap_protection_enabled ()) + pgmap->flags |=PGMAP_PROTECTION ; addr =devm_memremap_pages (dev, pgmap);if (IS_ERR (addr))return PTR_ERR (addr);
Signature · 1 line
2.28.0.rc0.12.gb6a658bd00c9