Skip to contentSkip to conversation

selftests/vm/pkeys: Bug fixes and a new test

PATCH Applied Dave Hansen 2 participants 6 messages Jun 11, 2021 – Jun 13, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover selftests/vm/pkeys: Bug fixes and a new test

There has been a lot of activity on the x86 front around the XSAVE architecture which is used to context-switch processor state (among other things). In addition, AMD has recently joined the protection keys club by adding processor support for PKU.

The AMD implementation helped uncover a kernel bug around the PKRU "init state", which actually applied to Intel's implementation but was just harder to hit. This series adds a test which is expected to help find this class of bug both on AMD and Intel. All the work around pkeys on x86 also uncovered a few bugs in the selftest.

Any testing of this new code (especially from my powerpc friends) would be appreciated.

Cc Thomas Gleixner <[email protected]>
Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>
#1 lore raw

PATCH 1/4 selftests/vm/pkeys: Fix alloc_random_pkey() to make it really, really random

From: Dave Hansen <[email protected]>

The "random" pkey allocation code currently does the good old:

	srand((unsigned int)time(NULL));

*But*, it unfortunately does this on every random pkey allocation.

There may be thousands of these a second. time() has a one second resolution. So, each time alloc_random_pkey() is called, the PRNG is *RESET* to time(). This is nasty. Normally, if you do:

	srand(<ANYTHING>);
	foo = rand();
	bar = rand();

You'll be quite guaranteed that 'foo' and 'bar' are different. But, if you do:

	srand(1);
	foo = rand();
	srand(1);
	bar = rand();

You are quite guaranteed that 'foo' and 'bar' are the *SAME*. The recent "fix" effectively forced the test case to use the same "random" pkey for the whole test, unless the test run crossed a second boundary.

Only run srand() once at program startup.

This explains some very odd and persistent test failures I've been seeing.

Fixes 6e373263ce07 ("selftests/vm/pkeys: fix alloc_random_pkey() to make it really random")
Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>

---

 b/tools/testing/selftests/vm/protection_keys.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/vm/protection_keys.c renamed from tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_Fix_alloc_random_pkey_to_make_it_really_really_random-1 +2 -1
diff -puN tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_Fix_alloc_random_pkey_to_make_it_really_really_random-1 tools/testing/selftests/vm/protection_keys.c
@@ -561,7 +561,6 @@ int alloc_random_pkey(void)
 	int nr_alloced = 0;
 	int random_index;
 	memset(alloced_pkeys, 0, sizeof(alloced_pkeys));
-	srand((unsigned int)time(NULL));
 
 	/* allocate every possible key and make a note of which ones we got */
 	max_nr_pkey_allocs = NR_PKEYS;
@@ -1552,6 +1551,8 @@ int main(void)
 	int nr_iterations = 22;
 	int pkeys_supported = is_pkeys_supported();
 
+	srand((unsigned int)time(NULL));
+
 	setup_handlers();
 
 	printf("has pkeys: %d\n", pkeys_supported);

_

#2 lore raw

PATCH 2/4 selftests/vm/pkeys: Handle negative sys_pkey_alloc() return code

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

The alloc_pkey() sefltest function wraps the sys_pkey_alloc() system call. On success, it updates its "shadow" register value because sys_pkey_alloc() updates the real register.

But, the success check is wrong. pkey_alloc() considers any non-zero return code to indicate success where the pkey register will be modified. This fails to take negative return codes into account.

Consider only a positive return value as a successful call.

Fixes 5f23f6d082a9 ("x86/pkeys: Add self-tests")
Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>

---

 b/tools/testing/selftests/vm/protection_keys.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/vm/protection_keys.c renamed from tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_ret-code +1 -1
diff -puN tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_ret-code tools/testing/selftests/vm/protection_keys.c
@@ -510,7 +510,7 @@ int alloc_pkey(void)
 			" shadow: 0x%016llx\n",
 			__func__, __LINE__, ret, __read_pkey_reg(),
 			shadow_pkey_reg);
-	if (ret) {
+	if (ret > 0) {
 		/* clear both the bits: */
 		shadow_pkey_reg = set_pkey_bits(shadow_pkey_reg, ret,
 						~PKEY_MASK);

_

#3 lore raw

PATCH 3/4 selftests/vm/pkeys: Refill shadow register after implicit kernel write

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

The pkey test code keeps a "shadow" of the pkey register around. This ensures that any bugs which might write to the register can be caught more quickly.

Generally, userspace has a good idea when the kernel is going to write to the register. For instance, alloc_pkey() is passed a permission mask. The caller of alloc_pkey() can update the shadow based on the return value and the mask.

But, the kernel can also modify the pkey register in a more sneaky way. For mprotect(PROT_EXEC) mappings, the kernel will allocate a pkey and write the pkey register to create an execute-only mapping. The kernel never tells userspace what key it uses for this.

This can cause the test to fail with messages like:

	protection_keys_64.2: pkey-helpers.h:132: _read_pkey_reg: Assertion `pkey_reg == shadow_pkey_reg' failed.

because the shadow was not updated with the new kernel-set value.

Forcibly update the shadow value immediately after an mprotect().

Fixes 6af17cf89e99 ("x86/pkeys/selftests: Add PROT_EXEC test")
Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>

---

 b/tools/testing/selftests/vm/protection_keys.c |    7 +++++++
 1 file changed, 7 insertions(+)
tools/testing/selftests/vm/protection_keys.c renamed from tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_Refill_shadow_register_after_implict_kernel_write-1 +7 -0
diff -puN tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_Refill_shadow_register_after_implict_kernel_write-1 tools/testing/selftests/vm/protection_keys.c
@@ -1448,6 +1448,13 @@ void test_implicit_mprotect_exec_only_me
 	ret = mprotect(p1, PAGE_SIZE, PROT_EXEC);
 	pkey_assert(!ret);
 
+	/*
+	 * Reset the shadow, assuming that the above mprotect()
+	 * correctly changed PKRU, but to an unknown value since
+	 * the actual alllocated pkey is unknown.
+	 */
+	shadow_pkey_reg = __read_pkey_reg();
+
 	dprintf2("pkey_reg: %016llx\n", read_pkey_reg());
 
 	/* Make sure this is an *instruction* fault */

_

#4 lore raw

PATCH 4/4 selftests/vm/pkeys: Exercise x86 XSAVE init state

In reply to Dave Hansen

From: Dave Hansen <[email protected]>

On x86, there is a set of instructions used to save and restore register state collectively known as the XSAVE architecture. There are about a dozen different features managed with XSAVE. The protection keys register, PKRU, is one of those features.

The hardware optimizes XSAVE by tracking when the state has not changed from its initial (init) state. In this case, it can avoid the cost of writing state to memory (it would usually just be a bunch of 0's).

When the pkey register is 0x0 the hardware optionally choose to track the register as being in the init state (optimize away the writes). AMD CPUs do this more aggressively compared to Intel.

On x86, PKRU is rarely in its (very permissive) init state. Instead, the value defaults to something very restrictive. It is not surprising that bugs have popped up in the rare cases when PKRU reaches its init state.

Add a protection key selftest which gets the protection keys register into its init state in a way that should work on Intel and AMD. Then, do a bunch of pkey register reads to watch for inadvertent changes.

This adds "-mxsave" to CFLAGS for all the x86 vm selftests in order to allow use of the XSAVE instruction __builtin functions. This will make the builtins available on all of the vm selftests, but is expected to be harmless.

Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>

---

 b/tools/testing/selftests/vm/Makefile          |    4 -
 b/tools/testing/selftests/vm/pkey-x86.h        |    1 
 b/tools/testing/selftests/vm/protection_keys.c |   73 +++++++++++++++++++++++++
 3 files changed, 76 insertions(+), 2 deletions(-)
tools/testing/selftests/vm/Makefile renamed from tools/testing/selftests/vm/Makefile~selftests_vm_pkeys_Exercise_x86_XSAVE_init_state-1 +2 -2
diff -puN tools/testing/selftests/vm/Makefile~selftests_vm_pkeys_Exercise_x86_XSAVE_init_state-1 tools/testing/selftests/vm/Makefile
@@ -100,7 +100,7 @@ $(1) $(1)_64: $(OUTPUT)/$(1)_64
 endef
 
 ifeq ($(CAN_BUILD_I386),1)
-$(BINARIES_32): CFLAGS += -m32
+$(BINARIES_32): CFLAGS += -m32 -mxsave
 $(BINARIES_32): LDLIBS += -lrt -ldl -lm
 $(BINARIES_32): $(OUTPUT)/%_32: %.c
 	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $(notdir $^) $(LDLIBS) -o $@
@@ -108,7 +108,7 @@ $(foreach t,$(TARGETS),$(eval $(call gen
 endif
 
 ifeq ($(CAN_BUILD_X86_64),1)
-$(BINARIES_64): CFLAGS += -m64
+$(BINARIES_64): CFLAGS += -m64 -mxsave
 $(BINARIES_64): LDLIBS += -lrt -ldl
 $(BINARIES_64): $(OUTPUT)/%_64: %.c
 	$(CC) $(CFLAGS) $(EXTRA_CFLAGS) $(notdir $^) $(LDLIBS) -o $@
tools/testing/selftests/vm/pkey-x86.h renamed from tools/testing/selftests/vm/pkey-x86.h~selftests_vm_pkeys_Exercise_x86_XSAVE_init_state-1 +1 -0
diff -puN tools/testing/selftests/vm/pkey-x86.h~selftests_vm_pkeys_Exercise_x86_XSAVE_init_state-1 tools/testing/selftests/vm/pkey-x86.h
@@ -126,6 +126,7 @@ static inline u32 pkey_bit_position(int
 
 #define XSTATE_PKEY_BIT	(9)
 #define XSTATE_PKEY	0x200
+#define XSTATE_BV_OFFSET	512
 
 int pkey_reg_xstate_offset(void)
 {
tools/testing/selftests/vm/protection_keys.c renamed from tools/testing/selftests/vm/protection_keys.c~selftests_vm_pkeys_Exercise_x86_XSAVE_init_state-1 +73 -0

85 lines not on this page · raw message

_

Tested-by on the v1 series
#5 lore raw
In reply to Dave Hansen
Quoted 5 lines · Dave Hansen

There has been a lot of activity on the x86 front around the XSAVE architecture which is used to context-switch processor state (among other things). In addition, AMD has recently joined the protection keys club by adding processor support for PKU.

Quoted 8 lines

The AMD implementation helped uncover a kernel bug around the PKRU "init state", which actually applied to Intel's implementation but was just harder to hit. This series adds a test which is expected to help find this class of bug both on AMD and Intel. All the work around pkeys on x86 also uncovered a few bugs in the selftest.

Any testing of this new code (especially from my powerpc friends) would be appreciated.

I tested this on ppc64.

Quoted 15 lines
Cc Thomas Gleixner <[email protected]>
Cc Ram Pai <[email protected]>
Cc Sandipan Das <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Florian Weimer <[email protected]>
Cc "Desnes A. Nunes do Rosario" <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Thiago Jung Bauermann <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Michal Suchanek <[email protected]>
Cc Shuah Khan <[email protected]>