Skip to contentSkip to conversation

KVM: selftests: Use TEST_REQUIRE() in nx_huge_pages_test

PATCH In review Oliver Upton 3 participants 5 messages Aug 12, 2022 – Sep 8, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: selftests: Use TEST_REQUIRE() in nx_huge_pages_test

Avoid boilerplate for checking test preconditions by using TEST_REQUIRE(). While at it, add a precondition for KVM_CAP_VM_DISABLE_NX_HUGE_PAGES to skip (instead of silently pass) on older kernels.

---
 .../selftests/kvm/x86_64/nx_huge_pages_test.c | 24 +++++--------------
 1 file changed, 6 insertions(+), 18 deletions(-)
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +6 -18
index cc6421716400..e19933ea34ca 100644
@@ -118,13 +118,6 @@ void run_test(int reclaim_period_ms, bool disable_nx_huge_pages,
 	vm = vm_create(1);
 
 	if (disable_nx_huge_pages) {
-		/*
-		 * Cannot run the test without NX huge pages if the kernel
-		 * does not support it.
-		 */
-		if (!kvm_check_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
-			return;
-
 		r = __vm_disable_nx_huge_pages(vm);
 		if (reboot_permissions) {
 			TEST_ASSERT(!r, "Disabling NX huge pages should succeed if process has reboot permissions");
@@ -248,18 +241,13 @@ int main(int argc, char **argv)
 		}
 	}
 
-	if (token != MAGIC_TOKEN) {
-		print_skip("This test must be run with the magic token %d.\n"
-			   "This is done by nx_huge_pages_test.sh, which\n"
-			   "also handles environment setup for the test.",
-			   MAGIC_TOKEN);
-		exit(KSFT_SKIP);
-	}
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES));
+	TEST_REQUIRE(reclaim_period_ms > 0);
 
-	if (!reclaim_period_ms) {
-		print_skip("The NX reclaim period must be specified and non-zero");
-		exit(KSFT_SKIP);
-	}
+	__TEST_REQUIRE(token == MAGIC_TOKEN,
+		       "This test must be run with the magic token %d.\n"
+		       "This is done by nx_huge_pages_test.sh, which\n"
+		       "also handles environment setup for the test.");
 
 	run_test(reclaim_period_ms, false, reboot_permissions);
 	run_test(reclaim_period_ms, true, reboot_permissions);

base-commit: 93472b79715378a2386598d6632c654a2223267b

Signature · 1 line
2.37.1.595.g718a3a8f04-goog
#1 lore raw
Quoted 30 lines · Oliver Upton

Avoid boilerplate for checking test preconditions by using TEST_REQUIRE(). While at it, add a precondition for KVM_CAP_VM_DISABLE_NX_HUGE_PAGES to skip (instead of silently pass) on older kernels.

---
 .../selftests/kvm/x86_64/nx_huge_pages_test.c | 24 +++++--------------
 1 file changed, 6 insertions(+), 18 deletions(-)
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +0 -7
index cc6421716400..e19933ea34ca 100644
@@ -118,13 +118,6 @@ void run_test(int reclaim_period_ms, bool disable_nx_huge_pages,
 	vm = vm_create(1);
 
 	if (disable_nx_huge_pages) {
-		/*
-		 * Cannot run the test without NX huge pages if the kernel
-		 * does not support it.
-		 */
-		if (!kvm_check_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
-			return;
-
 		r = __vm_disable_nx_huge_pages(vm);
 		if (reboot_permissions) {
 			TEST_ASSERT(!r, "Disabling NX huge pages should succeed if process has reboot permissions");
@@ -248,18 +241,13 @@ int main(int argc, char **argv)
 		}

tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c int main(int argc, char **argv) · line 244

Quoted 10 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +1 -7
 	}
 
-	if (token != MAGIC_TOKEN) {
-		print_skip("This test must be run with the magic token %d.\n"
-			   "This is done by nx_huge_pages_test.sh, which\n"
-			   "also handles environment setup for the test.",
-			   MAGIC_TOKEN);
-		exit(KSFT_SKIP);
-	}
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES));

This cap is only needed for run_test(..., true, ...) below so I don't think we should require it for the entire test.

That being said, it still might be good to inform the user that the test is being skipped. So perhaps something like this:

  ...
  run_test(reclaim_period_ms, false, reboot_permissions);

  if (kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
          run_test(reclaim_period_ms, true, reboot_permissions);
  else
          print_skip("KVM_CAP_VM_DISABLE_NX_HUGE_PAGES not supported");
  ...
Quoted 18 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +5 -4
+	TEST_REQUIRE(reclaim_period_ms > 0);
 
-	if (!reclaim_period_ms) {
-		print_skip("The NX reclaim period must be specified and non-zero");
-		exit(KSFT_SKIP);
-	}
+	__TEST_REQUIRE(token == MAGIC_TOKEN,
+		       "This test must be run with the magic token %d.\n"
+		       "This is done by nx_huge_pages_test.sh, which\n"
+		       "also handles environment setup for the test.");
 
 	run_test(reclaim_period_ms, false, reboot_permissions);
 	run_test(reclaim_period_ms, true, reboot_permissions);

base-commit: 93472b79715378a2386598d6632c654a2223267b

Signature · 1 line
2.37.1.595.g718a3a8f04-goog
#2 lore raw
Quoted 33 lines · David Matlack
Quoted 32 lines · Oliver Upton

Avoid boilerplate for checking test preconditions by using TEST_REQUIRE(). While at it, add a precondition for KVM_CAP_VM_DISABLE_NX_HUGE_PAGES to skip (instead of silently pass) on older kernels.

---
 .../selftests/kvm/x86_64/nx_huge_pages_test.c | 24 +++++--------------
 1 file changed, 6 insertions(+), 18 deletions(-)
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +0 -7
index cc6421716400..e19933ea34ca 100644
@@ -118,13 +118,6 @@ void run_test(int reclaim_period_ms, bool disable_nx_huge_pages,
 	vm = vm_create(1);
 
 	if (disable_nx_huge_pages) {
-		/*
-		 * Cannot run the test without NX huge pages if the kernel
-		 * does not support it.
-		 */
-		if (!kvm_check_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
-			return;
-
 		r = __vm_disable_nx_huge_pages(vm);
 		if (reboot_permissions) {
 			TEST_ASSERT(!r, "Disabling NX huge pages should succeed if process has reboot permissions");
@@ -248,18 +241,13 @@ int main(int argc, char **argv)
 		}
 	}
 
Quoted 10 lines

tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c int main(int argc, char **argv) · line 244

Quoted 8 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +1 -7
-	if (token != MAGIC_TOKEN) {
-		print_skip("This test must be run with the magic token %d.\n"
-			   "This is done by nx_huge_pages_test.sh, which\n"
-			   "also handles environment setup for the test.",
-			   MAGIC_TOKEN);
-		exit(KSFT_SKIP);
-	}
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES));

This cap is only needed for run_test(..., true, ...) below so I don't think we should require it for the entire test.

It has always seemed that the test preconditions are a way to pretty-print a failure/skip instead of having some random ioctl fail deeper in the test.

If we really see value in adding predicates for individual test cases then IMO it deserves first-class support in our framework. Otherwise the next test that comes along is bound to open-code the same thing.

Can't folks just update their kernel? :-)

--
Thanks,
Oliver

Quoted 29 lines

That being said, it still might be good to inform the user that the test is being skipped. So perhaps something like this:

  ...
  run_test(reclaim_period_ms, false, reboot_permissions);

  if (kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
          run_test(reclaim_period_ms, true, reboot_permissions);
  else
          print_skip("KVM_CAP_VM_DISABLE_NX_HUGE_PAGES not supported");
  ...
Quoted 18 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +5 -4
+	TEST_REQUIRE(reclaim_period_ms > 0);
 
-	if (!reclaim_period_ms) {
-		print_skip("The NX reclaim period must be specified and non-zero");
-		exit(KSFT_SKIP);
-	}
+	__TEST_REQUIRE(token == MAGIC_TOKEN,
+		       "This test must be run with the magic token %d.\n"
+		       "This is done by nx_huge_pages_test.sh, which\n"
+		       "also handles environment setup for the test.");
 
 	run_test(reclaim_period_ms, false, reboot_permissions);
 	run_test(reclaim_period_ms, true, reboot_permissions);

base-commit: 93472b79715378a2386598d6632c654a2223267b

Signature · 1 line
2.37.1.595.g718a3a8f04-goog
Reviewed-by
#3 lore raw
Quoted 45 lines · Oliver Upton
Quoted 33 lines · David Matlack
Quoted 32 lines · Oliver Upton

Avoid boilerplate for checking test preconditions by using TEST_REQUIRE(). While at it, add a precondition for KVM_CAP_VM_DISABLE_NX_HUGE_PAGES to skip (instead of silently pass) on older kernels.

---
 .../selftests/kvm/x86_64/nx_huge_pages_test.c | 24 +++++--------------
 1 file changed, 6 insertions(+), 18 deletions(-)
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +0 -7
index cc6421716400..e19933ea34ca 100644
@@ -118,13 +118,6 @@ void run_test(int reclaim_period_ms, bool disable_nx_huge_pages,
 	vm = vm_create(1);
 
 	if (disable_nx_huge_pages) {
-		/*
-		 * Cannot run the test without NX huge pages if the kernel
-		 * does not support it.
-		 */
-		if (!kvm_check_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
-			return;
-
 		r = __vm_disable_nx_huge_pages(vm);
 		if (reboot_permissions) {
 			TEST_ASSERT(!r, "Disabling NX huge pages should succeed if process has reboot permissions");
@@ -248,18 +241,13 @@ int main(int argc, char **argv)
 		}
 	}
 
Quoted 10 lines

tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c int main(int argc, char **argv) · line 244

Quoted 8 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +1 -7
-	if (token != MAGIC_TOKEN) {
-		print_skip("This test must be run with the magic token %d.\n"
-			   "This is done by nx_huge_pages_test.sh, which\n"
-			   "also handles environment setup for the test.",
-			   MAGIC_TOKEN);
-		exit(KSFT_SKIP);
-	}
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES));

This cap is only needed for run_test(..., true, ...) below so I don't think we should require it for the entire test.

Quoted 7 lines

It has always seemed that the test preconditions are a way to pretty-print a failure/skip instead of having some random ioctl fail deeper in the test.

If we really see value in adding predicates for individual test cases then IMO it deserves first-class support in our framework. Otherwise the next test that comes along is bound to open-code the same thing.

Fair point.

Quoted 2 lines

Can't folks just update their kernel? :-)

Consider my suggestion optional. If anyone is backporting this test to their kernel they'll also probably backport KVM_CAP_VM_DISABLE_NX_HUGE_PAGES ;). So I don't think there will be a huge benefit of making the test more flexible.

That aside,

Quoted 34 lines

--
Thanks,
Oliver

Quoted 29 lines

That being said, it still might be good to inform the user that the test is being skipped. So perhaps something like this:

  ...
  run_test(reclaim_period_ms, false, reboot_permissions);

  if (kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES))
          run_test(reclaim_period_ms, true, reboot_permissions);
  else
          print_skip("KVM_CAP_VM_DISABLE_NX_HUGE_PAGES not supported");
  ...
Quoted 18 lines
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +5 -4
+	TEST_REQUIRE(reclaim_period_ms > 0);
 
-	if (!reclaim_period_ms) {
-		print_skip("The NX reclaim period must be specified and non-zero");
-		exit(KSFT_SKIP);
-	}
+	__TEST_REQUIRE(token == MAGIC_TOKEN,
+		       "This test must be run with the magic token %d.\n"
+		       "This is done by nx_huge_pages_test.sh, which\n"
+		       "also handles environment setup for the test.");
 
 	run_test(reclaim_period_ms, false, reboot_permissions);
 	run_test(reclaim_period_ms, true, reboot_permissions);

base-commit: 93472b79715378a2386598d6632c654a2223267b

Signature · 1 line
2.37.1.595.g718a3a8f04-goog
reviewer Reviewed-by David Matlack
#4 lore raw
Quoted 7 lines · David Matlack
Quoted 6 lines · Oliver Upton
Quoted 5 lines · David Matlack
Quoted 4 lines · Oliver Upton

Avoid boilerplate for checking test preconditions by using TEST_REQUIRE(). While at it, add a precondition for KVM_CAP_VM_DISABLE_NX_HUGE_PAGES to skip (instead of silently pass) on older kernels.

...

Quoted 15 lines
Quoted 11 lines
Quoted 3 lines

tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c int main(int argc, char **argv) · line 244

Quoted 1 line
tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c +1 -0
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES));

This cap is only needed for run_test(..., true, ...) below so I don't think we should require it for the entire test.

It has always seemed that the test preconditions are a way to pretty-print a failure/skip instead of having some random ioctl fail deeper in the test.

If we really see value in adding predicates for individual test cases then IMO it deserves first-class support in our framework. Otherwise the next test that comes along is bound to open-code the same thing.

Fair point.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Can't folks just update their kernel? :-)

Consider my suggestion optional. If anyone is backporting this test to their kernel they'll also probably backport KVM_CAP_VM_DISABLE_NX_HUGE_PAGES ;). So I don't think there will be a huge benefit of making the test more flexible.

Yeah, I'm somewhat ambivalent as well. All things considered, I think it makes sense to skip the entire test. Like Oliver said, without first-class support, this will become a mess. And I'm not convinced that adding first-class support is a good idea, as that will inevitably lead to tests ballooning to include a big pile of subtests, a la KUT's VMX test. I would much rather steer selftests in a "one test per binary" direction; IMO it's easier to do filtering via scripts, and it also minimizes the chances of creating subtle dependencies between (sub)tests.

So, pushed to branch for_paolo/6.1 at:

https://github.com/sean-jc/linux.git

but with a rewritten shortlog+changelog to capture this conversation. And that's a good lesson for the future as well: when piggybacking a patch, making functional changes as the "opportunistic cleanup" is rarely the right thing to do.

This is what I ended up with, holler if anything about it bothers you.

KVM: selftests: Require DISABLE_NX_HUGE_PAGES cap for NX hugepage test

Require KVM_CAP_VM_DISABLE_NX_HUGE_PAGES for the entire NX hugepage test instead of skipping the "disable" subtest if the capability isn't supported by the host kernel. While the "enable" subtest does provide value when the capability isn't supported, silently providing only half the promised coveraged is undesirable, i.e. it's better to skip the test so that the user knows something.

Alternatively, the test could print something to alert the user instead of silently skipping the subtest, but that would encourage other tests to follow suit, and it's not clear that it's desirable to take selftests in that direction. And if selftests do head down the path of skipping subtests, such behavior needs first-class support in the framework.

Opportunistically convert other test preconditions to TEST_REQUIRE().

[sean: rewrote changelog to capture discussion about skipping the test]