Skip to contentSkip to conversation

KVM: selftests: Make sure kvm_create_max_vcpus test won't hit RLIMIT_NOFILE

PATCH Superseded Vitaly Kuznetsov 2 participants 4 messages Nov 22, 2021 – Nov 23, 2021

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 KVM: selftests: Make sure kvm_create_max_vcpus test won't hit RLIMIT_NOFILE

With the elevated 'KVM_CAP_MAX_VCPUS' value kvm_create_max_vcpus test may hit RLIMIT_NOFILE limits:

 # ./kvm_create_max_vcpus
 KVM_CAP_MAX_VCPU_ID: 4096
 KVM_CAP_MAX_VCPUS: 1024
 Testing creating 1024 vCPUs, with IDs 0...1023.
 /dev/kvm not available (errno: 24), skipping test

Adjust RLIMIT_NOFILE limits to make sure KVM_CAP_MAX_VCPUS fds can be opened. Note, raising hard limit ('rlim_max') requires CAP_SYS_RESOURCE capability which is generally not needed to run kvm selftests (but without raising the limit the test is doomed to fail anyway).

---
 .../selftests/kvm/kvm_create_max_vcpus.c      | 22 +++++++++++++++++++
 1 file changed, 22 insertions(+)
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +22 -0
index f968dfd4ee88..19198477a10e 100644
@@ -12,6 +12,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/resource.h>
 
 #include "test_util.h"
 
@@ -19,6 +20,9 @@
 #include "asm/kvm.h"
 #include "linux/kvm.h"
 
+/* 'Safe' number of open file descriptors in addition to vCPU fds needed */
+#define NOFD 16
+
 void test_vcpu_creation(int first_vcpu_id, int num_vcpus)
 {
 	struct kvm_vm *vm;
@@ -40,10 +44,28 @@ int main(int argc, char *argv[])
 {
 	int kvm_max_vcpu_id = kvm_check_cap(KVM_CAP_MAX_VCPU_ID);
 	int kvm_max_vcpus = kvm_check_cap(KVM_CAP_MAX_VCPUS);
+	struct rlimit rl;
 
 	pr_info("KVM_CAP_MAX_VCPU_ID: %d\n", kvm_max_vcpu_id);
 	pr_info("KVM_CAP_MAX_VCPUS: %d\n", kvm_max_vcpus);
 
+	/*
+	 * Creating KVM_CAP_MAX_VCPUS vCPUs require KVM_CAP_MAX_VCPUS open
+	 * file decriptors.
+	 */
+	TEST_ASSERT(!getrlimit(RLIMIT_NOFILE, &rl),
+		    "getrlimit() failed (errno: %d)", errno);
+
+	if (kvm_max_vcpus > rl.rlim_cur - NOFD) {
+		rl.rlim_cur = kvm_max_vcpus + NOFD;
+
+		if (kvm_max_vcpus > rl.rlim_max - NOFD)
+			rl.rlim_max = kvm_max_vcpus + NOFD;
+
+		TEST_ASSERT(!setrlimit(RLIMIT_NOFILE, &rl),
+			    "setrlimit() failed (errno: %d)", errno);
+	}
+
 	/*
 	 * Upstream KVM prior to 4.8 does not support KVM_CAP_MAX_VCPU_ID.
 	 * Userspace is supposed to use KVM_CAP_MAX_VCPUS as the maximum ID
Signature · 1 line
2.33.1
#1 lore raw
Quoted 32 lines · Vitaly Kuznetsov

With the elevated 'KVM_CAP_MAX_VCPUS' value kvm_create_max_vcpus test may hit RLIMIT_NOFILE limits:

 # ./kvm_create_max_vcpus
 KVM_CAP_MAX_VCPU_ID: 4096
 KVM_CAP_MAX_VCPUS: 1024
 Testing creating 1024 vCPUs, with IDs 0...1023.
 /dev/kvm not available (errno: 24), skipping test

Adjust RLIMIT_NOFILE limits to make sure KVM_CAP_MAX_VCPUS fds can be opened. Note, raising hard limit ('rlim_max') requires CAP_SYS_RESOURCE capability which is generally not needed to run kvm selftests (but without raising the limit the test is doomed to fail anyway).

---
 .../selftests/kvm/kvm_create_max_vcpus.c      | 22 +++++++++++++++++++
 1 file changed, 22 insertions(+)
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +1 -0
index f968dfd4ee88..19198477a10e 100644
@@ -12,6 +12,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/resource.h>
 
 #include "test_util.h"
 

tools/testing/selftests/kvm/kvm_create_max_vcpus.c line 24

Quoted 5 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +2 -0
@@ -19,6 +20,9 @@
 #include "asm/kvm.h"
 #include "linux/kvm.h"
 
+/* 'Safe' number of open file descriptors in addition to vCPU fds needed */
+#define NOFD 16

Any reason not to make this "buffer" extra large, e.g. 100+ to avoid having to debug this issue again in the future?

tools/testing/selftests/kvm/kvm_create_max_vcpus.c int main(int argc, char *argv[]) · line 46

Quoted 8 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +1 -0
+
 void test_vcpu_creation(int first_vcpu_id, int num_vcpus)
 {
 	struct kvm_vm *vm;
@@ -40,10 +44,28 @@ int main(int argc, char *argv[])
 {
 	int kvm_max_vcpu_id = kvm_check_cap(KVM_CAP_MAX_VCPU_ID);
 	int kvm_max_vcpus = kvm_check_cap(KVM_CAP_MAX_VCPUS);

Rather than a separate define that's hard to describe succintly, what about:

int nr_fds_wanted = kvm_max_vcpus + <arbitrary number>

and then the body becomes

	if (nr_fds_wanted > rl.rlim_cur) {
		rl.rlim_cur = nr_fds_wanted;
		rl.rlim_max = max(rl.rlim_max, nr_fds_wanted);

		...
	}

tools/testing/selftests/kvm/kvm_create_max_vcpus.c int main(int argc, char *argv[]) · line 57

Quoted 11 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +7 -0
+	struct rlimit rl;
 
 	pr_info("KVM_CAP_MAX_VCPU_ID: %d\n", kvm_max_vcpu_id);
 	pr_info("KVM_CAP_MAX_VCPUS: %d\n", kvm_max_vcpus);
 
+	/*
+	 * Creating KVM_CAP_MAX_VCPUS vCPUs require KVM_CAP_MAX_VCPUS open
+	 * file decriptors.
+	 */
+	TEST_ASSERT(!getrlimit(RLIMIT_NOFILE, &rl),
+		    "getrlimit() failed (errno: %d)", errno);

And strerror() output too?

Quoted 17 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +11 -0
+
+	if (kvm_max_vcpus > rl.rlim_cur - NOFD) {
+		rl.rlim_cur = kvm_max_vcpus + NOFD;
+
+		if (kvm_max_vcpus > rl.rlim_max - NOFD)
+			rl.rlim_max = kvm_max_vcpus + NOFD;
+
+		TEST_ASSERT(!setrlimit(RLIMIT_NOFILE, &rl),
+			    "setrlimit() failed (errno: %d)", errno);
+	}
+
 	/*
 	 * Upstream KVM prior to 4.8 does not support KVM_CAP_MAX_VCPU_ID.
 	 * Userspace is supposed to use KVM_CAP_MAX_VCPUS as the maximum ID
Signature · 1 line
2.33.1
#2 lore raw
Quoted 36 lines · Sean Christopherson
Quoted 35 lines · Vitaly Kuznetsov

With the elevated 'KVM_CAP_MAX_VCPUS' value kvm_create_max_vcpus test may hit RLIMIT_NOFILE limits:

 # ./kvm_create_max_vcpus
 KVM_CAP_MAX_VCPU_ID: 4096
 KVM_CAP_MAX_VCPUS: 1024
 Testing creating 1024 vCPUs, with IDs 0...1023.
 /dev/kvm not available (errno: 24), skipping test

Adjust RLIMIT_NOFILE limits to make sure KVM_CAP_MAX_VCPUS fds can be opened. Note, raising hard limit ('rlim_max') requires CAP_SYS_RESOURCE capability which is generally not needed to run kvm selftests (but without raising the limit the test is doomed to fail anyway).

---
 .../selftests/kvm/kvm_create_max_vcpus.c      | 22 +++++++++++++++++++
 1 file changed, 22 insertions(+)
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +1 -0
index f968dfd4ee88..19198477a10e 100644
@@ -12,6 +12,7 @@
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/resource.h>
 
 #include "test_util.h"
 
@@ -19,6 +20,9 @@
 #include "asm/kvm.h"
 #include "linux/kvm.h"
 
Quoted 6 lines

tools/testing/selftests/kvm/kvm_create_max_vcpus.c line 24

Quoted 2 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +2 -0
+/* 'Safe' number of open file descriptors in addition to vCPU fds needed */
+#define NOFD 16

Any reason not to make this "buffer" extra large, e.g. 100+ to avoid having to debug this issue again in the future?

No, not really. We could've avoided this ambiguity completely by checking how many fds are already open but all methods I can think of are 'too much'. In my testing I needed around 10 so I put '16' but '100' is even better.

Quoted 13 lines

tools/testing/selftests/kvm/kvm_create_max_vcpus.c int main(int argc, char *argv[]) · line 46

Quoted 8 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +1 -0
+
 void test_vcpu_creation(int first_vcpu_id, int num_vcpus)
 {
 	struct kvm_vm *vm;
@@ -40,10 +44,28 @@ int main(int argc, char *argv[])
 {
 	int kvm_max_vcpu_id = kvm_check_cap(KVM_CAP_MAX_VCPU_ID);
 	int kvm_max_vcpus = kvm_check_cap(KVM_CAP_MAX_VCPUS);

Rather than a separate define that's hard to describe succintly, what about:

int nr_fds_wanted = kvm_max_vcpus + <arbitrary number>

Quoted 8 lines

and then the body becomes

	if (nr_fds_wanted > rl.rlim_cur) {
		rl.rlim_cur = nr_fds_wanted;
		rl.rlim_max = max(rl.rlim_max, nr_fds_wanted);

		...
	}

Sure but a "succinct" comment will still be needed, either near the 'NOFD' define or above 'int nr_fds_wanted' :-)

Quoted 6 lines
Quoted 6 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +1 -0
+	struct rlimit rl;
 
 	pr_info("KVM_CAP_MAX_VCPU_ID: %d\n", kvm_max_vcpu_id);
 	pr_info("KVM_CAP_MAX_VCPUS: %d\n", kvm_max_vcpus);
 
Quoted 9 lines

tools/testing/selftests/kvm/kvm_create_max_vcpus.c int main(int argc, char *argv[]) · line 57

Quoted 6 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +6 -0
+	/*
+	 * Creating KVM_CAP_MAX_VCPUS vCPUs require KVM_CAP_MAX_VCPUS open
+	 * file decriptors.
+	 */
+	TEST_ASSERT(!getrlimit(RLIMIT_NOFILE, &rl),
+		    "getrlimit() failed (errno: %d)", errno);

And strerror() output too?

Sure, will add in v2.

Quoted 18 lines
Quoted 17 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +11 -0
+
+	if (kvm_max_vcpus > rl.rlim_cur - NOFD) {
+		rl.rlim_cur = kvm_max_vcpus + NOFD;
+
+		if (kvm_max_vcpus > rl.rlim_max - NOFD)
+			rl.rlim_max = kvm_max_vcpus + NOFD;
+
+		TEST_ASSERT(!setrlimit(RLIMIT_NOFILE, &rl),
+			    "setrlimit() failed (errno: %d)", errno);
+	}
+
 	/*
 	 * Upstream KVM prior to 4.8 does not support KVM_CAP_MAX_VCPU_ID.
 	 * Userspace is supposed to use KVM_CAP_MAX_VCPUS as the maximum ID
Signature · 1 line
2.33.1
Signature · 1 line
Vitaly
#3 lore raw
Quoted 11 lines · Vitaly Kuznetsov
Quoted 10 lines

tools/testing/selftests/kvm/kvm_create_max_vcpus.c int main(int argc, char *argv[]) · line 57

Quoted 7 lines
tools/testing/selftests/kvm/kvm_create_max_vcpus.c +6 -0
+	/*
+	 * Creating KVM_CAP_MAX_VCPUS vCPUs require KVM_CAP_MAX_VCPUS open
+	 * file decriptors.
+	 */
+	TEST_ASSERT(!getrlimit(RLIMIT_NOFILE, &rl),
+		    "getrlimit() failed (errno: %d)", errno);

And strerror() output too?

Quoted 2 lines

Sure, will add in v2.

Actually, there are two issues with the code above. First, TEST_ASSERT() already prints both errno and strerror() (setrlimit() counterpart which is easier to make fail):

KVM_CAP_MAX_VCPU_ID: 4096
KVM_CAP_MAX_VCPUS: 1024
==== Test Assertion Failure ====
kvm_create_max_vcpus.c:68: !setrlimit(RLIMIT_NOFILE, &rl) pid=344504 tid=344504 errno=1 - Operation not permitted

     1	0x0000000000402485: main at kvm_create_max_vcpus.c:68
     2	0x00007fcb2e8b4041: ?? ??:0
     3	0x000000000040254d: _start at ??:?

setrlimit() failed, errno: 0

Second, note "errno: 0" above. There's no guarantee that getrlimit() will be executed before evaluating 'errno' in C. I think I'll just drop redundant errno printout then.

Signature · 1 line
Vitaly