KVM: selftests: Fix maxnodes bugs and cleanup related code
Sean Christopherson
maintainercover KVM: selftests: Fix maxnodes bugs and cleanup related code
This is a spin-off of Shivank's series[*] to fix "flaws" in KVM selftests handling of @maxnode in various sycalls. In quote because the flaw is that KVM selftests followed the docs, and the kernel has a longstanding, poorly documented off-by-one bug.
Along the way, massage the xAPIC IPI test to automatically run its page migration testcases when possible, because that's the entire reason the test exists.
I didn't included the guest_memfd changes that were the focus of Shivank's series, as they are bigger and more involved (I'll let Shivank tackle those).
[*] https://lore.kernel.org/all/[email protected]
Sean Christopherson (5):
KVM: selftests: Account for kernel's off-by-one bug in NUMA node
syscalls
KVM: selftests: Compute node masks on-demand in xAPIC IPI test
KVM: selftests: Add common helper to get mask+number of usable memory
NUMA nodes
KVM: selftests: Automatically run xAPIC IPI migration test when
possible
KVM: selftests: Skip xAPIC IPI migration test when forced but
unsupported
Shivank Garg (2):
KVM: selftests: Fix maxnode argument to migrate_pages() in
xapic_ipi_test
KVM: selftests: use BITS_PER_TYPE() for NUMA masks
.../testing/selftests/kvm/guest_memfd_test.c | 2 +- tools/testing/selftests/kvm/include/numaif.h | 43 ++++++++++ .../selftests/kvm/x86/xapic_ipi_test.c | 78 +++++++++---------- 3 files changed, 83 insertions(+), 40 deletions(-)
base-commit: 76671054f9a1ff6abb976583cd8da37650acdc97
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Sean Christopherson
maintainerPATCH 1/7 KVM: selftests: Account for kernel's off-by-one bug in NUMA node syscalls
Add and use MAXNODE_FOR_MASK() to compute the "correct" maxnode value that is passed to various NUMA-related syscalls, e.g. get_mempolicy(), mbind(), migrate_pages(), etc. In quotes, because the kernel has an undocumented, longstanding off-by-one bug that requires userspace to specify the number of bits plus one, i.e. the max node plus two.
The kernel bug has been known since 2007[*]:
: And this is I think the reason why we can't change this now. I assume : numactl allocates 1024 bits (0 to 1023) and passes 1025 to make sure all : 1024 bits are processed. If we change it now, kernel will process 1025 : bits (0 to 1024) and overflow the allocated bitmask. If it happens to be : at the border of mmaped vma, it's a segfault...
But unfortunately the manpages haven't yet been updated, e.g.
: The maxnode argument is the maximum node number in the bit mask plus one
--- tools/testing/selftests/kvm/guest_memfd_test.c | 2 +- tools/testing/selftests/kvm/include/numaif.h | 11 +++++++++++ tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 2 +- 3 files changed, 13 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/guest_memfd_test.c +1 -1
@@ -80,7 +80,7 @@ static void test_mbind(int fd, size_t total_size) {const unsigned long nodemask_0 =1 ;/* nid: 0 */ unsigned long nodemask =0 ; -unsigned long maxnode =BITS_PER_TYPE (nodemask); +unsigned long maxnode =MAXNODE_FOR_MASK (nodemask);int policy;char *mem;int ret;
tools/testing/selftests/kvm/include/numaif.h +11 -0
@@ -6,6 +6,7 @@#include <dirent.h> +#include <linux/bitops.h> #include <linux/mempolicy.h> #include "kvm_syscalls.h"
@@ -30,6 +31,16 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode, const unsigned long *, nodemask, unsigned long, maxnode, unsigned int, flags); +/* +* Calculate the @maxnode param for the above syscalls given the mask that will +* be passed to the kernel, to account for a longstanding off-by-one bug in the +* kernel that isn't properly documented in the manpages. The manpages say +* that @maxnode is "the maximum node ID plus one", but the kernel's actual +* behavior is "the number of bits in the mask plus one", i.e. "the maximum +* node ID plus two". +*/ +#define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +static inline int get_max_numa_node (void ) {struct dirent *de;
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1 -1
@@ -248,7 +248,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, delay_usecs);/* Get set of first 64 numa nodes available */ -kvm_get_mempolicy (NULL , &nodemask,sizeof (nodemask) *8 , +kvm_get_mempolicy (NULL , &nodemask,MAXNODE_FOR_MASK (nodemask),0 ,MPOL_F_MEMS_ALLOWED ); fprintf(stderr, "Numa nodes found amongst first %lu possible nodes "
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Re: [PATCH 1/7] KVM: selftests: Account for kernel's off-by-one bug in NUMA node syscalls
Quoted 73 lines · Sean Christopherson
Add and use
MAXNODE_FOR_MASK()to compute the "correct" maxnode value that is passed to various NUMA-related syscalls, e.g.get_mempolicy(),mbind(),migrate_pages(), etc. In quotes, because the kernel has an undocumented, longstanding off-by-one bug that requires userspace to specify the number of bits plus one, i.e. the max node plus two.The kernel bug has been known since 2007[*]:
: And this is I think the reason why we can't change this now. I assume : numactl allocates 1024 bits (0 to 1023) and passes 1025 to make sure all : 1024 bits are processed. If we change it now, kernel will process 1025 : bits (0 to 1024) and overflow the allocated bitmask. If it happens to be : at the border of mmaped vma, it's a segfault...
But unfortunately the manpages haven't yet been updated, e.g.
: The maxnode argument is the maximum node number in the bit mask plus one
Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/guest_memfd_test.c | 2 +- tools/testing/selftests/kvm/include/numaif.h | 11 +++++++++++ tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 2 +- 3 files changed, 13 insertions(+), 2 deletions(-)tools/testing/selftests/kvm/guest_memfd_test.c +1
-1@@ -80,7 +80,7 @@ static void test_mbind(int fd, size_t total_size) {const unsigned long nodemask_0 =1 ;/* nid: 0 */ unsigned long nodemask =0 ; -unsigned long maxnode =BITS_PER_TYPE (nodemask); +unsigned long maxnode =MAXNODE_FOR_MASK (nodemask);int policy;char *mem;int ret;tools/testing/selftests/kvm/include/numaif.h +11
-0@@ -6,6 +6,7 @@#include <dirent.h> +#include <linux/bitops.h> #include <linux/mempolicy.h> #include "kvm_syscalls.h" @@ -30,6 +31,16 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode, const unsigned long *, nodemask, unsigned long, maxnode, unsigned int, flags); +/* +* Calculate the @maxnode param for the above syscalls given the mask that will +* be passed to the kernel, to account for a longstanding off-by-one bug in the +* kernel that isn't properly documented in the manpages. The manpages say +* that @maxnode is "the maximum node ID plus one", but the kernel's actual +* behavior is "the number of bits in the mask plus one", i.e. "the maximum +* node ID plus two". +*/ +#define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +static inline int get_max_numa_node (void ) {struct dirent *de;tools/testing/selftests/kvm/x86/xapic_ipi_test.c +0
-0
tools/testing/selftests/kvm/x86/xapic_ipi_test.c void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, · line 254
Quoted 8 lines
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1
-1@@ -248,7 +248,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, delay_usecs);/* Get set of first 64 numa nodes available */ -kvm_get_mempolicy (NULL , &nodemask,sizeof (nodemask) *8 , +kvm_get_mempolicy (NULL , &nodemask,MAXNODE_FOR_MASK (nodemask),0 ,MPOL_F_MEMS_ALLOWED ); fprintf(stderr, "Numa nodes found amongst first %lu possible nodes "
LGTM!
Thanks,
Shivank
Sean Christopherson
maintainerPATCH 2/7 KVM: selftests: Fix maxnode argument to migrate_pages() in xapic_ipi_test
In reply to Sean ChristophersonFrom: Shivank Garg <[email protected]>
Use MAXNODE_FOR_MASK() to compute the @maxnode argument when migrating pages in the xAPIC IPI test, as the current code incorrectly passes the number of bytes, not the number of bits, in the mask, and also fails to account for the kernel's off-by-one bug. I.e. the test sets maxnode to 8, and so the kernel only checks node IDs 0-6 even though the nodemask covers node IDs 0-63.
--- tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1 -1
@@ -291,7 +291,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,* KVM_CREATE_VCPU ioctl. If that assumption ever changes this * test may break or give a false positive signal. */ - pages_not_moved =migrate_pages (0 ,sizeof (nodemasks[from]), + pages_not_moved =migrate_pages (0 ,MAXNODE_FOR_MASK (nodemasks[from]), &nodemasks[from], &nodemasks[to]);if (pages_not_moved <0 )
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Sean Christopherson
maintainerPATCH 3/7 KVM: selftests: use BITS_PER_TYPE() for NUMA masks
In reply to Sean ChristophersonFrom: Shivank Garg <[email protected]>
Replace the open-coded sizeof() * 8 calculations in do_migrations() with BITS_PER_TYPE().
No functional change intended.
--- tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +3 -3
@@ -233,7 +233,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, {long pages_not_moved;unsigned long nodemask =0 ; -unsigned long nodemasks[sizeof (nodemask) *8 ]; +unsigned long nodemasks[BITS_PER_TYPE (nodemask)];int nodes =0 ;time_t start_time, last_update, now;time_t interval_secs =1 ;
@@ -253,13 +253,13 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,fprintf (stderr,"Numa nodes found amongst first %lu possible nodes " "(each 1-bit indicates node is present): %#lx\n" , -sizeof (nodemask) *8 , nodemask); +BITS_PER_TYPE (nodemask), nodemask);/* Init array of masks containing a single-bit in each, one for each * available node. migrate_pages called below requires specifying nodes * as bit masks. */ -for (i =0 , bit =1 ; i <sizeof (nodemask) *8 ; i++, bit <<=1 ) { +for (i =0 , bit =1 ; i <BITS_PER_TYPE (nodemask); i++, bit <<=1 ) {if (nodemask & bit) { nodemasks[nodes] = nodemask & bit; nodes++;
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Sean Christopherson
maintainerPATCH 4/7 KVM: selftests: Compute node masks on-demand in xAPIC IPI test
In reply to Sean ChristophersonCompute the node masks for the source (from) and destination (to) NUMA nodes in the xAPIC IPI test instead of pre-filling an array of masks with node per mask. Computing the mask on-demand is technically slower, but doesn't require a large-ish on-stack array, and more importantly allows the test to use a generic "get next NUMA node" API without having to commit all of KVM selftests to using a large array of single-bit nodemasks.
Implement said API as a common KVM NUMA API so that it can be used by other tests, e.g. in guest_memfd tests. Deliberately make @from "exclusive" as the anticipated usage in KVM selftests is to select the next, *different* node, i.e. so that users don't have to copy+paste code to assert that the found node is different than the starting node. The obvious downside is that implementing the exclusive logic forces callers to pass -1 instead of 0 when the goal is to find the first node in the mask, but that arguably yields more intuitive code anyways.
From an overall test functionality/coverage perspective, no functional change intended (the walking pattern of node migration should be unchanged).
--- tools/testing/selftests/kvm/include/numaif.h | 21 +++++++++++- .../selftests/kvm/x86/xapic_ipi_test.c | 32 ++++++------------- 2 files changed, 30 insertions(+), 23 deletions(-)
tools/testing/selftests/kvm/include/numaif.h +20 -1
@@ -6,7 +6,7 @@#include <dirent.h> -#include <linux/bitops.h> +#include <linux/bitmap.h> #include <linux/mempolicy.h> #include "kvm_syscalls.h"
@@ -41,6 +41,25 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,*/ #define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +/* +* Return the node ID of the next NUMA node in the mask, starting at @from+1. +* Guarantees a node is found, and that the found node is not @from. Pass -1 +* to find the first node in the mask. +*/ +static inline int kvm_get_next_numa_node (unsigned long nodemask,int from) +{ +const unsigned long nr_bits =BITS_PER_TYPE (nodemask); +int to; + + to =find_next_bit (&nodemask, nr_bits, from +1 ); +if (to == nr_bits) + to =find_next_bit (&nodemask, nr_bits,0 ); + +TEST_ASSERT (to != nr_bits && to != from, +"Unabled to find second NUMA node (from = %d, to = %d)" , from, to); +return to; +} +static inline int get_max_numa_node (void ) {struct dirent *de;
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +10 -22
@@ -233,13 +233,10 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, {long pages_not_moved;unsigned long nodemask =0 ; -unsigned long nodemasks[BITS_PER_TYPE (nodemask)];int nodes =0 ;time_t start_time, last_update, now;time_t interval_secs =1 ; -int i;int from, to; -unsigned long bit;u64 hlt_count;u64 wake_count;u64 ipis_sent;
@@ -255,24 +252,15 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,"(each 1-bit indicates node is present): %#lx\n" , BITS_PER_TYPE(nodemask), nodemask); -/* Init array of masks containing a single-bit in each, one for each -* available node. migrate_pages called below requires specifying nodes -* as bit masks. -*/ -for (i =0 , bit =1 ; i <BITS_PER_TYPE (nodemask); i++, bit <<=1 ) { -if (nodemask & bit) { - nodemasks[nodes] = nodemask & bit; - nodes++; - } - } - + nodes =__builtin_popcountl (nodemask);TEST_ASSERT (nodes >1 ,"Did not find at least 2 numa nodes. Can't do migration" );fprintf (stderr,"Migrating amongst %d nodes found\n" , nodes); - from =0 ; - to =1 ; + from =kvm_get_next_numa_node (nodemask,-1 ); + to =kvm_get_next_numa_node (nodemask, from); + start_time =time (NULL ); last_update = start_time;
@@ -281,6 +269,9 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, wake_count = data->wake_count;while ((int )(time (NULL ) - start_time) < run_secs) { +unsigned long from_mask =BIT (from); +unsigned long to_mask =BIT (to); + data->migrations_attempted++;/*
@@ -291,9 +282,8 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,* KVM_CREATE_VCPU ioctl. If that assumption ever changes this * test may break or give a false positive signal. */ - pages_not_moved =migrate_pages (0 ,MAXNODE_FOR_MASK (nodemasks[from]), - &nodemasks[from], - &nodemasks[to]); + pages_not_moved =migrate_pages (0 ,MAXNODE_FOR_MASK (from_mask), + &from_mask, &to_mask);if (pages_not_moved <0 )fprintf (stderr,"migrate_pages failed, errno=%d\n" , errno);
@@ -305,9 +295,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, data->migrations_completed++; from = to; - to++; -if (to == nodes) - to =0 ; + to =kvm_get_next_numa_node (nodemask, from); now =time (NULL ); if (((now - start_time) % interval_secs == 0) &&
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Re: [PATCH 4/7] KVM: selftests: Compute node masks on-demand in xAPIC IPI test
Quoted 48 lines · Sean Christopherson
Compute the node masks for the source (from) and destination (to) NUMA nodes in the xAPIC IPI test instead of pre-filling an array of masks with node per mask. Computing the mask on-demand is technically slower, but doesn't require a large-ish on-stack array, and more importantly allows the test to use a generic "get next NUMA node" API without having to commit all of KVM selftests to using a large array of single-bit nodemasks.
Implement said API as a common KVM NUMA API so that it can be used by other tests, e.g. in
guest_memfdtests. Deliberately make @from "exclusive" as the anticipated usage in KVM selftests is to select the next, *different* node, i.e. so that users don't have to copy+paste code to assert that the found node is different than the starting node. The obvious downside is that implementing the exclusive logic forces callers to pass -1 instead of 0 when the goal is to find the first node in the mask, but that arguably yields more intuitive code anyways.From an overall test functionality/coverage perspective, no functional change intended (the walking pattern of node migration should be unchanged).
Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/include/numaif.h | 21 +++++++++++- .../selftests/kvm/x86/xapic_ipi_test.c | 32 ++++++------------- 2 files changed, 30 insertions(+), 23 deletions(-)tools/testing/selftests/kvm/include/numaif.h +7
-1@@ -6,7 +6,7 @@#include <dirent.h> -#include <linux/bitops.h> +#include <linux/bitmap.h> #include <linux/mempolicy.h> #include "kvm_syscalls.h" @@ -41,6 +41,25 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,*/ #define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +/* +* Return the node ID of the next NUMA node in the mask, starting at @from+1. +* Guarantees a node is found, and that the found node is not @from. Pass -1 +* to find the first node in the mask. +*/ +static inline int kvm_get_next_numa_node (unsigned long nodemask,int from)
tools/testing/selftests/kvm/include/numaif.h KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode, · line 59
Quoted 10 lines
tools/testing/selftests/kvm/include/numaif.h +10
-0+{ +const unsigned long nr_bits =BITS_PER_TYPE (nodemask); +int to; + + to =find_next_bit (&nodemask, nr_bits, from +1 ); +if (to == nr_bits) + to =find_next_bit (&nodemask, nr_bits,0 ); + +TEST_ASSERT (to != nr_bits && to != from, +"Unabled to find second NUMA node (from = %d, to = %d)" , from, to);
typo: Unable
Quoted 76 lines
tools/testing/selftests/kvm/include/numaif.h +3
-0+return to; +} +static inline int get_max_numa_node (void ) {struct dirent *de;tools/testing/selftests/kvm/x86/xapic_ipi_test.c +9
-19@@ -233,13 +233,10 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, {long pages_not_moved;unsigned long nodemask =0 ; -unsigned long nodemasks[BITS_PER_TYPE (nodemask)];int nodes =0 ;time_t start_time, last_update, now;time_t interval_secs =1 ; -int i;int from, to; -unsigned long bit;u64 hlt_count;u64 wake_count;u64 ipis_sent;@@ -255,24 +252,15 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,"(each 1-bit indicates node is present): %#lx\n" , BITS_PER_TYPE(nodemask), nodemask); -/* Init array of masks containing a single-bit in each, one for each -* available node. migrate_pages called below requires specifying nodes -* as bit masks. -*/ -for (i =0 , bit =1 ; i <BITS_PER_TYPE (nodemask); i++, bit <<=1 ) { -if (nodemask & bit) { - nodemasks[nodes] = nodemask & bit; - nodes++; - } - } - + nodes =__builtin_popcountl (nodemask);TEST_ASSERT (nodes >1 ,"Did not find at least 2 numa nodes. Can't do migration" );fprintf (stderr,"Migrating amongst %d nodes found\n" , nodes); - from =0 ; - to =1 ; + from =kvm_get_next_numa_node (nodemask,-1 ); + to =kvm_get_next_numa_node (nodemask, from); + start_time =time (NULL ); last_update = start_time;@@ -281,6 +269,9 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, wake_count = data->wake_count;while ((int )(time (NULL ) - start_time) < run_secs) { +unsigned long from_mask =BIT (from); +unsigned long to_mask =BIT (to); + data->migrations_attempted++;/* @@ -291,9 +282,8 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,* KVM_CREATE_VCPU ioctl. If that assumption ever changes this * test may break or give a false positive signal. */ - pages_not_moved =migrate_pages (0 ,MAXNODE_FOR_MASK (nodemasks[from]), - &nodemasks[from], - &nodemasks[to]); + pages_not_moved =migrate_pages (0 ,MAXNODE_FOR_MASK (from_mask), + &from_mask, &to_mask);if (pages_not_moved <0 )fprintf (stderr,"migrate_pages failed, errno=%d\n" , errno);
tools/testing/selftests/kvm/x86/xapic_ipi_test.c void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, · line 301
Quoted 10 lines
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1
-3@@ -305,9 +295,7 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, data->migrations_completed++; from = to; - to++; -if (to == nodes) - to =0 ; + to =kvm_get_next_numa_node (nodemask, from); now =time (NULL ); if (((now - start_time) % interval_secs == 0) &&
LGTM
Thanks,
Shivank
Sean Christopherson
maintainerPATCH 5/7 KVM: selftests: Add common helper to get mask+number of usable memory NUMA nodes
In reply to Sean ChristophersonExtract and slightly adopt the xAPIC IPI test's logic for getting the mask and number of usable memory NUMA nodes into a common helper. To allow for friendlier behavior when the underlying kernel doesn't support NUMA, or the test was run without sufficient permissions, zero out the mask and return "zero nodes" if get_mempolicy() fails with ENOSYS or EPERM respectively. I.e. allow tests to skip (sub)tests instead of throwing an assert.
--- tools/testing/selftests/kvm/include/numaif.h | 13 +++++++++++++ tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 5 +---- 2 files changed, 14 insertions(+), 4 deletions(-)
tools/testing/selftests/kvm/include/numaif.h +13 -0
@@ -41,6 +41,19 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,*/ #define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +static inline int kvm_get_numa_memory_nodes (unsigned long *nodemask) +{ +int r; + + *nodemask =0 ; + + r =get_mempolicy (NULL , nodemask,MAXNODE_FOR_MASK (*nodemask),0 , +MPOL_F_MEMS_ALLOWED ); +TEST_ASSERT (!r || errno ==ENOSYS || errno ==EPERM , +"Unexpected get_mempolicy() failure" ); +return __builtin_popcountl (*nodemask); +} +/* * Return the node ID of the next NUMA node in the mask, starting at @from+1. * Guarantees a node is found, and that the found node is not @from. Pass -1
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1 -4
@@ -244,15 +244,12 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,fprintf (stderr,"Calling migrate_pages every %d microseconds\n" , delay_usecs); -/* Get set of first 64 numa nodes available */ -kvm_get_mempolicy (NULL , &nodemask,MAXNODE_FOR_MASK (nodemask), -0 ,MPOL_F_MEMS_ALLOWED ); + nodes =kvm_get_numa_memory_nodes (&nodemask);fprintf (stderr,"Numa nodes found amongst first %lu possible nodes " "(each 1-bit indicates node is present): %#lx\n" ,BITS_PER_TYPE (nodemask), nodemask); - nodes =__builtin_popcountl (nodemask);TEST_ASSERT (nodes >1 ,"Did not find at least 2 numa nodes. Can't do migration" );
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Re: [PATCH 5/7] KVM: selftests: Add common helper to get mask+number of usable memory NUMA nodes
Quoted 49 lines · Sean Christopherson
Extract and slightly adopt the xAPIC IPI test's logic for getting the mask and number of usable memory NUMA nodes into a common helper. To allow for friendlier behavior when the underlying kernel doesn't support NUMA, or the test was run without sufficient permissions, zero out the mask and return "zero nodes" if
get_mempolicy()fails with ENOSYS or EPERM respectively. I.e. allow tests to skip (sub)tests instead of throwing an assert.Suggested-by Shivank Garg <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/include/numaif.h | 13 +++++++++++++ tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 5 +---- 2 files changed, 14 insertions(+), 4 deletions(-)tools/testing/selftests/kvm/include/numaif.h +13
-0@@ -41,6 +41,19 @@ KVM_SYSCALL_DEFINE(mbind, 6, void *, addr, unsigned long, size, int, mode,*/ #define MAXNODE_FOR_MASK (mask) (BITS_PER_TYPE(mask) + 1) +static inline int kvm_get_numa_memory_nodes (unsigned long *nodemask) +{ +int r; + + *nodemask =0 ; + + r =get_mempolicy (NULL , nodemask,MAXNODE_FOR_MASK (*nodemask),0 , +MPOL_F_MEMS_ALLOWED ); +TEST_ASSERT (!r || errno ==ENOSYS || errno ==EPERM , +"Unexpected get_mempolicy() failure" ); +return __builtin_popcountl (*nodemask); +} +/* * Return the node ID of the next NUMA node in the mask, starting at @from+1. * Guarantees a node is found, and that the found node is not @from. Pass -1 tools/testing/selftests/kvm/x86/xapic_ipi_test.c +0
-3@@ -244,15 +244,12 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs,fprintf (stderr,"Calling migrate_pages every %d microseconds\n" , delay_usecs); -/* Get set of first 64 numa nodes available */ -kvm_get_mempolicy (NULL , &nodemask,MAXNODE_FOR_MASK (nodemask), -0 ,MPOL_F_MEMS_ALLOWED );
tools/testing/selftests/kvm/x86/xapic_ipi_test.c void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, · line 255
Quoted 10 lines
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +1
-1+ nodes =kvm_get_numa_memory_nodes (&nodemask);fprintf (stderr,"Numa nodes found amongst first %lu possible nodes " "(each 1-bit indicates node is present): %#lx\n" ,BITS_PER_TYPE (nodemask), nodemask); - nodes =__builtin_popcountl (nodemask);TEST_ASSERT (nodes >1 ,"Did not find at least 2 numa nodes. Can't do migration" );
Thanks,
Shivank
Sean Christopherson
maintainerPATCH 6/7 KVM: selftests: Automatically run xAPIC IPI migration test when possible
In reply to Sean ChristophersonRework the interface to the xAPIC IPI test to automatically run the migration testcase if at least two NUMA nodes are found. The test exists specifically to validate KVM's handling of migration of the APIC backing page, i.e. forcing end users to opt-in to running the test in migration mode largely defeats the purpose of the test.
Run both the "sleeping" and "migration" testcases by default, e.g. so that the more basic testcase will fail if KVM completely breaks IPI delivery.
Keep the -m / migration parameter so that infrastructure that wants to specifically test the migration case can do so and not get false passes.
--- .../selftests/kvm/x86/xapic_ipi_test.c | 40 +++++++++++++------ 1 file changed, 27 insertions(+), 13 deletions(-)
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +27 -13
@@ -245,14 +245,13 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, delay_usecs); nodes =kvm_get_numa_memory_nodes (&nodemask); +TEST_ASSERT (nodes >1 , +"NUMA nodes disappeared? nodemask = 0x%lx" , nodemask);fprintf (stderr,"Numa nodes found amongst first %lu possible nodes " "(each 1-bit indicates node is present): %#lx\n" ,BITS_PER_TYPE (nodemask), nodemask); -TEST_ASSERT (nodes >1 , -"Did not find at least 2 numa nodes. Can't do migration" ); -fprintf (stderr,"Migrating amongst %d nodes found\n" , nodes); from =kvm_get_next_numa_node (nodemask,-1 );
@@ -351,26 +350,17 @@ void get_cmdline_args(int argc, char *argv[], int *run_secs, } } -int main (int argc,char *argv[]) +static void test_xapic_ipi (int run_secs,int delay_usecs,bool migrate) {int wait_secs;const int max_halter_wait =10 ; -int run_secs =0 ; -int delay_usecs =0 ;struct test_data_page *data;gva_t test_data_page_gva; -bool migrate = false;pthread_t threads[2 ];struct thread_params params[2 ];struct kvm_vm *vm;u64 *pipis_rcvd; -get_cmdline_args (argc, argv, &run_secs, &migrate, &delay_usecs); -if (run_secs <=0 ) - run_secs =DEFAULT_RUN_SECS ; -if (delay_usecs <=0 ) - delay_usecs =DEFAULT_DELAY_USECS ; - vm =vm_create_with_one_vcpu (¶ms[0 ].vcpu, halter_guest_code);vm_install_exception_handler (vm,IPI_VECTOR , guest_ipi_handler);
@@ -458,5 +448,29 @@ int main(int argc, char *argv[])kvm_vm_free (vm); +} + +int main (int argc,char *argv[]) +{ +bool force_migrate = false; +unsigned long nodemask; +int run_secs =0 ; +int delay_usecs =0 ; + +get_cmdline_args (argc, argv, &run_secs, &force_migrate, &delay_usecs); +if (run_secs <=0 ) + run_secs =DEFAULT_RUN_SECS ; +if (delay_usecs <=0 ) + delay_usecs =DEFAULT_DELAY_USECS ; + +if (!force_migrate) +test_xapic_ipi (run_secs, delay_usecs, false); + +if (kvm_get_numa_memory_nodes (&nodemask) >1 ) +test_xapic_ipi (run_secs, delay_usecs, true); +else +TEST_ASSERT (!force_migrate, +"Did not find at least 2 numa nodes. Can't do migration" ); +return 0 ; }
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Re: [PATCH 6/7] KVM: selftests: Automatically run xAPIC IPI migration test when possible
Quoted 86 lines · Sean Christopherson
Rework the interface to the xAPIC IPI test to automatically run the migration testcase if at least two NUMA nodes are found. The test exists specifically to validate KVM's handling of migration of the APIC backing page, i.e. forcing end users to opt-in to running the test in migration mode largely defeats the purpose of the test.
Run both the "sleeping" and "migration" testcases by default, e.g. so that the more basic testcase will fail if KVM completely breaks IPI delivery.
Keep the -m / migration parameter so that infrastructure that wants to specifically test the migration case can do so and not get false passes.
Signed-off-by Sean Christopherson <[email protected]>--- .../selftests/kvm/x86/xapic_ipi_test.c | 40 +++++++++++++------ 1 file changed, 27 insertions(+), 13 deletions(-)tools/testing/selftests/kvm/x86/xapic_ipi_test.c +19
-13@@ -245,14 +245,13 @@ void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, delay_usecs); nodes =kvm_get_numa_memory_nodes (&nodemask); +TEST_ASSERT (nodes >1 , +"NUMA nodes disappeared? nodemask = 0x%lx" , nodemask);fprintf (stderr,"Numa nodes found amongst first %lu possible nodes " "(each 1-bit indicates node is present): %#lx\n" ,BITS_PER_TYPE (nodemask), nodemask); -TEST_ASSERT (nodes >1 , -"Did not find at least 2 numa nodes. Can't do migration" ); -fprintf (stderr,"Migrating amongst %d nodes found\n" , nodes); from =kvm_get_next_numa_node (nodemask,-1 );@@ -351,26 +350,17 @@ void get_cmdline_args(int argc, char *argv[], int *run_secs, } } -int main (int argc,char *argv[]) +static void test_xapic_ipi (int run_secs,int delay_usecs,bool migrate) {int wait_secs;const int max_halter_wait =10 ; -int run_secs =0 ; -int delay_usecs =0 ;struct test_data_page *data;gva_t test_data_page_gva; -bool migrate = false;pthread_t threads[2 ];struct thread_params params[2 ];struct kvm_vm *vm;u64 *pipis_rcvd; -get_cmdline_args (argc, argv, &run_secs, &migrate, &delay_usecs); -if (run_secs <=0 ) - run_secs =DEFAULT_RUN_SECS ; -if (delay_usecs <=0 ) - delay_usecs =DEFAULT_DELAY_USECS ; - vm =vm_create_with_one_vcpu (¶ms[0 ].vcpu, halter_guest_code);vm_install_exception_handler (vm,IPI_VECTOR , guest_ipi_handler);@@ -458,5 +448,29 @@ int main(int argc, char *argv[])kvm_vm_free (vm); +} + +int main (int argc,char *argv[]) +{ +bool force_migrate = false; +unsigned long nodemask; +int run_secs =0 ; +int delay_usecs =0 ; + +get_cmdline_args (argc, argv, &run_secs, &force_migrate, &delay_usecs); +if (run_secs <=0 ) + run_secs =DEFAULT_RUN_SECS ; +if (delay_usecs <=0 ) + delay_usecs =DEFAULT_DELAY_USECS ; + +if (!force_migrate)
tools/testing/selftests/kvm/x86/xapic_ipi_test.c int main(int argc, char *argv[]) · line 476
Quoted 10 lines
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +8
-0+test_xapic_ipi (run_secs, delay_usecs, false); + +if (kvm_get_numa_memory_nodes (&nodemask) >1 ) +test_xapic_ipi (run_secs, delay_usecs, true); +else +TEST_ASSERT (!force_migrate, +"Did not find at least 2 numa nodes. Can't do migration" ); +return 0 ; }
Thanks,
Shivank
Sean Christopherson
maintainerPATCH 7/7 KVM: selftests: Skip xAPIC IPI migration test when forced but unsupported
In reply to Sean ChristophersonSkip the xAPIC IPI migration test instead of failing outright if migration is forced by the user, but is unsupported due to lack of NUMA support. Failing the test is arguably "fine" for opt-in behavior, but in practice doesn't allow running the test in forced-migration mode in any kind of automated test environment.
--- tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +3 -2
@@ -469,8 +469,9 @@ int main(int argc, char *argv[])if (kvm_get_numa_memory_nodes (&nodemask) >1 )test_xapic_ipi (run_secs, delay_usecs, true);else -TEST_ASSERT (!force_migrate, -"Did not find at least 2 numa nodes. Can't do migration" ); +__TEST_REQUIRE (!force_migrate, +"Need at least 2 NUMA nodes to do migration (nodemask = 0x%lx)" , + nodemask);return 0 ; }
Signature · 1 line
2.55.0.970.g62bdec98f9-goog
Re: [PATCH 7/7] KVM: selftests: Skip xAPIC IPI migration test when forced but unsupported
Quoted 17 lines · Sean Christopherson
Skip the xAPIC IPI migration test instead of failing outright if migration is forced by the user, but is unsupported due to lack of NUMA support. Failing the test is arguably "fine" for opt-in behavior, but in practice doesn't allow running the test in forced-migration mode in any kind of automated test environment.
Signed-off-by Sean Christopherson <[email protected]>--- tools/testing/selftests/kvm/x86/xapic_ipi_test.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)tools/testing/selftests/kvm/x86/xapic_ipi_test.c +0
-0@@ -469,8 +469,9 @@ int main(int argc, char *argv[])if (kvm_get_numa_memory_nodes (&nodemask) >1 )
tools/testing/selftests/kvm/x86/xapic_ipi_test.c int main(int argc, char *argv[]) · line 477
Quoted 10 lines
tools/testing/selftests/kvm/x86/xapic_ipi_test.c +3
-2test_xapic_ipi (run_secs, delay_usecs, true); else - TEST_ASSERT(!force_migrate, - "Did not find at least 2 numa nodes. Can't do migration"); + __TEST_REQUIRE(!force_migrate, +"Need at least 2 NUMA nodes to do migration (nodemask = 0x%lx)" , + nodemask);return 0 ; }
Thanks,
Shivank
Sean Christopherson
maintainerQuoted 10 lines · Sean Christopherson
This is a spin-off of Shivank's series[*] to fix "flaws" in KVM selftests handling of @maxnode in various sycalls. In quote because the flaw is that KVM selftests followed the docs, and the kernel has a longstanding, poorly documented off-by-one bug.
Along the way, massage the xAPIC IPI test to automatically run its page migration testcases when possible, because that's the entire reason the test exists.
[...]
Applied to kvm-x86 selftests, thanks!
[1/7] KVM: selftests: Account for kernel's off-by-one bug in NUMA node syscalls
https://github.com/kvm-x86/linux/commit/be3c6888e1ab [2/7] KVM: selftests: Fix maxnode argument to migrate_pages() in xapic_ipi_test https://github.com/kvm-x86/linux/commit/8883d2410c45 [3/7] KVM: selftests: use BITS_PER_TYPE() for NUMA masks https://github.com/kvm-x86/linux/commit/d2dde5c387fc [4/7] KVM: selftests: Compute node masks on-demand in xAPIC IPI test https://github.com/kvm-x86/linux/commit/3a2a37d2d48e [5/7] KVM: selftests: Add common helper to get mask+number of usable memory NUMA nodes https://github.com/kvm-x86/linux/commit/0be096bbc9cc [6/7] KVM: selftests: Automatically run xAPIC IPI migration test when possible https://github.com/kvm-x86/linux/commit/92ed4d5c78fd [7/7] KVM: selftests: Skip xAPIC IPI migration test when forced but unsupported https://github.com/kvm-x86/linux/commit/d5aa06fbfec2