qtest: pick tests that require KVM at runtime
cover qtest: pick tests that require KVM at runtime
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'
Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.
PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.
Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs
root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM
tests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Signature · 1 line
2.27.0
RFC 1/3 tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM
From: root <[email protected]>
Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).
Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288
This patch introduces qtest_has_kvm() to let users check if KVM is available in advance and skip registering non run-able test-cases.
---
PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.
on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.
tests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 3 files changed, 28 insertions(+)
tests/qtest/libqos/libqtest.h +7 -0
@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);*/ const char *qtest_get_arch (void ); +/** +* qtest_has_kvm: +* +* Returns: True if the QEMU executable under test supports KVM +*/ +bool qtest_has_kvm (void ); +/** * qtest_add_func: * @str: Test case path.
meson.build +1 -0
@@ -75,6 +75,7 @@ elif cpu in ['mips', 'mips64'] else kvm_targets = [] endif +config_host_data.set_quoted('CONFIG_KVM_TARGETS', ' ,'.join(kvm_targets)) accelerator_targets = { 'CONFIG_KVM': kvm_targets } if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
tests/qtest/libqtest.c +20 -0
@@ -920,6 +920,26 @@ const char *qtest_get_arch(void)return end +1 ; } +bool qtest_has_kvm (void ) +{ +int i; +bool ret = false; +const char *arch =qtest_get_arch (); +const char *kvm_targets_str =CONFIG_KVM_TARGETS ; +gchar **targets =g_strsplit (kvm_targets_str," " ,-1 ); + +for (i =0 ; targets[i]; i++) { +if (!strncmp (targets[i], arch,strlen (arch))) { +if (access ("/dev/kvm" ,R_OK |W_OK )) { + ret = true; +break ; + } + } + } +g_strfreev (targets); +return ret; +} +bool qtest_get_irq (QTestState *s,int num) {/* dummy operation in order to make sure irq is up to date */
Signature · 1 line
2.27.0
RFC v2 1/3 tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM
Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).
Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288
This patch introduces qtest_has_kvm() to let users check if KVM is available in advance and skip registering non run-able test-cases.
PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.
on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.
---
v2:
- fix access() check.
s/access()/!access()/
- format C array items at meson.build time, and drop splitting targets string at runtime
tests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 2 ++ tests/qtest/libqtest.c | 18 ++++++++++++++++++ 3 files changed, 27 insertions(+)
tests/qtest/libqos/libqtest.h +7 -0
@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);*/ const char *qtest_get_arch (void ); +/** +* qtest_has_kvm: +* +* Returns: True if the QEMU executable under test supports KVM +*/ +bool qtest_has_kvm (void ); +/** * qtest_add_func: * @str: Test case path.
meson.build +2 -0
@@ -75,6 +75,8 @@ elif cpu in ['mips', 'mips64'] else kvm_targets = [] endif +kvm_targets_c = '"' + '" ,"'.join(kvm_targets) + '"' +config_host_data.set('CONFIG_KVM_TARGETS', kvm_targets_c) accelerator_targets = { 'CONFIG_KVM': kvm_targets } if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
tests/qtest/libqtest.c +18 -0
@@ -920,6 +920,24 @@ const char *qtest_get_arch(void)return end +1 ; } +bool qtest_has_kvm (void ) +{ +int i; +bool ret = false; +const char *arch =qtest_get_arch (); +const char *targets[] = {CONFIG_KVM_TARGETS }; + +for (i =0 ; i <ARRAY_SIZE (targets); i++) { +if (!strncmp (targets[i], arch,strlen (arch))) { +if (!access ("/dev/kvm" ,R_OK |W_OK )) { + ret = true; +break ; + } + } + } +return ret; +} +bool qtest_get_irq (QTestState *s,int num) {/* dummy operation in order to make sure irq is up to date */
Signature · 1 line
2.27.0
Re: [RFC v2 1/3] tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM
On Thu, 17 Jun 2021 06:00:31 -0400
Igor Mammedov <[email protected]> wrote:
Paolo,
Is it acceptable to (ab)use meson like in this patch?
Quoted 91 lines
Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).
Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288This patch introduces
qtest_has_kvm()to let users check if KVM is available in advance and skip registering non run-able test-cases.PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.
Signed-off-by Igor Mammedov <[email protected]>---
v2:
- fixaccess()check.s/access()/!access()/- format C array items at meson.build time, and drop splitting targets string at runtime
tests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 2 ++ tests/qtest/libqtest.c | 18 ++++++++++++++++++ 3 files changed, 27 insertions(+)tests/qtest/libqos/libqtest.h +7
-0@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);*/ const char *qtest_get_arch (void ); +/** +* qtest_has_kvm: +* +* Returns: True if the QEMU executable under test supports KVM +*/ +bool qtest_has_kvm (void ); +/** * qtest_add_func: * @str: Test case path. meson.build +2
-0@@ -75,6 +75,8 @@ elif cpu in ['mips', 'mips64'] else kvm_targets = [] endif +kvm_targets_c = '"' + '" ,"'.join(kvm_targets) + '"' +config_host_data.set('CONFIG_KVM_TARGETS', kvm_targets_c) accelerator_targets = { 'CONFIG_KVM': kvm_targets } if cpu in ['x86', 'x86_64', 'arm', 'aarch64']tests/qtest/libqtest.c +18
-0@@ -920,6 +920,24 @@ const char *qtest_get_arch(void)return end +1 ; } +bool qtest_has_kvm (void ) +{ +int i; +bool ret = false; +const char *arch =qtest_get_arch (); +const char *targets[] = {CONFIG_KVM_TARGETS }; + +for (i =0 ; i <ARRAY_SIZE (targets); i++) { +if (!strncmp (targets[i], arch,strlen (arch))) { +if (!access ("/dev/kvm" ,R_OK |W_OK )) { + ret = true; +break ; + } + } + } +return ret; +} +bool qtest_get_irq (QTestState *s,int num) {/* dummy operation in order to make sure irq is up to date */
RFC 2/3 tests: acpi: q35: test for x2APIC entries in SRAT
In reply to Igor MammedovSet -smp 1,maxcpus=288 to test for ACPI code that
deal with CPUs with large APIC ID (>255).
PS:
Test requires KVM and in-kernel irqchip support,
so skip test if KVM is not available.
--- tests/qtest/bios-tables-test-allowed-diff.h | 4 ++++ tests/data/acpi/q35/APIC.numamem | 0 tests/data/acpi/q35/FACP.numamem | 0 tests/qtest/bios-tables-test.c | 10 +++++++--- 4 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
tests/qtest/bios-tables-test-allowed-diff.h +4 -0
@@ -1 +1,5 @@/* List of comma-separated changed AML files to ignore */ +"tests/data/acpi/q35/DSDT.numamem" , +"tests/data/acpi/q35/SRAT.numamem" , +"tests/data/acpi/q35/FACP.numamem" , +"tests/data/acpi/q35/APIC.numamem",
tests/data/acpi/q35/APIC.numamem new file +0 -0
tests/data/acpi/q35/FACP.numamem new file +0 -0
tests/qtest/bios-tables-test.c +7 -3
@@ -1021,7 +1021,7 @@ static void test_acpi_piix4_tcg_nohpet(void)free_test_data (&data); } -static void test_acpi_q35_tcg_numamem (void ) +static void test_acpi_q35_kvm_numamem (void ) {test_data data;
@@ -1029,7 +1029,9 @@ static void test_acpi_q35_tcg_numamem(void) data.machine =MACHINE_Q35 ; data.variant =".numamem" ;test_acpi_one (" -object memory-backend-ram,id=ram0,size=128M" -" -numa node -numa node,memdev=ram0" , &data); +" -numa node -numa node,memdev=ram0" +" -machine kernel-irqchip=on -smp 1,maxcpus=288" + , &data);free_test_data (&data); }
@@ -1536,7 +1538,6 @@ int main(int argc, char *argv[])qtest_add_func ("acpi/piix4/memhp" , test_acpi_piix4_tcg_memhp);qtest_add_func ("acpi/q35/memhp" , test_acpi_q35_tcg_memhp);qtest_add_func ("acpi/piix4/numamem" , test_acpi_piix4_tcg_numamem); -qtest_add_func ("acpi/q35/numamem" , test_acpi_q35_tcg_numamem);qtest_add_func ("acpi/piix4/nosmm" , test_acpi_piix4_tcg_nosmm);qtest_add_func ("acpi/piix4/smm-compat" , test_acpi_piix4_tcg_smm_compat);
@@ -1561,6 +1562,9 @@ int main(int argc, char *argv[])if (strcmp (arch,"x86_64" ) ==0 ) {qtest_add_func ("acpi/microvm/pcie" , test_acpi_microvm_pcie_tcg); } +if (qtest_has_kvm ()) { +qtest_add_func ("acpi/q35/numamem" , test_acpi_q35_kvm_numamem); + } }else if (strcmp (arch,"aarch64" ) ==0 ) {qtest_add_func ("acpi/virt" , test_acpi_virt_tcg);qtest_add_func ("acpi/virt/numamem" , test_acpi_virt_tcg_numamem);
Signature · 1 line
2.27.0
RFC 3/3 tests: acpi: update expected tables blobs
In reply to Igor MammedovUpdate adds CPU entries to MADT/SRAT/FACP and DSDT to cover 288 CPUs. Notable changes are that CPUs with APIC ID 255 and higher use 'Processor Local x2APIC Affinity' structure in SRAT and "Device" element in DSDT.
FACP:
- Use APIC Cluster Model (V4) : 0
+ Use APIC Cluster Model (V4) : 1
SRAT:
...
+[1010h 4112 1] Subtable Type : 00 [Processor Local APIC/SAPIC Affinity]
+[1011h 4113 1] Length : 10
+
+[1012h 4114 1] Proximity Domain Low(8) : 00 +[1013h 4115 1] Apic ID : FE +[1014h 4116 4] Flags (decoded below) : 00000001 + Enabled : 1 +[1018h 4120 1] Local Sapic EID : 00 +[1019h 4121 3] Proximity Domain High(24) : 000000 +[101Ch 4124 4] Clock Domain : 00000000
+
+[1020h 4128 1] Subtable Type : 02 [Processor Local x2APIC Affinity]
+[1021h 4129 1] Length : 18
+
+[1022h 4130 2] Reserved1 : 0000 +[1024h 4132 4] Proximity Domain : 00000001 +[1028h 4136 4] Apic ID : 000000FF +[102Ch 4140 4] Flags (decoded below) : 00000001
+ Enabled : 1
+[1030h 4144 4] Clock Domain : 00000000
+[1034h 4148 4] Reserved2 : 00000000
...
+[1320h 4896 1] Subtable Type : 02 [Processor Local x2APIC Affinity]
+[1321h 4897 1] Length : 18
+
+[1322h 4898 2] Reserved1 : 0000 +[1324h 4900 4] Proximity Domain : 00000001 +[1328h 4904 4] Apic ID : 0000011F +[132Ch 4908 4] Flags (decoded below) : 00000001
+ Enabled : 1
+[1330h 4912 4] Clock Domain : 00000000
+[1334h 4916 4] Reserved2 : 00000000
DSDT:
...
+ Processor (C0FE, 0xFE, 0x00000000, 0x00)
+ {
...
+ }
+
+ Device (C0FF)
+ {
+ Name (_HID, "ACPI0007" /* Processor Device */) // _HID: Hardware ID
+ Name (_UID, 0xFF) // _UID: Unique ID
...
+ }
+ Device (C11F)
+ {
+ Name (_HID, "ACPI0007" /* Processor Device */) // _HID: Hardware ID
+ Name (_UID, 0x011F) // _UID: Unique ID
...
+ }
APIC:
+[034h 0052 1] Subtable Type : 00 [Processor Local APIC] +[035h 0053 1] Length : 08 +[036h 0054 1] Processor ID : 01 +[037h 0055 1] Local Apic ID : 01 +[038h 0056 4] Flags (decoded below) : 00000000
+ Processor Enabled : 0
...
+[81Ch 2076 1] Subtable Type : 00 [Processor Local APIC] +[81Dh 2077 1] Length : 08 +[81Eh 2078 1] Processor ID : FE +[81Fh 2079 1] Local Apic ID : FE +[820h 2080 4] Flags (decoded below) : 00000000 + Processor Enabled : 0 + +[824h 2084 1] Subtable Type : 09 [Processor Local x2APIC] +[825h 2085 1] Length : 10 +[826h 2086 2] Reserved : 0000 +[828h 2088 4] Processor x2Apic ID : 000000FF +[82Ch 2092 4] Flags (decoded below) : 00000000
+ Processor Enabled : 0
+[830h 2096 4] Processor UID : 000000FF
...
+[A24h 2596 1] Subtable Type : 09 [Processor Local x2APIC] +[A25h 2597 1] Length : 10 +[A26h 2598 2] Reserved : 0000 +[A28h 2600 4] Processor x2Apic ID : 0000011F +[A2Ch 2604 4] Flags (decoded below) : 00000000 + Processor Enabled : 0 +[A30h 2608 4] Processor UID : 0000011F + +[A34h 2612 1] Subtable Type : 01 [I/O APIC] +[A35h 2613 1] Length : 0C +[A36h 2614 1] I/O Apic ID : 00 +[A37h 2615 1] Reserved : 00 +[A38h 2616 4] Address : FEC00000 +[A3Ch 2620 4] Interrupt : 00000000 + +[A40h 2624 1] Subtable Type : 02 [Interrupt Source Override] +[A41h 2625 1] Length : 0A +[A42h 2626 1] Bus : 00 +[A43h 2627 1] Source : 00 +[A44h 2628 4] Interrupt : 00000002 +[A48h 2632 2] Flags (decoded below) : 0000
Polarity : 0
Trigger Mode : 0
-[04Ah 0074 1] Subtable Type : 02 [Interrupt Source Override] -[04Bh 0075 1] Length : 0A -[04Ch 0076 1] Bus : 00 -[04Dh 0077 1] Source : 05 -[04Eh 0078 4] Interrupt : 00000005 -[052h 0082 2] Flags (decoded below) : 000D +[A4Ah 2634 1] Subtable Type : 02 [Interrupt Source Override] +[A4Bh 2635 1] Length : 0A +[A4Ch 2636 1] Bus : 00 +[A4Dh 2637 1] Source : 05 +[A4Eh 2638 4] Interrupt : 00000005 +[A52h 2642 2] Flags (decoded below) : 000D
Polarity : 1
Trigger Mode : 3
-[054h 0084 1] Subtable Type : 02 [Interrupt Source Override] -[055h 0085 1] Length : 0A -[056h 0086 1] Bus : 00 -[057h 0087 1] Source : 09 -[058h 0088 4] Interrupt : 00000009 -[05Ch 0092 2] Flags (decoded below) : 000D +[A54h 2644 1] Subtable Type : 02 [Interrupt Source Override] +[A55h 2645 1] Length : 0A +[A56h 2646 1] Bus : 00 +[A57h 2647 1] Source : 09 +[A58h 2648 4] Interrupt : 00000009 +[A5Ch 2652 2] Flags (decoded below) : 000D
Polarity : 1
Trigger Mode : 3
-[05Eh 0094 1] Subtable Type : 02 [Interrupt Source Override] -[05Fh 0095 1] Length : 0A -[060h 0096 1] Bus : 00 -[061h 0097 1] Source : 0A -[062h 0098 4] Interrupt : 0000000A -[066h 0102 2] Flags (decoded below) : 000D +[A5Eh 2654 1] Subtable Type : 02 [Interrupt Source Override] +[A5Fh 2655 1] Length : 0A +[A60h 2656 1] Bus : 00 +[A61h 2657 1] Source : 0A +[A62h 2658 4] Interrupt : 0000000A +[A66h 2662 2] Flags (decoded below) : 000D
Polarity : 1
Trigger Mode : 3
-[068h 0104 1] Subtable Type : 02 [Interrupt Source Override] -[069h 0105 1] Length : 0A -[06Ah 0106 1] Bus : 00 -[06Bh 0107 1] Source : 0B -[06Ch 0108 4] Interrupt : 0000000B -[070h 0112 2] Flags (decoded below) : 000D +[A68h 2664 1] Subtable Type : 02 [Interrupt Source Override] +[A69h 2665 1] Length : 0A +[A6Ah 2666 1] Bus : 00 +[A6Bh 2667 1] Source : 0B +[A6Ch 2668 4] Interrupt : 0000000B +[A70h 2672 2] Flags (decoded below) : 000D
Polarity : 1
Trigger Mode : 3
-[072h 0114 1] Subtable Type : 04 [Local APIC NMI]
-[073h 0115 1] Length : 06
-[074h 0116 1] Processor ID : FF
-[075h 0117 2] Flags (decoded below) : 0000
+[A72h 2674 1] Subtable Type : 0A [Local x2APIC NMI]
+[A73h 2675 1] Length : 0C
+[A74h 2676 2] Flags (decoded below) : 0000
Polarity : 0
Trigger Mode : 0
-[077h 0119 1] Interrupt Input LINT : 01
+[A76h 2678 4] Processor UID : FFFFFFFF
+[A7Ah 2682 1] Interrupt Input LINT : 01
+[A7Bh 2683 3] Reserved : 000000--- tests/qtest/bios-tables-test-allowed-diff.h | 4 ---- tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes 5 files changed, 4 deletions(-)
tests/qtest/bios-tables-test-allowed-diff.h +0 -4
@@ -1,5 +1 @@/* List of comma-separated changed AML files to ignore */ -"tests/data/acpi/q35/DSDT.numamem" , -"tests/data/acpi/q35/SRAT.numamem" , -"tests/data/acpi/q35/FACP.numamem" , -"tests/data/acpi/q35/APIC.numamem",
tests/data/acpi/q35/APIC.numamem binary +0 -0
tests/data/acpi/q35/DSDT.numamem binary +0 -0
tests/data/acpi/q35/FACP.numamem binary +0 -0
tests/data/acpi/q35/SRAT.numamem binary +0 -0
Signature · 1 line
2.27.0
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series seems to have some coding style problems. See output below for more information:
Type: series
Message-id: [email protected]
Subject: [RFC 0/3] qtest: pick tests that require KVM at runtime
=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===
Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
- [tag update] patchew/[email protected] -> patchew/[email protected]
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
99b622f tests: acpi: update expected tables blobs
8aad11d tests: acpi: q35: test for x2APIC entries in SRAT
5bab45e tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM
=== OUTPUT BEGIN ===
1/3 Checking commit 5bab45eeb20a (tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM)
2/3 Checking commit 8aad11d7634d (tests: acpi: q35: test for x2APIC entries in SRAT)
ERROR: Do not add expected files together with tests, follow instructions in tests/qtest/bios-tables-test.c: both tests/data/acpi/q35/FACP.numamem and tests/qtest/bios-tables-test.c found
ERROR: Do not add expected files together with tests, follow instructions in tests/qtest/bios-tables-test.c: both tests/data/acpi/q35/FACP.numamem and tests/qtest/bios-tables-test.c found
total: 2 errors, 0 warnings, 39 lines checked
Patch 2/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.
3/3 Checking commit 99b622fa110f (tests: acpi: update expected tables blobs) === OUTPUT END ===
Test command exited with code: 1
The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?
Does this series work with --disable-kvm builds? (TCG-only builds?)
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 9 lines · Igor Mammedov
Quoted 6 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Quoted 7 lines
Quoted 5 lines
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?
Quoted 56 lines
Quoted 2 lines
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 15 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?
Quoted 8 lines
Quoted 2 lines
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to
qtest_has_accel()needing some update? You mention it is time consuming to useqtest_has_accel(), have you measured an important overhead? Withqtest_has_accel()not even being committed yet, is it already necessary to work around it because it's too slow?
Tests are already take a lot of time as is, so I'd try to avoid slowing them down.
proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets
It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.
In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.
qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().
Given a less expensive approach exists, the qtest_has_accel() in its current form might be not justifiable.
Quoted 56 lines
Quoted 55 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 42 lines · Igor Mammedov
Quoted 15 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?Quoted 8 lines
Quoted 2 lines
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to
qtest_has_accel()needing some update? You mention it is time consuming to useqtest_has_accel(), have you measured an important overhead? Withqtest_has_accel()not even being committed yet, is it already necessary to work around it because it's too slow?Tests are already take a lot of time as is, so I'd try to avoid slowing them down.
proposed
qtest_has_accel()requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targetsIt's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.
In current form
qtest_has_accel()is only marginally better functionality wise, as it reports all built in accelerators whileqtest_has_kvm()accounts only for KVM.
Quoted 5 lines
qtest_has_kvm()is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposedqtest_has_accel().
One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.
If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.
Quoted 61 lines
Given a less expensive approach exists, the
qtest_has_accel()in its current form might be not justifiable.Quoted 56 lines
Quoted 55 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Signature · 1 line
Alex Bennée
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?Quoted 8 lines
Quoted 2 lines
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to
qtest_has_accel()needing some update? You mention it is time consuming to useqtest_has_accel(), have you measured an important overhead? Withqtest_has_accel()not even being committed yet, is it already necessary to work around it because it's too slow?Tests are already take a lot of time as is, so I'd try to avoid slowing them down.
proposed
qtest_has_accel()requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targetsIt's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.
In current form
qtest_has_accel()is only marginally better functionality wise, as it reports all built in accelerators whileqtest_has_kvm()accounts only for KVM.Quoted 5 lines
qtest_has_kvm()is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposedqtest_has_accel().
Quoted 6 lines
One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.
This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.
Quoted 4 lines
If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.
Good idea.
Quoted 62 lines
Quoted 60 lines
Given a less expensive approach exists, the
qtest_has_accel()in its current form might be not justifiable.Quoted 56 lines
Quoted 55 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 55 lines · On Tue, 22 Jun 2021 10:22:19 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?Quoted 8 lines
Quoted 2 lines
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to
qtest_has_accel()needing some update? You mention it is time consuming to useqtest_has_accel(), have you measured an important overhead? Withqtest_has_accel()not even being committed yet, is it already necessary to work around it because it's too slow?Tests are already take a lot of time as is, so I'd try to avoid slowing them down.
proposed
qtest_has_accel()requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targetsIt's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.
In current form
qtest_has_accel()is only marginally better functionality wise, as it reports all built in accelerators whileqtest_has_kvm()accounts only for KVM.Quoted 5 lines
qtest_has_kvm()is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposedqtest_has_accel().Quoted 6 lines
One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.
Quoted 3 lines
This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.
Currently qtest is only a build only and hard tied to concrete release tests. And it's usually mercilessly altered to fit any QEMU interface changes new version brings, which in turn breaks idea of testing other QEMU versions with it.
I'd guess it would take *a lot* of effort to make and maintain it as external test harness for various QEMU versions.
I think build time qtest used as public CI and external test suite are conflicting goals, for the former we probably minimize used compute resources while the later is probably run by QA in a more controlled manner (unless one integrates that into another CI)
If I needed something that were to be usable as external test suite, I'd look towards acceptance tests which are less depended on QEMU internals and versions.
Quoted 6 lines
Quoted 4 lines
If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.
Good idea.
Yep, it should be less taxing, than the currently posted qtest_has_accel() version.
It will reduce worst complexity little bit to
probbing_time * #test_binaries * number_targets * #push_req (all_forks)
but that's still explosive path.
Quoted 63 lines
Quoted 62 lines
Quoted 60 lines
Given a less expensive approach exists, the
qtest_has_accel()in its current form might be not justifiable.Quoted 56 lines
Quoted 55 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
Quoted 50 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
+Daniel (sorry Daniel)
Quoted 61 lines · Igor Mammedov
Quoted 55 lines · On Tue, 22 Jun 2021 10:22:19 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
Hello,
It seems to me that we are constantly re-implementing the same feature with slight variations?
Didn't we have a generic series to introduce
qtest_has_accel()from Philippe before?Quoted 8 lines
Quoted 2 lines
It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.
Thought it was separate, but now I see that it uses query-accel underneath.
Seems strange to add another check to do the same thing, it may point to
qtest_has_accel()needing some update? You mention it is time consuming to useqtest_has_accel(), have you measured an important overhead? Withqtest_has_accel()not even being committed yet, is it already necessary to work around it because it's too slow?Tests are already take a lot of time as is, so I'd try to avoid slowing them down.
proposed
qtest_has_accel()requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targetsIt's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.
In current form
qtest_has_accel()is only marginally better functionality wise, as it reports all built in accelerators whileqtest_has_kvm()accounts only for KVM.Quoted 5 lines
qtest_has_kvm()is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposedqtest_has_accel().Quoted 6 lines
One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.
Quoted 3 lines
This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.
Quoted 7 lines
Currently qtest is only a build only and hard tied to concrete release tests. And it's usually mercilessly altered to fit any QEMU interface changes new version brings, which in turn breaks idea of testing other QEMU versions with it.
I'd guess it would take *a lot* of effort to make and maintain it as external test harness for various QEMU versions.
I had the understanding the outcome of the last 6 months discussions was "do not tie testing with built binary, we have a machine protocol to introspect the binary" and "tests should focus on the feature to test, let's remove the build system burden".
Quoted 4 lines
I think build time qtest used as public CI and external test suite are conflicting goals, for the former we probably minimize used compute resources while the later is probably run by QA in a more controlled manner (unless one integrates that into another CI)
I think we are mixing orthogonal topics here:
* CI time is a constraint, no doubt, we don't want to waste it.
* Tests being buildsys-agnostic
If CI can't run all our tests, it is certainly not a reason for not adding more tests... We should be able to add as many tests as we want to the repository. Then we should select which tests we want to run, because we can't run them all.
Otherwise let's declare no more tests can be added in tests/qtest/ because our CI time is already full ;)
Quoted 16 lines
If I needed something that were to be usable as external test suite, I'd look towards acceptance tests which are less depended on QEMU internals and versions.
Quoted 6 lines
Quoted 4 lines
If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.
Good idea.
Yep, it should be less taxing, than the currently posted
qtest_has_accel()version.
It will reduce worst complexity little bit to
probbing_time* #test_binaries *number_targets* #push_req (all_forks)
but that's still explosive path.
Quoted 2 lines · On Fri, 18 Jun 2021 13:26:47 +0200 Igor Mammedov <[email protected]> wrote:
On Thu, 17 Jun 2021 18:49:17 +0200
Claudio Fontana <[email protected]> wrote:
[...]
Quoted 3 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
it didn't work, built fine but still tries to execute KVM test. Fixed v3 is on the way
Quoted 53 lines
Quoted 51 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 6 lines · Igor Mammedov
Quoted 2 lines · On Fri, 18 Jun 2021 13:26:47 +0200 Igor Mammedov <[email protected]> wrote:
On Thu, 17 Jun 2021 18:49:17 +0200
Claudio Fontana <[email protected]> wrote:
Quoted 8 lines
[...]
Quoted 3 lines
Quoted 1 line
Does this series work with --disable-kvm builds? (TCG-only builds?)
I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).
it didn't work, built fine but still tries to execute KVM test. Fixed v3 is on the way
I am thinking, what about doing the checks we need to do at the beginning of the tests, and cache the results for all the tests, instead of checking every time?
This way we could use a more general implementation qtest_has_accel("kvm"), and mitigate its cost?
Thanks
C
Quoted 56 lines
Quoted 54 lines
Quoted 52 lines
Thanks,
CLaudio
Quoted 43 lines
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobsroot (1):
tests: qtest: addqtest_has_kvm()to check if tested bynary supports
KVMtests/qtest/libqos/libqtest.h | 7 +++++++ meson.build | 1 + tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes tests/qtest/bios-tables-test.c | 10 +++++++--- tests/qtest/libqtest.c | 20 ++++++++++++++++++++ 8 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 tests/data/acpi/q35/APIC.numamem create mode 100644 tests/data/acpi/q35/FACP.numamem
Quoted 12 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'
Quoted 3 lines
Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.
You also might want to update the check in tests/qtest/migration-test.c while you're at it.
Quoted 5 lines
PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared toqtest_has_kvm()and could be extended to take into account machine type.
Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...
Thanks,
Thomas
Quoted 24 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Quoted 3 lines
Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.You also might want to update the check in
tests/qtest/migration-test.cwhile you're at it.Quoted 4 lines
PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
Quoted 5 lines
Quoted 2 lines
which I think is more robust compared to
qtest_has_kvm()and could be extended to take into account machine type.Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...
I'm certainly fine if Igor wants to restart this effort :)
Maybe doing as Claudio suggested, start have qtest_has_accel() tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'Quoted 3 lines
Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.You also might want to update the check in
tests/qtest/migration-test.cwhile you're at it.Quoted 6 lines
PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/which I think is more robust compared to
qtest_has_kvm()and could be extended to take into account machine type.
Quoted 8 lines
Quoted 2 lines
Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...
I'm certainly fine if Igor wants to restart this effort :)
Maybe doing as Claudio suggested, start have
qtest_has_accel()tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?
Yeah, that's maybe a good idea ...
But I'm currently wondering whether we need query-accels at all? For detecting kvm, we already have "query-kvm" ... and we don't have tests for any other accelerators yet (hax, hvf, etc.) ... so it's likely just about having a way to detect whether TCG is compiled into the binary? Is there already another command that could be used to check for the availability of TCG?
Thomas
Quoted 15 lines · On Tue, 22 Jun 2021 09:59:48 +0200 Thomas Huth <[email protected]> wrote:
Quoted 14 lines · Philippe Mathieu-Daudé
Quoted 13 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov
Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.
For an example:
test q35 machine withintel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Series adds a lightweight
qtest_has_kvm()check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.You also might want to update the check in
tests/qtest/migration-test.cwhile you're at it.
sure (this series was just a discussion starter).
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/which I think is more robust compared to
qtest_has_kvm()and could be extended to take into account machine type.
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...
I'm certainly fine if Igor wants to restart this effort :)
Maybe doing as Claudio suggested, start have
qtest_has_accel()tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?
it should save time when shortpath result is positive, but won't help on fallback path.
Maybe we can add a build option to enable fallback, or even better to swap build time impl. with runtime impl., so ones who need qtest as external harness can build and use it on request.
Quoted 5 lines
Yeah, that's maybe a good idea ...
But I'm currently wondering whether we need query-accels at all? For detecting kvm, we already have "query-kvm" ... and we don't have tests for any other accelerators yet (hax, hvf, etc.) ... so it's likely just about having a way to detect whether TCG is compiled into the binary?
if we ignore other accelerators, it's probably not hard to add detection of TCG at build time, I'll look into it and also try to change API to qtest_has_accel() instead of kvm specific one. This way whatever solution we end up with, won't affect code that uses it.
Quoted 5 lines
Is there
already another command that could be used to check for the availability of TCG?Thomas