Skip to contentSkip to conversation

qtest: pick tests that require KVM at runtime

RFC Superseded Igor Mammedov 6 participants 21 messages Jun 16, 2021 – Jun 22, 2021 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

cover qtest: pick tests that require KVM at runtime

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
Signature · 1 line
2.27.0
#1 lore raw

RFC 1/3 tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM

From: root <[email protected]>

Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).

Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288

This patch introduces qtest_has_kvm() to let users check if KVM is available in advance and skip registering non run-able test-cases.

---
PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.

on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.

 tests/qtest/libqos/libqtest.h |  7 +++++++
 meson.build                   |  1 +
 tests/qtest/libqtest.c        | 20 ++++++++++++++++++++
 3 files changed, 28 insertions(+)
tests/qtest/libqos/libqtest.h +7 -0
index a68dcd79d4..bab0047117 100644
@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);
  */
 const char *qtest_get_arch(void);
 
+/**
+ * qtest_has_kvm:
+ *
+ * Returns: True if the QEMU executable under test supports KVM
+ */
+bool qtest_has_kvm(void);
+
 /**
  * qtest_add_func:
  * @str: Test case path.
meson.build +1 -0
index d2a9ce91f5..dbd31f9f6c 100644
@@ -75,6 +75,7 @@ elif cpu in ['mips', 'mips64']
 else
   kvm_targets = []
 endif
+config_host_data.set_quoted('CONFIG_KVM_TARGETS', ' ,'.join(kvm_targets))
 
 accelerator_targets = { 'CONFIG_KVM': kvm_targets }
 if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
tests/qtest/libqtest.c +20 -0
index 825b13a44c..d019ac88c9 100644
@@ -920,6 +920,26 @@ const char *qtest_get_arch(void)
     return end + 1;
 }
 
+bool qtest_has_kvm(void)
+{
+    int i;
+    bool ret = false;
+    const char *arch = qtest_get_arch();
+    const char *kvm_targets_str = CONFIG_KVM_TARGETS;
+    gchar **targets = g_strsplit(kvm_targets_str, " ", -1);
+
+    for (i = 0; targets[i]; i++) {
+        if (!strncmp(targets[i], arch, strlen(arch))) {
+            if (access("/dev/kvm", R_OK | W_OK)) {
+                ret = true;
+                break;
+            }
+        }
+    }
+    g_strfreev(targets);
+    return ret;
+}
+
 bool qtest_get_irq(QTestState *s, int num)
 {
     /* dummy operation in order to make sure irq is up to date */
Signature · 1 line
2.27.0
#2 lore raw

RFC v2 1/3 tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM

Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).

Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288

This patch introduces qtest_has_kvm() to let users check if KVM is available in advance and skip registering non run-able test-cases.

PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.

on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.

---
v2:
- fix access() check.

     s/access()/!access()/

- format C array items at meson.build time, and drop splitting targets string at runtime

 tests/qtest/libqos/libqtest.h |  7 +++++++
 meson.build                   |  2 ++
 tests/qtest/libqtest.c        | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+)
tests/qtest/libqos/libqtest.h +7 -0
index a68dcd79d4..bab0047117 100644
@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);
  */
 const char *qtest_get_arch(void);
 
+/**
+ * qtest_has_kvm:
+ *
+ * Returns: True if the QEMU executable under test supports KVM
+ */
+bool qtest_has_kvm(void);
+
 /**
  * qtest_add_func:
  * @str: Test case path.
meson.build +2 -0
index d2a9ce91f5..7fec4e8289 100644
@@ -75,6 +75,8 @@ elif cpu in ['mips', 'mips64']
 else
   kvm_targets = []
 endif
+kvm_targets_c = '"' + '" ,"'.join(kvm_targets) + '"'
+config_host_data.set('CONFIG_KVM_TARGETS', kvm_targets_c)
 
 accelerator_targets = { 'CONFIG_KVM': kvm_targets }
 if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
tests/qtest/libqtest.c +18 -0
index 825b13a44c..daa6d54059 100644
@@ -920,6 +920,24 @@ const char *qtest_get_arch(void)
     return end + 1;
 }
 
+bool qtest_has_kvm(void)
+{
+    int i;
+    bool ret = false;
+    const char *arch = qtest_get_arch();
+    const char *targets[] = { CONFIG_KVM_TARGETS };
+
+    for (i = 0; i < ARRAY_SIZE(targets); i++) {
+        if (!strncmp(targets[i], arch, strlen(arch))) {
+            if (!access("/dev/kvm", R_OK | W_OK)) {
+                ret = true;
+                break;
+            }
+        }
+    }
+    return ret;
+}
+
 bool qtest_get_irq(QTestState *s, int num)
 {
     /* dummy operation in order to make sure irq is up to date */
Signature · 1 line
2.27.0
#3 lore raw

Re: [RFC v2 1/3] tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM

On Thu, 17 Jun 2021 06:00:31 -0400
Igor Mammedov <[email protected]> wrote:

Paolo,
Is it acceptable to (ab)use meson like in this patch?

Quoted 91 lines

Currently it not possible to create tests that have KVM as a hard requirement on a host that doesn't support KVM for tested target binary (modulo going through the trouble of compiling out the offending test case).

Following scenario makes test fail when it's run on non x86 host:
qemu-system-x86_64 -enable-kvm -M q35,kernel-irqchip=on -smp 1,maxcpus=288

This patch introduces qtest_has_kvm() to let users check if KVM is available in advance and skip registering non run-able test-cases.

PS:
It's simplistic and not as versatile/precise as earlier proposed 'query-accels' series, but it get job done for simple cases.

on upside it's much cheaper to execute than the 'query-accels' as it doesn't need to run QEMU for probing.

---
v2:
- fix access() check.

     s/access()/!access()/

- format C array items at meson.build time, and drop splitting targets string at runtime

 tests/qtest/libqos/libqtest.h |  7 +++++++
 meson.build                   |  2 ++
 tests/qtest/libqtest.c        | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+)
tests/qtest/libqos/libqtest.h +7 -0
index a68dcd79d4..bab0047117 100644
@@ -588,6 +588,13 @@ bool qtest_big_endian(QTestState *s);
  */
 const char *qtest_get_arch(void);
 
+/**
+ * qtest_has_kvm:
+ *
+ * Returns: True if the QEMU executable under test supports KVM
+ */
+bool qtest_has_kvm(void);
+
 /**
  * qtest_add_func:
  * @str: Test case path.
meson.build +2 -0
index d2a9ce91f5..7fec4e8289 100644
@@ -75,6 +75,8 @@ elif cpu in ['mips', 'mips64']
 else
   kvm_targets = []
 endif
+kvm_targets_c = '"' + '" ,"'.join(kvm_targets) + '"'
+config_host_data.set('CONFIG_KVM_TARGETS', kvm_targets_c)
 
 accelerator_targets = { 'CONFIG_KVM': kvm_targets }
 if cpu in ['x86', 'x86_64', 'arm', 'aarch64']
tests/qtest/libqtest.c +18 -0
index 825b13a44c..daa6d54059 100644
@@ -920,6 +920,24 @@ const char *qtest_get_arch(void)
     return end + 1;
 }
 
+bool qtest_has_kvm(void)
+{
+    int i;
+    bool ret = false;
+    const char *arch = qtest_get_arch();
+    const char *targets[] = { CONFIG_KVM_TARGETS };
+
+    for (i = 0; i < ARRAY_SIZE(targets); i++) {
+        if (!strncmp(targets[i], arch, strlen(arch))) {
+            if (!access("/dev/kvm", R_OK | W_OK)) {
+                ret = true;
+                break;
+            }
+        }
+    }
+    return ret;
+}
+
 bool qtest_get_irq(QTestState *s, int num)
 {
     /* dummy operation in order to make sure irq is up to date */
#4 lore raw

RFC 2/3 tests: acpi: q35: test for x2APIC entries in SRAT

In reply to Igor Mammedov

Set -smp 1,maxcpus=288 to test for ACPI code that
deal with CPUs with large APIC ID (>255).

PS:
Test requires KVM and in-kernel irqchip support,
so skip test if KVM is not available.

---
 tests/qtest/bios-tables-test-allowed-diff.h |  4 ++++
 tests/data/acpi/q35/APIC.numamem            |  0
 tests/data/acpi/q35/FACP.numamem            |  0
 tests/qtest/bios-tables-test.c              | 10 +++++++---
 4 files changed, 11 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
tests/qtest/bios-tables-test-allowed-diff.h +4 -0
index dfb8523c8b..c4c132e49b 100644
@@ -1 +1,5 @@
 /* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/q35/DSDT.numamem",
+"tests/data/acpi/q35/SRAT.numamem",
+"tests/data/acpi/q35/FACP.numamem",
+"tests/data/acpi/q35/APIC.numamem",
tests/data/acpi/q35/APIC.numamem new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/data/acpi/q35/FACP.numamem new file +0 -0
new file mode 100644
index 0000000000..e69de29bb2
tests/qtest/bios-tables-test.c +7 -3
index 51d3a4e239..3ecf7d0d58 100644
@@ -1021,7 +1021,7 @@ static void test_acpi_piix4_tcg_nohpet(void)
     free_test_data(&data);
 }
 
-static void test_acpi_q35_tcg_numamem(void)
+static void test_acpi_q35_kvm_numamem(void)
 {
     test_data data;
 
@@ -1029,7 +1029,9 @@ static void test_acpi_q35_tcg_numamem(void)
     data.machine = MACHINE_Q35;
     data.variant = ".numamem";
     test_acpi_one(" -object memory-backend-ram,id=ram0,size=128M"
-                  " -numa node -numa node,memdev=ram0", &data);
+                  " -numa node -numa node,memdev=ram0"
+                  " -machine kernel-irqchip=on -smp 1,maxcpus=288"
+                   , &data);
     free_test_data(&data);
 }
 
@@ -1536,7 +1538,6 @@ int main(int argc, char *argv[])
         qtest_add_func("acpi/piix4/memhp", test_acpi_piix4_tcg_memhp);
         qtest_add_func("acpi/q35/memhp", test_acpi_q35_tcg_memhp);
         qtest_add_func("acpi/piix4/numamem", test_acpi_piix4_tcg_numamem);
-        qtest_add_func("acpi/q35/numamem", test_acpi_q35_tcg_numamem);
         qtest_add_func("acpi/piix4/nosmm", test_acpi_piix4_tcg_nosmm);
         qtest_add_func("acpi/piix4/smm-compat",
                        test_acpi_piix4_tcg_smm_compat);
@@ -1561,6 +1562,9 @@ int main(int argc, char *argv[])
         if (strcmp(arch, "x86_64") == 0) {
             qtest_add_func("acpi/microvm/pcie", test_acpi_microvm_pcie_tcg);
         }
+        if (qtest_has_kvm()) {
+            qtest_add_func("acpi/q35/numamem", test_acpi_q35_kvm_numamem);
+        }
     } else if (strcmp(arch, "aarch64") == 0) {
         qtest_add_func("acpi/virt", test_acpi_virt_tcg);
         qtest_add_func("acpi/virt/numamem", test_acpi_virt_tcg_numamem);
Signature · 1 line
2.27.0
#5 lore raw

RFC 3/3 tests: acpi: update expected tables blobs

In reply to Igor Mammedov

Update adds CPU entries to MADT/SRAT/FACP and DSDT to cover 288 CPUs. Notable changes are that CPUs with APIC ID 255 and higher use 'Processor Local x2APIC Affinity' structure in SRAT and "Device" element in DSDT.

FACP:
- Use APIC Cluster Model (V4) : 0
+ Use APIC Cluster Model (V4) : 1

SRAT:
...
+[1010h 4112 1] Subtable Type : 00 [Processor Local APIC/SAPIC Affinity]
+[1011h 4113 1] Length : 10
+

+[1012h 4114   1]      Proximity Domain Low(8) : 00
+[1013h 4115   1]                      Apic ID : FE
+[1014h 4116   4]        Flags (decoded below) : 00000001
+                                     Enabled : 1
+[1018h 4120   1]              Local Sapic EID : 00
+[1019h 4121   3]    Proximity Domain High(24) : 000000
+[101Ch 4124   4]                 Clock Domain : 00000000

+
+[1020h 4128 1] Subtable Type : 02 [Processor Local x2APIC Affinity]
+[1021h 4129 1] Length : 18
+

+[1022h 4130   2]                    Reserved1 : 0000
+[1024h 4132   4]             Proximity Domain : 00000001
+[1028h 4136   4]                      Apic ID : 000000FF
+[102Ch 4140   4]        Flags (decoded below) : 00000001

+ Enabled : 1
+[1030h 4144 4] Clock Domain : 00000000
+[1034h 4148 4] Reserved2 : 00000000

...

+[1320h 4896 1] Subtable Type : 02 [Processor Local x2APIC Affinity]
+[1321h 4897 1] Length : 18
+

+[1322h 4898   2]                    Reserved1 : 0000
+[1324h 4900   4]             Proximity Domain : 00000001
+[1328h 4904   4]                      Apic ID : 0000011F
+[132Ch 4908   4]        Flags (decoded below) : 00000001

+ Enabled : 1
+[1330h 4912 4] Clock Domain : 00000000
+[1334h 4916 4] Reserved2 : 00000000

DSDT:

...
+ Processor (C0FE, 0xFE, 0x00000000, 0x00)
+ {
...
+ }
+
+ Device (C0FF)
+ {
+ Name (_HID, "ACPI0007" /* Processor Device */) // _HID: Hardware ID
+ Name (_UID, 0xFF) // _UID: Unique ID
...
+ }

+ Device (C11F)
+ {
+ Name (_HID, "ACPI0007" /* Processor Device */) // _HID: Hardware ID
+ Name (_UID, 0x011F) // _UID: Unique ID
...
+ }

APIC:

+[034h 0052   1]                Subtable Type : 00 [Processor Local APIC]
+[035h 0053   1]                       Length : 08
+[036h 0054   1]                 Processor ID : 01
+[037h 0055   1]                Local Apic ID : 01
+[038h 0056   4]        Flags (decoded below) : 00000000

+ Processor Enabled : 0

...

+[81Ch 2076   1]                Subtable Type : 00 [Processor Local APIC]
+[81Dh 2077   1]                       Length : 08
+[81Eh 2078   1]                 Processor ID : FE
+[81Fh 2079   1]                Local Apic ID : FE
+[820h 2080   4]        Flags (decoded below) : 00000000
+                           Processor Enabled : 0
+
+[824h 2084   1]                Subtable Type : 09 [Processor Local x2APIC]
+[825h 2085   1]                       Length : 10
+[826h 2086   2]                     Reserved : 0000
+[828h 2088   4]          Processor x2Apic ID : 000000FF
+[82Ch 2092   4]        Flags (decoded below) : 00000000

+ Processor Enabled : 0
+[830h 2096 4] Processor UID : 000000FF

...

+[A24h 2596   1]                Subtable Type : 09 [Processor Local x2APIC]
+[A25h 2597   1]                       Length : 10
+[A26h 2598   2]                     Reserved : 0000
+[A28h 2600   4]          Processor x2Apic ID : 0000011F
+[A2Ch 2604   4]        Flags (decoded below) : 00000000
+                           Processor Enabled : 0
+[A30h 2608   4]                Processor UID : 0000011F
+
+[A34h 2612   1]                Subtable Type : 01 [I/O APIC]
+[A35h 2613   1]                       Length : 0C
+[A36h 2614   1]                  I/O Apic ID : 00
+[A37h 2615   1]                     Reserved : 00
+[A38h 2616   4]                      Address : FEC00000
+[A3Ch 2620   4]                    Interrupt : 00000000
+
+[A40h 2624   1]                Subtable Type : 02 [Interrupt Source Override]
+[A41h 2625   1]                       Length : 0A
+[A42h 2626   1]                          Bus : 00
+[A43h 2627   1]                       Source : 00
+[A44h 2628   4]                    Interrupt : 00000002
+[A48h 2632   2]        Flags (decoded below) : 0000

Polarity : 0
Trigger Mode : 0

-[04Ah 0074   1]                Subtable Type : 02 [Interrupt Source Override]
-[04Bh 0075   1]                       Length : 0A
-[04Ch 0076   1]                          Bus : 00
-[04Dh 0077   1]                       Source : 05
-[04Eh 0078   4]                    Interrupt : 00000005
-[052h 0082   2]        Flags (decoded below) : 000D
+[A4Ah 2634   1]                Subtable Type : 02 [Interrupt Source Override]
+[A4Bh 2635   1]                       Length : 0A
+[A4Ch 2636   1]                          Bus : 00
+[A4Dh 2637   1]                       Source : 05
+[A4Eh 2638   4]                    Interrupt : 00000005
+[A52h 2642   2]        Flags (decoded below) : 000D

Polarity : 1
Trigger Mode : 3

-[054h 0084   1]                Subtable Type : 02 [Interrupt Source Override]
-[055h 0085   1]                       Length : 0A
-[056h 0086   1]                          Bus : 00
-[057h 0087   1]                       Source : 09
-[058h 0088   4]                    Interrupt : 00000009
-[05Ch 0092   2]        Flags (decoded below) : 000D
+[A54h 2644   1]                Subtable Type : 02 [Interrupt Source Override]
+[A55h 2645   1]                       Length : 0A
+[A56h 2646   1]                          Bus : 00
+[A57h 2647   1]                       Source : 09
+[A58h 2648   4]                    Interrupt : 00000009
+[A5Ch 2652   2]        Flags (decoded below) : 000D

Polarity : 1
Trigger Mode : 3

-[05Eh 0094   1]                Subtable Type : 02 [Interrupt Source Override]
-[05Fh 0095   1]                       Length : 0A
-[060h 0096   1]                          Bus : 00
-[061h 0097   1]                       Source : 0A
-[062h 0098   4]                    Interrupt : 0000000A
-[066h 0102   2]        Flags (decoded below) : 000D
+[A5Eh 2654   1]                Subtable Type : 02 [Interrupt Source Override]
+[A5Fh 2655   1]                       Length : 0A
+[A60h 2656   1]                          Bus : 00
+[A61h 2657   1]                       Source : 0A
+[A62h 2658   4]                    Interrupt : 0000000A
+[A66h 2662   2]        Flags (decoded below) : 000D

Polarity : 1
Trigger Mode : 3

-[068h 0104   1]                Subtable Type : 02 [Interrupt Source Override]
-[069h 0105   1]                       Length : 0A
-[06Ah 0106   1]                          Bus : 00
-[06Bh 0107   1]                       Source : 0B
-[06Ch 0108   4]                    Interrupt : 0000000B
-[070h 0112   2]        Flags (decoded below) : 000D
+[A68h 2664   1]                Subtable Type : 02 [Interrupt Source Override]
+[A69h 2665   1]                       Length : 0A
+[A6Ah 2666   1]                          Bus : 00
+[A6Bh 2667   1]                       Source : 0B
+[A6Ch 2668   4]                    Interrupt : 0000000B
+[A70h 2672   2]        Flags (decoded below) : 000D

Polarity : 1
Trigger Mode : 3

-[072h 0114   1]                Subtable Type : 04 [Local APIC NMI]
-[073h 0115   1]                       Length : 06
-[074h 0116   1]                 Processor ID : FF
-[075h 0117   2]        Flags (decoded below) : 0000
+[A72h 2674   1]                Subtable Type : 0A [Local x2APIC NMI]
+[A73h 2675   1]                       Length : 0C
+[A74h 2676   2]        Flags (decoded below) : 0000
                                     Polarity : 0
                                 Trigger Mode : 0
-[077h 0119   1]         Interrupt Input LINT : 01
+[A76h 2678   4]                Processor UID : FFFFFFFF
+[A7Ah 2682   1]         Interrupt Input LINT : 01
+[A7Bh 2683   3]                     Reserved : 000000
---
 tests/qtest/bios-tables-test-allowed-diff.h |   4 ----
 tests/data/acpi/q35/APIC.numamem            | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem            | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem            | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem            | Bin 224 -> 5080 bytes
 5 files changed, 4 deletions(-)
tests/qtest/bios-tables-test-allowed-diff.h +0 -4
index c4c132e49b..dfb8523c8b 100644
@@ -1,5 +1 @@
 /* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/q35/DSDT.numamem",
-"tests/data/acpi/q35/SRAT.numamem",
-"tests/data/acpi/q35/FACP.numamem",
-"tests/data/acpi/q35/APIC.numamem",
tests/data/acpi/q35/APIC.numamem binary +0 -0
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..c1969c35aa12b61d25e0134bbb8d2187ba42d663 100644
GIT binary patch
tests/data/acpi/q35/DSDT.numamem binary +0 -0
index e4c4582e7f76b072ab1123c748b89ea33ea1db87..8458aad2484bc48a95f2df37109a4dbfe8cee893 100644
GIT binary patch
tests/data/acpi/q35/FACP.numamem binary +0 -0
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..2d3659c9c6753d07c3d48742343cb8e8cc034de7 100644
GIT binary patch
tests/data/acpi/q35/SRAT.numamem binary +0 -0
index 56da58e2699cb13fcd41b5e1667ceeddbdd443b5..097b11d8de2cc7aa8d76154a0082252ba5b735dd 100644
GIT binary patch
Signature · 1 line
2.27.0
#6 lore raw
In reply to Igor Mammedov
Patchew URL: https://patchew.org/QEMU/[email protected]/

Patchew URL: https://patchew.org/QEMU/[email protected]/

Hi,

This series seems to have some coding style problems. See output below for more information:

Type: series
Message-id: [email protected]
Subject: [RFC 0/3] qtest: pick tests that require KVM at runtime

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
- [tag update] patchew/[email protected] -> patchew/[email protected]
* [new tag] patchew/[email protected] -> patchew/[email protected]
Switched to a new branch 'test'
99b622f tests: acpi: update expected tables blobs
8aad11d tests: acpi: q35: test for x2APIC entries in SRAT
5bab45e tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM

=== OUTPUT BEGIN ===
1/3 Checking commit 5bab45eeb20a (tests: qtest: add qtest_has_kvm() to check if tested binary supports KVM)
2/3 Checking commit 8aad11d7634d (tests: acpi: q35: test for x2APIC entries in SRAT)
ERROR: Do not add expected files together with tests, follow instructions in tests/qtest/bios-tables-test.c: both tests/data/acpi/q35/FACP.numamem and tests/qtest/bios-tables-test.c found

ERROR: Do not add expected files together with tests, follow instructions in tests/qtest/bios-tables-test.c: both tests/data/acpi/q35/FACP.numamem and tests/qtest/bios-tables-test.c found

total: 2 errors, 0 warnings, 39 lines checked

Patch 2/3 has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS.

3/3 Checking commit 99b622fa110f (tests: acpi: update expected tables blobs) === OUTPUT END ===

Test command exited with code: 1

The full log is available at
http://patchew.org/logs/[email protected]/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]

#7 lore raw
In reply to Igor Mammedov
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Does this series work with --disable-kvm builds? (TCG-only builds?)

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#8 lore raw
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#9 lore raw
Quoted 9 lines · Igor Mammedov
Quoted 6 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Quoted 7 lines
Quoted 5 lines

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Quoted 56 lines
Quoted 2 lines

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#10 lore raw
Quoted 15 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Quoted 8 lines
Quoted 2 lines

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Tests are already take a lot of time as is, so I'd try to avoid slowing them down.

proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets

It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.

In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.

qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().

Given a less expensive approach exists, the qtest_has_accel() in its current form might be not justifiable.

Quoted 56 lines
Quoted 55 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#11 lore raw
Quoted 42 lines · Igor Mammedov
Quoted 15 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Quoted 8 lines
Quoted 2 lines

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Tests are already take a lot of time as is, so I'd try to avoid slowing them down.

proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets

It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.

In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.

Quoted 5 lines

qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().

One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.

If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.

Quoted 61 lines

Given a less expensive approach exists, the qtest_has_accel() in its current form might be not justifiable.

Quoted 56 lines
Quoted 55 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
Signature · 1 line
Alex Bennée
#12 lore raw
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Quoted 8 lines
Quoted 2 lines

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Tests are already take a lot of time as is, so I'd try to avoid slowing them down.

proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets

It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.

In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.

Quoted 5 lines

qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().

Quoted 6 lines

One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.

This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.

Quoted 4 lines

If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.

Good idea.

Quoted 62 lines
Quoted 60 lines

Given a less expensive approach exists, the qtest_has_accel() in its current form might be not justifiable.

Quoted 56 lines
Quoted 55 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#13 lore raw
Quoted 55 lines · On Tue, 22 Jun 2021 10:22:19 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Quoted 8 lines
Quoted 2 lines

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Tests are already take a lot of time as is, so I'd try to avoid slowing them down.

proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets

It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.

In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.

Quoted 5 lines

qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().

Quoted 6 lines

One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.

Quoted 3 lines

This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.

Currently qtest is only a build only and hard tied to concrete release tests. And it's usually mercilessly altered to fit any QEMU interface changes new version brings, which in turn breaks idea of testing other QEMU versions with it.

I'd guess it would take *a lot* of effort to make and maintain it as external test harness for various QEMU versions.

I think build time qtest used as public CI and external test suite are conflicting goals, for the former we probably minimize used compute resources while the later is probably run by QA in a more controlled manner (unless one integrates that into another CI)

If I needed something that were to be usable as external test suite, I'd look towards acceptance tests which are less depended on QEMU internals and versions.

Quoted 6 lines
Quoted 4 lines

If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.

Good idea.

Yep, it should be less taxing, than the currently posted qtest_has_accel() version.
It will reduce worst complexity little bit to
probbing_time * #test_binaries * number_targets * #push_req (all_forks)
but that's still explosive path.

Quoted 63 lines
Quoted 62 lines
Quoted 60 lines

Given a less expensive approach exists, the qtest_has_accel() in its current form might be not justifiable.

Quoted 56 lines
Quoted 55 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

Quoted 50 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#14 lore raw

+Daniel (sorry Daniel)

Quoted 61 lines · Igor Mammedov
Quoted 55 lines · On Tue, 22 Jun 2021 10:22:19 +0200 Philippe Mathieu-Daudé <[email protected]>…
Quoted 47 lines · Alex Bennée
Quoted 40 lines · Igor Mammedov
Quoted 14 lines · On Fri, 18 Jun 2021 14:43:46 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 13 lines · Igor Mammedov
Quoted 11 lines · On Thu, 17 Jun 2021 18:49:17 +0200 Claudio Fontana <[email protected]> wrote:
Quoted 4 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

Hello,

It seems to me that we are constantly re-implementing the same feature with slight variations?

Didn't we have a generic series to introduce qtest_has_accel() from Philippe before?

Quoted 8 lines
Quoted 2 lines

It's mentioned in cover letter (PS: part) and in [1/3] with rationale why this was posted.

Thought it was separate, but now I see that it uses query-accel underneath.

Seems strange to add another check to do the same thing, it may point to qtest_has_accel() needing some update? You mention it is time consuming to use qtest_has_accel(), have you measured an important overhead? With qtest_has_accel() not even being committed yet, is it already necessary to work around it because it's too slow?

Tests are already take a lot of time as is, so I'd try to avoid slowing them down.

proposed qtest_has_accel() requires spawning QEMU to probe, which is slow. Worst case would be:
= qemu startup time * number of checks * number of targets

It's fine to run occasionally, I can take a coffee break while tests run. But put it in context of CI and it multiplies by the number of push requests and starts to eat not only time but also limited CI resources.

In current form qtest_has_accel() is only marginally better functionality wise, as it reports all built in accelerators while qtest_has_kvm() accounts only for KVM.

Quoted 5 lines

qtest_has_kvm() is collecting info about built-in accelerators at configure/build time and that probably could be extended to other accelerators (not a thing that I'm interested in at the moment). So it could be extended to support the same accelerators as currently proposed qtest_has_accel().

Quoted 6 lines

One minor downside is this forever ties the tests to the build. I have spoken with people before about the idea of separating the test artefacts from the build so they can be used either as a) cached test objects or b) other testing environments, for example verifying the kernel has not regressed. However we don't do either of those things at the moment so it's not a major concern.

Quoted 3 lines

This is the feature that is interesting RedHat QE too, run the latest qtests on various released binaries to compare performances between releases.

Quoted 7 lines

Currently qtest is only a build only and hard tied to concrete release tests. And it's usually mercilessly altered to fit any QEMU interface changes new version brings, which in turn breaks idea of testing other QEMU versions with it.

I'd guess it would take *a lot* of effort to make and maintain it as external test harness for various QEMU versions.

I had the understanding the outcome of the last 6 months discussions was "do not tie testing with built binary, we have a machine protocol to introspect the binary" and "tests should focus on the feature to test, let's remove the build system burden".

Quoted 4 lines

I think build time qtest used as public CI and external test suite are conflicting goals, for the former we probably minimize used compute resources while the later is probably run by QA in a more controlled manner (unless one integrates that into another CI)

I think we are mixing orthogonal topics here:

* CI time is a constraint, no doubt, we don't want to waste it.

* Tests being buildsys-agnostic

If CI can't run all our tests, it is certainly not a reason for not adding more tests... We should be able to add as many tests as we want to the repository. Then we should select which tests we want to run, because we can't run them all.

Otherwise let's declare no more tests can be added in tests/qtest/ because our CI time is already full ;)

Quoted 16 lines

If I needed something that were to be usable as external test suite, I'd look towards acceptance tests which are less depended on QEMU internals and versions.

Quoted 6 lines
Quoted 4 lines

If the worry is about extending test times by having an extra round trip of a spawn and query step for every test could we not consider caching the information somewhere? Really any given binary should only need to be queried once per run, not per test.

Good idea.

Yep, it should be less taxing, than the currently posted qtest_has_accel() version.
It will reduce worst complexity little bit to
probbing_time * #test_binaries * number_targets * #push_req (all_forks)
but that's still explosive path.

#15 lore raw
In reply to Igor Mammedov
Quoted 2 lines · On Fri, 18 Jun 2021 13:26:47 +0200 Igor Mammedov <[email protected]> wrote:

On Thu, 17 Jun 2021 18:49:17 +0200
Claudio Fontana <[email protected]> wrote:

[...]

Quoted 3 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

it didn't work, built fine but still tries to execute KVM test. Fixed v3 is on the way

Quoted 53 lines
Quoted 51 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#16 lore raw
Quoted 6 lines · Igor Mammedov
Quoted 2 lines · On Fri, 18 Jun 2021 13:26:47 +0200 Igor Mammedov <[email protected]> wrote:

On Thu, 17 Jun 2021 18:49:17 +0200
Claudio Fontana <[email protected]> wrote:

Quoted 8 lines

[...]

Quoted 3 lines
Quoted 1 line

Does this series work with --disable-kvm builds? (TCG-only builds?)

I'll test. But on the first glance it should work without issues. (i.e. kvm only tests will be skipped).

it didn't work, built fine but still tries to execute KVM test. Fixed v3 is on the way

I am thinking, what about doing the checks we need to do at the beginning of the tests, and cache the results for all the tests, instead of checking every time?

This way we could use a more general implementation qtest_has_accel("kvm"), and mitigate its cost?

Thanks

C

Quoted 56 lines
Quoted 54 lines
Quoted 52 lines

Thanks,

CLaudio

Quoted 43 lines

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type. But it's more complex and what I dislike about it most, it requires execution of 'probing' QEMU instance to find execute 'query-accels' QMP command, which is rather resource consuming. So I'd use query-accels approach only when it's the only possible option to minimize load on CI systems.

Igor Mammedov (2):
tests: acpi: q35: test for x2APIC entries in SRAT
tests: acpi: update expected tables blobs

root (1):
tests: qtest: add qtest_has_kvm() to check if tested bynary supports
KVM

 tests/qtest/libqos/libqtest.h    |   7 +++++++
 meson.build                      |   1 +
 tests/data/acpi/q35/APIC.numamem | Bin 0 -> 2686 bytes
 tests/data/acpi/q35/DSDT.numamem | Bin 7865 -> 35222 bytes
 tests/data/acpi/q35/FACP.numamem | Bin 0 -> 244 bytes
 tests/data/acpi/q35/SRAT.numamem | Bin 224 -> 5080 bytes
 tests/qtest/bios-tables-test.c   |  10 +++++++---
 tests/qtest/libqtest.c           |  20 ++++++++++++++++++++
 8 files changed, 35 insertions(+), 3 deletions(-)
 create mode 100644 tests/data/acpi/q35/APIC.numamem
 create mode 100644 tests/data/acpi/q35/FACP.numamem
#17 lore raw
In reply to Igor Mammedov
Quoted 12 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

For an example:
test q35 machine with intel_iommu
This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Quoted 3 lines

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

You also might want to update the check in tests/qtest/migration-test.c while you're at it.

Quoted 5 lines

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/
which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type.

Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...

Thanks,
Thomas

#18 lore raw
Quoted 24 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

For an example:
  test q35 machine with intel_iommu
  This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Quoted 3 lines

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

You also might want to update the check in tests/qtest/migration-test.c while you're at it.

Quoted 4 lines

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

Quoted 5 lines
Quoted 2 lines

which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type.

Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...

I'm certainly fine if Igor wants to restart this effort :)

Maybe doing as Claudio suggested, start have qtest_has_accel() tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?

#19 lore raw
Quoted 28 lines · Philippe Mathieu-Daudé
Quoted 27 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

For an example:
  test q35 machine with intel_iommu
  This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Quoted 3 lines

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

You also might want to update the check in tests/qtest/migration-test.c while you're at it.

Quoted 6 lines

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type.

Quoted 8 lines
Quoted 2 lines

Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...

I'm certainly fine if Igor wants to restart this effort :)

Maybe doing as Claudio suggested, start have qtest_has_accel() tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?

Yeah, that's maybe a good idea ...
But I'm currently wondering whether we need query-accels at all? For detecting kvm, we already have "query-kvm" ... and we don't have tests for any other accelerators yet (hax, hvf, etc.) ... so it's likely just about having a way to detect whether TCG is compiled into the binary? Is there already another command that could be used to check for the availability of TCG?

Thomas

#20 lore raw
Quoted 15 lines · On Tue, 22 Jun 2021 09:59:48 +0200 Thomas Huth <[email protected]> wrote:
Quoted 14 lines · Philippe Mathieu-Daudé
Quoted 13 lines · Thomas Huth
Quoted 12 lines · Igor Mammedov

Sometimes it's necessary to execute a test that depends on KVM, however qtest is not aware if tested QEMU binary supports KVM on the host it the test is executed.

For an example:
  test q35 machine with intel_iommu
  This test will run only is KVM is available and fail to start QEMU if it fallsback to TCG, thus failing whole test. So if test is executed in VM where nested KVM is not enabled or on other than x86 host, it will break 'make check-qtest'

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Series adds a lightweight qtest_has_kvm() check, which abuses build system and should help to avoid running KVM only tests on hosts that do not support it.

You also might want to update the check in tests/qtest/migration-test.c while you're at it.

sure (this series was just a discussion starter).

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

PS:
there is an alternative 'query-accels' QMP command proposal
https://patchwork.kernel.org/project/qemu-devel/patch/[email protected]/

which I think is more robust compared to qtest_has_kvm() and could be extended to take into account machine type.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Could you please get in touch with Philippe directly and discuss which way to go? We certainly don't need two approaches in the end...

I'm certainly fine if Igor wants to restart this effort :)

Maybe doing as Claudio suggested, start have qtest_has_accel() tests accel with Igor's shortpath first, then fallback to 'query-accels' QMP command?

it should save time when shortpath result is positive, but won't help on fallback path.

Maybe we can add a build option to enable fallback, or even better to swap build time impl. with runtime impl., so ones who need qtest as external harness can build and use it on request.

Quoted 5 lines

Yeah, that's maybe a good idea ...
But I'm currently wondering whether we need query-accels at all? For detecting kvm, we already have "query-kvm" ... and we don't have tests for any other accelerators yet (hax, hvf, etc.) ... so it's likely just about having a way to detect whether TCG is compiled into the binary?

if we ignore other accelerators, it's probably not hard to add detection of TCG at build time, I'll look into it and also try to change API to qtest_has_accel() instead of kvm specific one. This way whatever solution we end up with, won't affect code that uses it.

Quoted 5 lines

Is there
already another command that could be used to check for the availability of TCG?

Thomas