Skip to contentSkip to conversation

qtests/libqos: Allow PCI tests to be run with virt-machine

PATCH Superseded Eric Auger 3 participants 8 messages Mar 7, 2022 – Mar 8, 2022 Arm/Amlogic Meson SoC support

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

v4 cover qtests/libqos: Allow PCI tests to be run with virt-machine

Up to now the virt-machine node only contains a virtio-mmio driver node but no driver that eventually produces any pci-bus interface.

Hence, PCI libqos tests cannot be run with aarch64 binary.

This series brings the pieces needed to be able to run PCI tests with the aarch64 binary: a generic-pcihost driver node gets instantiated by the machine. This later contains a pci-bus-generic driver which produces a pci-bus interface. Then all tests consuming the pci-bus interface can be run with the libqos arm virt machine.

One of the first goal was to be able to run the virtio-iommu-pci tests as the virtio-iommu was initially targetting ARM and it was awkard to be run the test with the pc machine. This is now possible.

Only the tests doing hotplug cannot be run yet as hotplug is not possible on the root bus. This will be dealt with separately by adding a root port to the object tree.

Also I have some trouble with 2 of the vhost-user-blk-tests. I am obliged to hack them in "tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing on aarch64". Looks like a memory allocation issue, which at first sight seems unrelated to the aarch64 pci libqos enablement but we are never sure. Calling for help on this issue, if some vhost-user specialists can dedicate some cycles on this. Otherwise I will try my best to further debug.

To reproduce the issue, revert the above hack and run

QTEST_QEMU_STORAGE_DAEMON_BINARY=build/storage-daemon/qemu-storage-daemon QTEST_QEMU_BINARY=build/aarch64-softmmu/qemu-system-aarch64 build/tests/qtest/qos-test

you should get:

    ERROR:../tests/qtest/libqos/virtio.c:224:qvirtio_wait_used_elem:
    assertion failed (got_desc_idx == desc_idx): (50331648 == 0)
    Bail out! ERROR:../tests/qtest/libqos/virtio.c:224: qvirtio_wait_used_elem:
    assertion failed (got_desc_idx == desc_idx): (50331648 == 0)

Best Regards

Eric

This series can be found at:
https://github.com/eauger/qemu/tree/libqos-pci-arm-v4

History

v3 -> v4:
- handle endianess when accessing the cfg space (fix PPC64 BE failure). Tested on such machine.

v2 -> v3:
- force -cpu=max along with aarch64/virt
- reduced the vhost-user-block-pci issue workaround to a single guest_alloc() instead of enabling MSIs. Call for help on this specific issue. The 2 tests which fail are:
test_basic and indirect.

v1 -> v2:
- copyright updated to 2022
- QPCIBusARM renamed into QGenericPCIBus
- QGenericPCIHost declarations and definitions moved in the same place as the generic pci implementation
- rename pci-arm.c/h in generic-pcihost.c/h and remove any ref to ARM there
- remove qos_node_produces_opts, qpci_new_arm, qpci_free_arm
- ecam_alloc_ptr now is a field of QGenericPCIBus and not QPCIBus
- new libqos_init to create generic-pcihost driver that contains pci-bus-generic
- QGenericPCIHost moved in the same place as the generic pci bindings
- collected Thomas A-b/R-b

Eric Auger (4):
tests/qtest/libqos/pci: Introduce pio_limit tests/qtest/libqos: Skip hotplug tests if pci root bus is not
hotpluggable
tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing
on aarch64
tests/qtest/libqos: Add generic pci host bridge in arm-virt machine

 tests/qtest/e1000e-test.c             |   6 +
 tests/qtest/libqos/arm-virt-machine.c |  19 ++-
 tests/qtest/libqos/generic-pcihost.c  | 231 ++++++++++++++++++++++++++
 tests/qtest/libqos/generic-pcihost.h  |  54 ++++++
 tests/qtest/libqos/meson.build        |   1 +
 tests/qtest/libqos/pci-pc.c           |   1 +
 tests/qtest/libqos/pci-spapr.c        |   1 +
 tests/qtest/libqos/pci.c              |  78 +++++----
 tests/qtest/libqos/pci.h              |   6 +-
 tests/qtest/vhost-user-blk-test.c     |  16 ++
 tests/qtest/virtio-blk-test.c         |   5 +
 tests/qtest/virtio-net-test.c         |   5 +
 tests/qtest/virtio-rng-test.c         |   5 +
 13 files changed, 393 insertions(+), 35 deletions(-)
 create mode 100644 tests/qtest/libqos/generic-pcihost.c
 create mode 100644 tests/qtest/libqos/generic-pcihost.h
Signature · 1 line
2.26.3
author
#1 lore raw

PATCH v4 1/4 tests/qtest/libqos/pci: Introduce pio_limit

At the moment the IO space limit is hardcoded to QPCI_PIO_LIMIT = 0x10000. When accesses are performed to a bar, the base address of this latter is compared against the limit to decide whether we perform an IO or a memory access.

On ARM, we cannot keep this PIO limit as the arm-virt machine uses [0x3eff0000, 0x3f000000 ] for the IO space map and we are mandated to allocate at 0x0.

Add a new flag in QPCIBar indicating whether it is an IO bar or a memory bar. This flag is set on QPCIBar allocation and provisionned based on the BAR configuration. Then the new flag is used in access functions and in iomap() function.

---
 tests/qtest/libqos/pci-pc.c    |  1 +
 tests/qtest/libqos/pci-spapr.c |  1 +
 tests/qtest/libqos/pci.c       | 78 ++++++++++++++++++++++------------
 tests/qtest/libqos/pci.h       |  5 +--
 4 files changed, 54 insertions(+), 31 deletions(-)
tests/qtest/libqos/pci-pc.c +1 -0
index f97844289f..8051a0881a 100644
@@ -150,6 +150,7 @@ void qpci_init_pc(QPCIBusPC *qpci, QTestState *qts, QGuestAllocator *alloc)
 
     qpci->bus.qts = qts;
     qpci->bus.pio_alloc_ptr = 0xc000;
+    qpci->bus.pio_limit = 0x10000;
     qpci->bus.mmio_alloc_ptr = 0xE0000000;
     qpci->bus.mmio_limit = 0x100000000ULL;
 
tests/qtest/libqos/pci-spapr.c +1 -0
index 262226985f..870ffdd8b5 100644
@@ -197,6 +197,7 @@ void qpci_init_spapr(QPCIBusSPAPR *qpci, QTestState *qts,
 
     qpci->bus.qts = qts;
     qpci->bus.pio_alloc_ptr = 0xc000;
+    qpci->bus.pio_limit = 0x10000;
     qpci->bus.mmio_alloc_ptr = qpci->mmio32.pci_base;
     qpci->bus.mmio_limit = qpci->mmio32.pci_base + qpci->mmio32.size;
 
tests/qtest/libqos/pci.c +50 -28

173 lines not on this page · raw message

tests/qtest/libqos/pci.h +2 -3
index becb800f9e..44f6806fe4 100644
@@ -16,8 +16,6 @@
 #include "libqtest.h"
 #include "qgraph.h"
 
-#define QPCI_PIO_LIMIT    0x10000
-
 #define QPCI_DEVFN(dev, fn) (((dev) << 3) | (fn))
 
 typedef struct QPCIDevice QPCIDevice;
@@ -51,7 +49,7 @@ struct QPCIBus {
                           uint8_t offset, uint32_t value);
 
     QTestState *qts;
-    uint16_t pio_alloc_ptr;
+    uint64_t pio_alloc_ptr, pio_limit;
     uint64_t mmio_alloc_ptr, mmio_limit;
     bool has_buggy_msi; /* TRUE for spapr, FALSE for pci */
 
@@ -59,6 +57,7 @@ struct QPCIBus {
 
 struct QPCIBar {
     uint64_t addr;
+    bool is_io;
 };
 
 struct QPCIDevice
Signature · 1 line
2.26.3
author
#2 lore raw

PATCH v4 2/4 tests/qtest/libqos: Skip hotplug tests if pci root bus is not hotpluggable

In reply to Eric Auger

ARM does not not support hotplug on pcie.0. Add a flag on the bus which tells if devices can be hotplugged and skip hotplug tests if the bus cannot be hotplugged. This is a temporary solution to enable the other pci tests on aarch64.

---

v1 ->v2:
- reword g_test_skip msg into "pci bus does not support hotplug"

---
 tests/qtest/e1000e-test.c         |  6 ++++++
 tests/qtest/libqos/pci.h          |  1 +
 tests/qtest/vhost-user-blk-test.c | 10 ++++++++++
 tests/qtest/virtio-blk-test.c     |  5 +++++
 tests/qtest/virtio-net-test.c     |  5 +++++
 tests/qtest/virtio-rng-test.c     |  5 +++++
 6 files changed, 32 insertions(+)
tests/qtest/e1000e-test.c +6 -0
index 0273fe4c15..48f3dbb0fd 100644
@@ -235,6 +235,12 @@ static void test_e1000e_multiple_transfers(void *obj, void *data,
 static void test_e1000e_hotplug(void *obj, void *data, QGuestAllocator * alloc)
 {
     QTestState *qts = global_qtest;  /* TODO: get rid of global_qtest here */
+    QE1000E_PCI *dev = obj;
+
+    if (dev->pci_dev.bus->not_hotpluggable) {
+        g_test_skip("pci bus does not support hotplug");
+        return;
+    }
 
     qtest_qmp_device_add(qts, "e1000e", "e1000e_net", "{'addr': '0x06'}");
     qpci_unplug_acpi_device_test(qts, "e1000e_net", 0x06);
tests/qtest/libqos/pci.h +1 -0
index 44f6806fe4..6a28b40522 100644
@@ -52,6 +52,7 @@ struct QPCIBus {
     uint64_t pio_alloc_ptr, pio_limit;
     uint64_t mmio_alloc_ptr, mmio_limit;
     bool has_buggy_msi; /* TRUE for spapr, FALSE for pci */
+    bool not_hotpluggable; /* TRUE if devices cannot be hotplugged */
 
 };
 
tests/qtest/vhost-user-blk-test.c +10 -0
index 62e670f39b..1316aae0fa 100644
@@ -676,6 +676,11 @@ static void pci_hotplug(void *obj, void *data, QGuestAllocator *t_alloc)
     QVirtioPCIDevice *dev;
     QTestState *qts = dev1->pdev->bus->qts;
 
+    if (dev1->pdev->bus->not_hotpluggable) {
+        g_test_skip("pci bus does not support hotplug");
+        return;
+    }
+
     /* plug secondary disk */
     qtest_qmp_device_add(qts, "vhost-user-blk-pci", "drv1",
                          "{'addr': %s, 'chardev': 'char2'}",
@@ -703,6 +708,11 @@ static void multiqueue(void *obj, void *data, QGuestAllocator *t_alloc)
     uint64_t features;
     uint16_t num_queues;
 
+    if (pdev1->pdev->bus->not_hotpluggable) {
+        g_test_skip("bus pci.0 does not support hotplug");
+        return;
+    }
+
     /*
      * The primary device has 1 queue and VIRTIO_BLK_F_MQ is not enabled. The
      * VIRTIO specification allows VIRTIO_BLK_F_MQ to be enabled when there is
tests/qtest/virtio-blk-test.c +5 -0
index 2a23698211..acb44c9fb8 100644
@@ -701,6 +701,11 @@ static void pci_hotplug(void *obj, void *data, QGuestAllocator *t_alloc)
     QVirtioPCIDevice *dev;
     QTestState *qts = dev1->pdev->bus->qts;
 
+    if (dev1->pdev->bus->not_hotpluggable) {
+        g_test_skip("pci bus does not support hotplug");
+        return;
+    }
+
     /* plug secondary disk */
     qtest_qmp_device_add(qts, "virtio-blk-pci", "drv1",
                          "{'addr': %s, 'drive': 'drive1'}",
tests/qtest/virtio-net-test.c +5 -0
index 8bf74e516c..af3027144f 100644
@@ -174,6 +174,11 @@ static void hotplug(void *obj, void *data, QGuestAllocator *t_alloc)
     QTestState *qts = dev->pdev->bus->qts;
     const char *arch = qtest_get_arch();
 
+    if (dev->pdev->bus->not_hotpluggable) {
+        g_test_skip("pci bus does not support hotplug");
+        return;
+    }
+
     qtest_qmp_device_add(qts, "virtio-net-pci", "net1",
                          "{'addr': %s}", stringify(PCI_SLOT_HP));
 
tests/qtest/virtio-rng-test.c +5 -0
index e6b8cd8e0c..5ce444ad72 100644
@@ -20,6 +20,11 @@ static void rng_hotplug(void *obj, void *data, QGuestAllocator *alloc)
     QVirtioPCIDevice *dev = obj;
     QTestState *qts = dev->pdev->bus->qts;
 
+   if (dev->pdev->bus->not_hotpluggable) {
+        g_test_skip("pci bus does not support hotplug");
+        return;
+    }
+
     const char *arch = qtest_get_arch();
 
     qtest_qmp_device_add(qts, "virtio-rng-pci", "rng1",
Signature · 1 line
2.26.3
author
#3 lore raw

PATCH v4 3/4 tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing on aarch64

In reply to Eric Auger

When run on ARM, basic and indirect tests currently fail with the following error:

ERROR:../tests/qtest/libqos/virtio.c:224:qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0) Bail out! ERROR:../tests/qtest/libqos/virtio.c:224: qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0)

I noticed it worked when I set up MSI and I further reduced the code to a simple guest_alloc() that removes the error. At the moment I am not able to identify where ths issue is and this blocks the whole pci/aarch64 enablement.

---
 tests/qtest/vhost-user-blk-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
tests/qtest/vhost-user-blk-test.c +6 -0
index 1316aae0fa..cf8057fb57 100644
@@ -225,6 +225,9 @@ static QVirtQueue *test_basic(QVirtioDevice *dev, QGuestAllocator *alloc)
     QTestState *qts = global_qtest;
     QVirtQueue *vq;
 
+    /* temporary hack to let the test pass on aarch64 */
+    guest_alloc(alloc, 4);
+
     features = qvirtio_get_features(dev);
     features = features & ~(QVIRTIO_F_BAD_FEATURE |
                     (1u << VIRTIO_RING_F_INDIRECT_DESC) |
@@ -469,6 +472,9 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)
     char *data;
     QTestState *qts = global_qtest;
 
+    /* temporary hack to let the test pass on aarch64 */
+    guest_alloc(t_alloc, 4);
+
     features = qvirtio_get_features(dev);
     g_assert_cmphex(features & (1u << VIRTIO_RING_F_INDIRECT_DESC), !=, 0);
     features = features & ~(QVIRTIO_F_BAD_FEATURE |
Signature · 1 line
2.26.3
Acked-by on v4 3/4
#4 lore raw

Re: [PATCH v4 3/4] tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing on aarch64

Quoted 13 lines · Eric Auger

When run on ARM, basic and indirect tests currently fail with the following error:

ERROR:../tests/qtest/libqos/virtio.c:224:qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0) Bail out! ERROR:../tests/qtest/libqos/virtio.c:224: qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0)

I noticed it worked when I set up MSI and I further reduced the code to a simple guest_alloc() that removes the error. At the moment I am not able to identify where ths issue is and this blocks the whole pci/aarch64 enablement.

Quoted 4 lines
---
 tests/qtest/vhost-user-blk-test.c | 6 ++++++
 1 file changed, 6 insertions(+)

What is the plan to identify the root cause?

author
#5 lore raw

Re: [PATCH v4 3/4] tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing on aarch64

Hi Stefan,

Quoted 14 lines · Stefan Hajnoczi
Quoted 13 lines · Eric Auger

When run on ARM, basic and indirect tests currently fail with the following error:

ERROR:../tests/qtest/libqos/virtio.c:224:qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0) Bail out! ERROR:../tests/qtest/libqos/virtio.c:224: qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0)

I noticed it worked when I set up MSI and I further reduced the code to a simple guest_alloc() that removes the error. At the moment I am not able to identify where ths issue is and this blocks the whole pci/aarch64 enablement.

Quoted 5 lines
Quoted 4 lines
---
 tests/qtest/vhost-user-blk-test.c | 6 ++++++
 1 file changed, 6 insertions(+)

What is the plan to identify the root cause?

To be honest, I have no precise plan yet. However I commit to work on it within the next weeks.

If maintainers prefer to delay the introduction of qtest libqos pci/aarch64 tests until this gets fixed, I would understand though.

Eric

Quoted 2 lines
#6 lore raw

Re: [PATCH v4 3/4] tests/qtest/vhost-user-blk-test: Temporary hack to get tests passing on aarch64

Quoted 21 lines · Eric Auger

Hi Stefan,

Quoted 18 lines · Stefan Hajnoczi
Quoted 17 lines · Eric Auger

When run on ARM, basic and indirect tests currently fail with the following error:

ERROR:../tests/qtest/libqos/virtio.c:224:qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0) Bail out! ERROR:../tests/qtest/libqos/virtio.c:224: qvirtio_wait_used_elem:
assertion failed (got_desc_idx == desc_idx): (50331648 == 0)

I noticed it worked when I set up MSI and I further reduced the code to a simple guest_alloc() that removes the error. At the moment I am not able to identify where ths issue is and this blocks the whole pci/aarch64 enablement.

---
 tests/qtest/vhost-user-blk-test.c | 6 ++++++
 1 file changed, 6 insertions(+)
Quoted 8 lines
Quoted 1 line

What is the plan to identify the root cause?

To be honest, I have no precise plan yet. However I commit to work on it within the next weeks.

If maintainers prefer to delay the introduction of qtest libqos pci/aarch64 tests until this gets fixed, I would understand though.

Eric

Given soft freeze sounds like a good plan.

Quoted 2 lines
Quoted 2 lines
author
#7 lore raw

PATCH v4 4/4 tests/qtest/libqos: Add generic pci host bridge in arm-virt machine

In reply to Eric Auger

Up to now the virt-machine node contains a virtio-mmio node. However no driver produces any PCI interface node. Hence, PCI tests cannot be run with aarch64 binary.

Add a GPEX driver node that produces a pci interface node. This latter then can be consumed by all the pci tests. One of the first motivation was to be able to run the virtio-iommu-pci tests.

We still face an issue with pci hotplug tests as hotplug cannot happen on the pcie root bus and require a generic root port. This will be addressed later on.

We force cpu=max along with aarch64/virt machine as some PCI tests require high MMIO regions to be available.

---
v3 -> v4:
- properly handle le/cpu conversions on config access

v2 -> v3:
- force cpu=max with aarch64/virt

v1 -> v2:
- copyright updated to 2022
- QPCIBusARM renamed into QGenericPCIBus
- QGenericPCIHost declarations and definitions moved in the same place as the generic pci implementation
- rename pci-arm.c/h in generic-pcihost.c/h and remove any ref to ARM there
- remove qos_node_produces_opts, qpci_new_arm, qpci_free_arm
- ecam_alloc_ptr now is a field of QGenericPCIBus and not QPCIBus
- new libqos_init to create generic-pcihost driver that contains pci-bus-generic
- QGenericPCIHost moved in the same place as the generic pci bindings

---
 tests/qtest/libqos/arm-virt-machine.c |  19 ++-
 tests/qtest/libqos/generic-pcihost.c  | 231 ++++++++++++++++++++++++++
 tests/qtest/libqos/generic-pcihost.h  |  54 ++++++
 tests/qtest/libqos/meson.build        |   1 +
 4 files changed, 301 insertions(+), 4 deletions(-)
 create mode 100644 tests/qtest/libqos/generic-pcihost.c
 create mode 100644 tests/qtest/libqos/generic-pcihost.h
tests/qtest/libqos/arm-virt-machine.c +15 -4
index e0f5932284..96da0dde54 100644
@@ -22,6 +22,8 @@
 #include "malloc.h"
 #include "qgraph.h"
 #include "virtio-mmio.h"
+#include "generic-pcihost.h"
+#include "hw/pci/pci_regs.h"
 
 #define ARM_PAGE_SIZE               4096
 #define VIRTIO_MMIO_BASE_ADDR       0x0A003E00
@@ -35,6 +37,7 @@ struct QVirtMachine {
     QOSGraphObject obj;
     QGuestAllocator alloc;
     QVirtioMMIODevice virtio_mmio;
+    QGenericPCIHost bridge;
 };
 
 static void virt_destructor(QOSGraphObject *obj)
@@ -57,11 +60,13 @@ static void *virt_get_driver(void *object, const char *interface)
 static QOSGraphObject *virt_get_device(void *obj, const char *device)
 {
     QVirtMachine *machine = obj;
-    if (!g_strcmp0(device, "virtio-mmio")) {
+    if (!g_strcmp0(device, "generic-pcihost")) {
+        return &machine->bridge.obj;
+    } else if (!g_strcmp0(device, "virtio-mmio")) {
         return &machine->virtio_mmio.obj;
     }
 
-    fprintf(stderr, "%s not present in arm/virtio\n", device);
+    fprintf(stderr, "%s not present in arm/virt\n", device);
     g_assert_not_reached();
 }
 
@@ -76,16 +81,22 @@ static void *qos_create_machine_arm_virt(QTestState *qts)
     qvirtio_mmio_init_device(&machine->virtio_mmio, qts, VIRTIO_MMIO_BASE_ADDR,
                              VIRTIO_MMIO_SIZE);
 
+    qos_create_generic_pcihost(&machine->bridge, qts, &machine->alloc);
+
     machine->obj.get_device = virt_get_device;
     machine->obj.get_driver = virt_get_driver;
     machine->obj.destructor = virt_destructor;
     return machine;
 }
 
-static void virtio_mmio_register_nodes(void)
+static void virt_machine_register_nodes(void)
 {
     qos_node_create_machine("arm/virt", qos_create_machine_arm_virt);
     qos_node_contains("arm/virt", "virtio-mmio", NULL);
+
+    qos_node_create_machine_args("aarch64/virt", qos_create_machine_arm_virt,
+                                 " -cpu max");
+    qos_node_contains("aarch64/virt", "generic-pcihost", NULL);
 }
 
-libqos_init(virtio_mmio_register_nodes);
+libqos_init(virt_machine_register_nodes);
tests/qtest/libqos/generic-pcihost.c new file +231 -0

231 lines not on this page · raw message

tests/qtest/libqos/generic-pcihost.h new file +54 -0

54 lines not on this page · raw message

tests/qtest/libqos/meson.build +1 -0
index e988d15791..8c8ee15553 100644
@@ -42,6 +42,7 @@ libqos_srcs = files('../libqtest.c',
         'virtio-scsi.c',
         'virtio-serial.c',
         'virtio-iommu.c',
+        'generic-pcihost.c',
 
         # qgraph machines:
         'aarch64-xlnx-zcu102-machine.c',
Signature · 1 line
2.26.3