libqos: add VIRTIO PCI 1.0 support
v4 cover libqos: add VIRTIO PCI 1.0 support
v4:
* Introduce bool d->features_negotiated so that tests can negotiate a 0 feature bit set in Legacy mode [Thomas]
* Make the FEATURES_OK code change in qvirtio_set_driver_ok() clearer and mention it in the commit description [Thomas]
* Fix indentation in qvring_init() [Thomas]
v3:
* Now implements VIRTIO 1.0 fully (vring, device initialization). This required several new patches to address the following issues:
1. Tests that do not negotiate features (non-compliant!)
2. Tests that access configuration space before feature negotiation (non-compliant!)
3. Tests that set up virtqueues before feature negotiation (non-compliant!)
4. vring accesses require byte-swapping when VIRTIO 1.0 is used with a big-endian guest because the qtest_readX/writeX() API automatically converts to guest-endian
5. VIRTIO 1.0 has an additional FEATURES_OK step during Device Initialization
* Change uint8_t bar_idx to int [Thomas]
* Document qpci_find_capability() [Thomas]
* Every commit tested with arm, ppc64, and x86_64 targets using:
git rebase -i origin/master -x 'make tests/qos-test && QTEST_QEMU_BINARY=ppc64-softmmu/qemu-system-ppc64 tests/qos-test && QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 tests/qos-test' QTEST_QEMU_BINARY=arm-softmmu/qemu-system-arm tests/qos-test'
v2:
* Fix checkpatch.pl issues, except MAINTAINERS file warning. libqos already has maintainers and the new virtio-pci-modern.[ch] files don't need extra entries since they are already covered by the existing libqos/ entry.
New VIRTIO devices are Non-Transitional. This means they only expose the VIRTIO 1.0 interface, not the Legacy interface.
The libqos only supports Legacy and Transitional devices (in Legacy mode). This patch series adds VIRTIO 1.0 support so that tests can run against Non-Transitional devices too.
The virtio-fs device is Non-Transitional, so this is a prerequisite for virtio-fs qos tests.
Stefan Hajnoczi (16):
tests/virtio-blk-test: read config space after feature negotiation
libqos: read QVIRTIO_MMIO_VERSION register
libqos: extend feature bits to 64-bit
virtio-scsi-test: add missing feature negotiation tests/virtio-blk-test: set up virtqueue after feature negotiation
libqos: add missing virtio-9p feature negotiation
libqos: enforce Device Initialization order
libqos: implement VIRTIO 1.0 FEATURES_OK step
libqos: access VIRTIO 1.0 vring in little-endian
libqos: add iteration support to qpci_find_capability()
libqos: pass full QVirtQueue to set_queue_address()
libqos: add MSI-X callbacks to QVirtioPCIDevice
libqos: expose common virtqueue setup/cleanup functions
libqos: make the virtio-pci BAR index configurable
libqos: extract Legacy virtio-pci.c code
libqos: add VIRTIO PCI 1.0 support
tests/Makefile.include | 1 + tests/libqos/pci.h | 2 +- tests/libqos/virtio-mmio.h | 1 + tests/libqos/virtio-pci-modern.h | 17 ++ tests/libqos/virtio-pci.h | 34 ++- tests/libqos/virtio.h | 19 +- tests/libqos/pci.c | 30 ++- tests/libqos/virtio-9p.c | 6 + tests/libqos/virtio-mmio.c | 38 ++- tests/libqos/virtio-net.c | 6 +- tests/libqos/virtio-pci-modern.c | 443 +++++++++++++++++++++++++++++++ tests/libqos/virtio-pci.c | 105 +++++--- tests/libqos/virtio.c | 160 ++++++++--- tests/virtio-blk-test.c | 66 +++-- tests/virtio-scsi-test.c | 8 + 15 files changed, 802 insertions(+), 134 deletions(-) create mode 100644 tests/libqos/virtio-pci-modern.h create mode 100644 tests/libqos/virtio-pci-modern.c
Signature · 1 line
2.21.0
PATCH v4 1/16 tests/virtio-blk-test: read config space after feature negotiation
The VIRTIO Configuration Space cannot be accessed before device feature bits have been read because a driver doesn't know the endianness until it has checked VIRTIO_F_VERSION_1.
Fix this problem in preparation for VIRTIO 1.0 support.
--- tests/virtio-blk-test.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
tests/virtio-blk-test.c +20 -13
@@ -125,10 +125,6 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc,char *data;QTestState *qts = global_qtest; - capacity =qvirtio_config_readq (dev,0 ); - - g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); - features =qvirtio_get_features (dev); features = features & ~(QVIRTIO_F_BAD_FEATURE | (1u << VIRTIO_RING_F_INDIRECT_DESC) |
@@ -136,6 +132,9 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc, (1u << VIRTIO_BLK_F_SCSI));qvirtio_set_features (dev, features); + capacity =qvirtio_config_readq (dev,0 ); + g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); +qvirtio_set_driver_ok (dev);/* Write and read with 3 descriptor layout */
@@ -359,9 +358,6 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)char *data;QTestState *qts = global_qtest; - capacity =qvirtio_config_readq (dev,0 ); - g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); - features =qvirtio_get_features (dev); g_assert_cmphex(features & (1u << VIRTIO_RING_F_INDIRECT_DESC), !=, 0); features = features & ~(QVIRTIO_F_BAD_FEATURE |
@@ -369,6 +365,9 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc) (1u << VIRTIO_BLK_F_SCSI));qvirtio_set_features (dev, features); + capacity =qvirtio_config_readq (dev,0 ); + g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); + vq =qvirtqueue_setup (dev, t_alloc,0 );qvirtio_set_driver_ok (dev);
@@ -434,8 +433,16 @@ static void config(void *obj, void *data, QGuestAllocator *t_alloc)QVirtioBlk *blk_if = obj;QVirtioDevice *dev = blk_if->vdev;int n_size =TEST_IMAGE_SIZE /2 ; +uint64_t features;uint64_t capacity; + features =qvirtio_get_features (dev); + features = features & ~(QVIRTIO_F_BAD_FEATURE | + (1u <<VIRTIO_RING_F_INDIRECT_DESC ) | + (1u <<VIRTIO_RING_F_EVENT_IDX ) | + (1u <<VIRTIO_BLK_F_SCSI )); +qvirtio_set_features (dev, features); + capacity =qvirtio_config_readq (dev,0 ); g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512);
@@ -475,9 +482,6 @@ static void msix(void *obj, void *u_data, QGuestAllocator *t_alloc)qpci_msix_enable (pdev->pdev);qvirtio_pci_set_msix_configuration_vector (pdev, t_alloc,0 ); - capacity =qvirtio_config_readq (dev,0 ); - g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); - features =qvirtio_get_features (dev); features = features & ~(QVIRTIO_F_BAD_FEATURE | (1u << VIRTIO_RING_F_INDIRECT_DESC) |
@@ -485,6 +489,9 @@ static void msix(void *obj, void *u_data, QGuestAllocator *t_alloc) (1u << VIRTIO_BLK_F_SCSI));qvirtio_set_features (dev, features); + capacity =qvirtio_config_readq (dev,0 ); + g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); + vq =qvirtqueue_setup (dev, t_alloc,0 );qvirtqueue_pci_msix_setup (pdev, (QVirtQueuePCI *)vq, t_alloc,1 );
@@ -584,9 +591,6 @@ static void idx(void *obj, void *u_data, QGuestAllocator *t_alloc)qpci_msix_enable (pdev->pdev);qvirtio_pci_set_msix_configuration_vector (pdev, t_alloc,0 ); - capacity =qvirtio_config_readq (dev,0 ); - g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); - features =qvirtio_get_features (dev); features = features & ~(QVIRTIO_F_BAD_FEATURE | (1u << VIRTIO_RING_F_INDIRECT_DESC) |
@@ -594,6 +598,9 @@ static void idx(void *obj, void *u_data, QGuestAllocator *t_alloc) (1u << VIRTIO_BLK_F_SCSI));qvirtio_set_features (dev, features); + capacity =qvirtio_config_readq (dev,0 ); + g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); + vq =qvirtqueue_setup (dev, t_alloc,0 );qvirtqueue_pci_msix_setup (pdev, (QVirtQueuePCI *)vq, t_alloc,1 );
Signature · 1 line
2.21.0
PATCH v4 2/16 libqos: read QVIRTIO_MMIO_VERSION register
In reply to Stefan HajnocziThere was no real virtio-mmio ABI change between Legacy and VIRTIO 1.0 except that the Version field was incremented from 1 to 2.
However, QEMU does not allow Legacy drivers to perform VIRTIO 1.0 operations like accessing 64-bit feature bits. Since we will introduce 64-bit feature bit support we need a way to differentiate between virtio-mmio Version 1 and 2 to avoid upsetting QEMU when we operate in Legacy mode.
Stash away the Version field so later patches can change behavior depending on the version.
--- tests/libqos/virtio-mmio.h | 1 + tests/libqos/virtio-mmio.c | 3 +++ 2 files changed, 4 insertions(+)
tests/libqos/virtio-mmio.h +1 -0
@@ -40,6 +40,7 @@ typedef struct QVirtioMMIODevice {uint64_t addr;uint32_t page_size;uint32_t features;/* As it cannot be read later, save it */ +uint32_t version; } QVirtioMMIODevice;extern const QVirtioBus qvirtio_mmio;
tests/libqos/virtio-mmio.c +3 -0
@@ -223,6 +223,9 @@ void qvirtio_mmio_init_device(QVirtioMMIODevice *dev, QTestState *qts, magic =qtest_readl (qts, addr +QVIRTIO_MMIO_MAGIC_VALUE );g_assert (magic == ('v' |'i' <<8 |'r' <<16 |'t' <<24 )); + dev->version =qtest_readl (qts, addr +QVIRTIO_MMIO_VERSION ); +g_assert (dev->version ==1 || dev->version ==2 ); + dev->qts = qts; dev->addr = addr; dev->page_size = page_size;
Signature · 1 line
2.21.0
PATCH v4 3/16 libqos: extend feature bits to 64-bit
In reply to Stefan HajnocziIn VIRTIO 1.0 feature bits changed from 32-bit to 64-bit. (In fact, the transports allow even more feature bits but nothing uses more than 64 bits today.)
Add 64-bit feature bit support to virtio-mmio and virtio-pci. This will be necessary for VIRTIO 1.0 support.
--- tests/libqos/virtio.h | 12 ++++++------ tests/libqos/virtio-mmio.c | 28 ++++++++++++++++++++++------ tests/libqos/virtio-net.c | 6 +++--- tests/libqos/virtio-pci.c | 12 ++++++------ tests/libqos/virtio.c | 4 ++-- tests/virtio-blk-test.c | 8 ++++---- 6 files changed, 43 insertions(+), 27 deletions(-)
tests/libqos/virtio.h +6 -6
@@ -13,7 +13,7 @@#include "libqos/malloc.h" #include "standard-headers/linux/virtio_ring.h" -#define QVIRTIO_F_BAD_FEATURE 0x40000000 +#define QVIRTIO_F_BAD_FEATURE 0x40000000ulltypedef struct QVirtioBus QVirtioBus ;
@@ -52,13 +52,13 @@ struct QVirtioBus {uint64_t (*config_readq)(QVirtioDevice *d,uint64_t addr);/* Get features of the device */ -uint32_t (*get_features)(QVirtioDevice *d); +uint64_t (*get_features)(QVirtioDevice *d);/* Set features of the device */ -void (*set_features)(QVirtioDevice *d,uint32_t features); +void (*set_features)(QVirtioDevice *d,uint64_t features);/* Get features of the guest */ -uint32_t (*get_guest_features)(QVirtioDevice *d); +uint64_t (*get_guest_features)(QVirtioDevice *d);/* Get status of the device */ uint8_t (*get_status)(QVirtioDevice *d);
@@ -103,8 +103,8 @@ uint8_t qvirtio_config_readb(QVirtioDevice *d, uint64_t addr);uint16_t qvirtio_config_readw (QVirtioDevice *d,uint64_t addr);uint32_t qvirtio_config_readl (QVirtioDevice *d,uint64_t addr);uint64_t qvirtio_config_readq (QVirtioDevice *d,uint64_t addr); -uint32_t qvirtio_get_features (QVirtioDevice *d); -void qvirtio_set_features (QVirtioDevice *d,uint32_t features); +uint64_t qvirtio_get_features (QVirtioDevice *d); +void qvirtio_set_features (QVirtioDevice *d,uint64_t features);bool qvirtio_is_big_endian (QVirtioDevice *d);void qvirtio_reset (QVirtioDevice *d);
tests/libqos/virtio-mmio.c +22 -6
@@ -40,22 +40,38 @@ static uint64_t qvirtio_mmio_config_readq(QVirtioDevice *d, uint64_t off)return qtest_readq (dev->qts, dev->addr +QVIRTIO_MMIO_DEVICE_SPECIFIC + off); } -static uint32_t qvirtio_mmio_get_features (QVirtioDevice *d) +static uint64_t qvirtio_mmio_get_features (QVirtioDevice *d) {QVirtioMMIODevice *dev =container_of (d, QVirtioMMIODevice, vdev); +uint64_t lo; +uint64_t hi =0 ; +qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_HOST_FEATURES_SEL ,0 ); -return qtest_readl (dev->qts, dev->addr +QVIRTIO_MMIO_HOST_FEATURES ); + lo =qtest_readl (dev->qts, dev->addr +QVIRTIO_MMIO_HOST_FEATURES ); + +if (dev->version >=2 ) { +qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_HOST_FEATURES_SEL ,1 ); + hi =qtest_readl (dev->qts, dev->addr +QVIRTIO_MMIO_HOST_FEATURES ); + } + +return (hi <<32 ) | lo; } -static void qvirtio_mmio_set_features (QVirtioDevice *d,uint32_t features) +static void qvirtio_mmio_set_features (QVirtioDevice *d,uint64_t features) {QVirtioMMIODevice *dev =container_of (d, QVirtioMMIODevice, vdev); dev->features = features;qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_GUEST_FEATURES_SEL ,0 );qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_GUEST_FEATURES , features); + +if (dev->version >=2 ) { +qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_GUEST_FEATURES_SEL ,1 ); +qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_GUEST_FEATURES , + features >>32 ); + } } -static uint32_t qvirtio_mmio_get_guest_features (QVirtioDevice *d) +static uint64_t qvirtio_mmio_get_guest_features (QVirtioDevice *d) {QVirtioMMIODevice *dev =container_of (d, QVirtioMMIODevice, vdev);return dev->features;
@@ -149,8 +165,8 @@ static QVirtQueue *qvirtio_mmio_virtqueue_setup(QVirtioDevice *d, vq->free_head =0 ; vq->num_free = vq->size; vq->align = dev->page_size; - vq->indirect = (dev->features & (1u <<VIRTIO_RING_F_INDIRECT_DESC )) !=0 ; - vq->event = (dev->features & (1u <<VIRTIO_RING_F_EVENT_IDX )) !=0 ; + vq->indirect = dev->features & (1ull <<VIRTIO_RING_F_INDIRECT_DESC ); + vq->event = dev->features & (1ull <<VIRTIO_RING_F_EVENT_IDX );qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_QUEUE_NUM , vq->size);
tests/libqos/virtio-net.c +3 -3
@@ -44,11 +44,11 @@ static void virtio_net_setup(QVirtioNet *interface) features =qvirtio_get_features (vdev); features &= ~(QVIRTIO_F_BAD_FEATURE | - (1u <<VIRTIO_RING_F_INDIRECT_DESC ) | - (1u <<VIRTIO_RING_F_EVENT_IDX )); + (1ull <<VIRTIO_RING_F_INDIRECT_DESC ) | + (1ull <<VIRTIO_RING_F_EVENT_IDX ));qvirtio_set_features (vdev, features); -if (features & (1u <<VIRTIO_NET_F_MQ )) { +if (features & (1ull <<VIRTIO_NET_F_MQ )) { interface->n_queues =qvirtio_config_readw (vdev,8 ) *2 ; }else { interface->n_queues =2 ;
tests/libqos/virtio-pci.c +6 -6
@@ -96,19 +96,19 @@ static uint64_t qvirtio_pci_config_readq(QVirtioDevice *d, uint64_t off)return val; } -static uint32_t qvirtio_pci_get_features (QVirtioDevice *d) +static uint64_t qvirtio_pci_get_features (QVirtioDevice *d) {QVirtioPCIDevice *dev =container_of (d, QVirtioPCIDevice, vdev);return qpci_io_readl (dev->pdev, dev->bar,VIRTIO_PCI_HOST_FEATURES ); } -static void qvirtio_pci_set_features (QVirtioDevice *d,uint32_t features) +static void qvirtio_pci_set_features (QVirtioDevice *d,uint64_t features) {QVirtioPCIDevice *dev =container_of (d, QVirtioPCIDevice, vdev);qpci_io_writel (dev->pdev, dev->bar,VIRTIO_PCI_GUEST_FEATURES , features); } -static uint32_t qvirtio_pci_get_guest_features (QVirtioDevice *d) +static uint64_t qvirtio_pci_get_guest_features (QVirtioDevice *d) {QVirtioPCIDevice *dev =container_of (d, QVirtioPCIDevice, vdev);return qpci_io_readl (dev->pdev, dev->bar,VIRTIO_PCI_GUEST_FEATURES );
@@ -208,7 +208,7 @@ static void qvirtio_pci_set_queue_address(QVirtioDevice *d, uint32_t pfn)static QVirtQueue *qvirtio_pci_virtqueue_setup (QVirtioDevice *d,QGuestAllocator *alloc,uint16_t index) { -uint32_t feat; +uint64_t feat;uint64_t addr;QVirtQueuePCI *vqpci;QVirtioPCIDevice *qvpcidev =container_of (d, QVirtioPCIDevice, vdev);
@@ -222,8 +222,8 @@ static QVirtQueue *qvirtio_pci_virtqueue_setup(QVirtioDevice *d, vqpci->vq.free_head =0 ; vqpci->vq.num_free = vqpci->vq.size; vqpci->vq.align =VIRTIO_PCI_VRING_ALIGN ; - vqpci->vq.indirect = (feat & (1u <<VIRTIO_RING_F_INDIRECT_DESC )) !=0 ; - vqpci->vq.event = (feat & (1u <<VIRTIO_RING_F_EVENT_IDX )) !=0 ; + vqpci->vq.indirect = feat & (1ull <<VIRTIO_RING_F_INDIRECT_DESC ); + vqpci->vq.event = feat & (1ull <<VIRTIO_RING_F_EVENT_IDX ); vqpci->msix_entry =-1 ; vqpci->msix_addr =0 ;
tests/libqos/virtio.c +2 -2
@@ -33,12 +33,12 @@ uint64_t qvirtio_config_readq(QVirtioDevice *d, uint64_t addr)return d->bus->config_readq (d, addr); } -uint32_t qvirtio_get_features (QVirtioDevice *d) +uint64_t qvirtio_get_features (QVirtioDevice *d) {return d->bus->get_features (d); } -void qvirtio_set_features (QVirtioDevice *d,uint32_t features) +void qvirtio_set_features (QVirtioDevice *d,uint64_t features) { d->features = features; d->bus->set_features (d, features);
tests/virtio-blk-test.c +4 -4
@@ -119,7 +119,7 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc,QVirtioBlkReq req;uint64_t req_addr;uint64_t capacity; -uint32_t features; +uint64_t features;uint32_t free_head;uint8_t status;char *data;
@@ -352,7 +352,7 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)QVRingIndirectDesc *indirect;uint64_t req_addr;uint64_t capacity; -uint32_t features; +uint64_t features;uint32_t free_head;uint8_t status;char *data;
@@ -467,7 +467,7 @@ static void msix(void *obj, void *u_data, QGuestAllocator *t_alloc)int n_size =TEST_IMAGE_SIZE /2 ;uint64_t req_addr;uint64_t capacity; -uint32_t features; +uint64_t features;uint32_t free_head;uint8_t status;char *data;
@@ -574,7 +574,7 @@ static void idx(void *obj, void *u_data, QGuestAllocator *t_alloc)QVirtioBlkReq req;uint64_t req_addr;uint64_t capacity; -uint32_t features; +uint64_t features;uint32_t free_head;uint32_t write_head;uint32_t desc_idx;
Signature · 1 line
2.21.0
PATCH v4 4/16 virtio-scsi-test: add missing feature negotiation
In reply to Stefan HajnocziVIRTIO Device Initialization requires feature negotiation. Currently virtio-scsi-test.c is non-compliant.
libqos tests acknowledge all feature bits advertised by the device, except VIRTIO_F_BAD_FEATURE (which devices use to detect broken drivers!) and VIRTIO_RING_F_EVENT_IDX (which is not implemented in libqos and accepting it would break notifications).
--- tests/virtio-scsi-test.c | 8 ++++++++ 1 file changed, 8 insertions(+)
tests/virtio-scsi-test.c +8 -0
@@ -123,10 +123,16 @@ static QVirtioSCSIQueues *qvirtio_scsi_init(QVirtioDevice *dev)QVirtioSCSIQueues *vs;const uint8_t test_unit_ready_cdb[VIRTIO_SCSI_CDB_SIZE ] = {};struct virtio_scsi_cmd_resp resp; +uint64_t features;int i; vs =g_new0 (QVirtioSCSIQueues,1 ); vs->dev = dev; + + features =qvirtio_get_features (dev); + features &= ~(QVIRTIO_F_BAD_FEATURE | (1ull <<VIRTIO_RING_F_EVENT_IDX )); +qvirtio_set_features (dev, features); + vs->num_queues =qvirtio_config_readl (dev,0 ); g_assert_cmpint(vs->num_queues, <, MAX_NUM_QUEUES);
@@ -135,6 +141,8 @@ static QVirtioSCSIQueues *qvirtio_scsi_init(QVirtioDevice *dev) vs->vq[i] =qvirtqueue_setup (dev, alloc, i); } +qvirtio_set_driver_ok (dev); +/* Clear the POWER ON OCCURRED unit attention */ g_assert_cmpint(virtio_scsi_do_command(vs, test_unit_ready_cdb, NULL, 0, NULL, 0, &resp),
Signature · 1 line
2.21.0
Re: [PATCH v4 04/16] virtio-scsi-test: add missing feature negotiation
----- Original Message -----
Quoted 11 lines
From: "Stefan Hajnoczi" <[email protected]>
Sent: Wednesday, October 23, 2019 12:04:13 PMVIRTIO Device Initialization requires feature negotiation. Currently
virtio-scsi-test.cis non-compliant.libqos tests acknowledge all feature bits advertised by the device, except
VIRTIO_F_BAD_FEATURE(which devices use to detect broken drivers!) andVIRTIO_RING_F_EVENT_IDX(which is not implemented in libqos and accepting it would break notifications).
Quoted 4 lines
Signed-off-by Stefan Hajnoczi <[email protected]>--- tests/virtio-scsi-test.c | 8 ++++++++ 1 file changed, 8 insertions(+)
PATCH v4 5/16 tests/virtio-blk-test: set up virtqueue after feature negotiation
In reply to Stefan HajnocziVIRTIO Device Initialization requires that feature negotiation has completed before virtqueues are set up. This makes sense because the driver must know whether it is operating in Legacy or VIRTIO 1.0 mode before it can access vring fields with the correct endianness.
--- tests/virtio-blk-test.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-)
tests/virtio-blk-test.c +10 -7
@@ -113,8 +113,8 @@ static uint64_t virtio_blk_request(QGuestAllocator *alloc, QVirtioDevice *d,return addr; } -static void test_basic (QVirtioDevice *dev,QGuestAllocator *alloc, -QVirtQueue *vq) +/* Returns the request virtqueue so the caller can perform further tests */ +static QVirtQueue *test_basic (QVirtioDevice *dev,QGuestAllocator *alloc) {QVirtioBlkReq req;uint64_t req_addr;
@@ -124,6 +124,7 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc,uint8_t status;char *data;QTestState *qts = global_qtest; +QVirtQueue *vq; features =qvirtio_get_features (dev); features = features & ~(QVIRTIO_F_BAD_FEATURE |
@@ -135,6 +136,8 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc, capacity =qvirtio_config_readq (dev,0 ); g_assert_cmpint(capacity, ==, TEST_IMAGE_SIZE / 512); + vq =qvirtqueue_setup (dev, alloc,0 ); +qvirtio_set_driver_ok (dev);/* Write and read with 3 descriptor layout */
@@ -331,14 +334,16 @@ static void test_basic(QVirtioDevice *dev, QGuestAllocator *alloc,guest_free (alloc, req_addr); } + +return vq; }static void basic (void *obj,void *data,QGuestAllocator *t_alloc) {QVirtioBlk *blk_if = obj;QVirtQueue *vq; - vq =qvirtqueue_setup (blk_if->vdev, t_alloc,0 ); -test_basic (blk_if->vdev, t_alloc, vq); + + vq =test_basic (blk_if->vdev, t_alloc);qvirtqueue_cleanup (blk_if->vdev->bus, vq, t_alloc); }
@@ -746,9 +751,7 @@ static void resize(void *obj, void *data, QGuestAllocator *t_alloc)QVirtQueue *vq;QTestState *qts = global_qtest; - vq =qvirtqueue_setup (dev, t_alloc,0 ); - -test_basic (dev, t_alloc, vq); + vq =test_basic (dev, t_alloc);qmp_discard_response ("{ 'execute': 'block_resize', " " 'arguments': { 'device': 'drive0', "
Signature · 1 line
2.21.0
PATCH v4 6/16 libqos: add missing virtio-9p feature negotiation
In reply to Stefan HajnocziVIRTIO Device Initialization requires feature negotiation. The libqos virtio-9p driver lacks feature negotiation and is therefore non-compliant.
libqos tests acknowledge all feature bits advertised by the device, except VIRTIO_F_BAD_FEATURE (which devices use to detect broken drivers!) and VIRTIO_RING_F_EVENT_IDX (which is not implemented in libqos and accepting it would break notifications).
--- tests/libqos/virtio-9p.c | 6 ++++++ 1 file changed, 6 insertions(+)
tests/libqos/virtio-9p.c +6 -0
@@ -32,6 +32,12 @@ static void virtio_9p_cleanup(QVirtio9P *interface)static void virtio_9p_setup (QVirtio9P *interface) { +uint64_t features; + + features =qvirtio_get_features (interface->vdev); + features &= ~(QVIRTIO_F_BAD_FEATURE | (1ull <<VIRTIO_RING_F_EVENT_IDX )); +qvirtio_set_features (interface->vdev, features); + interface->vq =qvirtqueue_setup (interface->vdev, alloc,0 );qvirtio_set_driver_ok (interface->vdev); }
Signature · 1 line
2.21.0
Re: [PATCH v4 06/16] libqos: add missing virtio-9p feature negotiation
----- Original Message -----
Quoted 12 lines
From: "Stefan Hajnoczi" <[email protected]>
Sent: Wednesday, October 23, 2019 12:04:15 PMVIRTIO Device Initialization requires feature negotiation. The libqos virtio-9p driver lacks feature negotiation and is therefore non-compliant.
libqos tests acknowledge all feature bits advertised by the device, except
VIRTIO_F_BAD_FEATURE(which devices use to detect broken drivers!) andVIRTIO_RING_F_EVENT_IDX(which is not implemented in libqos and accepting it would break notifications).
Quoted 4 lines
Signed-off-by Stefan Hajnoczi <[email protected]>--- tests/libqos/virtio-9p.c | 6 ++++++ 1 file changed, 6 insertions(+)
PATCH v4 7/16 libqos: enforce Device Initialization order
In reply to Stefan HajnocziAccording to VIRTIO 1.1 "3.1.1 Driver Requirements: Device Initialization", configuration space and virtqueues cannot be accessed before features have been negotiated. Enforce this requirement.
---
v4:
* Introduce bool d->features_negotiated so that tests can negotiate a 0 feature bit set in Legacy mode [Thomas]
--- tests/libqos/virtio.h | 1 + tests/libqos/virtio.c | 7 +++++++ 2 files changed, 8 insertions(+)
tests/libqos/virtio.h +1 -0
@@ -23,6 +23,7 @@ typedef struct QVirtioDevice {uint16_t device_type;uint64_t features;bool big_endian; +bool features_negotiated; } QVirtioDevice; typedef struct QVirtQueue {
tests/libqos/virtio.c +7 -0
@@ -15,21 +15,25 @@uint8_t qvirtio_config_readb (QVirtioDevice *d,uint64_t addr) { +g_assert_true (d->features_negotiated);return d->bus->config_readb (d, addr); }uint16_t qvirtio_config_readw (QVirtioDevice *d,uint64_t addr) { +g_assert_true (d->features_negotiated);return d->bus->config_readw (d, addr); }uint32_t qvirtio_config_readl (QVirtioDevice *d,uint64_t addr) { +g_assert_true (d->features_negotiated);return d->bus->config_readl (d, addr); }uint64_t qvirtio_config_readq (QVirtioDevice *d,uint64_t addr) { +g_assert_true (d->features_negotiated);return d->bus->config_readq (d, addr); }
@@ -42,11 +46,13 @@ void qvirtio_set_features(QVirtioDevice *d, uint64_t features) { d->features = features; d->bus->set_features (d, features); + d->features_negotiated = true; }QVirtQueue *qvirtqueue_setup (QVirtioDevice *d,QGuestAllocator *alloc,uint16_t index) { +g_assert_true (d->features_negotiated);return d->bus->virtqueue_setup (d, alloc, index); }
@@ -60,6 +66,7 @@ void qvirtio_reset(QVirtioDevice *d) { d->bus->set_status (d,0 ); g_assert_cmphex(d->bus->get_status(d), ==, 0); + d->features_negotiated = false; }void qvirtio_set_acknowledge (QVirtioDevice *d)
Signature · 1 line
2.21.0
Re: [PATCH v4 07/16] libqos: enforce Device Initialization order
----- Original Message -----
Quoted 8 lines
From: "Stefan Hajnoczi" <[email protected]>
Sent: Wednesday, October 23, 2019 12:04:16 PMAccording to VIRTIO 1.1 "3.1.1 Driver Requirements: Device Initialization", configuration space and virtqueues cannot be accessed before features have been negotiated. Enforce this requirement.
Signed-off-by Stefan Hajnoczi <[email protected]>
Quoted 8 lines
---
v4:
* Introduce boold->features_negotiatedso that tests can negotiate a 0 feature bit set in Legacy mode [Thomas]--- tests/libqos/virtio.h | 1 + tests/libqos/virtio.c | 7 +++++++ 2 files changed, 8 insertions(+)
PATCH v4 8/16 libqos: implement VIRTIO 1.0 FEATURES_OK step
In reply to Stefan HajnocziDevice initialization has an extra step in VIRTIO 1.0. The FEATURES_OK status bit is set to indicate that feature negotiation has completed. The driver then reads the status register again to check that the device agrees with the final features.
Implement this step as part of qvirtio_set_features() instead of introducing a separate function. This way all existing code works without modifications.
The check in qvirtio_set_driver_ok() needs to be updated because FEATURES_OK will be set for VIRTIO 1.0 devices.
---
v4:
* Make FEATURES_OK change in qvirtio_set_driver_ok() clearer and mention it in the commit description [Thomas]
--- tests/libqos/virtio.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
tests/libqos/virtio.c +17 -1
@@ -46,6 +46,20 @@ void qvirtio_set_features(QVirtioDevice *d, uint64_t features) { d->features = features; d->bus->set_features (d, features); + +/* +* This could be a separate function for drivers that want to access +* configuration space before setting FEATURES_OK, but no existing users +* need that and it's less code for callers if this is done implicitly. +*/ +if (features & (1ull <<VIRTIO_F_VERSION_1 )) { +uint8_t status = d->bus->get_status (d) | +VIRTIO_CONFIG_S_FEATURES_OK ; + + d->bus->set_status (d, status); + g_assert_cmphex(d->bus->get_status(d), ==, status); + } + d->features_negotiated = true; }
@@ -86,7 +100,9 @@ void qvirtio_set_driver_ok(QVirtioDevice *d) { d->bus->set_status (d, d->bus->get_status (d) |VIRTIO_CONFIG_S_DRIVER_OK ); g_assert_cmphex(d->bus->get_status(d), ==, VIRTIO_CONFIG_S_DRIVER_OK | -VIRTIO_CONFIG_S_DRIVER |VIRTIO_CONFIG_S_ACKNOWLEDGE ); +VIRTIO_CONFIG_S_DRIVER |VIRTIO_CONFIG_S_ACKNOWLEDGE | + (d->features & (1ull <<VIRTIO_F_VERSION_1 ) ? +VIRTIO_CONFIG_S_FEATURES_OK :0 )); } void qvirtio_wait_queue_isr(QTestState *qts, QVirtioDevice *d,
Signature · 1 line
2.21.0
Re: [PATCH v4 08/16] libqos: implement VIRTIO 1.0 FEATURES_OK step
----- Original Message -----
Quoted 15 lines
From: "Stefan Hajnoczi" <[email protected]>
Sent: Wednesday, October 23, 2019 12:04:17 PMDevice initialization has an extra step in VIRTIO 1.0. The
FEATURES_OKstatus bit is set to indicate that feature negotiation has completed. The driver then reads the status register again to check that the device agrees with the final features.Implement this step as part of
qvirtio_set_features()instead of introducing a separate function. This way all existing code works without modifications.The check in
qvirtio_set_driver_ok()needs to be updated becauseFEATURES_OKwill be set for VIRTIO 1.0 devices.
Quoted 8 lines
Signed-off-by Stefan Hajnoczi <[email protected]>---
v4:
* MakeFEATURES_OKchange inqvirtio_set_driver_ok()clearer and mention it in the commit description [Thomas]--- tests/libqos/virtio.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-)
PATCH v4 9/16 libqos: access VIRTIO 1.0 vring in little-endian
In reply to Stefan HajnocziVIRTIO 1.0 uses little-endian for the vring. Legacy VIRTIO uses guest endianness. Adjust the code to handle both.
Note that qvirtio_readq() is not defined because it has no users. All the other accessors are really needed.
---
v4:
* Fixed indentation in qvring_init() [Thomas]
--- tests/libqos/virtio.h | 4 +- tests/libqos/virtio-mmio.c | 1 + tests/libqos/virtio-pci.c | 1 + tests/libqos/virtio.c | 131 +++++++++++++++++++++++++++---------- tests/virtio-blk-test.c | 8 +-- 5 files changed, 106 insertions(+), 39 deletions(-)
tests/libqos/virtio.h +3 -1
@@ -27,6 +27,7 @@ typedef struct QVirtioDevice { } QVirtioDevice; typedef struct QVirtQueue { + QVirtioDevice *vdev; uint64_t desc; /* This points to an array of struct vring_desc */ uint64_t avail; /* This points to a struct vring_avail */ uint64_t used; /* This points to a struct vring_used */
@@ -135,7 +136,8 @@ void qvring_init(QTestState *qts, const QGuestAllocator *alloc, QVirtQueue *vq,QVRingIndirectDesc *qvring_indirect_desc_setup (QTestState *qs,QVirtioDevice *d,QGuestAllocator *alloc,uint16_t elem); -void qvring_indirect_desc_add (QTestState *qts,QVRingIndirectDesc *indirect, +void qvring_indirect_desc_add (QVirtioDevice *d,QTestState *qts, +QVRingIndirectDesc *indirect,uint64_t data,uint32_t len,bool write);uint32_t qvirtqueue_add (QTestState *qts,QVirtQueue *vq,uint64_t data,uint32_t len,bool write,bool next);
tests/libqos/virtio-mmio.c +1 -0
@@ -157,6 +157,7 @@ static QVirtQueue *qvirtio_mmio_virtqueue_setup(QVirtioDevice *d,uint64_t addr; vq =g_malloc0 (sizeof (*vq)); + vq->vdev = d;qvirtio_mmio_queue_select (d, index);qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_QUEUE_ALIGN , dev->page_size);
tests/libqos/virtio-pci.c +1 -0
@@ -217,6 +217,7 @@ static QVirtQueue *qvirtio_pci_virtqueue_setup(QVirtioDevice *d, feat =qvirtio_pci_get_guest_features (d);qvirtio_pci_queue_select (d, index); + vqpci->vq.vdev = d; vqpci->vq.index = index; vqpci->vq.size =qvirtio_pci_get_queue_size (d); vqpci->vq.free_head =0 ;
tests/libqos/virtio.c +97 -34
237 lines not on this page · raw message
tests/virtio-blk-test.c +4 -4
@@ -388,8 +388,8 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)g_free (req.data); indirect =qvring_indirect_desc_setup (qts, dev, t_alloc,2 ); -qvring_indirect_desc_add (qts, indirect, req_addr,528 , false); -qvring_indirect_desc_add (qts, indirect, req_addr +528 ,1 , true); +qvring_indirect_desc_add (dev, qts, indirect, req_addr,528 , false); +qvring_indirect_desc_add (dev, qts, indirect, req_addr +528 ,1 , true); free_head =qvirtqueue_add_indirect (qts, vq, indirect);qvirtqueue_kick (qts, dev, vq, free_head);
@@ -413,8 +413,8 @@ static void indirect(void *obj, void *u_data, QGuestAllocator *t_alloc)g_free (req.data); indirect =qvring_indirect_desc_setup (qts, dev, t_alloc,2 ); -qvring_indirect_desc_add (qts, indirect, req_addr,16 , false); -qvring_indirect_desc_add (qts, indirect, req_addr +16 ,513 , true); +qvring_indirect_desc_add (dev, qts, indirect, req_addr,16 , false); +qvring_indirect_desc_add (dev, qts, indirect, req_addr +16 ,513 , true); free_head =qvirtqueue_add_indirect (qts, vq, indirect);qvirtqueue_kick (qts, dev, vq, free_head);
Signature · 1 line
2.21.0
PATCH v4 10/16 libqos: add iteration support to qpci_find_capability()
In reply to Stefan HajnocziVIRTIO 1.0 PCI devices have multiple PCI_CAP_ID_VNDR capabilities so we need a way to iterate over them. Extend qpci_find_capability() to take the last address.
--
v3:
* Document qpci_find_capability()
--- tests/libqos/pci.h | 2 +- tests/libqos/pci.c | 30 ++++++++++++++++++++++++------ 2 files changed, 25 insertions(+), 7 deletions(-)
tests/libqos/pci.h +1 -1
@@ -86,7 +86,7 @@ bool qpci_has_buggy_msi(QPCIDevice *dev);bool qpci_check_buggy_msi (QPCIDevice *dev);void qpci_device_enable (QPCIDevice *dev); -uint8_t qpci_find_capability (QPCIDevice *dev,uint8_t id); +uint8_t qpci_find_capability (QPCIDevice *dev,uint8_t id,uint8_t start_addr);void qpci_msix_enable (QPCIDevice *dev);void qpci_msix_disable (QPCIDevice *dev);bool qpci_msix_pending (QPCIDevice *dev,uint16_t entry);
tests/libqos/pci.c +24 -6
@@ -115,10 +115,28 @@ void qpci_device_enable(QPCIDevice *dev) g_assert_cmphex(cmd & PCI_COMMAND_MASTER, ==, PCI_COMMAND_MASTER); } -uint8_t qpci_find_capability (QPCIDevice *dev,uint8_t id) +/** +* qpci_find_capability: +* @dev: the PCI device +* @id: the PCI Capability ID (PCI_CAP_ID_*) +* @start_addr: 0 to begin iteration or the last return value to continue +* iteration +* +* Iterate over the PCI Capabilities List. +* +* Returns: PCI Configuration Space offset of the capabililty structure or +* 0 if no further matching capability is found +*/ +uint8_t qpci_find_capability (QPCIDevice *dev,uint8_t id,uint8_t start_addr) {uint8_t cap; -uint8_t addr =qpci_config_readb (dev,PCI_CAPABILITY_LIST ); +uint8_t addr; + +if (start_addr) { + addr =qpci_config_readb (dev, start_addr +PCI_CAP_LIST_NEXT ); + }else { + addr =qpci_config_readb (dev,PCI_CAPABILITY_LIST ); + } do { cap = qpci_config_readb(dev, addr);
@@ -138,7 +156,7 @@ void qpci_msix_enable(QPCIDevice *dev)uint8_t bir_table;uint8_t bir_pba; - addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ); + addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ,0 ); g_assert_cmphex(addr, !=, 0); val =qpci_config_readw (dev, addr +PCI_MSIX_FLAGS );
@@ -167,7 +185,7 @@ void qpci_msix_disable(QPCIDevice *dev)uint16_t val;g_assert (dev->msix_enabled); - addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ); + addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ,0 ); g_assert_cmphex(addr, !=, 0); val =qpci_config_readw (dev, addr +PCI_MSIX_FLAGS ); qpci_config_writew(dev, addr + PCI_MSIX_FLAGS,
@@ -203,7 +221,7 @@ bool qpci_msix_masked(QPCIDevice *dev, uint16_t entry)uint64_t vector_off = dev->msix_table_off + entry *PCI_MSIX_ENTRY_SIZE ;g_assert (dev->msix_enabled); - addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ); + addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ,0 ); g_assert_cmphex(addr, !=, 0); val =qpci_config_readw (dev, addr +PCI_MSIX_FLAGS );
@@ -221,7 +239,7 @@ uint16_t qpci_msix_table_size(QPCIDevice *dev)uint8_t addr;uint16_t control; - addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ); + addr =qpci_find_capability (dev,PCI_CAP_ID_MSIX ,0 ); g_assert_cmphex(addr, !=, 0); control =qpci_config_readw (dev, addr +PCI_MSIX_FLAGS );
Signature · 1 line
2.21.0
PATCH v4 11/16 libqos: pass full QVirtQueue to set_queue_address()
In reply to Stefan HajnocziInstead of just passing the vring page frame number, pass the full QVirtQueue. This will allow the VIRTIO 1.0 transport to program the fine-grained vring address registers in the future.
--- tests/libqos/virtio.h | 2 +- tests/libqos/virtio-mmio.c | 6 ++++-- tests/libqos/virtio-pci.c | 6 ++++-- 3 files changed, 9 insertions(+), 5 deletions(-)
tests/libqos/virtio.h +1 -1
@@ -81,7 +81,7 @@ struct QVirtioBus {uint16_t (*get_queue_size)(QVirtioDevice *d);/* Set the address of the selected queue */ -void (*set_queue_address)(QVirtioDevice *d,uint32_t pfn); +void (*set_queue_address)(QVirtioDevice *d,QVirtQueue *vq);/* Setup the virtqueue specified by index */ QVirtQueue *(*virtqueue_setup)(QVirtioDevice *d, QGuestAllocator *alloc,
tests/libqos/virtio-mmio.c +4 -2
@@ -143,9 +143,11 @@ static uint16_t qvirtio_mmio_get_queue_size(QVirtioDevice *d)return (uint16_t )qtest_readl (dev->qts, dev->addr +QVIRTIO_MMIO_QUEUE_NUM_MAX ); } -static void qvirtio_mmio_set_queue_address (QVirtioDevice *d,uint32_t pfn) +static void qvirtio_mmio_set_queue_address (QVirtioDevice *d,QVirtQueue *vq) {QVirtioMMIODevice *dev =container_of (d, QVirtioMMIODevice, vdev); +uint64_t pfn = vq->desc / dev->page_size; +qtest_writel (dev->qts, dev->addr +QVIRTIO_MMIO_QUEUE_PFN , pfn); }
@@ -179,7 +181,7 @@ static QVirtQueue *qvirtio_mmio_virtqueue_setup(QVirtioDevice *d, addr =guest_alloc (alloc,qvring_size (vq->size, dev->page_size));qvring_init (dev->qts, alloc, vq, addr); -qvirtio_mmio_set_queue_address (d, vq->desc / dev->page_size); +qvirtio_mmio_set_queue_address (d, vq);return vq; }
tests/libqos/virtio-pci.c +4 -2
@@ -199,9 +199,11 @@ static uint16_t qvirtio_pci_get_queue_size(QVirtioDevice *d)return qpci_io_readw (dev->pdev, dev->bar,VIRTIO_PCI_QUEUE_NUM ); } -static void qvirtio_pci_set_queue_address (QVirtioDevice *d,uint32_t pfn) +static void qvirtio_pci_set_queue_address (QVirtioDevice *d,QVirtQueue *vq) {QVirtioPCIDevice *dev =container_of (d, QVirtioPCIDevice, vdev); +uint64_t pfn = vq->desc /VIRTIO_PCI_VRING_ALIGN ; +qpci_io_writel (dev->pdev, dev->bar,VIRTIO_PCI_QUEUE_PFN , pfn); }
@@ -239,7 +241,7 @@ static QVirtQueue *qvirtio_pci_virtqueue_setup(QVirtioDevice *d, addr =guest_alloc (alloc,qvring_size (vqpci->vq.size,VIRTIO_PCI_VRING_ALIGN ));qvring_init (qvpcidev->pdev->bus->qts, alloc, &vqpci->vq, addr); -qvirtio_pci_set_queue_address (d, vqpci->vq.desc /VIRTIO_PCI_VRING_ALIGN ); +qvirtio_pci_set_queue_address (d, &vqpci->vq);return &vqpci->vq; }
Signature · 1 line
2.21.0
PATCH v4 12/16 libqos: add MSI-X callbacks to QVirtioPCIDevice
In reply to Stefan HajnocziThe MSI-X vectors are programmed differently in the VIRTIO 1.0 and Legacy interfaces. Introduce callbacks so different implementations can be used depending on the interface version.
--- tests/libqos/virtio-pci.h | 12 ++++++++++++ tests/libqos/virtio-pci.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 40 insertions(+), 9 deletions(-)
tests/libqos/virtio-pci.h +12 -0
@@ -14,16 +14,28 @@#include "libqos/pci.h" #include "libqos/qgraph.h" +typedef struct QVirtioPCIMSIXOps QVirtioPCIMSIXOps ; +typedef struct QVirtioPCIDevice {QOSGraphObject obj;QVirtioDevice vdev;QPCIDevice *pdev;QPCIBar bar; +const QVirtioPCIMSIXOps *msix_ops;uint16_t config_msix_entry;uint64_t config_msix_addr;uint32_t config_msix_data; }QVirtioPCIDevice ; +struct QVirtioPCIMSIXOps { +/* Set the Configuration Vector for MSI-X */ +void (*set_config_vector)(QVirtioPCIDevice *d,uint16_t entry); + +/* Set the Queue Vector for MSI-X */ +void (*set_queue_vector)(QVirtioPCIDevice *d,uint16_t vq_idx, +uint16_t entry); +}; + typedef struct QVirtQueuePCI { QVirtQueue vq; uint16_t msix_entry;
tests/libqos/virtio-pci.c +28 -9
@@ -281,6 +281,31 @@ const QVirtioBus qvirtio_pci = { .virtqueue_kick = qvirtio_pci_virtqueue_kick, }; +static void qvirtio_pci_set_config_vector (QVirtioPCIDevice *d,uint16_t entry) +{ +uint16_t vector; + +qpci_io_writew (d->pdev, d->bar,VIRTIO_MSI_CONFIG_VECTOR , entry); + vector =qpci_io_readw (d->pdev, d->bar,VIRTIO_MSI_CONFIG_VECTOR ); + g_assert_cmphex(vector, !=, VIRTIO_MSI_NO_VECTOR); +} + +static void qvirtio_pci_set_queue_vector (QVirtioPCIDevice *d,uint16_t vq_idx, +uint16_t entry) +{ +uint16_t vector; + +qvirtio_pci_queue_select (&d->vdev, vq_idx); +qpci_io_writew (d->pdev, d->bar,VIRTIO_MSI_QUEUE_VECTOR , entry); + vector =qpci_io_readw (d->pdev, d->bar,VIRTIO_MSI_QUEUE_VECTOR ); + g_assert_cmphex(vector, !=, VIRTIO_MSI_NO_VECTOR); +} + +static const QVirtioPCIMSIXOps qvirtio_pci_msix_ops_legacy = { + .set_config_vector = qvirtio_pci_set_config_vector, + .set_queue_vector = qvirtio_pci_set_queue_vector, +}; +void qvirtio_pci_device_enable (QVirtioPCIDevice *d) {qpci_device_enable (d->pdev);
@@ -295,7 +320,6 @@ void qvirtio_pci_device_disable(QVirtioPCIDevice *d)void qvirtqueue_pci_msix_setup (QVirtioPCIDevice *d,QVirtQueuePCI *vqpci,QGuestAllocator *alloc,uint16_t entry) { -uint16_t vector;uint32_t control;uint64_t off;
@@ -321,16 +345,12 @@ void qvirtqueue_pci_msix_setup(QVirtioPCIDevice *d, QVirtQueuePCI *vqpci, off +PCI_MSIX_ENTRY_VECTOR_CTRL , control & ~PCI_MSIX_ENTRY_CTRL_MASKBIT); -qvirtio_pci_queue_select (&d->vdev, vqpci->vq.index); -qpci_io_writew (d->pdev, d->bar,VIRTIO_MSI_QUEUE_VECTOR , entry); - vector =qpci_io_readw (d->pdev, d->bar,VIRTIO_MSI_QUEUE_VECTOR ); - g_assert_cmphex(vector, !=, VIRTIO_MSI_NO_VECTOR); + d->msix_ops->set_queue_vector (d, vqpci->vq.index, entry); }void qvirtio_pci_set_msix_configuration_vector (QVirtioPCIDevice *d,QGuestAllocator *alloc,uint16_t entry) { -uint16_t vector;uint32_t control;uint64_t off;
@@ -358,9 +378,7 @@ void qvirtio_pci_set_msix_configuration_vector(QVirtioPCIDevice *d, off +PCI_MSIX_ENTRY_VECTOR_CTRL , control & ~PCI_MSIX_ENTRY_CTRL_MASKBIT); -qpci_io_writew (d->pdev, d->bar,VIRTIO_MSI_CONFIG_VECTOR , entry); - vector =qpci_io_readw (d->pdev, d->bar,VIRTIO_MSI_CONFIG_VECTOR ); - g_assert_cmphex(vector, !=, VIRTIO_MSI_NO_VECTOR); + d->msix_ops->set_config_vector (d, entry); }void qvirtio_pci_destructor (QOSGraphObject *obj)
@@ -383,6 +401,7 @@ static void qvirtio_pci_init_from_pcidev(QVirtioPCIDevice *dev, QPCIDevice *pci_ dev->vdev.device_type =qpci_config_readw (pci_dev,PCI_SUBSYSTEM_ID ); dev->config_msix_entry =-1 ; + dev->msix_ops = &qvirtio_pci_msix_ops_legacy; dev->vdev.bus = &qvirtio_pci; dev->vdev.big_endian =qvirtio_pci_is_big_endian (dev);
Signature · 1 line
2.21.0
PATCH v4 13/16 libqos: expose common virtqueue setup/cleanup functions
In reply to Stefan HajnocziThe VIRTIO 1.0 code will need to perform additional steps but it will reuse the common virtqueue setup/cleanup code. Make these functions public.
Make sure to invoke callbacks via QVirtioBus instead of directly calling the virtio-pci Legacy versions of these functions.
--- tests/libqos/virtio-pci.h | 8 ++++++++ tests/libqos/virtio-pci.c | 19 ++++++++++--------- 2 files changed, 18 insertions(+), 9 deletions(-)
tests/libqos/virtio-pci.h +8 -0
@@ -65,4 +65,12 @@ void qvirtio_pci_set_msix_configuration_vector(QVirtioPCIDevice *d, QGuestAllocator *alloc, uint16_t entry);void qvirtqueue_pci_msix_setup (QVirtioPCIDevice *d,QVirtQueuePCI *vqpci,QGuestAllocator *alloc,uint16_t entry); + +/* Used by Legacy and Modern virtio-pci code */ +QVirtQueue *qvirtio_pci_virtqueue_setup_common (QVirtioDevice *d, +QGuestAllocator *alloc, +uint16_t index); +void qvirtio_pci_virtqueue_cleanup_common (QVirtQueue *vq, +QGuestAllocator *alloc); +#endif
tests/libqos/virtio-pci.c +10 -9
@@ -207,8 +207,9 @@ static void qvirtio_pci_set_queue_address(QVirtioDevice *d, QVirtQueue *vq)qpci_io_writel (dev->pdev, dev->bar,VIRTIO_PCI_QUEUE_PFN , pfn); } -static QVirtQueue *qvirtio_pci_virtqueue_setup (QVirtioDevice *d, -QGuestAllocator *alloc,uint16_t index) +QVirtQueue *qvirtio_pci_virtqueue_setup_common (QVirtioDevice *d, +QGuestAllocator *alloc, +uint16_t index) {uint64_t feat;uint64_t addr;
@@ -216,12 +217,12 @@ static QVirtQueue *qvirtio_pci_virtqueue_setup(QVirtioDevice *d,QVirtioPCIDevice *qvpcidev =container_of (d, QVirtioPCIDevice, vdev); vqpci =g_malloc0 (sizeof (*vqpci)); - feat =qvirtio_pci_get_guest_features (d); + feat = d->bus->get_guest_features (d); -qvirtio_pci_queue_select (d, index); + d->bus->queue_select (d, index); vqpci->vq.vdev = d; vqpci->vq.index = index; - vqpci->vq.size =qvirtio_pci_get_queue_size (d); + vqpci->vq.size = d->bus->get_queue_size (d); vqpci->vq.free_head =0 ; vqpci->vq.num_free = vqpci->vq.size; vqpci->vq.align =VIRTIO_PCI_VRING_ALIGN ;
@@ -241,12 +242,12 @@ static QVirtQueue *qvirtio_pci_virtqueue_setup(QVirtioDevice *d, addr =guest_alloc (alloc,qvring_size (vqpci->vq.size,VIRTIO_PCI_VRING_ALIGN ));qvring_init (qvpcidev->pdev->bus->qts, alloc, &vqpci->vq, addr); -qvirtio_pci_set_queue_address (d, &vqpci->vq); + d->bus->set_queue_address (d, &vqpci->vq);return &vqpci->vq; } -static void qvirtio_pci_virtqueue_cleanup (QVirtQueue *vq, +void qvirtio_pci_virtqueue_cleanup_common (QVirtQueue *vq,QGuestAllocator *alloc) {QVirtQueuePCI *vqpci =container_of (vq, QVirtQueuePCI, vq);
@@ -276,8 +277,8 @@ const QVirtioBus qvirtio_pci = { .queue_select = qvirtio_pci_queue_select, .get_queue_size = qvirtio_pci_get_queue_size, .set_queue_address = qvirtio_pci_set_queue_address, - .virtqueue_setup = qvirtio_pci_virtqueue_setup, - .virtqueue_cleanup = qvirtio_pci_virtqueue_cleanup, + .virtqueue_setup = qvirtio_pci_virtqueue_setup_common, + .virtqueue_cleanup = qvirtio_pci_virtqueue_cleanup_common, .virtqueue_kick = qvirtio_pci_virtqueue_kick, };
Signature · 1 line
2.21.0
PATCH v4 14/16 libqos: make the virtio-pci BAR index configurable
In reply to Stefan HajnocziThe Legacy virtio-pci interface always uses BAR 0. VIRTIO 1.0 may need to use a different BAR index, so make it configurable.
---
v3:
* Change uint8_t bar_idx to int [Thomas]
--- tests/libqos/virtio-pci.h | 2 ++ tests/libqos/virtio-pci.c | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)
tests/libqos/virtio-pci.h +2 -0
@@ -25,6 +25,8 @@ typedef struct QVirtioPCIDevice {uint16_t config_msix_entry;uint64_t config_msix_addr;uint32_t config_msix_data; + +int bar_idx; } QVirtioPCIDevice;struct QVirtioPCIMSIXOps {
tests/libqos/virtio-pci.c +2 -1
@@ -310,7 +310,7 @@ static const QVirtioPCIMSIXOps qvirtio_pci_msix_ops_legacy = {void qvirtio_pci_device_enable (QVirtioPCIDevice *d) {qpci_device_enable (d->pdev); - d->bar =qpci_iomap (d->pdev,0 ,NULL ); + d->bar =qpci_iomap (d->pdev, d->bar_idx,NULL ); }void qvirtio_pci_device_disable (QVirtioPCIDevice *d)
@@ -400,6 +400,7 @@ static void qvirtio_pci_init_from_pcidev(QVirtioPCIDevice *dev, QPCIDevice *pci_ { dev->pdev = pci_dev; dev->vdev.device_type =qpci_config_readw (pci_dev,PCI_SUBSYSTEM_ID ); + dev->bar_idx =0 ; dev->config_msix_entry =-1 ; dev->msix_ops = &qvirtio_pci_msix_ops_legacy;
Signature · 1 line
2.21.0
PATCH v4 15/16 libqos: extract Legacy virtio-pci.c code
In reply to Stefan HajnocziThe current libqos virtio-pci.c code implements the VIRTIO Legacy interface. Extract existing code in preparation for VIRTIO 1.0 support.
--- tests/libqos/virtio-pci.h | 2 -- tests/libqos/virtio-pci.c | 29 ++++++++++++----------------- 2 files changed, 12 insertions(+), 19 deletions(-)
tests/libqos/virtio-pci.h +0 -2
@@ -45,8 +45,6 @@ typedef struct QVirtQueuePCI {uint32_t msix_data; } QVirtQueuePCI; -extern const QVirtioBus qvirtio_pci; -void virtio_pci_init (QVirtioPCIDevice *dev,QPCIBus *bus,QPCIAddress * addr);QVirtioPCIDevice *virtio_pci_new (QPCIBus *bus,QPCIAddress * addr);
tests/libqos/virtio-pci.c +12 -17
@@ -35,14 +35,6 @@* original qvirtio_pci_destructor and qvirtio_pci_start_hw. */ -static inline bool qvirtio_pci_is_big_endian (QVirtioPCIDevice *dev) -{ -QPCIBus *bus = dev->pdev->bus; - -/* FIXME: virtio 1.0 is always little-endian */ -return qtest_big_endian (bus->qts); -} -#define CONFIG_BASE (dev) (VIRTIO_PCI_CONFIG_OFF((dev)->pdev->msix_enabled))static uint8_t qvirtio_pci_config_readb (QVirtioDevice *d,uint64_t off)
@@ -55,8 +47,7 @@ static uint8_t qvirtio_pci_config_readb(QVirtioDevice *d, uint64_t off)* but virtio ( < 1.0) is in guest order * so with a big-endian guest the order has been reversed, * reverse it again -* virtio-1.0 is always little-endian, like PCI, but this -* case will be managed inside qvirtio_pci_is_big_endian() +* virtio-1.0 is always little-endian, like PCI */ static uint16_t qvirtio_pci_config_readw (QVirtioDevice *d,uint64_t off)
@@ -262,7 +253,7 @@ static void qvirtio_pci_virtqueue_kick(QVirtioDevice *d, QVirtQueue *vq)qpci_io_writew (dev->pdev, dev->bar,VIRTIO_PCI_QUEUE_NOTIFY , vq->index); } -const QVirtioBus qvirtio_pci = { +static const QVirtioBus qvirtio_pci_legacy = { .config_readb = qvirtio_pci_config_readb, .config_readw = qvirtio_pci_config_readw, .config_readl = qvirtio_pci_config_readl,
@@ -396,17 +387,21 @@ void qvirtio_pci_start_hw(QOSGraphObject *obj)qvirtio_start_device (&dev->vdev); } +static void qvirtio_pci_init_legacy (QVirtioPCIDevice *dev) +{ + dev->vdev.device_type =qpci_config_readw (dev->pdev,PCI_SUBSYSTEM_ID ); + dev->bar_idx =0 ; + dev->vdev.bus = &qvirtio_pci_legacy; + dev->msix_ops = &qvirtio_pci_msix_ops_legacy; + dev->vdev.big_endian =qtest_big_endian (dev->pdev->bus->qts); +} +static void qvirtio_pci_init_from_pcidev (QVirtioPCIDevice *dev,QPCIDevice *pci_dev) { dev->pdev = pci_dev; - dev->vdev.device_type =qpci_config_readw (pci_dev,PCI_SUBSYSTEM_ID ); - dev->bar_idx =0 ; - dev->config_msix_entry =-1 ; - dev->msix_ops = &qvirtio_pci_msix_ops_legacy; - dev->vdev.bus = &qvirtio_pci; - dev->vdev.big_endian =qvirtio_pci_is_big_endian (dev); +qvirtio_pci_init_legacy (dev);/* each virtio-xxx-pci device should override at least this function */ dev->obj.get_driver =NULL ;
Signature · 1 line
2.21.0
PATCH v4 16/16 libqos: add VIRTIO PCI 1.0 support
In reply to Stefan HajnocziImplement the VIRTIO 1.0 virtio-pci interface. The main change here is that the register layout is no longer a fixed layout in BAR 0. Instead we have to iterate of PCI Capabilities to find descriptions of where various registers are located. The vring registers are also more fine-grained, allowing for more flexible vring layouts, but we don't take advantage of that.
--- tests/Makefile.include | 1 + tests/libqos/virtio-pci-modern.h | 17 ++ tests/libqos/virtio-pci.h | 10 + tests/libqos/virtio-pci-modern.c | 443 +++++++++++++++++++++++++++++++ tests/libqos/virtio-pci.c | 6 +- 5 files changed, 476 insertions(+), 1 deletion(-) create mode 100644 tests/libqos/virtio-pci-modern.h create mode 100644 tests/libqos/virtio-pci-modern.c
tests/Makefile.include +1 -0
@@ -715,6 +715,7 @@ qos-test-obj-y += tests/libqos/virtio-blk.o qos-test-obj-y += tests/libqos/virtio-mmio.o qos-test-obj-y += tests/libqos/virtio-net.o qos-test-obj-y += tests/libqos/virtio-pci.o +qos-test-obj-y += tests/libqos/virtio-pci-modern.o qos-test-obj-y += tests/libqos/virtio-rng.o qos-test-obj-y += tests/libqos/virtio-scsi.o qos-test-obj-y += tests/libqos/virtio-serial.o
tests/libqos/virtio-pci-modern.h new file +17 -0
@@ -0,0 +1,17 @@ +/* +* libqos virtio PCI VIRTIO 1.0 definitions +* +* Copyright (c) 2019 Red Hat, Inc +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifndef LIBQOS_VIRTIO_PCI_MODERN_H +#define LIBQOS_VIRTIO_PCI_MODERN_H + +#include "virtio-pci.h" + +bool qvirtio_pci_init_virtio_1 (QVirtioPCIDevice *dev); + +#endif /* LIBQOS_VIRTIO_PCI_MODERN_H */
tests/libqos/virtio-pci.h +10 -0
@@ -27,6 +27,13 @@ typedef struct QVirtioPCIDevice {uint32_t config_msix_data;int bar_idx; + +/* VIRTIO 1.0 */ +uint32_t common_cfg_offset; +uint32_t notify_cfg_offset; +uint32_t notify_off_multiplier; +uint32_t isr_cfg_offset; +uint32_t device_cfg_offset; } QVirtioPCIDevice;struct QVirtioPCIMSIXOps {
@@ -43,6 +50,9 @@ typedef struct QVirtQueuePCI {uint16_t msix_entry;uint64_t msix_addr;uint32_t msix_data; + +/* VIRTIO 1.0 */ +uint64_t notify_offset; } QVirtQueuePCI;void virtio_pci_init (QVirtioPCIDevice *dev,QPCIBus *bus,QPCIAddress * addr);
tests/libqos/virtio-pci-modern.c new file +443 -0
443 lines not on this page · raw message
tests/libqos/virtio-pci.c +5 -1
@@ -22,6 +22,8 @@#include "hw/pci/pci.h" #include "hw/pci/pci_regs.h" +#include "virtio-pci-modern.h" +/* virtio-pci is a superclass of all virtio-xxx-pci devices; * the relation between virtio-pci and virtio-xxx-pci is implicit, * and therefore virtio-pci does not produce virtio and is not
@@ -401,7 +403,9 @@ static void qvirtio_pci_init_from_pcidev(QVirtioPCIDevice *dev, QPCIDevice *pci_ dev->pdev = pci_dev; dev->config_msix_entry =-1 ; -qvirtio_pci_init_legacy (dev); +if (!qvirtio_pci_init_virtio_1 (dev)) { +qvirtio_pci_init_legacy (dev); + }/* each virtio-xxx-pci device should override at least this function */ dev->obj.get_driver =NULL ;
Signature · 1 line
2.21.0
----- Original Message -----
Quoted 12 lines
From: "Stefan Hajnoczi" <[email protected]>
Sent: Wednesday, October 23, 2019 12:04:25 PMImplement the VIRTIO 1.0 virtio-pci interface. The main change here is that the register layout is no longer a fixed layout in BAR 0. Instead we have to iterate of PCI Capabilities to find descriptions of where various registers are located. The vring registers are also more fine-grained, allowing for more flexible vring layouts, but we don't take advantage of that.
Signed-off-by Stefan Hajnoczi <[email protected]>Reviewed-by Sergio Lopez <[email protected]>
Quoted 9 lines
--- tests/Makefile.include | 1 + tests/libqos/virtio-pci-modern.h | 17 ++ tests/libqos/virtio-pci.h | 10 + tests/libqos/virtio-pci-modern.c | 443 +++++++++++++++++++++++++++++++ tests/libqos/virtio-pci.c | 6 +- 5 files changed, 476 insertions(+), 1 deletion(-) create mode 100644 tests/libqos/virtio-pci-modern.h create mode 100644 tests/libqos/virtio-pci-modern.c
Quoted 30 lines · Stefan Hajnoczi
v4:
* Introduce boold->features_negotiatedso that tests can negotiate a 0 feature bit set in Legacy mode [Thomas]
* Make theFEATURES_OKcode change inqvirtio_set_driver_ok()clearer and mention it in the commit description [Thomas]
* Fix indentation inqvring_init()[Thomas]
v3:
* Now implements VIRTIO 1.0 fully (vring, device initialization). This required several new patches to address the following issues:
1. Tests that do not negotiate features (non-compliant!)
2. Tests that access configuration space before feature negotiation (non-compliant!)
3. Tests that set up virtqueues before feature negotiation (non-compliant!)
4. vring accesses require byte-swapping when VIRTIO 1.0 is used with a big-endian guest because the qtest_readX/writeX() API automatically converts to guest-endian
5. VIRTIO 1.0 has an additionalFEATURES_OKstep during Device Initialization
* Changeuint8_tbar_idxto int [Thomas]
* Documentqpci_find_capability()[Thomas]
* Every commit tested with arm, ppc64, andx86_64targets using:git rebase -i origin/master -x 'make tests/qos-test && QTEST_QEMU_BINARY=ppc64-softmmu/qemu-system-ppc64 tests/qos-test && QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 tests/qos-test' QTEST_QEMU_BINARY=arm-softmmu/qemu-system-arm tests/qos-test'v2:
* Fix checkpatch.pl issues, exceptMAINTAINERSfile warning. libqos already has maintainers and the new virtio-pci-modern.[ch] files don't need extra entries since they are already covered by the existing libqos/ entry.
Quoted 6 lines
New VIRTIO devices are Non-Transitional. This means they only expose the VIRTIO 1.0 interface, not the Legacy interface.
The libqos only supports Legacy and Transitional devices (in Legacy mode). This patch series adds VIRTIO 1.0 support so that tests can run against Non-Transitional devices too.
Very nice, thanks!
I'll queue this up in my tree.
Quoted 42 lines
The virtio-fs device is Non-Transitional, so this is a prerequisite for virtio-fs qos tests.
Stefan Hajnoczi (16):
tests/virtio-blk-test: read config space after feature negotiation
libqos: readQVIRTIO_MMIO_VERSIONregister
libqos: extend feature bits to 64-bit
virtio-scsi-test: add missing feature negotiation tests/virtio-blk-test: set up virtqueue after feature negotiation
libqos: add missing virtio-9p feature negotiation
libqos: enforce Device Initialization order
libqos: implement VIRTIO 1.0FEATURES_OKstep
libqos: access VIRTIO 1.0 vring in little-endian
libqos: add iteration support toqpci_find_capability()
libqos: pass full QVirtQueue toset_queue_address()
libqos: add MSI-X callbacks to QVirtioPCIDevice
libqos: expose common virtqueue setup/cleanup functions
libqos: make the virtio-pci BAR index configurable
libqos: extract Legacyvirtio-pci.ccode
libqos: add VIRTIO PCI 1.0 supporttests/Makefile.include | 1 + tests/libqos/pci.h | 2 +- tests/libqos/virtio-mmio.h | 1 + tests/libqos/virtio-pci-modern.h | 17 ++ tests/libqos/virtio-pci.h | 34 ++- tests/libqos/virtio.h | 19 +- tests/libqos/pci.c | 30 ++- tests/libqos/virtio-9p.c | 6 + tests/libqos/virtio-mmio.c | 38 ++- tests/libqos/virtio-net.c | 6 +- tests/libqos/virtio-pci-modern.c | 443 +++++++++++++++++++++++++++++++ tests/libqos/virtio-pci.c | 105 +++++--- tests/libqos/virtio.c | 160 ++++++++--- tests/virtio-blk-test.c | 66 +++-- tests/virtio-scsi-test.c | 8 + 15 files changed, 802 insertions(+), 134 deletions(-) create mode 100644 tests/libqos/virtio-pci-modern.h create mode 100644 tests/libqos/virtio-pci-modern.cSignature · 1 line
2.21.0