virtio-blk-ccw: tweak the default for num_queues
PATCH 1/1 virtio-blk-ccw: tweak the default for num_queues
Currently the default value of num_queues is effectively 1 for virtio-blk-ccw. Recently 9445e1e15e ("virtio-blk-pci: default num_queues to -smp N") changed the default for pci to the number of vcpus, citing interrupt better locality and better performance as a rationale.
While virtio-blk-ccw does not yet benefit from better interrupt locality, measurements have shown that for secure VMs multiqueue does help with performance. Since the bounce buffering happens with the queue lock held (in the guest) this is hardly a surprise.
As for non-secure VMs the extra queues shouldn't hurt too much.
---
We would prefer to land this for 5.2. If we do then commit 9445e1e15e ("virtio-blk-pci: default num_queues to -smp N") took care of all the necessary compat handling.
If that's not possible, I will send a version that does the necessary compat handling.
--- hw/s390x/virtio-ccw-blk.c | 6 ++++++ 1 file changed, 6 insertions(+)
hw/s390x/virtio-ccw-blk.c +6 -0
@@ -10,6 +10,7 @@*/ #include "qemu/osdep.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/virtio.h" #include "qapi/error.h"
@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }qdev_realize (vdev,BUS (&ccw_dev->bus), errp); }
base-commit: 2a190a7256a3e0563b29ffd67e0164097b4a6dac
Signature · 1 line
2.17.1
Quoted 11 lines · Halil Pasic
Currently the default value of
num_queuesis effectively 1 for virtio-blk-ccw. Recently9445e1e15e("virtio-blk-pci: defaultnum_queuesto -smp N") changed the default for pci to the number of vcpus, citing interrupt better locality and better performance as a rationale.While virtio-blk-ccw does not yet benefit from better interrupt locality, measurements have shown that for secure VMs multiqueue does help with performance. Since the bounce buffering happens with the queue lock held (in the guest) this is hardly a surprise.
As for non-secure VMs the extra queues shouldn't hurt too much.
In fact we have also seen improvments for multiqueues for non secure guests as one virtqueue seems to have a hard capping in terms of bandwidth that can be smaller than newer storage devices.
Quoted 11 lines
Suggested-by Christian Borntraeger <[email protected]>Signed-off-by Halil Pasic <[email protected]>---
We would prefer to land this for 5.2. If we do then commit
9445e1e15e("virtio-blk-pci: defaultnum_queuesto -smp N") took care of all the necessary compat handling.If that's not possible, I will send a version that does the necessary compat handling.
Right. At the moment the original patch has all the necessary compat handling for 5.1 for all platforms. Adding multi-queue for s390x for > 5.2 would mean that we would need to have additional platform specific compat handling and it would increase the patch size unnecessarily.
In the end you could consider this an enhancement (not calling it fix) of the original patch.
Quoted 31 lines
--- hw/s390x/virtio-ccw-blk.c | 6 ++++++ 1 file changed, 6 insertions(+)hw/s390x/virtio-ccw-blk.c +6
-0@@ -10,6 +10,7 @@*/ #include "qemu/osdep.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/virtio.h" #include "qapi/error.h" @@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }qdev_realize (vdev,BUS (&ccw_dev->bus), errp); }base-commit:
2a190a7256a3e0563b29ffd67e0164097b4a6dac
Quoted 14 lines · On Mon, 9 Nov 2020 16:48:31 +0100 Halil Pasic <[email protected]> wrote:
Currently the default value of
num_queuesis effectively 1 for virtio-blk-ccw. Recently9445e1e15e("virtio-blk-pci: defaultnum_queuesto -smp N") changed the default for pci to the number of vcpus, citing interrupt better locality and better performance as a rationale.While virtio-blk-ccw does not yet benefit from better interrupt locality, measurements have shown that for secure VMs multiqueue does help with performance. Since the bounce buffering happens with the queue lock held (in the guest) this is hardly a surprise.
As for non-secure VMs the extra queues shouldn't hurt too much.
Suggested-by Christian Borntraeger <[email protected]>Signed-off-by Halil Pasic <[email protected]>
Quoted 8 lines
---
We would prefer to land this for 5.2. If we do then commit
9445e1e15e("virtio-blk-pci: defaultnum_queuesto -smp N") took care of all the necessary compat handling.If that's not possible, I will send a version that does the necessary compat handling.
I think we can still get this into 5.2, and that would indeed be less hassle.
Quoted 17 lines
--- hw/s390x/virtio-ccw-blk.c | 6 ++++++ 1 file changed, 6 insertions(+)hw/s390x/virtio-ccw-blk.c +1
-0@@ -10,6 +10,7 @@*/ #include "qemu/osdep.h" +#include "hw/boards.h" #include "hw/qdev-properties.h" #include "hw/virtio/virtio.h" #include "qapi/error.h"
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 8 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }
I would like to have a comment explaining the numbers here, however.
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Do we also want something similar for virtio-scsi (and vhost-scsi)?
Quoted 5 lines
hw/s390x/virtio-ccw-blk.c +0
-0qdev_realize (vdev,BUS (&ccw_dev->bus), errp); }base-commit:
2a190a7256a3e0563b29ffd67e0164097b4a6dac
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?
Regards,
Halil
Quoted 23 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?
My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
So we can
- go with num queues = num cpus. But this will consume memory
for guests with lots of CPUs.
- go with the proposed logic of min(4,vcpus) and accept that future compat handling is harder
- defer this change
Quoted 28 lines · On Tue, 10 Nov 2020 09:47:51 +0100 Christian Borntraeger…
Quoted 27 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
Quoted 7 lines
Quoted 3 lines
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Yes, 4 does not look like a bad number, but I still don't feel really comfortable with it without at least some data.
What about large guests with slow vs. fast storage?
Quoted 4 lines
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
Yes, I fear that will be messy. Just picking a value later will need compat handling, but not a really complicated one.
Quoted 4 lines
So we can
- go with num queues = num cpus. But this will consume memory for guests with lots of CPUs.
I'm not sure that would be a good choice, as we don't have the benefits that pci has.
Quoted 2 lines
- go with the proposed logic of
min(4,vcpus)and accept that future compat handling is harder
With a bit more data, I'd be way more comfortable. Might still be ok for the next rc.
Quoted 1 line
- defer this change
We might end up with that, given the timing :( (not blaming anyone)
Quoted 33 lines · On Tue, 10 Nov 2020 09:47:51 +0100 Christian Borntraeger…
Quoted 23 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?
Quoted 8 lines
My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
Regarding compat handling, I believe we would need a new property for
virtio-blk-ccw: something like def_num_queues_max. Then logic would morph to MIN(def_num_queues_max, current_machine->smp.cpus), and we could relatively freely do compat stuff on def_num_queues_max.
IMHO not pretty but certainly doable.
Quoted 4 lines
So we can
- go with num queues = num cpus. But this will consume memory for guests with lots of CPUs.
In absence of data that showcases the benefit outweighing the obvious detriment, I lean towards finding this option the least favorable.
Quoted 2 lines
- go with the proposed logic of
min(4,vcpus)and accept that future compat handling is harder
IMHO not a bad option, but I think I would still feel better about a more informed decision. In the end, the end user can already specify the num_queues explicitly, so I don't think this is urgent.
Quoted 1 line
- defer this change
So I tend to lean towards deferring.
Another thought is, provided the load is about evenly spread on the different virtqueues, if the game is about vCPU locality, one could think of decreasing the size of each individual virtqueue while increasing their number, with the idea of not paying much more in terms of memory. The queue size however needs to be a power of 2, so there is a limit on the granularity.
Regarding secure VMs, currently we have to cramp at least the swiotlb and the virtqueues into ZONE_DMA. So increasing the number of virtqueues heavily may get us into new trouble with exotic configs.
Regards,
Halil
Quoted 63 lines · Halil Pasic
Quoted 33 lines · On Tue, 10 Nov 2020 09:47:51 +0100 Christian Borntraeger…
Quoted 23 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Quoted 8 lines
My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
Regarding compat handling, I believe we would need a new property for
virtio-blk-ccw: something likedef_num_queues_max. Then logic would morph toMIN(def_num_queues_max, current_machine->smp.cpus), and we could relatively freely do compat stuff ondef_num_queues_max.IMHO not pretty but certainly doable.
Quoted 4 lines
So we can
- go with num queues = num cpus. But this will consume memory for guests with lots of CPUs.In absence of data that showcases the benefit outweighing the obvious detriment, I lean towards finding this option the least favorable.
Quoted 2 lines
- go with the proposed logic of
min(4,vcpus)and accept that future compat handling is harder
Quoted 7 lines
IMHO not a bad option, but I think I would still feel better about a more informed decision. In the end, the end user can already specify the
num_queuesexplicitly, so I don't think this is urgent.Quoted 1 line
- defer this change
So I tend to lean towards deferring.
Yes, I was pushing this for 5.2 to avoid compat handling. But maybe it is better to wait and do it later. But we should certainly continue the discussion to have something for the next release.
Quoted 15 lines
Another thought is, provided the load is about evenly spread on the different virtqueues, if the game is about vCPU locality, one could think of decreasing the size of each individual virtqueue while increasing their number, with the idea of not paying much more in terms of memory. The queue size however needs to be a power of 2, so there is a limit on the granularity.
Regarding secure VMs, currently we have to cramp at least the swiotlb and the virtqueues into
ZONE_DMA. So increasing the number of virtqueues heavily may get us into new trouble with exotic configs.Regards,
Halil
Quoted 68 lines · On Tue, 10 Nov 2020 14:18:39 +0100 Christian Borntraeger…
Quoted 67 lines · Halil Pasic
Quoted 33 lines · On Tue, 10 Nov 2020 09:47:51 +0100 Christian Borntraeger…
Quoted 23 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Quoted 8 lines
My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
Regarding compat handling, I believe we would need a new property for
virtio-blk-ccw: something likedef_num_queues_max. Then logic would morph toMIN(def_num_queues_max, current_machine->smp.cpus), and we could relatively freely do compat stuff ondef_num_queues_max.IMHO not pretty but certainly doable.
Quoted 4 lines
So we can
- go with num queues = num cpus. But this will consume memory for guests with lots of CPUs.In absence of data that showcases the benefit outweighing the obvious detriment, I lean towards finding this option the least favorable.
Quoted 2 lines
- go with the proposed logic of
min(4,vcpus)and accept that future compat handling is harderIMHO not a bad option, but I think I would still feel better about a more informed decision. In the end, the end user can already specify the
num_queuesexplicitly, so I don't think this is urgent.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
- defer this change
So I tend to lean towards deferring.
Yes, I was pushing this for 5.2 to avoid compat handling. But maybe it is better to wait and do it later. But we should certainly continue the discussion to have something for the next release.
<going through older mails>
Do we have a better idea now about which values would make sense here?
Quoted 17 lines
Quoted 16 lines
Another thought is, provided the load is about evenly spread on the different virtqueues, if the game is about vCPU locality, one could think of decreasing the size of each individual virtqueue while increasing their number, with the idea of not paying much more in terms of memory. The queue size however needs to be a power of 2, so there is a limit on the granularity.
Regarding secure VMs, currently we have to cramp at least the swiotlb and the virtqueues into
ZONE_DMA. So increasing the number of virtqueues heavily may get us into new trouble with exotic configs.Regards,
Halil
Quoted 75 lines · On Tue, 15 Dec 2020 09:26:56 +0100 Cornelia Huck <[email protected]> wrote:
Quoted 72 lines · On Tue, 10 Nov 2020 14:18:39 +0100 Christian Borntraeger…
Quoted 70 lines · Halil Pasic
Quoted 33 lines · On Tue, 10 Nov 2020 09:47:51 +0100 Christian Borntraeger…
Quoted 23 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
Review static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
(quoted diff) +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) { VirtIOBlkCcw *dev = VIRTIO_BLK_CCW(ccw_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = MIN(4, current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Quoted 8 lines
My point was to find a balance between performance gain and memory usage. As a matter of fact, virtqueue do consume memory. So 4 looked like a reasonable default for me for large guests as long as we do not have directed interrupts.
Now, thinking about this again: If we want to change the default to something else in the future (e.g. to num vcpus) then the compat handling will get really complicated.
Regarding compat handling, I believe we would need a new property for
virtio-blk-ccw: something likedef_num_queues_max. Then logic would morph toMIN(def_num_queues_max, current_machine->smp.cpus), and we could relatively freely do compat stuff ondef_num_queues_max.IMHO not pretty but certainly doable.
Quoted 4 lines
So we can
- go with num queues = num cpus. But this will consume memory for guests with lots of CPUs.In absence of data that showcases the benefit outweighing the obvious detriment, I lean towards finding this option the least favorable.
Quoted 2 lines
- go with the proposed logic of
min(4,vcpus)and accept that future compat handling is harderIMHO not a bad option, but I think I would still feel better about a more informed decision. In the end, the end user can already specify the
num_queuesexplicitly, so I don't think this is urgent.Quoted 1 line
- defer this change
So I tend to lean towards deferring.
Quoted 8 lines
Quoted 3 lines
Yes, I was pushing this for 5.2 to avoid compat handling. But maybe it is better to wait and do it later. But we should certainly continue the discussion to have something for the next release.
<going through older mails>
Do we have a better idea now about which values would make sense here?
Hi Conny!
I have nothing new since then. Capping at 4 queues still looks like a reasonable compromise to me. @Mimu: anything new since then?
If you like I can spin a new version (we need compat handling now).
Halil
Quoted 6 lines · On Tue, 15 Dec 2020 12:33:34 +0100 Halil Pasic <[email protected]> wrote:
Quoted 2 lines · On Tue, 15 Dec 2020 09:26:56 +0100 Cornelia Huck <[email protected]> wrote:
Do we have a better idea now about which values would make sense here?
Quoted 9 lines
Hi Conny!
I have nothing new since then. Capping at 4 queues still looks like a reasonable compromise to me. @Mimu: anything new since then?
If you like I can spin a new version (we need compat handling now).
Halil
If your tests show that 4 queues are a reasonable value, this is fine by me (with an explanatory comment in the code.)
Feel free to respin.
Quoted 19 lines · Halil Pasic
Quoted 12 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) · line 28
Quoted 9 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static void virtio_ccw_blk_realize(VirtioCcwDevice *ccw_dev, Error **errp) {VirtIOBlkCcw *dev =VIRTIO_BLK_CCW (ccw_dev);DeviceState *vdev =DEVICE (&dev->vdev); +VirtIOBlkConf *conf = &dev->vdev.conf; + +if (conf->num_queues ==VIRTIO_BLK_AUTO_NUM_QUEUES ) { + conf->num_queues =MIN (4 , current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Quoted 7 lines
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
I'm talking with the perf team tomorrow. They have done some measurements with multiqueue for PV guests and I asked for a comparison to non PV guests as well.
Michael
Quoted 8 lines
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Regards,
Halil
Signature · 8 lines
Mit freundlichen Grüßen / Kind regards Michael Müller IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen Registergericht: Amtsgericht Stuttgart, HRB 243294
Quoted 27 lines · Michael Mueller
Quoted 24 lines · Halil Pasic
Quoted 13 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void · line 29
Quoted 10 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static voidvirtio_ccw_blk_realize (VirtioCcwDevice *ccw_dev, Error **errp) { VirtIOBlkCcw *dev = VIRTIO_BLK_CCW(ccw_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = MIN(4, current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 5 lines
Quoted 3 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
I'm talking with the perf team tomorrow. They have done some measurements with multiqueue for PV guests and I asked for a comparison to non PV guests as well.
The perf team has performed measurements for us that show that a *PV KVM guest* benefits in terms of throughput for random read, random write and sequential read (no difference for sequential write) by a multi queue setup. CPU cost are reduced as well due to reduced spinlock contention.
For a *standard KVM guest* it currently has no throughput effect. No benefit and no harm. I have asked them to finalize their measurements by comparing the CPU cost as well. I will receive that information on Friday.
Michael
Quoted 12 lines
Michael
Quoted 8 lines
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Regards,
Halil
Quoted 36 lines · On Wed, 11 Nov 2020 13:26:11 +0100 Michael Mueller <[email protected]> wrote:
Quoted 27 lines · Michael Mueller
Quoted 24 lines · Halil Pasic
Quoted 13 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void · line 29
Quoted 10 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static voidvirtio_ccw_blk_realize (VirtioCcwDevice *ccw_dev, Error **errp) { VirtIOBlkCcw *dev = VIRTIO_BLK_CCW(ccw_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = MIN(4, current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Quoted 3 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
I'm talking with the perf team tomorrow. They have done some measurements with multiqueue for PV guests and I asked for a comparison to non PV guests as well.
Quoted 5 lines
The perf team has performed measurements for us that show that a *PV KVM guest* benefits in terms of throughput for random read, random write and sequential read (no difference for sequential write) by a multi queue setup. CPU cost are reduced as well due to reduced spinlock contention.
Just to be clear, that was with 4 queues?
Quoted 5 lines
For a *standard KVM guest* it currently has no throughput effect. No benefit and no harm. I have asked them to finalize their measurements by comparing the CPU cost as well. I will receive that information on Friday.
Thank you for checking!
Quoted 18 lines
Michael
Quoted 12 lines
Michael
Quoted 8 lines
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Regards,
Halil
Quoted 39 lines · Cornelia Huck
Quoted 36 lines · On Wed, 11 Nov 2020 13:26:11 +0100 Michael Mueller <[email protected]> wrote:
Quoted 34 lines · Michael Mueller
Quoted 20 lines · Halil Pasic
Quoted 13 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void · line 29
Quoted 10 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static voidvirtio_ccw_blk_realize (VirtioCcwDevice *ccw_dev, Error **errp) { VirtIOBlkCcw *dev = VIRTIO_BLK_CCW(ccw_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = MIN(4, current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Quoted 7 lines
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
I'm talking with the perf team tomorrow. They have done some measurements with multiqueue for PV guests and I asked for a comparison to non PV guests as well.
Quoted 7 lines
Quoted 5 lines
The perf team has performed measurements for us that show that a *PV KVM guest* benefits in terms of throughput for random read, random write and sequential read (no difference for sequential write) by a multi queue setup. CPU cost are reduced as well due to reduced spinlock contention.
Just to be clear, that was with 4 queues?
Yes, we have seen it with 4 and also with 9 queues.
Halil,
still I would like to know what the exact memory consumption per queue is that you are talking about. Have you made a calculation? Thanks.
Quoted 28 lines
Quoted 6 lines
For a *standard KVM guest* it currently has no throughput effect. No benefit and no harm. I have asked them to finalize their measurements by comparing the CPU cost as well. I will receive that information on Friday.
Thank you for checking!
Quoted 18 lines
Michael
Quoted 12 lines
Michael
Quoted 8 lines
The number 4 was suggested by Christian, maybe Christian does have some readily available measurement data for the normal VM case. @Christian:
can you help me out?Regards,
Halil
Quoted 4 lines · On Wed, 11 Nov 2020 13:49:08 +0100 Michael Mueller <[email protected]> wrote:
Halil,
still I would like to know what the exact memory consumption per queue is that you are talking about. Have you made a calculation? Thanks.
Hi!
The default size for virtio-blk seems to be 256 ring entries, which translates to 6668 bytes for the split ring(s). The queue size is user configurable, and guest, in theory, can decide to have a smaller queue. The indirect descriptors are allocated separately, and bounced via swiotlb in case of secure guests.
Regards,
Halil
Quoted 57 lines · On Wed, 11 Nov 2020 13:38:15 +0100 Cornelia Huck <[email protected]> wrote:
Tags: inv, me
From: Cornelia Huck <[email protected]>
To: Michael Mueller <[email protected]>
Cc: Thomas Huth <[email protected]>, David Hildenbrand <[email protected]>, "Michael S. Tsirkin" <[email protected]>, [email protected], Halil Pasic <[email protected]>, Christian Borntraeger <[email protected]>, [email protected]
Subject: Re: [PATCH 1/1] virtio-blk-ccw: tweak the default fornum_queues
Date: Wed, 11 Nov 2020 13:38:15 +0100
Sender: "Qemu-devel" <[email protected]>
Organization: Red Hat GmbHQuoted 36 lines · On Wed, 11 Nov 2020 13:26:11 +0100 Michael Mueller <[email protected]> wrote:
Quoted 27 lines · Michael Mueller
Quoted 24 lines · Halil Pasic
Quoted 13 lines · On Mon, 9 Nov 2020 17:06:16 +0100 Cornelia Huck <[email protected]> wrote:
hw/s390x/virtio-ccw-blk.c static void · line 29
Quoted 10 lines
hw/s390x/virtio-ccw-blk.c +5
-0@@ -20,6 +21,11 @@ static voidvirtio_ccw_blk_realize (VirtioCcwDevice *ccw_dev, Error **errp) { VirtIOBlkCcw *dev = VIRTIO_BLK_CCW(ccw_dev); DeviceState *vdev = DEVICE(&dev->vdev); + VirtIOBlkConf *conf = &dev->vdev.conf; + + if (conf->num_queues == VIRTIO_BLK_AUTO_NUM_QUEUES) { + conf->num_queues = MIN(4, current_machine->smp.cpus); + }I would like to have a comment explaining the numbers here, however.
Quoted 3 lines
virtio-pci has a pretty good explanation (use 1:1 for vqs:vcpus if possible, apply some other capping). 4 seems to be a bit arbitrary without explanation, although I'm sure you did some measurements :)
Frankly, I don't have any measurements yet. For the secure case, I think Mimu has assessed the impact of multiqueue, hence adding Mimu to the cc list. @Mimu can you help us out.
Quoted 7 lines
Quoted 3 lines
Regarding the normal non-protected VMs I'm in a middle of producing some measurement data. This was admittedly a bit rushed because of where we are in the cycle. Sorry to disappoint you.
I'm talking with the perf team tomorrow. They have done some measurements with multiqueue for PV guests and I asked for a comparison to non PV guests as well.
Quoted 5 lines
The perf team has performed measurements for us that show that a *PV KVM guest* benefits in terms of throughput for random read, random write and sequential read (no difference for sequential write) by a multi queue setup. CPU cost are reduced as well due to reduced spinlock contention.
Just to be clear, that was with 4 queues?
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
For a *standard KVM guest* it currently has no throughput effect. No benefit and no harm. I have asked them to finalize their measurements by comparing the CPU cost as well. I will receive that information on Friday.
Thank you for checking!
The results of my measurements (normal case only) are consistent with these findings.
My setup looks like this: A guest with 6 vcpus and an attached virtio-blk-ccw disk backed by a raw image on a tmpfs (i.e. backed by ram, because we are interested in virtio and not in the scsi disk). The performance was evaluated with fio, randrw, queuedepth=1 and bs=1M. I scaled the number of virtqueues from 1 to 5, and collected 30 data points each.
The full fio command line I used is at the end of this mail.
For a nicer table, please see the attached png. Regarding the difference in averages, it's little about 1,2 percent.
The percentages are with respect to average over
all queues values.
queues
1 Average - write_iops 99.45%
Average - write_bw 99.45% Average - read_iops 99.44% Average - read_bw 99.44% 2 Average - write_iops 99.93% Average - write_bw 99.93% Average - read_iops 99.92% Average - read_bw 99.92% 3 Average - write_iops 100.02% Average - write_bw 100.02% Average - read_iops 100.02% Average - read_bw 100.02% 4 Average - write_iops 100.64% Average - write_bw 100.64% Average - read_iops 100.64% Average - read_bw 100.64% 5 Average - write_iops 99.97% Average - write_bw 99.97% Average - read_iops 99.97% Average - read_bw 99.97%
Total Average - write_iops 100.00%
Total Average - write_bw 100.00%
Total Average - read_iops 100.00%
Total Average - read_bw 100.00%
fio --ramp_time=30s --output-format=json --bs=1M --ioengine=libaio --readwrite=randrw --runtime=120s --size=1948m --name=measurement --gtod_reduce=1 --direct=1 --iodepth=1 --filename=/dev/vda --time_based