Skip to content

RFC Implement multiqueue virtio-net

Krishna Kumar kvm netdev 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 RFC Latest4 patches 30 repliesIn review
  2. v2 RFC4 patches 12 repliesSuperseded
  3. v1 RFC4 patches 38 repliesSuperseded

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4Change virtqueue structuredrivers/virtio/virtio_pci.c include/linux/virtio.h+4 -70 replies
2/4Changes for virtio-netdrivers/net/virtio_net.c include/linux/virtio_net.h+185 -550 replies
3/4Changes for vhostdrivers/vhost/net.c drivers/vhost/vhost.c drivers/vhost/vhost.h+430 -1710 replies
4/4qemu changeshw/vhost.c hw/vhost.h hw/vhost_net.c +7 more+107 -290 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v3 yet.

Read the v3 thread

Cover letter [v3 RFC PATCH 0/4] Implement multiqueue virtio-net

Following set of patches implement transmit MQ in virtio-net. Also included is the user qemu changes. MQ is disabled by default unless qemu specifies it.

Changes from rev2:
------------------
1. Define (in virtio_net.h) the maximum send txqs; and use in virtio-net and vhost-net.
2. vi->sq[i] is allocated individually, resulting in cache line aligned sq[0] to sq[n]. Another option was to define 'send_queue' as:

       struct send_queue {
               struct virtqueue *svq;
               struct scatterlist tx_sg[MAX_SKB_FRAGS + 2];
       } ____cacheline_aligned_in_smp;

and to statically allocate 'VIRTIO_MAX_SQ' of those. I hope the submitted method is preferable.
3. Changed vhost model such that vhost[0] handles RX and vhost[1-MAX] handles TX[0-n].
4. Further change TX handling such that vhost[0] handles both RX/TX for single stream case.

Enabling MQ on virtio:
-----------------------
When following options are passed to qemu:
- smp > 1
- vhost=on
- mq=on (new option, default:off)
then #txqueues = #cpus. The #txqueues can be changed by using an optional 'numtxqs' option. e.g. for a smp=4 guest:

        vhost=on                   ->   #txqueues = 1
        vhost=on,mq=on             ->   #txqueues = 4
        vhost=on,mq=on,numtxqs=2   ->   #txqueues = 2
        vhost=on,mq=on,numtxqs=8   ->   #txqueues = 8
                   Performance (guest -> local host):
                   -----------------------------------

System configuration:
Host: 8 Intel Xeon, 8 GB memory
Guest: 4 cpus, 2 GB memory
Test: Each test case runs for 60 secs, sum over three runs (except when number of netperf sessions is 1, which has 10 runs of 12 secs each). No tuning (default netperf) other than taskset vhost's to cpus 0-3. numtxqs=32 gave the best results though the guest had only 4 vcpus (I haven't tried beyond that).

______________ numtxqs=2, vhosts=3 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1          4.46    -1.96     .19     -12.50   -6.06
2          4.93    -1.16    2.10      0       -2.38
4          46.17    64.77   33.72     19.51   -2.48
8          47.89    70.00   36.23     41.46    13.35
16         48.97    80.44   40.67     21.11   -5.46
24         49.03    78.78   41.22     20.51   -4.78
32         51.11    77.15   42.42     15.81   -6.87
40         51.60    71.65   42.43     9.75    -8.94
48         50.10    69.55   42.85     11.80   -5.81
64         46.24    68.42   42.67     14.18   -3.28
80         46.37    63.13   41.62     7.43    -6.73
96         46.40    63.31   42.20     9.36    -4.78
128        50.43    62.79   42.16     13.11   -1.23

________________________________________________________
BW: 37.2%, CPU/RCPU: 66.3%,41.6%, SD/RSD: 11.5%,-3.7%

______________ numtxqs=8, vhosts=5 ____________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -.76    -1.56     2.33      0        3.03
2           17.41    11.11    11.41     0       -4.76
4           42.12    55.11    30.20     19.51    .62
8           54.69    80.00    39.22     24.39    -3.88
16          54.77    81.62    40.89     20.34    -6.58
24          54.66    79.68    41.57     15.49    -8.99
32          54.92    76.82    41.79     17.59    -5.70
40          51.79    68.56    40.53     15.31    -3.87
48          51.72    66.40    40.84     9.72     -7.13
64          51.11    63.94    41.10     5.93     -8.82
80          46.51    59.50    39.80     9.33     -4.18
96          47.72    57.75    39.84     4.20     -7.62
128         54.35    58.95    40.66     3.24     -8.63

________________________________________________________
BW: 38.9%, CPU/RCPU: 63.0%,40.1%, SD/RSD: 6.0%,-7.4%

______________ numtxqs=16, vhosts=5 ___________________
#sessions BW% CPU% RCPU% SD% RSD%
________________________________________________________

1           -1.43    -3.52    1.55      0          3.03
2           33.09     21.63   20.12    -10.00     -9.52
4           67.17     94.60   44.28     19.51     -11.80
8           75.72     108.14  49.15     25.00     -10.71
16          80.34     101.77  52.94     25.93     -4.49
24          70.84     93.12   43.62     27.63     -5.03
32          69.01     94.16   47.33     29.68     -1.51
40          58.56     63.47   25.91    -3.92      -25.85
48          61.16     74.70   34.88     .89       -22.08
64          54.37     69.09   26.80    -6.68      -30.04
80          36.22     22.73   -2.97    -8.25      -27.23
96          41.51     50.59   13.24     9.84      -16.77
128         48.98     38.15   6.41     -.33       -22.80

________________________________________________________
BW: 46.2%, CPU/RCPU: 55.2%,18.8%, SD/RSD: 1.2%,-22.0%

______________ numtxqs=32, vhosts=5 ___________________
# BW% CPU% RCPU% SD% RSD%
________________________________________________________

1            7.62     -38.03   -26.26  -50.00   -33.33
2            28.95     20.46    21.62   0       -7.14
4            84.05     60.79    45.74  -2.43    -12.42
8            86.43     79.57    50.32   15.85   -3.10
16           88.63     99.48    58.17   9.47    -13.10
24           74.65     80.87    41.99  -1.81    -22.89
32           63.86     59.21    23.58  -18.13   -36.37
40           64.79     60.53    22.23  -15.77   -35.84
48           49.68     26.93    .51    -36.40   -49.61
64           54.69     36.50    5.41   -26.59   -43.23
80           45.06     12.72   -13.25  -37.79   -52.08
96           40.21    -3.16    -24.53  -39.92   -52.97
128          36.33    -33.19   -43.66  -5.68    -20.49

________________________________________________________
BW: 49.3%, CPU/RCPU: 15.5%,-8.2%, SD/RSD: -22.2%,-37.0%

---