doc: clarify virtio PMD path selection
PATCH v3 1/1 doc: clarify virtio PMD path selection
From: Wang Yinan <[email protected]>
add virtio paths selection and usage introduction for better virtio usability.
--- doc/guides/howto/index.rst | 1 + .../virtio_paths_selection_and_usage.rst | 142 ++++++++++++++++++ 2 files changed, 143 insertions(+) create mode 100644 doc/guides/howto/virtio_paths_selection_and_usage.rst
doc/guides/howto/index.rst +1 -0
@@ -16,6 +16,7 @@ HowTo Guides vfd virtio_user_for_container_networking virtio_user_as_exceptional_path + virtio_paths_selection_and_usage packet_capture_framework telemetry debug_troubleshoot
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +142 -0
142 lines not on this page · raw message
Signature · 1 line
2.17.1
Hi, yinan
Minor nit, please use '--in-reply-to' when using git-send-email, it helps tracking all history about this patch together.
Thanks,
Xiaolong
Quoted 175 lines · Yinan
From: Wang Yinan <[email protected]>
add virtio paths selection and usage introduction for better virtio usability.
Signed-off-by Wang Yinan <[email protected]>---
doc/guides/howto/index.rst| 1 +.../virtio_paths_selection_and_usage.rst| 142 ++++++++++++++++++2 files changed, 143 insertions(+)create mode 100644
doc/guides/howto/virtio_paths_selection_and_usage.rstdoc/guides/howto/index.rst +1
-0@@ -16,6 +16,7 @@ HowTo Guides vfd virtio_user_for_container_networking virtio_user_as_exceptional_path + virtio_paths_selection_and_usage packet_capture_framework telemetry debug_troubleshootdoc/guides/howto/virtio_paths_selection_and_usage.rst new file +142
-0@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation. + +Virtio paths Selection and Usage +================================ + +Logically virtio-PMD has 9 paths based on the combination of virtio features +(Rx mergeable, In-order, Packed virtqueue), below is an introduction of virtio +common features: + +* `Rx mergeable <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-700009>`_: With this feature negotiated, device + can receive larger packets by combining individual descriptors. +* `In-order <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-690008>`_: Some devices always use descriptors + in the same order in which they have been made available, these + devices can offer the VIRTIO_F_IN_ORDER feature. If this feature negotiated, + driver will use descriptors in order. Meanwhile, this knowledge allows device + operate used ring in batches and driver operate available ring in batches and + such can decrease cache miss rate. +* `Packed virtqueue <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-610007>`_: The structure of packed virtqueue is + different from split virtqueue, split virtqueue is composed of available ring, + used ring and descriptor table, while packed virtqueue is composed of descriptor + ring, driver event suppression and device event suppression. The idea behind + this is to improve performance by avoiding cache misses and and make it easier + for devices to implement. + +Virtio paths Selection +---------------------- + +If packed virtqueue is not negotiated, below split virtqueue paths can be selected +according to below configuration: + +#. Split virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature is + not negotiated, this path will be selected. +#. Split virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, also Rx offload(s) are requested, this path can be selected. +#. Split virtqueue in-order mergeable path: If in-order feature and Rx mergeable are + both negotiated, this path can be selected. +#. Split virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path can be selected. +#. Split virtqueue vectorized RX path: If Rx mergeable is disabled and no Rx offload + requested, this path can be selected. + +If packed virtqueue is negotiated, below packed virtqueue paths can be selected +according to below configuration: + +#. Packed virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature + is not negotiated, this path will be selected. +#. Packed virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, also Rx offload(s) are requested, this path will be selected. +#. Packed virtqueue in-order mergeable path: If in-order feature and Rx mergeable are + both negotiated, this path will be selected. +#. Packed virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path will be selected. + +Rx/Tx callbacks of each Virtio path +----------------------------------- + +Refer to above descriptions, virtio path and Rx/TX callbacks are auto selected by +different parameters of vdev and workloads. Rx callbacks and Tx callbacks name for +each Virtio Path are shown in following tables:: + + +----------------------------------------------------------------------------------------------------------+ + | Virtio path | Rx callbacks | TX callbacks | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue mergeable path |virtio_recv_mergeable_pkts | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue non-mergeable path | virtio_recv_pkts | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order mergeable path | virtio_recv_pkts_inorder | virtio_xmit_pkts_inorder| + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order non-mergeable path | virtio_recv_pkts_inorder | virtio_xmit_pkts_inorder| + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue vectorized RX path | virtio_recv_pkts_vec | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue mergeable path | virtio_recv_mergeable_pkts_packed| virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue normal path | virtio_recv_pkts_packed | virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order mergeable path | virtio_recv_mergeable_pkts_packed| virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order normal path | virtio_recv_pkts_packed | virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + +Virtio paths Support Status from Release to Release +--------------------------------------------------- + +Virtio feature implementation: + +* In-order feature implemented in DPDK 18.08 by adding new Rx/TX callbacks + ``virtio_recv_pkts_inorder`` and ``virtio_xmit_pkts_inorder``. +* Packed virtqueue implemented in DPDK 19.02 by adding new Rx/TX callbacks + ``virtio_recv_pkts_packed`` , ``virtio_recv_mergeable_pkts_packed`` and + ``virtio_xmit_pkts_packed``. + +Virtio path number changes from release to release, all virtio paths support +status are shown in below table:: + + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Virtio path\ DPDK version | v16.11 | v17.02 | v17.05 | v17.08 | v17.11 | v18.02 | v18.05 | v18.08 | v18.11 | v19.02 | v19.05 | v19.08 | + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue mergebale path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue non-mergeable path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue vectorized RX path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue simple TX path | Y | Y | Y | Y | Y | Y | Y | N | N | N | N | N | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order non-mergeable path | | | | | | | | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order mergeable path | | | | | | | | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue mergeable path | | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue non-mergeable path | | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order mergeable path | | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order non-mergeable path| | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + +QEMU Support Status +------------------- + +* Qemu now support three paths of split virtqueue: Split virtqueue mergebale path, + Split virtqueue no-mergeable path, Split virtqueue vectorized RX path. +* Since qemu 4.2.0, Packed virtqueue mergeable path and Packed virtqueue non-mergeable + path can be supported. + +How to Debug +------------ + +If you meet performance drop or some other issues after upgrading the driver +or configuration, below steps can help you identify which path you selected and +root cause faster. + +#. Run vhost/virtio test case; +#. Run "perf top" and check virtio Rx/tx callback names; +#. Identify which virtio path is selected refer to above table.Signature · 1 line
2.17.1
Hi Yinan,
Thanks for the work! Will take a closer look ASAP. Please see some quick comments inline.
Quoted 7 lines · Yinan
From: Wang Yinan <[email protected]>
add virtio paths selection and usage introduction for better virtio usability.
Signed-off-by Wang Yinan <[email protected]>---
When sending new versions, please use --in-reply-to [1]. And it would be better if you can add a brief changelog.
[1] https://core.dpdk.org/contribute/#send
Quoted 25 lines
doc/guides/howto/index.rst | 1 + .../virtio_paths_selection_and_usage.rst | 142 ++++++++++++++++++ 2 files changed, 143 insertions(+) create mode 100644 doc/guides/howto/virtio_paths_selection_and_usage.rstdoc/guides/howto/index.rst +1
-0@@ -16,6 +16,7 @@ HowTo Guides vfd virtio_user_for_container_networking virtio_user_as_exceptional_path + virtio_paths_selection_and_usage packet_capture_framework telemetry debug_troubleshootdoc/guides/howto/virtio_paths_selection_and_usage.rst new file +2
-0@@ -0,0 +1,142 @@ +.. SPDX-License-Identifier: BSD-3-Clause + Copyright(c) 2019 Intel Corporation.
doc/guides/howto/virtio_paths_selection_and_usage.rst line 12
Quoted 10 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +10
-0+ +Virtio paths Selection and Usage +================================ + +Logically virtio-PMD has 9 paths based on the combination of virtio features +(Rx mergeable, In-order, Packed virtqueue), below is an introduction of virtio +common features: + +* `Rx mergeable <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-700009>`_: With this feature negotiated, device
Instead of referring to "Multi-buffer requests", below section is more related:
https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-2140004
Quoted 16 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +16
-0+ can receive larger packets by combining individual descriptors. +* `In-order <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-690008>`_: Some devices always use descriptors + in the same order in which they have been made available, these + devices can offer the VIRTIO_F_IN_ORDER feature. If this feature negotiated, + driver will use descriptors in order. Meanwhile, this knowledge allows device + operate used ring in batches and driver operate available ring in batches and + such can decrease cache miss rate. +* `Packed virtqueue <https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/ + virtio-v1.1-cs01.html#x1-610007>`_: The structure of packed virtqueue is + different from split virtqueue, split virtqueue is composed of available ring, + used ring and descriptor table, while packed virtqueue is composed of descriptor + ring, driver event suppression and device event suppression. The idea behind + this is to improve performance by avoiding cache misses and and make it easier + for devices to implement. +
doc/guides/howto/virtio_paths_selection_and_usage.rst line 38
Quoted 10 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +10
-0+Virtio paths Selection +---------------------- + +If packed virtqueue is not negotiated, below split virtqueue paths can be selected +according to below configuration: + +#. Split virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature is + not negotiated, this path will be selected. +#. Split virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, also Rx offload(s) are requested, this path can be selected.
s/can be/will be/
This applies to other similar cases as well.
Quoted 4 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +4
-0+#. Split virtqueue in-order mergeable path: If in-order feature and Rx mergeable are + both negotiated, this path can be selected. +#. Split virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path can be selected.
doc/guides/howto/virtio_paths_selection_and_usage.rst line 52
Quoted 10 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +10
-0+#. Split virtqueue vectorized RX path: If Rx mergeable is disabled and no Rx offload + requested, this path can be selected. + +If packed virtqueue is negotiated, below packed virtqueue paths can be selected +according to below configuration: + +#. Packed virtqueue mergeable path: If Rx mergeable is negotiated, in-order feature + is not negotiated, this path will be selected. +#. Packed virtqueue non-mergeable path: If Rx mergeable and in-order feature are not + negotiated, also Rx offload(s) are requested, this path will be selected.
Currently, requesting Rx offload or not doesn't matter in this case.
Quoted 42 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +42
-0+#. Packed virtqueue in-order mergeable path: If in-order feature and Rx mergeable are + both negotiated, this path will be selected. +#. Packed virtqueue in-order non-mergeable path: If in-order feature is negotiated and + Rx mergeable is not negotiated, this path will be selected. + +Rx/Tx callbacks of each Virtio path +----------------------------------- + +Refer to above descriptions, virtio path and Rx/TX callbacks are auto selected by +different parameters of vdev and workloads. Rx callbacks and Tx callbacks name for +each Virtio Path are shown in following tables:: + + +----------------------------------------------------------------------------------------------------------+ + | Virtio path | Rx callbacks | TX callbacks | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue mergeable path |virtio_recv_mergeable_pkts | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue non-mergeable path | virtio_recv_pkts | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order mergeable path | virtio_recv_pkts_inorder | virtio_xmit_pkts_inorder| + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order non-mergeable path | virtio_recv_pkts_inorder | virtio_xmit_pkts_inorder| + +----------------------------------------------------------------------------------------------------------+ + |Split virtqueue vectorized RX path | virtio_recv_pkts_vec | virtio_xmit_pkts | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue mergeable path | virtio_recv_mergeable_pkts_packed| virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue normal path | virtio_recv_pkts_packed | virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order mergeable path | virtio_recv_mergeable_pkts_packed| virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order normal path | virtio_recv_pkts_packed | virtio_xmit_pkts_packed | + +----------------------------------------------------------------------------------------------------------+ + +Virtio paths Support Status from Release to Release +--------------------------------------------------- + +Virtio feature implementation: + +* In-order feature implemented in DPDK 18.08 by adding new Rx/TX callbacks + ``virtio_recv_pkts_inorder`` and ``virtio_xmit_pkts_inorder``. +* Packed virtqueue implemented in DPDK 19.02 by adding new Rx/TX callbacks
doc/guides/howto/virtio_paths_selection_and_usage.rst line 104
Quoted 10 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +10
-0+ ``virtio_recv_pkts_packed`` , ``virtio_recv_mergeable_pkts_packed`` and + ``virtio_xmit_pkts_packed``. + +Virtio path number changes from release to release, all virtio paths support +status are shown in below table:: + + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Virtio path\ DPDK version | v16.11 | v17.02 | v17.05 | v17.08 | v17.11 | v18.02 | v18.05 | v18.08 | v18.11 | v19.02 | v19.05 | v19.08 | + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue mergebale path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y |
s/mergebale/mergeable/
Quoted 14 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +14
-0+ +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue non-mergeable path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | + +--------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue vectorized RX path | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue simple TX path | Y | Y | Y | Y | Y | Y | Y | N | N | N | N | N | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order non-mergeable path | | | | | | | | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Split virtqueue in-order mergeable path | | | | | | | | Y | Y | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue mergeable path | | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue non-mergeable path | | | | | | | | | | Y | Y | Y |
doc/guides/howto/virtio_paths_selection_and_usage.rst line 128
Quoted 10 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +10
-0+ ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order mergeable path | | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + |Packed virtqueue in-order non-mergeable path| | | | | | | | | | Y | Y | Y | + ---------------------------------------------------------------------------------------------------------------------------------------------------------+ + +QEMU Support Status +------------------- + +* Qemu now support three paths of split virtqueue: Split virtqueue mergebale path,
s/support/supports/
s/mergebale/mergeable/
doc/guides/howto/virtio_paths_selection_and_usage.rst line 129
Quoted 1 line
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +1
-0+ Split virtqueue no-mergeable path, Split virtqueue vectorized RX path.
s/no-mergeable/non-mergeable/
Quoted 16 lines
doc/guides/howto/virtio_paths_selection_and_usage.rst new file +13
-0+* Since qemu 4.2.0, Packed virtqueue mergeable path and Packed virtqueue non-mergeable + path can be supported. + +How to Debug +------------ + +If you meet performance drop or some other issues after upgrading the driver +or configuration, below steps can help you identify which path you selected and +root cause faster. + +#. Run vhost/virtio test case; +#. Run "perf top" and check virtio Rx/tx callback names; +#. Identify which virtio path is selected refer to above table.Signature · 1 line
2.17.1