virtio-net: define UDP tunnel offload
v6 cover virtio-net: define UDP tunnel offload
UDP tunnel usage is ubiquitous in container deployment, and the ability to offload UDP encapsulated GSO traffic impacts greatly the performances and the CPU utilization of such use cases.
This series introduces separate features to handle both UDP tunnel segmentation offload (patch 1) and UDP tunnel outer checksum offload (patch 2)
Changes from v5:
- split in 2 patches
- dropped outer_mh_offset field
- many csum related clarification
Paolo Abeni (2):
virtio-net: define UDP tunnel segmentation offload feature
virtio-net: define UDP tunnel checksum offload feature
device-types/net/description.tex | 227 +++++++++++++++++++++++++++++-- 1 file changed, 218 insertions(+), 9 deletions(-)
Signature · 1 line
2.45.2
PATCH v6 1/2 virtio-net: define UDP tunnel segmentation offload feature
The VIRTIO_NET_HDR_GSO_UDP_TUNNEL is a gso_type flag allowing GSO over UDP tunnel. It can be negotiated on both the host and guest sides.
One constraint addressed here is that the virtio side (either device or driver) receiving a UDP tunneled GSO packet must be able to reconstruct completely the inner and outer headers offset - to allow for later GSO.
To accommodate such need, new fields are introduced in the virtio_net
header: outer_th_offset, inner_protocol_type and inner_nh_offset. They map directly to the corresponding header information. The inner transport header is implied by the (inner) checksum offload.
Those fields are required because a virtio device H/W implementation performing segmentation for UDP tunneled packet will need to touch the outer transport protocol (for the UDP length filed), the inner network protocol (for the total length field, in the IPv4 case).
Note that segmentation will additionally need to touch
the outer network protocol and the inner transport protocol. The first is implied/easily found with trivial parsing, the latter is identified by the existing csum_start field.
Note that there is no concept of UDP tunnel type negotiation (e.g. vxlan, geneve, vxlan-gpe, etc.), as a virtio device H/W implementation can perform segmentation for every possible UDP-tunnel given the specified new fields.
In the reverse direction, if a virtio device H/W implementation receives some traffic for an unknown or unsupported UDP tunnel, it will simply not aggregate the wire packet in GSO one.
---
v5 -> v6:
- split in 2 patches
- dropped the unneeded outer_mh_offset field
https://lore.kernel.org/virtio-comment/56ccba136272593d0d2af0303600c06f26bd84a1.1718621522.git.pabeni@redhat.com/
v4 -> v5:
- add separate negotiation for UDP_TUNNEL_CSUM
- much more verbose specification of outer csum handling
- UDP_TUNNEL_CSUM requires GSO_UDP_TUNNEL
https://lore.kernel.org/virtio-comment/cb89d3d6b6a7e4a7f13e60dd3fded5e950727890.1716448766.git.pabeni@redhat.com/
v3 -> v4:
- new virtio_net_hdr fields depends on F_HOST_UDP_TUNNEL_GSO or F_GUEST_UDP_TUNNEL_GSO (Stefano)
- Extended the changelog to answer Jason's questions.
- Clarified outer csum handling (Jason)
https://lore.kernel.org/virtio-dev/f0bf2db203f8061a3ecb50b7a48cf26d8f3c7f68.1716193086.git.pabeni@redhat.com
v2 -> v3:
- UDP_TUNNEL -> UDP_TUNNEL_GSO
- add explicit fields for the inner meta-data
- more verbose changelog
https://lists.oasis-open.org/archives/virtio-dev/202206/msg00026.html
v1 -> v2:
- explicitly state that the outer header probing is mandatory
- explicitly state that GSO_UDP is not allowed with GSO_UDP_TUNNEL
- clarify hdr_len usage
- clarify UDP_TUNNEL_CSUM bit usage
- fix a few typos
https://lists.oasis-open.org/archives/virtio-dev/202205/msg00037.html
--- device-types/net/description.tex | 118 +++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 6 deletions(-)
device-types/net/description.tex +112 -6
@@ -88,6 +88,12 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO (47)] Driver can receive GSO packets + carried by a UDP tunnel. + +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO (49)] Device can receive GSO packets + carried by a UDP tunnel. + \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets. \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
@@ -133,12 +139,16 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_UFO] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO4] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO6] Requires VIRTIO_NET_F_GUEST_CSUM. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_ECN] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_HOST_UFO] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_USO] Requires VIRTIO_NET_F_CSUM. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.
@@ -375,6 +385,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features. +\item If any TCP or UDP segmentation feature is negotiated, a driver can + use TCP segmentation or UDP segmentation on top of UDP encapsulation + offload, when the outer header does not require checksumming - e.g. + the outer UDP checksum is zero - by negotiating the + VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature. + \item The converse features are also available: a driver can save the virtual device some work by negotiating these features.\note{For example, a network packet transported between two guests on the same system might not need checksumming at all, nor segmentation,
@@ -382,8 +398,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev The VIRTIO_NET_F_GUEST_CSUM feature indicates that partially checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, - VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4 - and VIRTIO_NET_F_GUEST_USO6 are the input equivalents of the features described above. + VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4, + VIRTIO_NET_F_GUEST_USO6 and VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO + are the input equivalents of the features described above. See \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~\nameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} and
@@ -413,6 +430,7 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O #define VIRTIO_NET_HDR_GSO_UDP 3 #define VIRTIO_NET_HDR_GSO_TCPV6 4 #define VIRTIO_NET_HDR_GSO_UDP_L4 5 +#define VIRTIO_NET_HDR_GSO_UDP_TUNNEL 0x40 #define VIRTIO_NET_HDR_GSO_ECN 0x80 u8 gso_type; le16 hdr_len;
@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) }; \end{lstlisting}
@@ -480,6 +501,8 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De followed by the TCP header (with the TCP checksum field 16 bytes into that header). \field{csum_start} will be 14+20 = 34 (the TCP checksum includes the header), and \field{csum_offset} will be 16. +If the given packet has the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit set, +the above checksum fields refer to the inner header checksum. \end{note} \item If the driver negotiated
@@ -516,6 +539,26 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De specifically in the protocol.}. \end{itemize} +\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero. + The other tunnel-related fields indicate how to replicate the packet + headers to cut it into smaller packets: + + \begin{itemize} + \item \field{outer_th_offset} field indicates the outer transport header within + the packet. Note that such field differs from \field{csum_start} as the latter + points to the inner transport header within the packet. + + \item \field{inner_protocol_type} field indicates the ethernet type of the inner + protocol. + + \item \field{inner_nh_offset} field indicates the inner network header within + the packet. + \end{itemize} + \item \field{num_buffers} is set to zero. This field is unused on transmitted packets. \item The header and packet are added as one output descriptor to the
@@ -557,6 +600,18 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De driver MUST set the VIRTIO_NET_HDR_GSO_ECN bit in \field{gso_type}. +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation offload, unless the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is +negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL +bit in \field{gso_type}. + +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation and outer UDP checksum offload. + +The driver MUST NOT set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL together with +VIRTIO_NET_HDR_GSO_UDP, as the latter is deprecated in favor of UDP_L4 +and no new features will support it. + If the VIRTIO_NET_F_CSUM feature has been negotiated, the driver MAY set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in \field{flags}, if so:
@@ -598,6 +653,28 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De header. \end{itemize} +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has been negotiated, the +driver MAY set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}, if so: +\begin{itemize} +\item the driver MUST set \field{outer_th_offset} to the outer UDP header + offset, \field{inner_protocol_type} to the ethernet type of the inner + protocol, and \field{inner_nh_offset} to the inner network header offset. +\end{itemize} + +If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header. + +\begin{note} +The valid complete checksum of the outer UDP header of individual segments +can be computed by the driver prior to segmentation only if the GSO packet +size is a multiple of \field{gso_size}. +\end{note} + +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has not +been negotiated, the driver MUST NOT set the VIRTIO_NET_HDR_F_UDP_TUNNEL bit +in \field{gso_type}. + The driver SHOULD accept the VIRTIO_NET_F_GUEST_HDRLEN feature if it has been offered, and if it's able to provide the exact header length.
@@ -633,6 +710,10 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De \end{note} \end{itemize} +If VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} is not set, the +device MUST NOT use the \field{outer_th_offset}, \field{inner_protocol_type} + and \field{inner_nh_offset}. + If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT rely on the packet checksum being correct. \paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt}
@@ -727,8 +808,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network has been validated. \end{enumerate} -Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN -features enable receive checksum, large receive offload and ECN +Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP, UDP_TUNNEL +and ECN features enable receive checksum, large receive offload and ECN support which are the input equivalents of the transmit checksum, transmit segmentation offloading and ECN features, as described in \ref{sec:Device Types / Network Device / Device Operation /
@@ -750,8 +831,14 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network from \field{csum_start} and any preceding checksums have been validated. The checksum on the packet is incomplete and if bit VIRTIO_NET_HDR_F_RSC_INFO is not set in \field{flags}, - then \field{csum_start} and \field{csum_offset} indicate how to calculate it - (see Packet Transmission point 1). + then \field{csum_start} and \field{csum_offset} indicate how to calculate it. + If the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit is set, the \field{csum_start} field + refers to the inner transport header offset (see Packet Transmission point 1). +\item If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO option was negotiated and + \field{gso_type} is not VIRTIO_NET_HDR_GSO_NONE, the VIRTIO_NET_HDR_GSO_UDP_TUNNEL + bit MAY be set. In such case the \field{outer_th_offset}, \field{inner_protocol_type}, + and \field{inner_nh_offset} fields indicate the corresponding + headers information. \end{enumerate}
@@ -796,6 +883,9 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If none of VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6 have been negotiated, the device MUST NOT set \field{gso_type} to VIRTIO_NET_HDR_GSO_UDP_L4. +If VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO is not negotiated the device MUST NOT set +the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}. + The device SHOULD NOT send to the driver TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_GUEST_ECN feature is negotiated, in which case the
@@ -819,6 +909,17 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network fully checksummed packet; \end{enumerate} +The device MUST NOT send to the driver TCP or UDP GSO packets encapsulated in UDP +tunnel and requiring segmentation offload, unless the +VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO is negotiated, in which case the device MUST set +the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} and MUST set the +\field{outer_th_offset}, \field{inner_protocol_type} and +\field{inner_nh_offset} fields to the corresponding header information, and the outer +UDP header MUST NOT require checksum offload. + +The device MUST NOT send the driver TCP or UDP GSO packets encapsulated in UDP +tunnel and requiring segmentation and outer checksum offload. + If none of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have been negotiated, the device MUST set \field{gso_type} to VIRTIO_NET_HDR_GSO_NONE.
@@ -867,6 +968,10 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network VIRTIO_NET_HDR_F_DATA_VALID is set, the driver MUST NOT rely on the packet checksum being correct. +If the VIRTIO_NET_HDR_GSO_UDP_TUNNEL_GSO bit in \field{gso_type} is not set, +the driver MUST NOT use the \field{outer_th_offset}, \field{inner_protocol_type}, +\field{inner_mac_offset} and \field{inner_nh_offset}. + \paragraph{Hash calculation for incoming packets} \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets}
@@ -1624,6 +1729,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi #define VIRTIO_NET_F_GUEST_TSO6 8 #define VIRTIO_NET_F_GUEST_ECN 9 #define VIRTIO_NET_F_GUEST_UFO 10 +#define VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO 47 #define VIRTIO_NET_F_GUEST_USO4 54 #define VIRTIO_NET_F_GUEST_USO6 55
Signature · 1 line
2.45.2
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 32 lines · Paolo Abeni
The
VIRTIO_NET_HDR_GSO_UDP_TUNNELis agso_typeflag allowing GSO over UDP tunnel. It can be negotiated on both the host and guest sides.One constraint addressed here is that the virtio side (either device or driver) receiving a UDP tunneled GSO packet must be able to reconstruct completely the inner and outer headers offset - to allow for later GSO.
To accommodate such need, new fields are introduced in the
virtio_net
header:outer_th_offset,inner_protocol_typeandinner_nh_offset. They map directly to the corresponding header information. The inner transport header is implied by the (inner) checksum offload.Those fields are required because a virtio device H/W implementation performing segmentation for UDP tunneled packet will need to touch the outer transport protocol (for the UDP length filed), the inner network protocol (for the total length field, in the IPv4 case).
Note that segmentation will additionally need to touch
the outer network protocol and the inner transport protocol. The first is implied/easily found with trivial parsing, the latter is identified by the existingcsum_startfield.Note that there is no concept of UDP tunnel type negotiation (e.g. vxlan, geneve, vxlan-gpe, etc.), as a virtio device H/W implementation can perform segmentation for every possible UDP-tunnel given the specified new fields.
In the reverse direction, if a virtio device H/W implementation receives some traffic for an unknown or unsupported UDP tunnel, it will simply not aggregate the wire packet in GSO one.
Quoted 1 line
Signed-off-by Paolo Abeni <[email protected]>
Overall this looks good to me. Thanks for splitting the patches.
You have a working software implementation, right? Will this be sent (as RFC) to the public netdev or so mailing list before this becomes a standard? I assume that there is a standard process for virtio that you are following. I'd just be more confident in a thumbs up if we also review the code that implements the text.
device-types/net/description.tex \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 8 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };
In relation to the above: I recall that expanding the virtio_net_hdr was non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 546
Quoted 5 lines
device-types/net/description.tex +5
-0+\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero.
nit: "e.g." should be "i.e.," here
and maybe must be a positive zero (0x0) checksum as defined in UDP RFC 768.
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 666
Quoted 3 lines
device-types/net/description.tex +3
-0+If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header.
MUST NOT require checksum validation. [..] or the validated complete checksum
(nit-picking at this point, all are suggestions, feel free to ignore).
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 672
Quoted 6 lines
device-types/net/description.tex +6
-0+ +\begin{note} +The valid complete checksum of the outer UDP header of individual segments +can be computed by the driver prior to segmentation only if the GSO packet +size is a multiple of \field{gso_size}. +\end{note}
Might be helpful to explain why. E.g., ", because then all segments have the same size and thus all data included in the outer UDP checksum is the same for every segment. These pre-computed segment length and checksum fields are different from those of the GSO packet."
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 36 lines · Willem de Bruijn
Quoted 33 lines · Paolo Abeni
The
VIRTIO_NET_HDR_GSO_UDP_TUNNELis agso_typeflag allowing GSO over UDP tunnel. It can be negotiated on both the host and guest sides.One constraint addressed here is that the virtio side (either device or driver) receiving a UDP tunneled GSO packet must be able to reconstruct completely the inner and outer headers offset - to allow for later GSO.
To accommodate such need, new fields are introduced in the
virtio_net
header:outer_th_offset,inner_protocol_typeandinner_nh_offset. They map directly to the corresponding header information. The inner transport header is implied by the (inner) checksum offload.Those fields are required because a virtio device H/W implementation performing segmentation for UDP tunneled packet will need to touch the outer transport protocol (for the UDP length filed), the inner network protocol (for the total length field, in the IPv4 case).
Note that segmentation will additionally need to touch
the outer network protocol and the inner transport protocol. The first is implied/easily found with trivial parsing, the latter is identified by the existingcsum_startfield.Note that there is no concept of UDP tunnel type negotiation (e.g. vxlan, geneve, vxlan-gpe, etc.), as a virtio device H/W implementation can perform segmentation for every possible UDP-tunnel given the specified new fields.
In the reverse direction, if a virtio device H/W implementation receives some traffic for an unknown or unsupported UDP tunnel, it will simply not aggregate the wire packet in GSO one.Signed-off-by Paolo Abeni <[email protected]>
Quoted 12 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 672
Quoted 6 lines
device-types/net/description.tex +6
-0+ +\begin{note} +The valid complete checksum of the outer UDP header of individual segments +can be computed by the driver prior to segmentation only if the GSO packet +size is a multiple of \field{gso_size}. +\end{note}Might be helpful to explain why. E.g., ", because then all segments have the same size and thus all data included in the outer UDP checksum is the same for every segment. These pre-computed segment length and checksum fields are different from those of the GSO packet."
Does this part belong in patch 2/2?
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 50 lines · Willem de Bruijn
Quoted 36 lines · Willem de Bruijn
Quoted 33 lines · Paolo Abeni
The
VIRTIO_NET_HDR_GSO_UDP_TUNNELis agso_typeflag allowing GSO over UDP tunnel. It can be negotiated on both the host and guest sides.One constraint addressed here is that the virtio side (either device or driver) receiving a UDP tunneled GSO packet must be able to reconstruct completely the inner and outer headers offset - to allow for later GSO.
To accommodate such need, new fields are introduced in the
virtio_net
header:outer_th_offset,inner_protocol_typeandinner_nh_offset. They map directly to the corresponding header information. The inner transport header is implied by the (inner) checksum offload.Those fields are required because a virtio device H/W implementation performing segmentation for UDP tunneled packet will need to touch the outer transport protocol (for the UDP length filed), the inner network protocol (for the total length field, in the IPv4 case).
Note that segmentation will additionally need to touch
the outer network protocol and the inner transport protocol. The first is implied/easily found with trivial parsing, the latter is identified by the existingcsum_startfield.Note that there is no concept of UDP tunnel type negotiation (e.g. vxlan, geneve, vxlan-gpe, etc.), as a virtio device H/W implementation can perform segmentation for every possible UDP-tunnel given the specified new fields.
In the reverse direction, if a virtio device H/W implementation receives some traffic for an unknown or unsupported UDP tunnel, it will simply not aggregate the wire packet in GSO one.Signed-off-by Paolo Abeni <[email protected]>Quoted 12 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 672
Quoted 6 lines
device-types/net/description.tex +6
-0+ +\begin{note} +The valid complete checksum of the outer UDP header of individual segments +can be computed by the driver prior to segmentation only if the GSO packet +size is a multiple of \field{gso_size}. +\end{note}Might be helpful to explain why. E.g., ", because then all segments have the same size and thus all data included in the outer UDP checksum is the same for every segment. These pre-computed segment length and checksum fields are different from those of the GSO packet."
Quoted 1 line
Does this part belong in patch 2/2?
Possibly is a bit border-line.
IMHO it could belong to this patch, too. The idea is to specify one of the scenarios what will not require checksum offload.
Thanks,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Willem de BruijnQuoted 5 lines · Willem de Bruijn
You have a working software implementation, right? Will this be sent (as RFC) to the public netdev or so mailing list before this becomes a standard? I assume that there is a standard process for virtio that you are following. I'd just be more confident in a thumbs up if we also review the code that implements the text.
My plan is to write the S/W virtio_net implementation, too, but it's currently not ready. I had a working PoC for the first draft of this changeset, long time ago. Things are changed enough that I'll need a rewrite. I can try to shared such a thing on netdev ASAP (where "soon" here is a very relative terms ;)
Also I don't know the standardization process. I hope for some guidance from Jason and/or Michael.
Quoted 16 lines
device-types/net/description.tex \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 9 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.
Do you mean, by adding 'padding' or 'unused' fields?
Side note: the above is not 32 bit aligned, do we need such aligment?
If we would have VIRTIO_NET_HDR_GSO_UDPv4_L4 and VIRTIO_NET_HDR_GSO_UDPv6_L4 variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.
Quoted 8 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 546
Quoted 5 lines
device-types/net/description.tex +5
-0+\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero.nit: "e.g." should be "i.e.," here
Quoted 7 lines
and maybe must be a positive zero (
0x0) checksum as defined in UDP RFC 768.device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 666
Quoted 3 lines
device-types/net/description.tex +3
-0+If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header.MUST NOT require checksum validation. [..] or the validated complete checksum
Ack to all the above.
Thanks!
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 8 lines · Paolo Abeni
Quoted 5 lines · Willem de Bruijn
You have a working software implementation, right? Will this be sent (as RFC) to the public netdev or so mailing list before this becomes a standard? I assume that there is a standard process for virtio that you are following. I'd just be more confident in a thumbs up if we also review the code that implements the text.
Quoted 8 lines
My plan is to write the S/W
virtio_netimplementation, too, but it's currently not ready. I had a working PoC for the first draft of this changeset, long time ago. Things are changed enough that I'll need a rewrite. I can try to shared such a thing on netdev ASAP (where "soon" here is a very relative terms ;)Also I don't know the standardization process. I hope for some guidance from Jason and/or Michael.
It should be something like this. When there's no future comment, open an issue https://github.com/oasis-tcs/virtio-spec/issues and ask for a vote. When it passes the voting, it would be merged.
Quoted 18 lines
Quoted 17 lines
device-types/net/description.tex \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 10 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.
Yes, but it looks like an independent issue for this.
Quoted 3 lines
Do you mean, by adding 'padding' or 'unused' fields?
Side note: the above is not 32 bit aligned, do we need such aligment?
Maybe, but it should be an independent topic.
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.
I may miss something, but can it be done by just introducing a new gso_type without vnet header extension?
Quoted 17 lines
Quoted 9 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 546
Quoted 6 lines
device-types/net/description.tex +5
-0+\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero.nit: "e.g." should be "i.e.," here
Quoted 7 lines
and maybe must be a positive zero (
0x0) checksum as defined in UDP RFC 768.device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 666
Quoted 3 lines
device-types/net/description.tex +3
-0+If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header.MUST NOT require checksum validation. [..] or the validated complete checksum
Quoted 6 lines
Ack to all the above.
Thanks!
Paolo
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 10 lines · Jason Wang
Quoted 9 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 8 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };
[...]
Quoted 7 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.I may miss something, but can it be done by just introducing a new
gso_typewithout vnet header extension?
Yep, that is what I mean above: we could replace the
'inner_protocol_type' field - which carries a single bit information, inner ipv4 vs ipv6 - with some (inner) GSO-type related info.
Unfortunately that last step does not look 110% 'clean' to me, as the inner GSO type is already specified by VIRTIO_NET_HDR_GSO_TCPV4, VIRTIO_NET_HDR_GSO_TCPV6 or VIRTIO_NET_HDR_GSO_UDP_L4. In case of inner TCP, the inner network protocol is already specified, but it's not in case of UDP.
I'm wondering: why the single VIRTIO_NET_HDR_GSO_UDP_L4 value has been preferred to a pair of ipv4/ipv6 variants? Could we add such variants now? Note that I would like such option very much, as it will make this change even more complex, but looks IMHO quite clean.
Otherwise another option would be to allocate another bit in the GSO type field, set such field when VIRTIO_NET_HDR_GSO_UDP_TUNNEL is set, too, and use such field specify the inner header network protocol. I don't like much even this option because the new bit will carry duplicate information in case of TCP, and it's implementation looks bug prone.
I don't see other options, any suggestions/preferences welcome!
Thanks,
Paolo
Quoted 27 lines
Quoted 18 lines
Quoted 10 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 546
Quoted 7 lines
+\item If the driver negotiated the
VIRTIO_NET_F_HOST_UDP_TUNNEL_GSOfeature,device-types/net/description.tex +4
-0+ the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero.nit: "e.g." should be "i.e.," here
Quoted 7 lines
and maybe must be a positive zero (
0x0) checksum as defined in UDP RFC 768.device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 666
Quoted 3 lines
device-types/net/description.tex +3
-0+If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header.MUST NOT require checksum validation. [..] or the validated complete checksum
Quoted 6 lines
Ack to all the above.
Thanks!
Paolo
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 32 lines · Paolo Abeni
Quoted 10 lines · Jason Wang
Quoted 9 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 8 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };[...]
Quoted 7 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.I may miss something, but can it be done by just introducing a new
gso_typewithout vnet header extension?Yep, that is what I mean above: we could replace the '
inner_protocol_type' field - which carries a single bit information, inner ipv4 vs ipv6 - with some (inner) GSO-type related info.Unfortunately that last step does not look 110% 'clean' to me, as the inner GSO type is already specified by
VIRTIO_NET_HDR_GSO_TCPV4,VIRTIO_NET_HDR_GSO_TCPV6orVIRTIO_NET_HDR_GSO_UDP_L4. In case of inner TCP, the inner network protocol is already specified, but it's not in case of UDP.
Quoted 4 lines
I'm wondering: why the single
VIRTIO_NET_HDR_GSO_UDP_L4value has been preferred to a pair of ipv4/ipv6 variants? Could we add such variants now? Note that I would like such option very much, as it will make this change even more complex, but looks IMHO quite clean.
After more pondering, I think implementing this option would be quite a mess. E.g. we would need a feature bit for VIRTIO_NET_HDR_GSO_UDP{4,6}_L4 different from VIRTIO_NET_F_{GUEST,HOST}_USO{4,6,}...
I guess we are better off excluding such option.
Quoted 6 lines
Otherwise another option would be to allocate another bit in the GSO type field, set such field when
VIRTIO_NET_HDR_GSO_UDP_TUNNELis set, too, and use such field specify the inner header network protocol. I don't like much even this option because the new bit will carry duplicate information in case of TCP, and it's implementation looks bug prone.
In practice it would replace 'inner_protocol_type' with:
#define VIRTIO_NET_HDR_GSO_INNER_IPv6 0x20
And set/clearing such bit depending on the inner network protocol type, for VIRTIO_NET_HDR_GSO_UDP_TUNNEL packets.
Processing the virtio net hdr will require validating such bit value with WRT the carried GSO type, i.e. must be set when
gso_type & (VIRTIO_NET_HDR_GSO_TCPV6 | VIRTIO_NET_HDR_GSO_UDP_TUNNEL) == VIRTIO_NET_HDR_GSO_TCPV6 | VIRTIO_NET_HDR_GSO_UDP_TUNNEL
must be cleared in all the other cases
Cheers,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Paolo AbeniQuoted 33 lines · Paolo Abeni
Quoted 10 lines · Jason Wang
Quoted 9 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 8 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };[...]
Quoted 7 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.I may miss something, but can it be done by just introducing a new
gso_typewithout vnet header extension?Yep, that is what I mean above: we could replace the '
inner_protocol_type' field - which carries a single bit information, inner ipv4 vs ipv6 - with some (inner) GSO-type related info.Unfortunately that last step does not look 110% 'clean' to me, as the inner GSO type is already specified by
VIRTIO_NET_HDR_GSO_TCPV4,VIRTIO_NET_HDR_GSO_TCPV6orVIRTIO_NET_HDR_GSO_UDP_L4. In case of inner TCP, the inner network protocol is already specified, but it's not in case of UDP.
Quoted 4 lines
I'm wondering: why the single
VIRTIO_NET_HDR_GSO_UDP_L4value has been preferred to a pair of ipv4/ipv6 variants? Could we add such variants now? Note that I would like such option very much, as it will make this change even more complex, but looks IMHO quite clean.
Actually, I point out that when it is proposed. And I was told that it is because Linux doesn't differ that.
Quoted 8 lines
Otherwise another option would be to allocate another bit in the GSO type field, set such field when
VIRTIO_NET_HDR_GSO_UDP_TUNNELis set, too, and use such field specify the inner header network protocol. I don't like much even this option because the new bit will carry duplicate information in case of TCP, and it's implementation looks bug prone.
Quoted 5 lines
I don't see other options, any suggestions/preferences welcome!
Thanks,
Paolo
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Jason WangQuoted 16 lines · Jason Wang
Quoted 14 lines · Paolo Abeni
Quoted 5 lines · Willem de Bruijn
You have a working software implementation, right? Will this be sent (as RFC) to the public netdev or so mailing list before this becomes a standard? I assume that there is a standard process for virtio that you are following. I'd just be more confident in a thumbs up if we also review the code that implements the text.
My plan is to write the S/W
virtio_netimplementation, too, but it's currently not ready. I had a working PoC for the first draft of this changeset, long time ago. Things are changed enough that I'll need a rewrite. I can try to shared such a thing on netdev ASAP (where "soon" here is a very relative terms ;)
Quoted 6 lines
Quoted 2 lines
Also I don't know the standardization process. I hope for some guidance from Jason and/or Michael.
It should be something like this. When there's no future comment, open an issue https://github.com/oasis-tcs/virtio-spec/issues and ask for a vote. When it passes the voting, it would be merged.
So changes to the spec can be merged, because we know that they can be implemented reasonably?
IETF always requires (two, because interoperable protocol definitions) working implementations.
It sounds a bit risky to me to skip this, as you come across the interesting non-obvious details during implementation.
Since implementation is planned anyway, doing at least a rough proof of concept first does not introduce extra work, but reduces risk that the spec contains errors.
Unless it is easy to update the spec before it is numbered and finalized. Then issues can be fixed up.
Anyway, you have gone through this many times, so whatever the process is, I trust that it works. Just sharing my thought as a somewhat uniformed reader.
Quoted 20 lines
Quoted 19 lines
Quoted 18 lines
device-types/net/description.tex \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 11 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) };In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.
Quoted 1 line
Yes, but it looks like an independent issue for this.
If this change requires new headroom, then it depends on headroom expansion.
My suggestion is to then expand it significantly immediately, to minimize the number of times we have to change the virtio_net code to support new lengths.
We already have structs virtio_net_hdr, virtio_net_hdr_mrg_rxbuf, virtio_net_hdr_v1, virtio_net_hdr_v1_hash.
Adding a new struct with every new field added just adds lots of churn in the code.
Just a thought, feel free to skip if the maintainers disagree.
Quoted 2 lines
Quoted 2 lines
Do you mean, by adding 'padding' or 'unused' fields?
Yes.
Quoted 9 lines
Quoted 2 lines
Side note: the above is not 32 bit aligned, do we need such aligment?
Maybe, but it should be an independent topic.
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.
But that's too late now? Probably don't want to make this work depending on such a change.
I don't entirely recall why we did not do that immediately. Perhaps it is related to also only having SKB_GSO_UDP_L4.
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 26 lines · Willem de Bruijn
Quoted 18 lines · Jason Wang
Quoted 16 lines · Paolo Abeni
Quoted 7 lines · Willem de Bruijn
Quoted 7 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated)Quoted 8 lines
Review \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 1 line
(quoted diff) +0
-0};In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.Quoted 1 line
Yes, but it looks like an independent issue for this.
If this change requires new headroom, then it depends on headroom expansion.
My suggestion is to then expand it significantly immediately, to minimize the number of times we have to change the
virtio_netcode to support new lengths.
Quoted 7 lines
We already have structs
virtio_net_hdr,virtio_net_hdr_mrg_rxbuf,virtio_net_hdr_v1,virtio_net_hdr_v1_hash.Adding a new struct with every new field added just adds lots of churn in the code.
Just a thought, feel free to skip if the maintainers disagree.
I just read the old thread WRT virtio net timestamping.
If we reserve additional headroom space for timestamping here, such room will be available only if UDP_TUNNEL is negotiated, which sounds a bit counterintuitive to me.
I *think* ideally we want a separate virtio net feature for that?
Regarding the virtio_net_hdr parsing complexity increase, I think we should add to the uAPI a struct containing only the new fields, e.g. for UDP tunnel:
struct virtio_net_udp_tunnel {
__virtio16 outer_th_offset;
// eventually __virtio16 inner_protocol_type;
__virtio16 inner_nh_offset;
};At feature negotiation time both the drivers and the device compute the offset inside the virtio buffer where such struct is placed:
if VIRTIO_NET_F_HASH_REPORT is negotiated such offset is sizeof(struct virtio_net_hdr_v1_hash) elsewhere is sizeof(struct virtio_net_hdr_v1)
At rx time, struct virtio_net_udp_tunnel() is parsed from the above offset.
With all the above in place, for every additional later feature we add just a struct definition and the virtio_net header parsing complexity scales linearly.
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.But that's too late now? Probably don't want to make this work depending on such a change.
I don't entirely recall why we did not do that immediately. Perhaps it is related to also only having
SKB_GSO_UDP_L4.
I reached the same conclusion here:
What about reserving an additional bit in gso_type to specify the inner network header protocol?
Thanks,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 36 lines · Paolo Abeni
Quoted 26 lines · Willem de Bruijn
Quoted 18 lines · Jason Wang
Quoted 16 lines · Paolo Abeni
Quoted 7 lines · Willem de Bruijn
Quoted 7 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated)Quoted 8 lines
Review \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 1 line
(quoted diff) +0
-0};In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.Quoted 1 line
Yes, but it looks like an independent issue for this.
If this change requires new headroom, then it depends on headroom expansion.
My suggestion is to then expand it significantly immediately, to minimize the number of times we have to change the
virtio_netcode to support new lengths.Quoted 7 lines
We already have structs
virtio_net_hdr,virtio_net_hdr_mrg_rxbuf,virtio_net_hdr_v1,virtio_net_hdr_v1_hash.Adding a new struct with every new field added just adds lots of churn in the code.
Just a thought, feel free to skip if the maintainers disagree.
Quoted 7 lines
I just read the old thread WRT virtio net timestamping.
If we reserve additional headroom space for timestamping here, such room will be available only if
UDP_TUNNELis negotiated, which sounds a bit counterintuitive to me.I *think* ideally we want a separate virtio net feature for that?
Fair point. Agreed that extra headroom should not be dependent on this feature.
Quoted 13 lines
Regarding the
virtio_net_hdrparsing complexity increase, I think we should add to the uAPI a struct containing only the new fields, e.g. for UDP tunnel:struct virtio_net_udp_tunnel { __virtio16 outer_th_offset; // eventually __virtio16 inner_protocol_type; __virtio16 inner_nh_offset; };At feature negotiation time both the drivers and the device compute the offset inside the virtio buffer where such struct is placed:
Quoted 8 lines
if
VIRTIO_NET_F_HASH_REPORTis negotiated such offset issizeof(struct virtio_net_hdr_v1_hash)elsewhere issizeof(struct virtio_net_hdr_v1)At rx time, struct
virtio_net_udp_tunnel()is parsed from the above offset.With all the above in place, for every additional later feature we add just a struct definition and the
virtio_netheader parsing complexity scales linearly.
Sounds good to me.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.But that's too late now? Probably don't want to make this work depending on such a change.
I don't entirely recall why we did not do that immediately. Perhaps it is related to also only having
SKB_GSO_UDP_L4.
Quoted 6 lines
I reached the same conclusion here:
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?
Similar to VIRTIO_NET_HDR_GSO_ECN?
Is that preferable over VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4 and .._UDP_TUNNEL_IP6 variants?
And to parsing just this one byte (or nibble) from packet data using inner_nh_offset?
If it's the best of those options, no objections from me. Definitely no need for a u16 if the only options are IPPROTO_IP and IPPROTO_IPV6.
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 83 lines · Willem de Bruijn
Quoted 36 lines · Paolo Abeni
Quoted 26 lines · Willem de Bruijn
Quoted 18 lines · Jason Wang
Quoted 16 lines · Paolo Abeni
Quoted 7 lines · Willem de Bruijn
Quoted 7 lines
(quoted diff) +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated)Quoted 8 lines
Review \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 1 line
(quoted diff) +0
-0};In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.Quoted 1 line
Yes, but it looks like an independent issue for this.
If this change requires new headroom, then it depends on headroom expansion.
My suggestion is to then expand it significantly immediately, to minimize the number of times we have to change the
virtio_netcode to support new lengths.Quoted 7 lines
We already have structs
virtio_net_hdr,virtio_net_hdr_mrg_rxbuf,virtio_net_hdr_v1,virtio_net_hdr_v1_hash.Adding a new struct with every new field added just adds lots of churn in the code.
Just a thought, feel free to skip if the maintainers disagree.
Quoted 7 lines
I just read the old thread WRT virtio net timestamping.
If we reserve additional headroom space for timestamping here, such room will be available only if
UDP_TUNNELis negotiated, which sounds a bit counterintuitive to me.I *think* ideally we want a separate virtio net feature for that?
Fair point. Agreed that extra headroom should not be dependent on this feature.
Quoted 13 lines
Regarding the
virtio_net_hdrparsing complexity increase, I think we should add to the uAPI a struct containing only the new fields, e.g. for UDP tunnel:struct virtio_net_udp_tunnel { __virtio16 outer_th_offset; // eventually __virtio16 inner_protocol_type; __virtio16 inner_nh_offset; };At feature negotiation time both the drivers and the device compute the offset inside the virtio buffer where such struct is placed:
Quoted 8 lines
if
VIRTIO_NET_F_HASH_REPORTis negotiated such offset issizeof(struct virtio_net_hdr_v1_hash)elsewhere issizeof(struct virtio_net_hdr_v1)At rx time, struct
virtio_net_udp_tunnel()is parsed from the above offset.With all the above in place, for every additional later feature we add just a struct definition and the
virtio_netheader parsing complexity scales linearly.Sounds good to me.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
If we would have
VIRTIO_NET_HDR_GSO_UDPv4_L4andVIRTIO_NET_HDR_GSO_UDPv6_L4variants, we could avoid entirely 'inner_protocol_type', as the network protocol will be implied by the GSO type itself.But that's too late now? Probably don't want to make this work depending on such a change.
I don't entirely recall why we did not do that immediately. Perhaps it is related to also only having
SKB_GSO_UDP_L4.
Quoted 8 lines
Quoted 6 lines
I reached the same conclusion here:
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?Similar to
VIRTIO_NET_HDR_GSO_ECN?
I think so.
Quoted 10 lines
Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?And to parsing just this one byte (or nibble) from packet data using
inner_nh_offset?If it's the best of those options, no objections from me. Definitely no need for a u16 if the only options are
IPPROTO_IPandIPPROTO_IPV6.
Exactly.
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Willem de BruijnQuoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?Similar to
VIRTIO_NET_HDR_GSO_ECN?Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?
Should be basically the same, as VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4 VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6 will be 2 separate bits in gso_type, right?
Quoted 2 lines
And to parsing just this one byte (or nibble) from packet data using
inner_nh_offset?
I think one of the state goal is to avoid parsing. More importantly, accessing the inner network header content this early will cause possibly avoidable (or at least mitigable via prefetch) cache misses.
Quoted 2 lines
If it's the best of those options, no objections from me. Definitely no need for a u16 if the only options are
IPPROTO_IPandIPPROTO_IPV6.
Wrapping all the above I'll go for the additional bit for the inner network header type. If I misread something, please LMK:)
Thanks,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 6 lines · Paolo Abeni
Quoted 4 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?
Quoted 8 lines
Quoted 4 lines
Similar to
VIRTIO_NET_HDR_GSO_ECN?Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?Should be basically the same, as
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6will be 2 separate bits ingso_type, right?
One difference is that we have to update other code to use a bitmask.
I'd prefer the two types for that reason.
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 12 lines · Willem de Bruijn
Quoted 11 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?Similar to
VIRTIO_NET_HDR_GSO_ECN?Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?
Quoted 7 lines
Quoted 3 lines
Should be basically the same, as
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6will be 2 separate bits ingso_type, right?One difference is that we have to update other code to use a bitmask.
I'd prefer the two types for that reason.
Please pardon me for the high-frequency spamming. I'd love to have this topic well defined.
I'm unsure how to read your suggestion exactly.
VIRTIO_NET_HDR_GSO_UDP_TUNNEL needs to be "combined" (set together) with VIRTIO_NET_HDR_GSO_TCPV4, VIRTIO_NET_HDR_GSO_TCPV6 or VIRTIO_NET_HDR_GSO_UDP_L4, so it's currently defined as a bit flag:
#define VIRTIO_NET_HDR_GSO_UDP_TUNNEL 0x40
From an implementation perspective this allow us reusing the existing code for to fill inner headers related field, masking such bit in the gso_type before invoking the existing virtio_hdr parsing helper - it will be used unmodified.
Due to all the above, I thought the IPv4 and IPv6 variant would be defined as:
#define VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4 0x20
#define VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6 0x40
How would you define VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4 and VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6?
Thanks,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 38 lines · Paolo Abeni
Quoted 12 lines · Willem de Bruijn
Quoted 11 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?Similar to
VIRTIO_NET_HDR_GSO_ECN?Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?Quoted 7 lines
Quoted 3 lines
Should be basically the same, as
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6will be 2 separate bits ingso_type, right?One difference is that we have to update other code to use a bitmask.
I'd prefer the two types for that reason.
Please pardon me for the high-frequency spamming. I'd love to have this topic well defined.
I'm unsure how to read your suggestion exactly.
VIRTIO_NET_HDR_GSO_UDP_TUNNELneeds to be "combined" (set together) withVIRTIO_NET_HDR_GSO_TCPV4,VIRTIO_NET_HDR_GSO_TCPV6orVIRTIO_NET_HDR_GSO_UDP_L4, so it's currently defined as a bit flag:#define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL0x40From an implementation perspective this allow us reusing the existing
code for to fill inner headers related field, masking such bit in thegso_typebefore invoking the existingvirtio_hdrparsing helper - it will be used unmodified.
Quoted 8 lines
Due to all the above, I thought the IPv4 and IPv6 variant would be defined as:
#define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP40x20
#defineVIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP60x40How would you define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4andVIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6?
Sorry for the confusion. I agree with your proposal.
I had forgotten that it already adds a bit that will have to be masked out in any branch that only deals with the inner type.
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Paolo AbeniQuoted 38 lines · Paolo Abeni
Quoted 12 lines · Willem de Bruijn
Quoted 11 lines · Paolo Abeni
Quoted 8 lines · Willem de Bruijn
Quoted 2 lines · Paolo Abeni
What about reserving an additional bit in
gso_typeto specify the inner network header protocol?Similar to
VIRTIO_NET_HDR_GSO_ECN?Is that preferable over
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4and .._UDP_TUNNEL_IP6 variants?Quoted 7 lines
Quoted 3 lines
Should be basically the same, as
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6will be 2 separate bits ingso_type, right?One difference is that we have to update other code to use a bitmask.
I'd prefer the two types for that reason.
Please pardon me for the high-frequency spamming. I'd love to have this topic well defined.
I'm unsure how to read your suggestion exactly.
VIRTIO_NET_HDR_GSO_UDP_TUNNELneeds to be "combined" (set together) withVIRTIO_NET_HDR_GSO_TCPV4,VIRTIO_NET_HDR_GSO_TCPV6orVIRTIO_NET_HDR_GSO_UDP_L4, so it's currently defined as a bit flag:#define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL0x40From an implementation perspective this allow us reusing the existing
code for to fill inner headers related field, masking such bit in thegso_typebefore invoking the existingvirtio_hdrparsing helper - it will be used unmodified.
Quoted 5 lines
Due to all the above, I thought the IPv4 and IPv6 variant would be defined as:
#define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP40x20
#defineVIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP60x40
Yes, I think this makes sense.
Quoted 8 lines
How would you define
VIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP4andVIRTIO_NET_HDR_GSO_UDP_TUNNEL_IP6?Thanks,
Paolo
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Willem de BruijnQuoted 21 lines · Willem de Bruijn
Quoted 19 lines · Jason Wang
Quoted 16 lines · Paolo Abeni
Quoted 5 lines · Willem de Bruijn
You have a working software implementation, right? Will this be sent (as RFC) to the public netdev or so mailing list before this becomes a standard? I assume that there is a standard process for virtio that you are following. I'd just be more confident in a thumbs up if we also review the code that implements the text.
My plan is to write the S/W
virtio_netimplementation, too, but it's currently not ready. I had a working PoC for the first draft of this changeset, long time ago. Things are changed enough that I'll need a rewrite. I can try to shared such a thing on netdev ASAP (where "soon" here is a very relative terms ;)Also I don't know the standardization process. I hope for some guidance from Jason and/or Michael.
Quoted 6 lines
Quoted 3 lines
It should be something like this. When there's no future comment, open an issue https://github.com/oasis-tcs/virtio-spec/issues and ask for a vote. When it passes the voting, it would be merged.
So changes to the spec can be merged, because we know that they can be implemented reasonably?
This seems to be how it works now.
Quoted 10 lines
IETF always requires (two, because interoperable protocol definitions) working implementations.
It sounds a bit risky to me to skip this, as you come across the interesting non-obvious details during implementation.
Since implementation is planned anyway, doing at least a rough proof of concept first does not introduce extra work, but reduces risk that the spec contains errors.
Exactly and we have already had qemu which fits for that purpose. A lot of hardware was prototyped in that way (for example, rocker fbnic etc).
Quoted 7 lines
Unless it is easy to update the spec before it is numbered and finalized. Then issues can be fixed up.
Anyway, you have gone through this many times, so whatever the process is, I trust that it works. Just sharing my thought as a somewhat uniformed reader.
I fully agree with you. In the long debate of the live migration series, I point out we need a prototype for features that requires huge changes. But I don't get sufficient feedback for that.
Quoted 28 lines
Quoted 21 lines
Quoted 4 lines
Quoted 4 lines
Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
device-types/net/description.tex +3
-0@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated)Quoted 8 lines
Review \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O · line 447
Quoted 1 line
(quoted diff) +0
-0};In relation to the above: I recall that expanding the
virtio_net_hdrwas non-trivial when I last tried to add fields ("[rfc,2/3]
virtio-net: support receive timestamp"). If that is the experience again, it might be worthwhile to expand with enough space to also accommodate a few subsequent extensions. Headroom is not that expensive.Quoted 1 line
Yes, but it looks like an independent issue for this.
If this change requires new headroom, then it depends on headroom expansion.
My suggestion is to then expand it significantly immediately, to minimize the number of times we have to change the
virtio_netcode to support new lengths.
Quoted 8 lines
We already have structs
virtio_net_hdr,virtio_net_hdr_mrg_rxbuf,virtio_net_hdr_v1,virtio_net_hdr_v1_hash.Adding a new struct with every new field added just adds lots of churn in the code.
Just a thought, feel free to skip if the maintainers disagree.
I think I agree with that, but I meant it's better to not make UDP tunnel series depending on that.
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Paolo AbeniQuoted 86 lines · Paolo Abeni
The
VIRTIO_NET_HDR_GSO_UDP_TUNNELis agso_typeflag allowing GSO over UDP tunnel. It can be negotiated on both the host and guest sides.One constraint addressed here is that the virtio side (either device or completely the inner and outer headers offset - to allow for later GSO.
To accommodate such need, new fields are introduced in the
virtio_net
header:outer_th_offset,inner_protocol_typeandinner_nh_offset. They map directly to the corresponding header information. The inner transport header is implied by the (inner) checksum offload.Those fields are required because a virtio device H/W implementation performing segmentation for UDP tunneled packet will need to touch the outer transport protocol (for the UDP length filed), the inner network protocol (for the total length field, in the IPv4 case).
Note that segmentation will additionally need to touch
the outer network protocol and the inner transport protocol. The first is implied/easily found with trivial parsing, the latter is identified by the existingcsum_startfield.Note that there is no concept of UDP tunnel type negotiation (e.g. vxlan, geneve, vxlan-gpe, etc.), as a virtio device H/W implementation can perform segmentation for every possible UDP-tunnel given the specified new fields.
In the reverse direction, if a virtio device H/W implementation receives some traffic for an unknown or unsupported UDP tunnel, it will simply not aggregate the wire packet in GSO one.Signed-off-by Paolo Abeni <[email protected]>---
v5 -> v6:
- split in 2 patches
- dropped the unneededouter_mh_offsetfield
https://lore.kernel.org/virtio-comment/56ccba136272593d0d2af0303600c06f26bd84a1.1718621522.git.pabeni@redhat.com/v4 -> v5:
- add separate negotiation forUDP_TUNNEL_CSUM
- much more verbose specification of outer csum handling
-UDP_TUNNEL_CSUMrequiresGSO_UDP_TUNNEL
https://lore.kernel.org/virtio-comment/cb89d3d6b6a7e4a7f13e60dd3fded5e950727890.1716448766.git.pabeni@redhat.com/v3 -> v4:
- newvirtio_net_hdrfields depends onF_HOST_UDP_TUNNEL_GSOorF_GUEST_UDP_TUNNEL_GSO(Stefano)
- Extended the changelog to answer Jason's questions.
- Clarified outer csum handling (Jason)
https://lore.kernel.org/virtio-dev/f0bf2db203f8061a3ecb50b7a48cf26d8f3c7f68.1716193086.git.pabeni@redhat.comv2 -> v3:
-UDP_TUNNEL->UDP_TUNNEL_GSO
- add explicit fields for the inner meta-data
- more verbose changelog
https://lists.oasis-open.org/archives/virtio-dev/202206/msg00026.htmlv1 -> v2:
- explicitly state that the outer header probing is mandatory
- explicitly state thatGSO_UDPis not allowed withGSO_UDP_TUNNEL
- clarifyhdr_lenusage
- clarifyUDP_TUNNEL_CSUMbit usage
- fix a few typos
https://lists.oasis-open.org/archives/virtio-dev/202205/msg00037.html--- device-types/net/description.tex | 118 +++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 6 deletions(-)device-types/net/description.tex +6
-0@@ -88,6 +88,12 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO (47)] Driver can receive GSO packets + carried by a UDP tunnel. + +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO (49)] Device can receive GSO packets + carried by a UDP tunnel. + \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets. \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
device-types/net/description.tex \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device · line 143
Quoted 5 lines
device-types/net/description.tex +2
-0@@ -133,12 +139,16 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_UFO] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO4] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO6] Requires VIRTIO_NET_F_GUEST_CSUM. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6.
I think VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO requires all of the four?
Quoted 12 lines
device-types/net/description.tex +2
-0\item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_ECN] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_HOST_UFO] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_USO] Requires VIRTIO_NET_F_CSUM. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.
device-types/net/description.tex \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 388
Quoted 4 lines
device-types/net/description.tex +1
-0@@ -375,6 +385,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features. +\item If any TCP or UDP segmentation feature is negotiated,
Not a native speaker but this seems to conflict with the above. For example, does "only VIRTIO_NET_F_GUEST_TSO4 is negotiated" mean any TCP segmentation feature?
Quoted 54 lines
a driver can
device-types/net/description.tex +16
-2+ use TCP segmentation or UDP segmentation on top of UDP encapsulation + offload, when the outer header does not require checksumming - e.g. + the outer UDP checksum is zero - by negotiating the + VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature. + \item The converse features are also available: a driver can save the virtual device some work by negotiating these features.\note{For example, a network packet transported between two guests on the same system might not need checksumming at all, nor segmentation,@@ -382,8 +398,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev The VIRTIO_NET_F_GUEST_CSUM feature indicates that partially checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, - VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4 - and VIRTIO_NET_F_GUEST_USO6 are the input equivalents of the features described above. + VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4, + VIRTIO_NET_F_GUEST_USO6 and VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO + are the input equivalents of the features described above. See \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~\nameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} and@@ -413,6 +430,7 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O #define VIRTIO_NET_HDR_GSO_UDP 3 #define VIRTIO_NET_HDR_GSO_TCPV6 4 #define VIRTIO_NET_HDR_GSO_UDP_L4 5 +#define VIRTIO_NET_HDR_GSO_UDP_TUNNEL 0x40 #define VIRTIO_NET_HDR_GSO_ECN 0x80 u8 gso_type; le16 hdr_len;@@ -423,6 +441,9 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O le32 hash_value; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 hash_report; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) le16 padding_reserved; (Only if VIRTIO_NET_F_HASH_REPORT negotiated) + le16 outer_th_offset (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_protocol_type;(Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) + le16 inner_nh_offset; (Only if VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO negotiated) }; \end{lstlisting}@@ -480,6 +501,8 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De followed by the TCP header (with the TCP checksum field 16 bytes into that header). \field{csum_start} will be 14+20 = 34 (the TCP checksum includes the header), and \field{csum_offset} will be 16. +If the given packet has the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit set, +the above checksum fields refer to the inner header checksum. \end{note} \item If the driver negotiated@@ -516,6 +539,26 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De specifically in the protocol.}. \end{itemize} +\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 553
Quoted 10 lines
device-types/net/description.tex +10
-0+ the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero. + The other tunnel-related fields indicate how to replicate the packet + headers to cut it into smaller packets: + + \begin{itemize} + \item \field{outer_th_offset} field indicates the outer transport header within + the packet. Note that such field differs from \field{csum_start} as the latter + points to the inner transport header within the packet
I wonder if it's better to stick to outer_csum_start, and say it's offset within the packet to begin checksumming for outer packet. This follows the existing name and could be reused for encapsulation other than UDP tunnel.
Quoted 12 lines
device-types/net/description.tex +8
-0+ + \item \field{inner_protocol_type} field indicates the ethernet type of the inner + protocol. + + \item \field{inner_nh_offset} field indicates the inner network header within + the packet. + \end{itemize} + \item \field{num_buffers} is set to zero. This field is unused on transmitted packets. \item The header and packet are added as one output descriptor to the
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 606
Quoted 7 lines
device-types/net/description.tex +4
-0@@ -557,6 +600,18 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De driver MUST set the VIRTIO_NET_HDR_GSO_ECN bit in \field{gso_type}. +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation offload, unless the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is +negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL +bit in \field{gso_type}.
Is this better to follow the existing style of the driver normative? For example, we had
"If VIRTIO_NET_F_HOST_TSO6 is negotiated, the driver MAY set gso_type to VIRTIO_NET_HDR_GSO_TCPV6 to request TCPv6 segmentation, otherwise the driver MUST NOT set gso_type to VIRTIO_NET_HDR_GSO_TCPV6."
We could do the same
"If VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is negotiated, the driver MAY ... otherwise the driver MUST NOT ..."
Quoted 12 lines
device-types/net/description.tex +8
-0+ +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation and outer UDP checksum offload. + +The driver MUST NOT set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL together with +VIRTIO_NET_HDR_GSO_UDP, as the latter is deprecated in favor of UDP_L4 +and no new features will support it. + If the VIRTIO_NET_F_CSUM feature has been negotiated, the driver MAY set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in \field{flags}, if so:
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 662
Quoted 10 lines
device-types/net/description.tex +7
-0@@ -598,6 +653,28 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De header. \end{itemize} +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has been negotiated, the +driver MAY set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}, if so: +\begin{itemize} +\item the driver MUST set \field{outer_th_offset} to the outer UDP header + offset, \field{inner_protocol_type} to the ethernet type of the inner + protocol, and \field{inner_nh_offset} to the inner network header offset. +\end{itemize}
Do we need to say then csum_start and csum_offset is for the inner packet here?
Quoted 68 lines
device-types/net/description.tex +32
-4+ +If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer +UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +value must be 0, or the valid complete checksum for such header. + +\begin{note} +The valid complete checksum of the outer UDP header of individual segments +can be computed by the driver prior to segmentation only if the GSO packet +size is a multiple of \field{gso_size}. +\end{note} + +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has not +been negotiated, the driver MUST NOT set the VIRTIO_NET_HDR_F_UDP_TUNNEL bit +in \field{gso_type}. + The driver SHOULD accept the VIRTIO_NET_F_GUEST_HDRLEN feature if it has been offered, and if it's able to provide the exact header length.@@ -633,6 +710,10 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De \end{note} \end{itemize} +If VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} is not set, the +device MUST NOT use the \field{outer_th_offset}, \field{inner_protocol_type} + and \field{inner_nh_offset}. + If VIRTIO_NET_HDR_F_NEEDS_CSUM is not set, the device MUST NOT rely on the packet checksum being correct. \paragraph{Packet Transmission Interrupt}\label{sec:Device Types / Network Device / Device Operation / Packet Transmission / Packet Transmission Interrupt}@@ -727,8 +808,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network has been validated. \end{enumerate} -Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP and ECN -features enable receive checksum, large receive offload and ECN +Additionally, VIRTIO_NET_F_GUEST_CSUM, TSO4, TSO6, UDP, UDP_TUNNEL +and ECN features enable receive checksum, large receive offload and ECN support which are the input equivalents of the transmit checksum, transmit segmentation offloading and ECN features, as described in \ref{sec:Device Types / Network Device / Device Operation /@@ -750,8 +831,14 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network from \field{csum_start} and any preceding checksums have been validated. The checksum on the packet is incomplete and if bit VIRTIO_NET_HDR_F_RSC_INFO is not set in \field{flags}, - then \field{csum_start} and \field{csum_offset} indicate how to calculate it - (see Packet Transmission point 1). + then \field{csum_start} and \field{csum_offset} indicate how to calculate it. + If the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit is set, the \field{csum_start} field + refers to the inner transport header offset (see Packet Transmission point 1). +\item If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO option was negotiated and + \field{gso_type} is not VIRTIO_NET_HDR_GSO_NONE, the VIRTIO_NET_HDR_GSO_UDP_TUNNEL + bit MAY be set. In such case the \field{outer_th_offset}, \field{inner_protocol_type}, + and \field{inner_nh_offset} fields indicate the corresponding + headers information. \end{enumerate}@@ -796,6 +883,9 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If none of VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6 have been negotiated, the device MUST NOT set \field{gso_type} to VIRTIO_NET_HDR_GSO_UDP_L4. +If VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO is not negotiated the device MUST NOT set +the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}. + The device SHOULD NOT send to the driver TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_GUEST_ECN feature is negotiated, in which case the
device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 918
Quoted 10 lines
device-types/net/description.tex +7
-0@@ -819,6 +909,17 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network fully checksummed packet; \end{enumerate} +The device MUST NOT send to the driver TCP or UDP GSO packets encapsulated in UDP +tunnel and requiring segmentation offload, unless the +VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO is negotiated, in which case the device MUST set +the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} and MUST set the +\field{outer_th_offset}, \field{inner_protocol_type} and +\field{inner_nh_offset} fields to the corresponding header information, and the outer +UDP header MUST NOT require checksum offload.
Similarly, do we need to say csum_start|offset is for inner?
Quoted 27 lines
device-types/net/description.tex +9
-1+ +The device MUST NOT send the driver TCP or UDP GSO packets encapsulated in UDP +tunnel and requiring segmentation and outer checksum offload. + If none of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have been negotiated, the device MUST set \field{gso_type} to VIRTIO_NET_HDR_GSO_NONE.@@ -867,6 +968,10 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network VIRTIO_NET_HDR_F_DATA_VALID is set, the driver MUST NOT rely on the packet checksum being correct. +If the VIRTIO_NET_HDR_GSO_UDP_TUNNEL_GSO bit in \field{gso_type} is not set, +the driver MUST NOT use the \field{outer_th_offset}, \field{inner_protocol_type}, +\field{inner_mac_offset} and \field{inner_nh_offset}. + \paragraph{Hash calculation for incoming packets} \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets}@@ -1624,6 +1729,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi #define VIRTIO_NET_F_GUEST_TSO6 8 #define VIRTIO_NET_F_GUEST_ECN 9 #define VIRTIO_NET_F_GUEST_UFO 10 +#define VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO 47 #define VIRTIO_NET_F_GUEST_USO4 54 #define VIRTIO_NET_F_GUEST_USO6 55 --
Quoted 2 lines
2.45.2
Thanks
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 8 lines · Jason Wang
device-types/net/description.tex \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device · line 143
Quoted 6 lines
device-types/net/description.tex +2
-0@@ -133,12 +139,16 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_UFO] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO4] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO6] Requires VIRTIO_NET_F_GUEST_CSUM. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6.I think
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSOrequires all of the four?
Functionally speaking is not strictly necessary: the UDP tunnel feature needs just a single inner GSO type to assemble and process an UDP tunneled packet.
On the flip side I think is reasonable that modern implementations support all the mentioned inner GSO type and requiring all of them would possibly make the implementation simpler and/or less bug prone.
So I guess I can change the requirement as you say above.
Quoted 14 lines
Quoted 14 lines
device-types/net/description.tex +2
-0\item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_ECN] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_HOST_UFO] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_USO] Requires VIRTIO_NET_F_CSUM. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.@@ -375,6 +385,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features.
Quoted 5 lines
Review \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 388
Quoted 1 line
(quoted diff) +1
-0+\item If any TCP or UDP segmentation feature is negotiated,Not a native speaker but this seems to conflict with the above. For example, does "only
VIRTIO_NET_F_GUEST_TSO4is negotiated" mean any TCP segmentation feature?
As per the previois comment, I'll change this in the next revision to:
"""
If the VIRTIO_NET_F_HOST_TSO6, VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_HDR_GSO_UDP_L4 features are negotiated """>> @@ -516,6 +539,26 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De
Quoted 16 lines
Quoted 5 lines
device-types/net/description.tex +2
-0specifically in the protocol.}. \end{itemize} +\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates thatdevice-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De
Quoted 10 lines
device-types/net/description.tex +10
-0+ the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero. + The other tunnel-related fields indicate how to replicate the packet + headers to cut it into smaller packets: + + \begin{itemize} + \item \field{outer_th_offset} field indicates the outer transport header within + the packet. Note that such field differs from \field{csum_start} as the latter + points to the inner transport header within the packet
Quoted 4 lines
I wonder if it's better to stick to
outer_csum_start, and say it's offset within the packet to begin checksumming for outer packet. This follows the existing name and could be reused for encapsulation other than UDP tunnel.
I thought we agreed on this name in the previous iteration?!? It feels strange to introduce a csum related field in a change NOT introducing a checksum related feature. And we need this filed here, for plain segmentation.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 606
Quoted 8 lines
device-types/net/description.tex +4
-0@@ -557,6 +600,18 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De driver MUST set the VIRTIO_NET_HDR_GSO_ECN bit in \field{gso_type}. +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation offload, unless the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is +negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL +bit in \field{gso_type}.Is this better to follow the existing style of the driver normative? For example, we had
Quoted 8 lines
"If
VIRTIO_NET_F_HOST_TSO6is negotiated, the driver MAY setgso_typetoVIRTIO_NET_HDR_GSO_TCPV6to request TCPv6 segmentation, otherwise the driver MUST NOT setgso_typetoVIRTIO_NET_HDR_GSO_TCPV6."We could do the same
"If
VIRTIO_NET_F_HOST_UDP_TUNNEL_GSOis negotiated, the driver MAY ... otherwise the driver MUST NOT ..."
ok, I'll do that in the next revision
Quoted 4 lines
Quoted 4 lines
device-types/net/description.tex +0
-0@@ -598,6 +653,28 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De header. \end{itemize}
Quoted 9 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 662
Quoted 7 lines
device-types/net/description.tex +7
-0+If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has been negotiated, the +driver MAY set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}, if so: +\begin{itemize} +\item the driver MUST set \field{outer_th_offset} to the outer UDP header + offset, \field{inner_protocol_type} to the ethernet type of the inner + protocol, and \field{inner_nh_offset} to the inner network header offset. +\end{itemize}Do we need to say then
csum_startandcsum_offsetis for the inner packet here?
I thought such statement was already everywhere, but no objection to add it here, too.
Somewhat related, I have a doubt WRT the csum_start/csum_offset fields "presence" with plain GSO packets. AFAICS, the specs require the csum offload feature as a pre-req for the plain segmentation feature, but also allow the reception of a GSO packet without the VIRTIO_NET_HDR_F_NEEDS_CSUM bit set, see the recent linux patch from Willem:
https://lore.kernel.org/netdev/[email protected]/
I think it will make the thing simpler if we always require such bit to be set for GSO packets. I guess we can't change the current status for the existing GSO types, but for UDP tunnel we could add that requirement from the start, both for the driver -> device direction and for the device -> driver direction.
Even when the device sets the VIRTIO_NET_HDR_F_DATA_VALID, so can always expect a valid csum_start/csum_offset pair, and make things hopefully more robust. When VIRTIO_NET_HDR_F_DATA_VALID is set, the pair will NOT be used to validated the csum, just to point to the inner transport header and avoid parsing. WDYT?
Thanks,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
Quoted 21 lines · Paolo Abeni
Quoted 14 lines · Jason Wang
Quoted 14 lines
\item[VIRTIO_NET_F_HOST_TSO4] Requires
VIRTIO_NET_F_CSUM.
\item[VIRTIO_NET_F_HOST_TSO6] RequiresVIRTIO_NET_F_CSUM.
\item[VIRTIO_NET_F_HOST_ECN] RequiresVIRTIO_NET_F_HOST_TSO4orVIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_HOST_UFO] RequiresVIRTIO_NET_F_CSUM.
\item[VIRTIO_NET_F_HOST_USO] RequiresVIRTIO_NET_F_CSUM.
+\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires bothVIRTIO_NET_F_HOST_TSO4andVIRTIO_NET_F_HOST_TSO6, ordevice-types/net/description.tex +1
-0+ both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.@@ -375,6 +385,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features.Quoted 5 lines
Review \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 388
Quoted 1 line
(quoted diff) +1
-0+\item If any TCP or UDP segmentation feature is negotiated,Not a native speaker but this seems to conflict with the above. For example, does "only
VIRTIO_NET_F_GUEST_TSO4is negotiated" mean any TCP segmentation feature?
Quoted 6 lines
As per the previois comment, I'll change this in the next revision to:
"""
If theVIRTIO_NET_F_HOST_TSO6,VIRTIO_NET_F_HOST_TSO4andVIRTIO_NET_HDR_GSO_UDP_L4features are negotiated """
Typos above, should be:
"""
If the VIRTIO_NET_F_HOST_TSO6, VIRTIO_NET_F_HOST_TSO4, VIRTIO_NET_F_HOST_USO features are negotiated """
Cheers,
Paolo
Re: [PATCH v6 1/2] virtio-net: define UDP tunnel segmentation offload feature
In reply to Paolo AbeniQuoted 11 lines · Paolo Abeni
Quoted 8 lines · Jason Wang
device-types/net/description.tex \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device · line 143
Quoted 6 lines
device-types/net/description.tex +2
-0@@ -133,12 +139,16 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_UFO] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO4] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_USO6] Requires VIRTIO_NET_F_GUEST_CSUM. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6.I think
VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSOrequires all of the four?
Quoted 7 lines
Functionally speaking is not strictly necessary: the UDP tunnel feature needs just a single inner GSO type to assemble and process an UDP tunneled packet.
On the flip side I think is reasonable that modern implementations support all the mentioned inner GSO type and requiring all of them would possibly make the implementation simpler and/or less bug prone.
Yes.
Quoted 2 lines
So I guess I can change the requirement as you say above.
That would be great.
Quoted 21 lines
Quoted 15 lines
Quoted 15 lines
device-types/net/description.tex +2
-0\item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_ECN] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_HOST_UFO] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_USO] Requires VIRTIO_NET_F_CSUM. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6, or + both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.@@ -375,6 +385,12 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev TCP), VIRTIO_NET_F_HOST_TSO6 (IPv6 TCP), VIRTIO_NET_F_HOST_UFO (UDP fragmentation) and VIRTIO_NET_F_HOST_USO (UDP segmentation) features.Quoted 5 lines
Review \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 388
Quoted 1 line
(quoted diff) +1
-0+\item If any TCP or UDP segmentation feature is negotiated,Not a native speaker but this seems to conflict with the above. For example, does "only
VIRTIO_NET_F_GUEST_TSO4is negotiated" mean any TCP segmentation feature?
Quoted 6 lines
As per the previois comment, I'll change this in the next revision to:
"""
If theVIRTIO_NET_F_HOST_TSO6,VIRTIO_NET_F_HOST_TSO4andVIRTIO_NET_HDR_GSO_UDP_L4features are negotiated """
Ok.
Quoted 18 lines
Review \subsubsection{Packet
Quoted 1 line
(quoted diff) +0
-0@@ -516,6 +539,26 @@ \subsubsection{PacketTransmission}\label{sec:Device Types / Network Device / De
Quoted 16 lines
Quoted 15 lines
device-types/net/description.tex +12
-0specifically in the protocol.}. \end{itemize} +\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that + the GSO protocol is encapsulated in a UDP tunnel. + The outer UDP header must not require checksumming, e.g. the outer UDP checksum + must be zero. + The other tunnel-related fields indicate how to replicate the packet + headers to cut it into smaller packets: + + \begin{itemize} + \item \field{outer_th_offset} field indicates the outer transport header within + the packet. Note that such field differs from \field{csum_start} as the latter + points to the inner transport header within the packet
Quoted 6 lines
Quoted 4 lines
I wonder if it's better to stick to
outer_csum_start, and say it's offset within the packet to begin checksumming for outer packet. This follows the existing name and could be reused for encapsulation other than UDP tunnel.I thought we agreed on this name in the previous iteration?!?
Sorry if I do that, the switch loses some context :(
Quoted 4 lines
It feels
strange to introduce a csum related field in a change NOT introducing a checksum related feature. And we need this filed here, for plain segmentation.
Ok, fine.
Quoted 38 lines
Quoted 13 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 606
Quoted 9 lines
device-types/net/description.tex +4
-0@@ -557,6 +600,18 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De driver MUST set the VIRTIO_NET_HDR_GSO_ECN bit in \field{gso_type}. +The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation offload, unless the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is +negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL +bit in \field{gso_type}.Is this better to follow the existing style of the driver normative? For example, we had
Quoted 8 lines
"If
VIRTIO_NET_F_HOST_TSO6is negotiated, the driver MAY setgso_typetoVIRTIO_NET_HDR_GSO_TCPV6to request TCPv6 segmentation, otherwise the driver MUST NOT setgso_typetoVIRTIO_NET_HDR_GSO_TCPV6."We could do the same
"If
VIRTIO_NET_F_HOST_UDP_TUNNEL_GSOis negotiated, the driver MAY ... otherwise the driver MUST NOT ..."ok, I'll do that in the next revision
Quoted 4 lines
Quoted 4 lines
device-types/net/description.tex +0
-0@@ -598,6 +653,28 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De header. \end{itemize}Quoted 9 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 662
Quoted 7 lines
device-types/net/description.tex +7
-0+If the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO option has been negotiated, the +driver MAY set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}, if so: +\begin{itemize} +\item the driver MUST set \field{outer_th_offset} to the outer UDP header + offset, \field{inner_protocol_type} to the ethernet type of the inner + protocol, and \field{inner_nh_offset} to the inner network header offset. +\end{itemize}Do we need to say then
csum_startandcsum_offsetis for the inner packet here?
Quoted 2 lines
I thought such statement was already everywhere, but no objection to add it here, too.
I think we need some tweaks. E.g with this patch applied, in "Packet Transmission" subsection, we still have something which is not clear:
"""
\item \field{csum_start} is set to the offset within the packet to
begin checksumming,
and
\item \field{csum_offset} indicates how many bytes after the csum_start the
new (16 bit ones' complement) checksum is placed by the device.
"""
I wonder if it's more clear to just describe the outer and inner metadata there?
And in the end of this description
"""
\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature,
the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that the GSO protocol is encapsulated in a UDP tunnel.
The outer UDP header must not require checksumming, e.g. the outer
UDP checksum
must be zero.
The other tunnel-related fields indicate how to replicate the packet headers to cut it into smaller packets:
\begin{itemize}
\item \field{outer_th_offset} field indicates the outer transport
header within
the packet. Note that such field differs from \field{csum_start}as the latter
points to the inner transport header within the packet.
\item \field{inner_protocol_type} field indicates the ethernet type
of the inner
protocol.
\item \field{inner_nh_offset} field indicates the inner network header within
the packet.
\end{itemize}
"""
And it would be even nicer if an example could be give there, as we have a non tunnel example for setting metatdata:
"""
For example, consider a partially checksummed TCP (IPv4) packet. It will have a 14 byte ethernet header and 20 byte IP header followed by the TCP header (with the TCP checksum field 16 bytes into that header). \field{csum_start} will be 14+20 = 34 (the TCP checksum includes the header), and \field{csum_offset} will be 16. """
Quoted 11 lines
Somewhat related, I have a doubt WRT the csum_start/csum_offset fields "presence" with plain GSO packets. AFAICS, the specs require the csum offload feature as a pre-req for the plain segmentation feature, but also allow the reception of a GSO packet without the
VIRTIO_NET_HDR_F_NEEDS_CSUMbit set, see the recent linux patch from Willem:https://lore.kernel.org/netdev/[email protected]/
I think it will make the thing simpler if we always require such bit to be set for GSO packets.
Did you mean forbid VIRTIO_NET_HDR_F_DATA_VALID? I think it means we will prevent vendors from implementing GRO_HW for UDP tunnels.
Quoted 8 lines
I guess we can't change the current status for
the existing GSO types, but for UDP tunnel we could add that requirement from the start, both for the driver -> device direction and for the device -> driver direction.Even when the device sets the
VIRTIO_NET_HDR_F_DATA_VALID, so can always expect a valid csum_start/csum_offset pair, and make things hopefully more robust.
Ok, so you meant you want to mandate csum_start/csum_offset even in the case of VIRTIO_NET_HDR_F_DATA_VALID.
Quoted 3 lines
When
VIRTIO_NET_HDR_F_DATA_VALIDis set, the pair will NOT be used to validated the csum, just to point to the inner transport header and avoid parsing. WDYT?
It should be fine, but I think it's better to be an independent feature (e.g a new feature flag as it is not specific to tunnels).
If we add too many dependencies for UDP tunnel, it would take too long to be converged.
Thanks
Quoted 5 lines
Thanks,
Paolo
PATCH v6 2/2 virtio-net: define UDP tunnel checksum offload feature
In reply to Paolo AbeniThis complements the previous change, additionally introducing the UDP tunnel checksum offload feature.
Differently from the plain checksum offload feature, this depends on UDP tunnel segmentation being available, as outer checksum computation for non GSO packets is cheap and H/W implementation of such feature is complex.
UDP tunnel checksum offload does not introduce additional fields, instead leverages the outer transport offset introduced by the UDP tunnel segmentation feature to locate the outer checksum inside the packet.
When UDP tunnel checksum offload is enabled, the VIRTIO_NET_HDR_F_DATA_VALID bit semantic is extended, in the device -> driver direction, to cover both the inner and the outer checksum validation.
--- device-types/net/description.tex | 123 ++++++++++++++++++++++++++++--- 1 file changed, 113 insertions(+), 10 deletions(-)
device-types/net/description.tex +113 -10
@@ -91,9 +91,15 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO (47)] Driver can receive GSO packets carried by a UDP tunnel. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM (48)] Driver handles packets + carried by a UDP tunnel with partial csum for the outer header. + \item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO (49)] Device can receive GSO packets carried by a UDP tunnel. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM (50)] Device handles packets + carried by a UDP tunnel with partial csum for the outer header. + \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets. \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
@@ -141,6 +147,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_USO6] Requires VIRTIO_NET_F_GUEST_CSUM. \item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_GUEST_TSO4 and VIRTIO_NET_F_GUEST_TSO6, or both VIRTIO_NET_F_GUEST_USO4 and VIRTIO_NET_F_GUEST_USO6. +\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM] Requires VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO \item[VIRTIO_NET_F_HOST_TSO4] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_TSO6] Requires VIRTIO_NET_F_CSUM.
@@ -149,6 +156,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_HOST_USO] Requires VIRTIO_NET_F_CSUM. \item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO] Requires both VIRTIO_NET_F_HOST_TSO4 and VIRTIO_NET_F_HOST_TSO6, or both VIRTIO_NET_F_GUEST_USO4 or VIRTIO_NET_F_GUEST_USO6. ++\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM] Requires VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO \item[VIRTIO_NET_F_CTRL_RX] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_VLAN] Requires VIRTIO_NET_F_CTRL_VQ.
@@ -162,6 +170,13 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT. \end{description} +\begin{note} +The dependency between UDP_TUNNEL_CSUM and UDP_TUNNEL_GSO is intentionally +in the opposite direction with respect to the plain GSO features and the plain +checksum offload because UDP tunnel checksum offload gives very little gain +for non GSO packets is quite complex to implement in H/W. +\end{note} + \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} \begin{description} \item[VIRTIO_NET_F_GSO (6)] Device handles packets with any GSO type. This was supposed to indicate segmentation offload support, but
@@ -391,6 +406,11 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev the outer UDP checksum is zero - by negotiating the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature. +\item If VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO is negotiated, a driver can + additionally use TCP segmentation or UDP segmentation on top of UDP + encapsulation with the outer header requiring checksum offload, + negotiating the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM feature. + \item The converse features are also available: a driver can save the virtual device some work by negotiating these features.\note{For example, a network packet transported between two guests on the same system might not need checksumming at all, nor segmentation,
@@ -399,8 +419,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4, - VIRTIO_NET_F_GUEST_USO6 and VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO - are the input equivalents of the features described above. + VIRTIO_NET_F_GUEST_USO6 VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO and + VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM are the input equivalents of + the features described above. See \ref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers}~\nameref{sec:Device Types / Network Device / Device Operation / Setting Up Receive Buffers} and
@@ -424,6 +445,7 @@ \subsection{Device Operation}\label{sec:Device Types / Network Device / Device O #define VIRTIO_NET_HDR_F_NEEDS_CSUM 1 #define VIRTIO_NET_HDR_F_DATA_VALID 2 #define VIRTIO_NET_HDR_F_RSC_INFO 4 +#define VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM 8 u8 flags; #define VIRTIO_NET_HDR_GSO_NONE 0 #define VIRTIO_NET_HDR_GSO_TCPV4 1
@@ -490,7 +512,7 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De new (16 bit ones' complement) checksum is placed by the device. \item The TCP checksum field in the packet is set to the sum - of the TCP pseudo header, so that replacing it by the ones' + of the TCP pseudo header, so that replacing it with the ones' complement checksum of the TCP header and body will give the correct result. \end{itemize}
@@ -542,8 +564,11 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De \item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO feature, the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type} indicates that the GSO protocol is encapsulated in a UDP tunnel. - The outer UDP header must not require checksumming, e.g. the outer UDP checksum - must be zero. + If the outer UDP header requires checksumming, the driver must have + additionally negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM feature + and offloaded the outer checksum accordingly, otherwise + the outer UDP header must not require checksumming, e.g. the outer + UDP checksum must be zero. The other tunnel-related fields indicate how to replicate the packet headers to cut it into smaller packets:
@@ -559,6 +584,20 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De the packet. \end{itemize} +\item If the driver negotiated the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM feature, + the transmitted packet is a GSO one encapsulated in a UDP tunnel, and + the outer UDP header requires checksumming, the driver can skip checksumming + the outer header: + + \begin{itemize} + \item \field{flags} has the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM set, + + \item The outer UDP checksum field in the packet is set to the sum + of the UDP pseudo header, so that replacing it by the ones' + complement checksum of the outer UDP header and payload will give the + correct result. + \end{itemize} + \item \field{num_buffers} is set to zero. This field is unused on transmitted packets. \item The header and packet are added as one output descriptor to the
@@ -606,12 +645,23 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De bit in \field{gso_type}. The driver MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel +requiring segmentation and outer UDP checksum offload, unless both the +VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO and VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM features +are negotiated, in which case the driver MUST set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL +bit in the \field{gso_type} and the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in the \field{flags}. + +If VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM is not negotiated, the driver MUST not set +the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in the \field{flags} and +MUST NOT send to the device TCP or UDP GSO packets over UDP tunnel requiring segmentation and outer UDP checksum offload. The driver MUST NOT set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL together with VIRTIO_NET_HDR_GSO_UDP, as the latter is deprecated in favor of UDP_L4 and no new features will support it. +The driver MUST NOT set the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit \field{flags} +without setting the VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO bit in \field{gso_type}. + If the VIRTIO_NET_F_CSUM feature has been negotiated, the driver MAY set the VIRTIO_NET_HDR_F_NEEDS_CSUM bit in \field{flags}, if so:
@@ -661,8 +711,27 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De protocol, and \field{inner_nh_offset} to the inner network header offset. \end{itemize} -If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the outer -UDP header MUST NOT require checksumming. That is the, the outer UDP checksum +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM feature has been negotiated, and the +bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set, the +driver MAY set the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in +\field{flags}, if so: +\begin{enumerate} +\item the driver MUST validate the packet checksum at + offset 6 from \field{outer_th_offset} as well as all + preceding offsets; +\item the driver MUST set the packet outer UDP header checksum + to the outer UDP pseudo header; +\end{enumerate} + +\begin{note} +calculating a ones' complement checksum from \field{outer_th_offset} +up until the end of the packet and storing the result at offset 6 +from \field{outer_th_offset} will result in a fully checksummed outer UDP packet; +\end{note} + +If the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL in \field{gso_type} is set +and the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM feature has not been negotiated, the outer +UDP header MUST NOT require checksumming. That is, the outer UDP checksum value must be 0, or the valid complete checksum for such header. \begin{note}
@@ -675,6 +744,10 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De been negotiated, the driver MUST NOT set the VIRTIO_NET_HDR_F_UDP_TUNNEL bit in \field{gso_type}. +If the VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM option has not been negotiated, +the driver MUST NOT set the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit +in \field{flags}. + The driver SHOULD accept the VIRTIO_NET_F_GUEST_HDRLEN feature if it has been offered, and if it's able to provide the exact header length.
@@ -839,6 +912,12 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network bit MAY be set. In such case the \field{outer_th_offset}, \field{inner_protocol_type}, and \field{inner_nh_offset} fields indicate the corresponding headers information. +\item If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM feature was negotiated, and + the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit is set in \field{gso_type}, the + VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in the \field{flags} can be set: if so, + the outer UDP header checksum at offset 6 from \field{outer_th_offset} have + been validated. The checksum on the packet is incomplete and the + \field{outer_th_offset} indicates how to calculate it. \end{enumerate}
@@ -886,6 +965,9 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO is not negotiated the device MUST NOT set the VIRTIO_NET_HDR_GSO_UDP_TUNNEL bit in \field{gso_type}. +If VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM is not negotiated the device MUST NOT set +the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in \field{flags}. + The device SHOULD NOT send to the driver TCP packets requiring segmentation offload which have the Explicit Congestion Notification bit set, unless the VIRTIO_NET_F_GUEST_ECN feature is negotiated, in which case the
@@ -917,7 +999,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network \field{inner_nh_offset} fields to the corresponding header information, and the outer UDP header MUST NOT require checksum offload. -The device MUST NOT send the driver TCP or UDP GSO packets encapsulated in UDP +If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM feature has not been negotiated, +the device MUST NOT send the driver TCP or UDP GSO packets encapsulated in UDP tunnel and requiring segmentation and outer checksum offload. If none of the VIRTIO_NET_F_GUEST_TSO4, TSO6, UFO, USO4 or USO6 options have
@@ -943,8 +1026,27 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If the VIRTIO_NET_F_GUEST_CSUM feature has been negotiated, the device MAY set the VIRTIO_NET_HDR_F_DATA_VALID bit in \field{flags}, if so, the device MUST validate the packet -checksum (in case of multiple encapsulated protocols, one level -of checksums is validated). +checksum. If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM feature has been negotiated, +and the VIRTIO_NET_HDR_F_UDP_TUNNEL bit set, both the outer UDP checksum +and the inner transport checksum have been +validated, otherwise only one level of checksums (the inner one in case +of tunnels) has been validated. + +If the VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM feature has been negotiated, +the bit VIRTIO_NET_HDR_GSO_UDP_TUNNEL is set in \field{gso_type} +device MAY set the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM bit in +\field{flags}, if so: +\begin{enumerate} +\item the device MUST validate the packet checksum at + offset 6 from \field{outer_th_offset}; +\item the device MUST set the packet checksum stored in the + receive buffer to the outer UDP pseudo header; +\end{enumerate} + +Otherwise, if the VIRTIO_NET_HDR_GSO_UDP_TUNNEL is set in +\field{gso_type} and the bit VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM is cleared in +\field{flags}, the device MUST either provide a zero outer UDP header +checksum or a fully checksummed outer UDP header. \drivernormative{\paragraph}{Processing of Incoming Packets}{Device Types / Network Device / Device Operation /
@@ -1730,6 +1832,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi #define VIRTIO_NET_F_GUEST_ECN 9 #define VIRTIO_NET_F_GUEST_UFO 10 #define VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO 47 +#define VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM 48 #define VIRTIO_NET_F_GUEST_USO4 54 #define VIRTIO_NET_F_GUEST_USO6 55
Signature · 1 line
2.45.2
Re: [PATCH v6 2/2] virtio-net: define UDP tunnel checksum offload feature
Quoted 26 lines · Paolo Abeni
This complements the previous change, additionally introducing the UDP tunnel checksum offload feature.
Differently from the plain checksum offload feature, this depends on UDP tunnel segmentation being available, as outer checksum computation for non GSO packets is cheap and H/W implementation of such feature is complex.
UDP tunnel checksum offload does not introduce additional fields, instead leverages the outer transport offset introduced by the UDP tunnel segmentation feature to locate the outer checksum inside the packet.
When UDP tunnel checksum offload is enabled, the
VIRTIO_NET_HDR_F_DATA_VALIDbit semantic is extended, in the device -> driver direction, to cover both the inner and the outer checksum validation.+\item[VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM (48)] Driver handles packets
device-types/net/description.tex +3
-0+ carried by a UDP tunnel with partial csum for the outer header. + \item[VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO (49)] Device can receive GSO packets carried by a UDP tunnel. +\item[VIRTIO_NET_F_HOST_UDP_TUNNEL_CSUM (50)] Device handles packets
device-types/net/description.tex \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
Quoted 10 lines
device-types/net/description.tex +8
-0+ carried by a UDP tunnel with partial csum for the outer header. + \item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for encapsulated packets. +\begin{note} +The dependency between UDP_TUNNEL_CSUM and UDP_TUNNEL_GSO is intentionally +in the opposite direction with respect to the plain GSO features and the plain +checksum offload because UDP tunnel checksum offload gives very little gain +for non GSO packets is quite complex to implement in H/W. +\end{note}
Plan was to call this UDP_TUNNEL_GSO_CSUM, to make this implicit dependency more explicit? It really only is checksum offload for the outer checksum that this feature covers.
device-types/net/description.tex \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 424
Quoted 12 lines
\item The converse features are also available: a driver can save
the virtual device some work by negotiating these features.\note{For example, a network packet transported between two guests on
the same system might not need checksumming at all, nor segmentation,device-types/net/description.tex +3
-2@@ -399,8 +419,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4, - VIRTIO_NET_F_GUEST_USO6 and VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO - are the input equivalents of the features described above. + VIRTIO_NET_F_GUEST_USO6 VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO and + VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM are the input equivalents of + the features described above.
Minor, not specific to this feature and more to the spec maintainers:
Does it make sense to put each constant on its own line? To avoid having to edit existing lines when expanding. This is source text, so that does not show in the generated output anyway:
and if it can do that then the VIRTIO_NET_F_GUEST_TSO4,VIRTIO_NET_F_GUEST_TSO6,
VIRTIO_NET_F_GUEST_UFO,VIRTIO_NET_F_GUEST_ECN,VIRTIO_NET_F_GUEST_USO4,
VIRTIO_NET_F_GUEST_USO6,
+ VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO,
are the input equivalents of the features described above.
The trailing comma is admittedly a bit ugly. If fixing that up, the diff is still cleaner with one constant per line:
- VIRTIO_NET_F_GUEST_USO6
+ VIRTIO_NET_F_GUEST_USO6,
+ VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 515
Quoted 6 lines
device-types/net/description.tex +1
-1@@ -490,7 +512,7 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De new (16 bit ones' complement) checksum is placed by the device. \item The TCP checksum field in the packet is set to the sum - of the TCP pseudo header, so that replacing it by the ones' + of the TCP pseudo header, so that replacing it with the ones'
Unintentional change?
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 599
Quoted 8 lines
device-types/net/description.tex +8
-0+ \begin{itemize} + \item \field{flags} has the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM set, + + \item The outer UDP checksum field in the packet is set to the sum + of the UDP pseudo header, so that replacing it by the ones' + complement checksum of the outer UDP header and payload will give the + correct result. + \end{itemize}
"set to the sum of the UDP pseudo header [of each segment]"?
as it will not be the UDP length of the gso packet.
Re: [PATCH v6 2/2] virtio-net: define UDP tunnel checksum offload feature
Quoted 11 lines · Willem de Bruijn
device-types/net/description.tex \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device · line 178
Quoted 6 lines · Paolo Abeni
+\begin{note}
device-types/net/description.tex +5
-0+The dependency between UDP_TUNNEL_CSUM and UDP_TUNNEL_GSO is intentionally +in the opposite direction with respect to the plain GSO features and the plain +checksum offload because UDP tunnel checksum offload gives very little gain +for non GSO packets is quite complex to implement in H/W. +\end{note}Plan was to call this
UDP_TUNNEL_GSO_CSUM, to make this implicit dependency more explicit? It really only is checksum offload for the outer checksum that this feature covers.
Indeed. I lost that change in the split. I'll fix in the next revision.
Quoted 36 lines
device-types/net/description.tex \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev · line 424
Quoted 12 lines
\item The converse features are also available: a driver can save
the virtual device some work by negotiating these features.\note{For example, a network packet transported between two guests on
the same system might not need checksumming at all, nor segmentation,device-types/net/description.tex +3
-2@@ -399,8 +419,9 @@ \subsection{Device Initialization}\label{sec:Device Types / Network Device / Dev checksummed packets can be received, and if it can do that then the VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, VIRTIO_NET_F_GUEST_UFO, VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_USO4, - VIRTIO_NET_F_GUEST_USO6 and VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO - are the input equivalents of the features described above. + VIRTIO_NET_F_GUEST_USO6 VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO and + VIRTIO_NET_F_GUEST_UDP_TUNNEL_CSUM are the input equivalents of + the features described above.Minor, not specific to this feature and more to the spec maintainers:
Does it make sense to put each constant on its own line? To avoid having to edit existing lines when expanding. This is source text, so that does not show in the generated output anyway:
and if it can do that then the
VIRTIO_NET_F_GUEST_TSO4,VIRTIO_NET_F_GUEST_TSO6,
VIRTIO_NET_F_GUEST_UFO,VIRTIO_NET_F_GUEST_ECN,VIRTIO_NET_F_GUEST_USO4,
VIRTIO_NET_F_GUEST_USO6,
+VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO,
are the input equivalents of the features described above.The trailing comma is admittedly a bit ugly. If fixing that up, the diff is still cleaner with one constant per line:
-
VIRTIO_NET_F_GUEST_USO6
+VIRTIO_NET_F_GUEST_USO6,
+VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO
Quoted 8 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 515
Quoted 6 lines
device-types/net/description.tex +1
-1@@ -490,7 +512,7 @@ \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De new (16 bit ones' complement) checksum is placed by the device. \item The TCP checksum field in the packet is set to the sum - of the TCP pseudo header, so that replacing it by the ones' + of the TCP pseudo header, so that replacing it with the ones'Unintentional change?
Yep, I piped the patch in a grammar checker and while reflecting the suggestions on the patch itself I unintentionally touched the context, too.
Quoted 9 lines
device-types/net/description.tex \subsubsection{Packet Transmission}\label{sec:Device Types / Network Device / De · line 599
Quoted 8 lines
device-types/net/description.tex +8
-0+ \begin{itemize} + \item \field{flags} has the VIRTIO_NET_HDR_F_UDP_TUNNEL_CSUM set, + + \item The outer UDP checksum field in the packet is set to the sum + of the UDP pseudo header, so that replacing it by the ones' + complement checksum of the outer UDP header and payload will give the + correct result. + \end{itemize}
Quoted 3 lines
"set to the sum of the UDP pseudo header [of each segment]"?
as it will not be the UDP length of the gso packet.
AFAICS the Linux tunnel implementation uses the GSO length:
<each UDP tunnel> -> udp_tunnel_xmit_skb -> udp_set_csum
https://elixir.bootlin.com/linux/v6.10.2/source/net/ipv4/udp_tunnel_core.c#L170
https://elixir.bootlin.com/linux/v6.10.2/source/net/ipv4/udp.c#L890
Regardless of the Linux implementation, using the individual segment len will makes things difficult for the last segment (len < gso_size). The caller will have to touch the individual segment headers, which could be quite hard/costily.
Am I missing something?
Thanks,
Paolo