virtio-net: support inner header hash

PATCH Superseded Heng Qi 3 participants 8 messages Jun 5, 2023 – Jun 8, 2023

Share & tools

Heng Qi

author

2023-06-05 07:16 UTC #0 lore raw

PATCH v15 1/1 virtio-net: support inner header hash

1. Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. The same flow can traverse through different tunnels, resulting in the encapsulated packets being spread across multiple receive queues (refer to the figure below). However, in certain scenarios, we may need to direct these encapsulated packets of the same flow to a single receive queue. This facilitates the processing of the flow by the same CPU to improve performance (warm caches, less locking, etc.).

               client1                    client2
                  |        +-------+         |
                  +------->|tunnels|<--------+
                           +-------+
                              |  |
                              v  v
                      +-----------------+
                      | monitoring host |
                      +-----------------+

To achieve this, the device can calculate a symmetric hash based on the inner headers of the same flow.

2. For legacy systems, they may lack entropy fields which modern protocols have in the outer header, resulting in multiple flows with the same outer header but different inner headers being directed to the same receive queue. This results in poor receive performance.

To address this limitation, inner header hash can be used to enable the device to advertise the capability to calculate the hash for the inner packet, regaining better receive performance.

Signed-off-by Heng Qi <[email protected]>

Reviewed-by Xuan Zhuo <[email protected]>

---
v14->v15:
1. Add tunnel hash option suggested by @Michael S . Tsirkin
2. Adjust some descriptions.

v13->v14:
1. Move supported_hash_tunnel_types from config space into cvq command. @Parav Pandit
2. Rebase to master branch.
3. Some minor modifications.

v12->v13:
1. Add a GET command for hash_tunnel_types. @Parav Pandit
2. Add tunneling protocol explanation. @Jason Wang
3. Add comments on some usage scenarios for inner hash.

v11->v12:
1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
2. Refine the commit log. @Michael S . Tsirkin
3. Add some tunnel types.

v10->v11:
1. Revise commit log for clarity for readers.
2. Some modifications to avoid undefined terms. @Parav Pandit
3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
4. Add the normative statements. @Parav Pandit

v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

 device-types/net/description.tex        | 184 ++++++++++++++++++++++++
 device-types/net/device-conformance.tex |   1 +
 device-types/net/driver-conformance.tex |   1 +
 introduction.tex                        |  40 ++++++
 4 files changed, 226 insertions(+)

device-types/net/description.tex +184 -0

index 3030222..1709bc0 100644
@@ -88,6 +88,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(51)] Device supports inner header hash for tunnel-encapsulated packets.
+
 \item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
 
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.

@@ -147,6 +149,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS and/or VIRTIO_NET_F_HASH_REPORT.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -869,6 +872,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 If the feature VIRTIO_NET_F_RSS was negotiated:
 \begin{itemize}
 \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtnet_hash_tunnel_config_get structure as 'Enabled encapsulation hash types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
 \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
 \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
 \end{itemize}

@@ -891,6 +895,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 
 \subparagraph{Supported/enabled hash types}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]}, \hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
 Hash types applicable for IPv4 packets:
 \begin{lstlisting}
 #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)

@@ -1001,6 +1006,185 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
 \end{itemize}
 
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the driver can send commands VIRTIO_NET_CTRL_HASH_TUNNEL_SET
+and VIRTIO_NET_CTRL_HASH_TUNNEL_GET for the inner header hash configuration.
+
+struct virtnet_hash_tunnel_config_set {
+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];
+};
+struct virtnet_hash_tunnel_config_get {
+    le32 supported_hash_tunnel_types;
+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];
+};
+
+#define VIRTIO_NET_CTRL_HASH_TUNNEL 7
+ #define VIRTIO_NET_CTRL_HASH_TUNNEL_SET 0
+ #define VIRTIO_NET_CTRL_HASH_TUNNEL_GET 1
+
+Filed \field{supported_hash_tunnel_types} provided by the device indicates that the device is capable of supporting inner header hash for these encapsulation types.
+\field{supported_hash_tunnel_types} contains the bitmask of supported tunnel hash types. See \ref{sec:Device Types / Network Device / Device Operation / Processing
+of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
+
+Filed \field{hash_tunnel_types} contains the bitmask of configured tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
+
+Field \field{hash_option} is one byte containing 0 (off) or 1 (on).
+\begin{itemize}
+\item \field{hash_option} = 0: instruct the device to calculate the inner header hash for the configured \field{hash_tunnel_types}.
+\item \field{hash_option} = 1: instruct the device to calculate the hash on the source UDP port for all encapsulated and non-encapsulated UDP packets.
+\end{itemize}
+
+For most UDP-encapsulated packets, the outer source UDP port may identify a specific flow, such as \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]},
+\hyperref[intro:geneve]{[GENEVE]} and \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}.
+Therefore, the hash calculated on the outer source UDP port can allow the same flow passing through different tunnels to be directed to the same receive queue.
+
+Field \field{reserved} is reserved and is ignored by the device.
+
+The class VIRTIO_NET_CTRL_HASH_TUNNEL has the following commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_SET: set the tunnel hash configuration for the device using the virtnet_hash_tunnel_config_set structure, which is read-only for the driver.
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_GET: get the tunnel hash configuration from the device using the virtnet_hash_tunnel_config_get structure, which is write-only for the driver.
+\end{itemize}
+
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+
+A tunnel packet is encapsulated from the original packet based on the tunneling protocol (only a single level of
+encapsulation is currently supported). The encapsulated packet contains an outer header and an inner header, and
+the device calculates the hash over either the inner header or the outer header.
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and \field{hash_option} is set to 1 by the driver:
+\begin{itemize}
+\item the hash on the source UDP port is calculated for a received UDP packet.
+\item the inner header hash is calculated for the configured \field{hash_tunnel_types} without the outer UDP header.
+\end{itemize}
+
+If \field{hash_option} is 0, tunnel hashing behaves as follows:
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated packet's outer header matches
+one of the configured \field{hash_tunnel_types}, the hash of the inner header is calculated.
+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan]{[VXLAN]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:geneve]{[GENEVE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:ipip]{[IPIP]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:nvgre]{[NVGRE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\end{itemize}
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is not included in the configured \field{hash_tunnel_types},
+the hash of the outer header is calculated for the received encapsulated packet.
+
+The hash is calculated for the received non-encapsulated packet as if VIRTIO_NET_F_HASH_TUNNEL was not negotiated.
+
+\subparagraph{Supported/enabled encapsulation hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}
+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE        (1 << 0)
+\end{lstlisting}
+
+Supported encapsulation hash types:
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan]{[VXLAN]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 5)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 6)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:geneve]{[GENEVE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 7)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:ipip]{[IPIP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 8)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:nvgre]{[NVGRE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 9)
+\end{lstlisting}
+
+\subparagraph{Advice}
+Usage scenarios of inner header hash (but not limited to):
+\begin{itemize}
+\item Legacy tunneling protocols that lack entropy in the outer header use inner header hash to hash flows
+      with the same outer header but different inner headers to different queues for better-receiving performance.
+\item In scenarios where the same flow passing through different tunnels is expected to be received in the same queue,
+      warm caches, lessing locking, etc. are optimized to obtain receiving performance.
+\end{itemize}
+
+For scenarios with sufficient outer entropy or no inner header hash requirements, inner header hash may not be needed:
+A tunnel is often expected to isolate the external network from the internal one. By completely ignoring entropy
+in the outer header and replacing it with entropy from the inner header, for hash calculations, this expectation
+might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue
+selection, inner header hash may have quality of service (QoS) limitations.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+      to disable inner header hash for encapsulated packets.
+\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}
+
+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The device MUST calculate the hash on the source UDP port for all UDP packets if \field{hash_option} is 1.
+
+The device MUST calculate the inner header hash for the configured \field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+
+The device MUST calculate the hash on the outer header if \field{hash_option} is 0 and the type of the received
+encapsulated packet does not match any value of \field{hash_tunnel_types}.
+
+The device MUST respond to the VIRTIO_NET_CTRL_HASH_TUNNEL_SET command with VIRTIO_NET_ERR if the device received
+an unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ flag or an unsupported value for \field{hash_option}.
+
+The device MUST provide the values of \field{supported_tunnel_hash_types} if it offers the VIRTIO_NET_F_HASH_TUNNEL feature.
+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} and \field{hash_option} to 0.
+
+The device MUST ignore \field{reserved}.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The driver MUST have negotiated the VIRTIO_NET_F_HASH_TUNNEL feature when issuing commands VIRTIO_NET_CTRL_HASH_TUNNEL_SET and VIRTIO_NET_CTRL_HASH_TUNNEL_GET.
+
+The driver MUST ignore the values received from the VIRTIO_NET_CTRL_HASH_TUNNEL_GET command if the device responds with VIRTIO_NET_ERR.
+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags not supported by the device.
+
+The driver MUST NOT set \field{hash_option} to any value not supported by the device.
+
 \paragraph{Hash reporting for incoming packets}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}

device-types/net/device-conformance.tex +1 -0

index 54f6783..f88f48b 100644
@@ -14,4 +14,5 @@
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
 \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
 \end{itemize}

device-types/net/driver-conformance.tex +1 -0

index 97d0cc1..9d853d9 100644
@@ -14,4 +14,5 @@
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
 \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
 \end{itemize}

introduction.tex +40 -0

index b7155bf..3f34950 100644
@@ -102,6 +102,46 @@ \section{Normative References}\label{sec:Normative References}
     Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
 	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:gre_rfc2784}\textbf{[GRE_rfc2784]} &
+    Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol.
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:gre_rfc2890}\textbf{[GRE_rfc2890]} &
+    Key and Sequence Number Extensions to GRE \ref{intro:gre_rfc2784}. This protocol describes extensions by which two fields, Key and
+    Sequence Number, can be optionally carried in the GRE Header \ref{intro:gre_rfc2784}.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2890}\\
+	\phantomsection\label{intro:gre_rfc7676}\textbf{[GRE_rfc7676]} &
+    IPv6 Support for Generic Routing Encapsulation (GRE). This protocol is specified for IPv6 and used as either the payload or
+    delivery protocol. Note that this does not change the GRE header format or any behaviors specified by RFC 2784 or RFC 2890.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7676/}\\
+	\phantomsection\label{intro:gre_in_udp_rfc8086}\textbf{[GRE-in-UDP]} &
+    GRE-in-UDP Encapsulation. This specifies a method of encapsulating network protocol packets within GRE and UDP headers.
+    This GRE-in-UDP encapsulation allows the UDP source port field to be used as an entropy field. This protocol is specified for IPv4 and IPv6,
+    and used as either the payload or delivery protocol.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc8086}\\
+	\phantomsection\label{intro:vxlan}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:vxlan-gpe}\textbf{[VXLAN-GPE]} &
+    Generic Protocol Extension for VXLAN. This protocol describes extending Virtual eXtensible Local Area Network (VXLAN) via changes to the VXLAN header.
+	\newline\url{https://www.ietf.org/archive/id/draft-ietf-nvo3-vxlan-gpe-12.txt}\\
+	\phantomsection\label{intro:geneve}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation.
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:ipip}\textbf{[IPIP]} &
+    IP Encapsulation within IP.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:nvgre}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc791}\\
+	\phantomsection\label{intro:UDP}\textbf{[UDP]} &
+    User Datagram Protocol
+	\newline\url{https://www.rfc-editor.org/rfc/rfc768}\\
+	\phantomsection\label{intro:TCP}\textbf{[TCP]} &
+    TRANSMISSION CONTROL PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc793}\\
 \end{longtable}
 
 \section{Non-Normative References}

Signature · 18 lines

2.19.1.6.gb485710b


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Parav Pandit

2023-06-07 19:55 UTC #1 lore raw

Quoted 53 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 5, 2023 3:16 AM
\subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types /
Network Device / Feature bits / Legacy Interface: Feature bits} @@ -869,6 +872,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If the feature VIRTIO_NET_F_RSS was negotiated:
\begin{itemize}
\item The device uses \field{hash_types} of the virtio_net_rss_config structure
as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtnet_hash_tunnel_config_get structure as 'Enabled encapsulation hash types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
\item The device uses a key as defined in \field{hash_key_data} and
\field{hash_key_length} of the virtio_net_rss_config structure (see \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
\end{itemize}
(quoted diff) +0 -0
@@ -891,6 +895,7 @@ \subsubsection{Processing of Incoming
Packets}\label{sec:Device Types / Network
\subparagraph{Supported/enabled hash types} \label{sec:Device Types /
Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} +This paragraph relies on definitions from \hyperref[intro:IP]{[IP]}, \hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
Hash types applicable for IPv4 packets:
\begin{lstlisting}
#define VIRTIO_NET_HASH_TYPE_IPv4 (1 << 0)
(quoted diff) +0 -0
@@ -1001,6 +1006,185 @@ \subsubsection{Processing of Incoming
Packets}\label{sec:Device Types / Network (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
\end{itemize}
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation /
(quoted diff) +4 -0
+Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the driver can send
+commands VIRTIO_NET_CTRL_HASH_TUNNEL_SET and
VIRTIO_NET_CTRL_HASH_TUNNEL_GET for the inner header hash configuration.
device-types/net/description.tex +8 -0
+
+struct virtnet_hash_tunnel_config_set {
+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];
+};
+struct virtnet_hash_tunnel_config_get {
+    le32 supported_hash_tunnel_types;

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network

Quoted 10 lines

device-types/net/description.tex +10 -0

+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];
+};
+
+#define VIRTIO_NET_CTRL_HASH_TUNNEL 7
+ #define VIRTIO_NET_CTRL_HASH_TUNNEL_SET 0  #define
+VIRTIO_NET_CTRL_HASH_TUNNEL_GET 1
+
+Filed \field{supported_hash_tunnel_types} provided by the device indicates

s/Filed/Field

Quoted 1 line

that the device is capable of supporting inner header hash for these

s/the device is capable of supporting/the device supports/

Review

Quoted 7 lines

encapsulation types.
+\field{supported_hash_tunnel_types} contains the bitmask of supported +tunnel hash types. See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
(quoted diff) +2 -0
+
+Filed \field{hash_tunnel_types} contains the bitmask of configured tunnel hash

s/Filed/Field

Quoted 11 lines

types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
device-types/net/description.tex +4 -0
+
+Field \field{hash_option} is one byte containing 0 (off) or 1 (on).
+\begin{itemize}
+\item \field{hash_option} = 0: instruct the device to calculate the inner header
hash for the configured \field{hash_tunnel_types}.
+\item \field{hash_option} = 1: instruct the device to calculate the hash on the source UDP port for all encapsulated and non-encapsulated UDP packets.

Why non-encapsulated UDP packets also?
Shouldn't we use the regular RSS hashing for non-encapsulated UDP packets which uses all 4 fields of ip and port?

Quoted 34 lines

+\end{itemize}
(quoted diff) +5 -0
+
+For most UDP-encapsulated packets, the outer source UDP port may
+identify a specific flow, such as
+\hyperref[intro:vxlan_gpe]{[VXLAN-GPE]},
+\hyperref[intro:geneve]{[GENEVE]} and
\hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}.
+Therefore, the hash calculated on the outer source UDP port can allow the same flow passing through different tunnels to be directed to the same receive queue.
device-types/net/description.tex +6 -0
+
+Field \field{reserved} is reserved and is ignored by the device.
+
+The class VIRTIO_NET_CTRL_HASH_TUNNEL has the following commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_SET: set the tunnel hash
configuration for the device using the virtnet_hash_tunnel_config_set structure, which is read-only for the driver.
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_GET: get the tunnel hash configuration from the device using the virtnet_hash_tunnel_config_get structure, which is write-only for the driver.
+\end{itemize}
(quoted diff) +8 -0
+
+\subparagraph{Tunnel/Encapsulated packet} \label{sec:Device Types /
+Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+
+A tunnel packet is encapsulated from the original packet based on the
+tunneling protocol (only a single level of encapsulation is currently
+supported). The encapsulated packet contains an outer header and an inner
header, and the device calculates the hash over either the inner header or the outer header.
(quoted diff) +2 -0
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and \field{hash_option} is set to

Quoted 3 lines

1 by the driver:
+\begin{itemize}
+\item the hash on the source UDP port is calculated for a received UDP packet.

Need to mention here that the source UDP port is of outer or inner header.

Quoted 85 lines

+\item the inner header hash is calculated for the configured \field{hash_tunnel_types} without the outer UDP header. +\end{itemize}
device-types/net/description.tex +5 -0
+
+If \field{hash_option} is 0, tunnel hashing behaves as follows:
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated
+packet's outer header matches one of the configured
\field{hash_tunnel_types}, the hash of the inner header is calculated.
device-types/net/description.tex +4 -0
+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}: the outer header is over
IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan]{[VXLAN]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:geneve]{[GENEVE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:ipip]{[IPIP]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol. +\item \hyperref[intro:nvgre]{[NVGRE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\end{itemize}
(quoted diff) +3 -0
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is
+not included in the configured \field{hash_tunnel_types}, the hash of the outer
header is calculated for the received encapsulated packet.
device-types/net/description.tex +5 -0
+
+\subparagraph{Advice}
+Usage scenarios of inner header hash (but not limited to):
+\begin{itemize}
+\item Legacy tunneling protocols that lack entropy in the outer header use
inner header hash to hash flows
+ with the same outer header but different inner headers to different queues
for better-receiving performance.
+\item In scenarios where the same flow passing through different tunnels is expected to be received in the same queue,
+ warm caches, lessing locking, etc. are optimized to obtain receiving
performance.
+\end{itemize}
(quoted diff) +2 -0
+
+For scenarios with sufficient outer entropy or no inner header hash
requirements, inner header hash may not be needed:
(quoted diff) +4 -0
+A tunnel is often expected to isolate the external network from the
+internal one. By completely ignoring entropy in the outer header and
+replacing it with entropy from the inner header, for hash calculations,
+this expectation might be violated to a certain extent, depending on how the
hash is used. When the hash use is limited to RSS queue selection, inner header hash may have quality of service (QoS) limitations.
device-types/net/description.tex +4 -0
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the receive queue
from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+ to disable inner header hash for encapsulated packets.
+\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}
(quoted diff) +6 -0
+
+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types /
+Network Device / Device Operation / Control Virtqueue / Inner Header
+Hash}
+
+The device MUST calculate the hash on the source UDP port for all UDP
packets if \field{hash_option} is 1.
(quoted diff) +2 -0
+
+The device MUST calculate the inner header hash for the configured

Quoted 3 lines

\field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+

This I think conflicts with the current RSS where if this override (higher priority than RSS) or not. I do not understand the motivation for hash_option = 1 for non-encapsulated packet. Also it is probably rename from hash_option to hash_src_udp_port.

Quoted 4 lines

+The device MUST calculate the hash on the outer header if +\field{hash_option} is 0 and the type of the received encapsulated packet does not match any value of \field{hash_tunnel_types}.
+

[..]

Quoted 3 lines

+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags not supported by the device.
+

s/flags not supported/flags which are not supported/

Quoted 6 lines

+The driver MUST NOT set \field{hash_option} to any value not supported by the device.
+
\paragraph{Hash reporting for incoming packets} \label{sec:Device Types /
Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Heng Qi

author

2023-06-08 06:40 UTC #2 lore raw

Quoted 56 lines · Parav Pandit

Quoted 56 lines
From: Heng Qi <[email protected]>
Sent: Monday, June 5, 2023 3:16 AM
\subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types /
Network Device / Feature bits / Legacy Interface: Feature bits} @@ -869,6 +872,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network If the feature VIRTIO_NET_F_RSS was negotiated:
\begin{itemize}
\item The device uses \field{hash_types} of the virtio_net_rss_config structure
as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtnet_hash_tunnel_config_get structure as 'Enabled encapsulation hash types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
\item The device uses a key as defined in \field{hash_key_data} and
\field{hash_key_length} of the virtio_net_rss_config structure (see \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
\end{itemize}
(quoted diff) +0 -0
@@ -891,6 +895,7 @@ \subsubsection{Processing of Incoming
Packets}\label{sec:Device Types / Network
\subparagraph{Supported/enabled hash types} \label{sec:Device Types /
Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} +This paragraph relies on definitions from \hyperref[intro:IP]{[IP]}, \hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
Hash types applicable for IPv4 packets:
\begin{lstlisting}
#define VIRTIO_NET_HASH_TYPE_IPv4 (1 << 0)
(quoted diff) +0 -0
@@ -1001,6 +1006,185 @@ \subsubsection{Processing of Incoming
Packets}\label{sec:Device Types / Network (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
\end{itemize}
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation /
(quoted diff) +4 -0
+Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the driver can send
+commands VIRTIO_NET_CTRL_HASH_TUNNEL_SET and
VIRTIO_NET_CTRL_HASH_TUNNEL_GET for the inner header hash configuration.
device-types/net/description.tex +11 -0
+
+struct virtnet_hash_tunnel_config_set {
+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];
+};
+struct virtnet_hash_tunnel_config_get {
+    le32 supported_hash_tunnel_types;
+    le32 hash_tunnel_types;
+    u8 hash_option;
+    u8 reserved[3];

Quoted 8 lines

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
Quoted 7 lines
device-types/net/description.tex +7 -0
+};
+
+#define VIRTIO_NET_CTRL_HASH_TUNNEL 7
+ #define VIRTIO_NET_CTRL_HASH_TUNNEL_SET 0  #define
+VIRTIO_NET_CTRL_HASH_TUNNEL_GET 1
+
+Filed \field{supported_hash_tunnel_types} provided by the device indicates
s/Filed/Field

Will fix.

Quoted 4 lines

Quoted 2 lines
that the device is capable of supporting inner header hash for these
s/the device is capable of supporting/the device supports/

Will modify.

Quoted 9 lines

Review
Quoted 7 lines
encapsulation types.
+\field{supported_hash_tunnel_types} contains the bitmask of supported +tunnel hash types. See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
(quoted diff) +2 -0
+
+Filed \field{hash_tunnel_types} contains the bitmask of configured tunnel hash
s/Filed/Field

Quoted 13 lines

Quoted 11 lines
types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}.
device-types/net/description.tex +4 -0
+
+Field \field{hash_option} is one byte containing 0 (off) or 1 (on).
+\begin{itemize}
+\item \field{hash_option} = 0: instruct the device to calculate the inner header
hash for the configured \field{hash_tunnel_types}.
+\item \field{hash_option} = 1: instruct the device to calculate the hash on the source UDP port for all encapsulated and non-encapsulated UDP packets.
Why non-encapsulated UDP packets also?
Shouldn't we use the regular RSS hashing for non-encapsulated UDP packets which uses all 4 fields of ip and port?

I think this simplifies the work of the device parsing the packet, that is, if \field{hash_option} is set to 1, then the device only needs to parse to the outer transport header, without knowing what tunneling protocol the special dst port corresponds to or further checking the tunneling protocol information.

But this is really a bit rough for non-encapsulated packets, we can only support this operation for encapsulated packets.

Quoted 32 lines

Quoted 32 lines
+\end{itemize}
(quoted diff) +5 -0
+
+For most UDP-encapsulated packets, the outer source UDP port may
+identify a specific flow, such as
+\hyperref[intro:vxlan_gpe]{[VXLAN-GPE]},
+\hyperref[intro:geneve]{[GENEVE]} and
\hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}.
+Therefore, the hash calculated on the outer source UDP port can allow the same flow passing through different tunnels to be directed to the same receive queue.
device-types/net/description.tex +6 -0
+
+Field \field{reserved} is reserved and is ignored by the device.
+
+The class VIRTIO_NET_CTRL_HASH_TUNNEL has the following commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_SET: set the tunnel hash
configuration for the device using the virtnet_hash_tunnel_config_set structure, which is read-only for the driver.
+\item VIRTIO_NET_CTRL_HASH_TUNNEL_GET: get the tunnel hash configuration from the device using the virtnet_hash_tunnel_config_get structure, which is write-only for the driver.
+\end{itemize}
(quoted diff) +8 -0
+
+\subparagraph{Tunnel/Encapsulated packet} \label{sec:Device Types /
+Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+
+A tunnel packet is encapsulated from the original packet based on the
+tunneling protocol (only a single level of encapsulation is currently
+supported). The encapsulated packet contains an outer header and an inner
header, and the device calculates the hash over either the inner header or the

Quoted 8 lines

Quoted 6 lines
outer header.
(quoted diff) +2 -0
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and \field{hash_option} is set to
1 by the driver:
+\begin{itemize}
+\item the hash on the source UDP port is calculated for a received UDP packet.
Need to mention here that the source UDP port is of outer or inner header.

Yes, if we only support encapsulated packets for \field{hash_option} or \field{hash_src_udp_port}, that should be the source UDP port number.

Quoted 82 lines

Quoted 82 lines
+\item the inner header hash is calculated for the configured \field{hash_tunnel_types} without the outer UDP header. +\end{itemize}
device-types/net/description.tex +5 -0
+
+If \field{hash_option} is 0, tunnel hashing behaves as follows:
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated
+packet's outer header matches one of the configured
\field{hash_tunnel_types}, the hash of the inner header is calculated.
device-types/net/description.tex +4 -0
+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}: the outer header is over
IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan]{[VXLAN]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:geneve]{[GENEVE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header uses UDP as the transport protocol.
+\item \hyperref[intro:ipip]{[IPIP]}: the outer header is over IPv4 and the inner header is over IPv4. The outer header does not contain the transport protocol. +\item \hyperref[intro:nvgre]{[NVGRE]}: the outer header is over IPv4/IPv6 and the inner header is over IPv4/IPv6. The outer header does not contain the transport protocol.
+\end{itemize}
(quoted diff) +3 -0
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is
+not included in the configured \field{hash_tunnel_types}, the hash of the outer
header is calculated for the received encapsulated packet.
device-types/net/description.tex +5 -0
+
+\subparagraph{Advice}
+Usage scenarios of inner header hash (but not limited to):
+\begin{itemize}
+\item Legacy tunneling protocols that lack entropy in the outer header use
inner header hash to hash flows
+ with the same outer header but different inner headers to different queues
for better-receiving performance.
+\item In scenarios where the same flow passing through different tunnels is expected to be received in the same queue,
+ warm caches, lessing locking, etc. are optimized to obtain receiving
performance.
+\end{itemize}
(quoted diff) +2 -0
+
+For scenarios with sufficient outer entropy or no inner header hash
requirements, inner header hash may not be needed:
(quoted diff) +4 -0
+A tunnel is often expected to isolate the external network from the
+internal one. By completely ignoring entropy in the outer header and
+replacing it with entropy from the inner header, for hash calculations,
+this expectation might be violated to a certain extent, depending on how the
hash is used. When the hash use is limited to RSS queue selection, inner header hash may have quality of service (QoS) limitations.
device-types/net/description.tex +4 -0
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the receive queue
from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+ to disable inner header hash for encapsulated packets.
+\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}
(quoted diff) +5 -0
+
+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types /
+Network Device / Device Operation / Control Virtqueue / Inner Header
+Hash}
+

Quoted 8 lines

Quoted 7 lines
+The device MUST calculate the hash on the source UDP port for all UDP packets if \field{hash_option} is 1.
(quoted diff) +2 -0
+
+The device MUST calculate the inner header hash for the configured
\field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+
This I think conflicts with the current RSS where if this override (higher priority than RSS) or not.

VIRTIO_NET_F_HASH_TUNNEL instructs the device to calculate an inner header hash for configured tunnel types.
Then, which fields are used for the calculation of the inner header hash is determined by the RSS configuration.

Therefore, if the inner header hash uses \field{hash_option} to indicate that the outer source UDP port is used to calculate the hash, it overrides the RSS configuration (higher priority than RSS). It now not only indicates for which tunnel types the inner hash should be calculated, but also indicates which fields participate in the RSS hash calculation.

At this point, we can
1. continue to allow this behavior. That is, if \field{hash_option} is set to 1, the hash value calculated based on the outer source port indicated by VIRTIO_NET_F_HASH_TUNNEL is provided to the indirection table to select the rxq instead of the hash value provided by RSS.

2. hand over to RSS the ability to specify a hash based on the outer source port. That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4, v6, EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols.
This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.

Quoted 2 lines

I do not understand the motivation for hash_option = 1 for non-encapsulated packet. Also it is probably rename from hash_option to hash_src_udp_port.

Ok. Will rename.

Thanks!

Quoted 12 lines

Quoted 5 lines
+The device MUST calculate the hash on the outer header if +\field{hash_option} is 0 and the type of the received encapsulated packet does not match any value of \field{hash_tunnel_types}.
+
[..]
Quoted 3 lines
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags not supported by the device.
+
s/flags not supported/flags which are not supported/

Quoted 10 lines

Quoted 6 lines
+The driver MUST NOT set \field{hash_option} to any value not supported by the device.
+
\paragraph{Hash reporting for incoming packets} \label{sec:Device Types /
Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Parav Pandit

2023-06-08 14:26 UTC #3 lore raw

Quoted 3 lines

From: Heng Qi <[email protected]>
Sent: Thursday, June 8, 2023 2:41 AM

[..]

Quoted 24 lines

Quoted 5 lines
Quoted 4 lines
+The device MUST calculate the inner header hash for the configured \field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+
This I think conflicts with the current RSS where if this override (higher priority
than RSS) or not.
VIRTIO_NET_F_HASH_TUNNEL instructs the device to calculate an inner header hash for configured tunnel types.
Then, which fields are used for the calculation of the inner header hash is determined by the RSS configuration.
Therefore, if the inner header hash uses \field{hash_option} to indicate that the outer source UDP port is used to calculate the hash, it overrides the RSS configuration (higher priority than RSS). It now not only indicates for which tunnel types the inner hash should be calculated, but also indicates which fields participate in the RSS hash calculation.
At this point, we can
1. continue to allow this behavior. That is, if \field{hash_option} is set to 1, the hash value calculated based on the outer source port indicated by VIRTIO_NET_F_HASH_TUNNEL is provided to the indirection table to select the rxq instead of the hash value provided by RSS.

Quoted 7 lines

2. hand over to RSS the ability to specify a hash based on the outer source port. That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4, v6, EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols. This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.

#2 is preferred.

But I must be missing something.
Without VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X, today if one wants to do RSS based on the outer UDP ports, It is already supported by the RSS (even in presence of the inner header).

So why do have to introduce VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Michael S. Tsirkin

2023-06-08 14:41 UTC #4 lore raw

Quoted 36 lines · Parav Pandit

Quoted 3 lines
From: Heng Qi <[email protected]>
Sent: Thursday, June 8, 2023 2:41 AM
[..]
Quoted 24 lines
Quoted 5 lines
Quoted 4 lines
+The device MUST calculate the inner header hash for the configured \field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+
This I think conflicts with the current RSS where if this override (higher priority
than RSS) or not.
VIRTIO_NET_F_HASH_TUNNEL instructs the device to calculate an inner header hash for configured tunnel types.
Then, which fields are used for the calculation of the inner header hash is determined by the RSS configuration.
Therefore, if the inner header hash uses \field{hash_option} to indicate that the outer source UDP port is used to calculate the hash, it overrides the RSS configuration (higher priority than RSS). It now not only indicates for which tunnel types the inner hash should be calculated, but also indicates which fields participate in the RSS hash calculation.
At this point, we can
1. continue to allow this behavior. That is, if \field{hash_option} is set to 1, the hash value calculated based on the outer source port indicated by VIRTIO_NET_F_HASH_TUNNEL is provided to the indirection table to select the rxq instead of the hash value provided by RSS.
Quoted 7 lines
2. hand over to RSS the ability to specify a hash based on the outer source port. That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4, v6, EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols. This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.

Quoted 7 lines

#2 is preferred.
But I must be missing something.
Without VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X, today if one wants to do RSS based on the outer UDP ports, It is already supported by the RSS (even in presence of the inner header).
So why do have to introduce VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X?

they want an option to ignore all other header fields.

Signature · 6 lines

MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Parav Pandit

2023-06-08 14:45 UTC #5 lore raw

Quoted 17 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 8, 2023 10:42 AM
Quoted 1 line
Quoted 1 line
2. hand over to RSS the ability to specify a hash based on the outer source
port.
Quoted 1 line
Quoted 1 line
That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4,
v6,
Quoted 9 lines
Quoted 5 lines
EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols. This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.
#2 is preferred.
But I must be missing something.
Without VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X, today if one

Quoted 8 lines

wants
Quoted 1 line
to do RSS based on the outer UDP ports, It is already supported by the RSS
(even in presence of the inner header).
Quoted 2 lines
So why do have to introduce
VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X?
they want an option to ignore all other header fields.

Than that looks like rss config extension on which fields of the packet to use for RSS.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Michael S. Tsirkin

2023-06-08 14:47 UTC #6 lore raw

Quoted 19 lines · Parav Pandit

Quoted 19 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 8, 2023 10:42 AM
Quoted 1 line
Quoted 1 line
2. hand over to RSS the ability to specify a hash based on the outer source
port.
Quoted 1 line
Quoted 1 line
That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4,
v6,
Quoted 9 lines
Quoted 5 lines
EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols. This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.
#2 is preferred.
But I must be missing something.
Without VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X, today if one
wants
Quoted 1 line
to do RSS based on the outer UDP ports, It is already supported by the RSS

Quoted 8 lines

Quoted 6 lines
(even in presence of the inner header).
Quoted 2 lines
So why do have to introduce
VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X?
they want an option to ignore all other header fields.
Than that looks like rss config extension on which fields of the packet to use for RSS.

yes, but also, only for specific packet type and dest port #s.

Signature · 6 lines

MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Heng Qi

author

2023-06-08 15:29 UTC #7 lore raw

In reply to Parav Pandit

Quoted 27 lines · Parav Pandit

Quoted 2 lines
From: Heng Qi <[email protected]>
Sent: Thursday, June 8, 2023 2:41 AM
[..]
Quoted 24 lines
Quoted 5 lines
Quoted 4 lines
+The device MUST calculate the inner header hash for the configured \field{hash_tunnel_types} without the outer UDP header if \field{hash_option} is 1.
+
This I think conflicts with the current RSS where if this override (higher priority
than RSS) or not.
VIRTIO_NET_F_HASH_TUNNEL instructs the device to calculate an inner header hash for configured tunnel types.
Then, which fields are used for the calculation of the inner header hash is determined by the RSS configuration.
Therefore, if the inner header hash uses \field{hash_option} to indicate that the outer source UDP port is used to calculate the hash, it overrides the RSS configuration (higher priority than RSS). It now not only indicates for which tunnel types the inner hash should be calculated, but also indicates which fields participate in the RSS hash calculation.
At this point, we can
1. continue to allow this behavior. That is, if \field{hash_option} is set to 1, the hash value calculated based on the outer source port indicated by VIRTIO_NET_F_HASH_TUNNEL is provided to the indirection table to select the rxq instead of the hash value provided by RSS.

Quoted 8 lines

Quoted 7 lines
2. hand over to RSS the ability to specify a hash based on the outer source port. That is, we can add VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_{v4, v6, EX} in \field{supported_hash_types} to indicate that the hash should be calculated using the outer source port for all UDP tunneling protocols. This approach removes the need for \field{hash_option} and makes the line between VIRTIO_NET_F_HASH_TUNNEL and VIRTIO_NET_F_RSS clearer.
#2 is preferred.

Ok.

Quoted 6 lines

But I must be missing something.
Without VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X, today if one wants to do RSS based on the outer UDP ports, It is already supported by the RSS (even in presence of the inner header).
So why do have to introduce VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X?

For legacy tunneling protocols like IPIP/GRE_2784/GRE_2890 that do not have outer transport headers, it is worth calculating the inner header hash for these protocols.

For modern tunneling protocols with outer transport headers, although they can provide entropy through the outer source udp port, sometimes we need to perform symmetric hashing on these tunneling protocols.
Then,
1. We can achieve this through the inner header, but we need a way to cope with the addition of new tunneling protocols in the future;
2. The outer source udp port can identify the same flow in modern tunneling protocols such as VXLAN/VXLAN-GPE/GENEVE/NVGRE. But this is marked as "SHOULD" or "RECOMMEND" in various RFCs, so it is not reliable. This method ( adding VIRTIO_NET_RSS_HASH_TUNNEL_SRC_PORT_UDP_X) can deal with the needs of new tunneling protocols in the future for symmetric hashing.

If #2 is preferred, I'll update it in the next version.
(this also doesn't affect the migration, and maybe we shouldn't need a new feature to extend new values for \field{supported_hash_types}.)

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.