virtio-net: support inner header hash

1. Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. The same flow can traverse through different tunnels, resulting in the encapsulated packets being spread across multiple receive queues (refer to the figure below). However, in certain scenarios, we may need to direct these encapsulated packets of the same flow to a single receive queue. This facilitates the processing of the flow by the same CPU to improve performance (warm caches, less locking, etc.).

                client1                    client2
                   |        +-------+         |
                   +------->|tunnels|<--------+
                            +-------+
                               |  |
                               v  v
                       +-----------------+
                       | monitoring host |
                       +-----------------+

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

2. For legacy systems, they may lack entropy fields which modern protocols have in the outer header, resulting in multiple flows with the same outer header but different inner headers being directed to the same receive queue. This results in poor receive performance.

To address this limitation, inner header hash can be used to enable the device to advertise the capability to calculate the hash for the inner packet, regaining better receive performance.

Signed-off-by Heng Qi <[email protected]>

Reviewed-by Xuan Zhuo <[email protected]>

---
v11->v12:
1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
2. Refine the commit log. @Michael S . Tsirkin
3. Add some tunnel types.

v10->v11:
1. Revise commit log for clarity for readers.
2. Some modifications to avoid undefined terms. @Parav Pandit
3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
4. Add the normative statements. @Parav Pandit

v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex        | 143 ++++++++++++++++++++++++
  device-types/net/device-conformance.tex |   1 +
  device-types/net/driver-conformance.tex |   1 +
  introduction.tex                        |  44 ++++++++
  4 files changed, 189 insertions(+)

device-types/net/description.tex +143 -0

index 0500bb6..121585e 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS and/or VIRTIO_NET_F_HASH_REPORT.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -198,6 +202,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
  The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.

@@ -212,6 +217,12 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+Field \field{supported_tunnel_hash_types} only exists if the device
+supports inner header hash, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Filed \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types} for details of supported tunnel hash types.
+
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
  The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,

@@ -848,6 +859,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
  \end{itemize}

@@ -855,6 +867,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
  \end{itemize}

@@ -870,6 +883,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 
  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)

@@ -980,6 +995,133 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports inner header hash and
+the driver can configure the inner header hash calculation for encapsulated packets \ref{sec:Device Types / Network Device / Device OperatiHn / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure virtio_net_hash_tunnel_config.
+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+
+Filed \field{hash_tunnel_types} contains a bitmask of supported hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+
+A tunnel packet is encapsulated from the original packet based on the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated packet's
+outer header matches one of the supported \field{hash_tunnel_types}, the hash
+of the inner header is calculated.
+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}
+\item \hyperref[intro:vxlan]{[VXLAN]}
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}
+\item \hyperref[intro:geneve]{[GENEVE]}
+\item \hyperref[intro:ipip]{[IPIP]}
+\item \hyperref[intro:nvgre]{[NVGRE]}
+\item \hyperref[intro:sit]{[STT]}
+\end{itemize}
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is not included in
+\field{hash_tunnel_types}, the hash of the outer header is calculated for the received encapsulated packet.
+
+The hash is calculated for the received non-encapsulated packet as if VIRTIO_NET_F_HASH_TUNNEL was not negotiated.
+
+\subparagraph{Supported/enabled encapsulation hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}
+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE        (1 << 0)
+\end{lstlisting}
+
+Supported encapsulation hash types:
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan]{[VXLAN]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 5)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 6)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:geneve]{[GENEVE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 7)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:ipip]{[IPIP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 8)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:nvgre]{[NVGRE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 9)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:stt]{[STT]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_STT         (1 << 10)
+\end{lstlisting}
+
+\subparagraph{Tunnel QoS limitation}
+When a specific receive queue is shared by multiple tunnels to receive encapsulated packets,
+there is no quality of service (QoS) for these packets. For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+      to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}
+
+The limitations mentioned above exist with/without the inner header hash.
+
+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The device MUST calculate the outer header hash if the received encapsulated packet has an encapsulation type not in \field{supported_tunnel_hash_types}.
+
+The device MUST respond to the VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG command with VIRTIO_NET_ERR if the device
+received an unrecognized or unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ flag.
+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} to 0.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The driver MUST have negotiated the feature VIRTIO_NET_F_HASH_TUNNEL when issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that are not supported by the device.
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
 

@@ -1308,6 +1450,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET        0 (for automatic receive steering)
   #define VIRTIO_NET_CTRL_MQ_RSS_CONFIG          1 (for configurable receive steering)
   #define VIRTIO_NET_CTRL_MQ_HASH_CONFIG         2 (for configurable hash calculation)
+ #define VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG       3 (for configurable inner header hash)
  \end{lstlisting}
 
  If more than one multiqueue mode is negotiated, the resulting device configuration is defined by the last command sent by the driver.

device-types/net/device-conformance.tex +1 -0

index 54f6783..f88f48b 100644
@@ -14,4 +14,5 @@
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
  \end{itemize}

device-types/net/driver-conformance.tex +1 -0

index 97d0cc1..9d853d9 100644
@@ -14,4 +14,5 @@
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
  \end{itemize}

introduction.tex +37 -0

index 287c5fc..36b620f 100644
@@ -99,6 +99,50 @@ \section{Normative References}\label{sec:Normative References}
      Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
  	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:gre_rfc2784}\textbf{[GRE_rfc2784]} &
+    Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol.
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:gre_rfc2890}\textbf{[GRE_rfc2890]} &
+    Key and Sequence Number Extensions to GRE \ref{intro:gre_rfc2784}. This protocol describes extensions by which two fields, Key and
+    Sequence Number, can be optionally carried in the GRE Header \ref{intro:gre_rfc2784}.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2890}\\
+	\phantomsection\label{intro:gre_rfc7676}\textbf{[GRE_rfc7676]} &
+    IPv6 Support for Generic Routing Encapsulation (GRE). This protocol is specified for IPv6 and used as either the payload or
+    delivery protocol. Note that this does not change the GRE header format or any behaviors specified by RFC 2784 or RFC 2890.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7676/}\\
+	\phantomsection\label{intro:gre_in_udp_rfc8086}\textbf{[GRE-in-UDP]} &
+    GRE-in-UDP Encapsulation. This specifies a method of encapsulating network protocol packets within GRE and UDP headers.
+    This GRE-in-UDP encapsulation allows the UDP source port field to be used as an entropy field. This protocol is specified for IPv4 and IPv6,
+    and used as either the payload or delivery protocol.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc8086}\\
+	\phantomsection\label{intro:vxlan}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:vxlan-gpe}\textbf{[VXLAN-GPE]} &
+    Generic Protocol Extension for VXLAN. This protocol describes extending Virtual eXtensible Local Area Network (VXLAN) via changes to the VXLAN header.
+	\newline\url{https://www.ietf.org/archive/id/draft-ietf-nvo3-vxlan-gpe-12.txt}\\
+	\phantomsection\label{intro:geneve}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation.
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:ipip}\textbf{[IPIP]} &
+    IP Encapsulation within IP.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:nvgre}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\phantomsection\label{intro:stt}\textbf{[STT]} &
+    Stateless Transport Tunneling. STT is particularly useful when some tunnel endpoints are in end-systems, as it utilizes the capabilities
+    of the network interface card to improve performance.
+	\newline\url{https://www.ietf.org/archive/id/draft-davie-stt-08.txt}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL

1. Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. The same flow can traverse through different tunnels, resulting in the encapsulated packets being spread across multiple receive queues (refer to the figure below). However, in certain scenarios, we may need to direct these encapsulated packets of the same flow to a single receive queue. This facilitates the processing of the flow by the same CPU to improve performance (warm caches, less locking, etc.).

                client1                    client2
                   |        +-------+         |
                   +------->|tunnels|<--------+
                            +-------+
                               |  |
                               v  v
                       +-----------------+
                       | monitoring host |
                       +-----------------+

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

2. For legacy systems, they may lack entropy fields which modern protocols have in the outer header, resulting in multiple flows with the same outer header but different inner headers being directed to the same receive queue. This results in poor receive performance.

To address this limitation, inner header hash can be used to enable the device to advertise the capability to calculate the hash for the inner packet, regaining better receive performance.

Signed-off-by Heng Qi <[email protected]>

Reviewed-by Xuan Zhuo <[email protected]>

---
v11->v12:
1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
2. Refine the commit log. @Michael S . Tsirkin
3. Add some tunnel types.

v10->v11:
1. Revise commit log for clarity for readers.
2. Some modifications to avoid undefined terms. @Parav Pandit
3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
4. Add the normative statements. @Parav Pandit

v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex        | 143 ++++++++++++++++++++++++
  device-types/net/device-conformance.tex |   1 +
  device-types/net/driver-conformance.tex |   1 +
  introduction.tex                        |  44 ++++++++
  4 files changed, 189 insertions(+)

device-types/net/description.tex +143 -0

index 0500bb6..121585e 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS and/or VIRTIO_NET_F_HASH_REPORT.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -198,6 +202,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
  The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.

@@ -212,6 +217,12 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+Field \field{supported_tunnel_hash_types} only exists if the device
+supports inner header hash, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Filed \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types} for details of supported tunnel hash types.
+
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
  The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,

@@ -848,6 +859,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
  \end{itemize}

@@ -855,6 +867,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
  \end{itemize}

@@ -870,6 +883,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 
  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)

@@ -980,6 +995,133 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports inner header hash and
+the driver can configure the inner header hash calculation for encapsulated packets \ref{sec:Device Types / Network Device / Device OperatiHn / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure virtio_net_hash_tunnel_config.
+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+
+Filed \field{hash_tunnel_types} contains a bitmask of supported hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+
+A tunnel packet is encapsulated from the original packet based on the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.
+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated packet's
+outer header matches one of the supported \field{hash_tunnel_types}, the hash
+of the inner header is calculated.
+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}
+\item \hyperref[intro:vxlan]{[VXLAN]}
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}
+\item \hyperref[intro:geneve]{[GENEVE]}
+\item \hyperref[intro:ipip]{[IPIP]}
+\item \hyperref[intro:nvgre]{[NVGRE]}
+\item \hyperref[intro:sit]{[STT]}
+\end{itemize}
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is not included in
+\field{hash_tunnel_types}, the hash of the outer header is calculated for the received encapsulated packet.
+
+The hash is calculated for the received non-encapsulated packet as if VIRTIO_NET_F_HASH_TUNNEL was not negotiated.
+
+\subparagraph{Supported/enabled encapsulation hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}
+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE        (1 << 0)
+\end{lstlisting}
+
+Supported encapsulation hash types:
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan]{[VXLAN]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 5)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 6)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:geneve]{[GENEVE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 7)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:ipip]{[IPIP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 8)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:nvgre]{[NVGRE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 9)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:stt]{[STT]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_STT         (1 << 10)
+\end{lstlisting}
+
+\subparagraph{Tunnel QoS limitation}
+When a specific receive queue is shared by multiple tunnels to receive encapsulated packets,
+there is no quality of service (QoS) for these packets. For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+      to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}
+
+The limitations mentioned above exist with/without the inner header hash.
+
+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The device MUST calculate the outer header hash if the received encapsulated packet has an encapsulation type not in \field{supported_tunnel_hash_types}.
+
+The device MUST respond to the VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG command with VIRTIO_NET_ERR if the device
+received an unrecognized or unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ flag.
+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} to 0.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The driver MUST have negotiated the feature VIRTIO_NET_F_HASH_TUNNEL when issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that are not supported by the device.
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
 

@@ -1308,6 +1450,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET        0 (for automatic receive steering)
   #define VIRTIO_NET_CTRL_MQ_RSS_CONFIG          1 (for configurable receive steering)
   #define VIRTIO_NET_CTRL_MQ_HASH_CONFIG         2 (for configurable hash calculation)
+ #define VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG       3 (for configurable inner header hash)
  \end{lstlisting}
 
  If more than one multiqueue mode is negotiated, the resulting device configuration is defined by the last command sent by the driver.

device-types/net/device-conformance.tex +1 -0

index 54f6783..f88f48b 100644
@@ -14,4 +14,5 @@
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}
  \item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
  \end{itemize}

device-types/net/driver-conformance.tex +1 -0

index 97d0cc1..9d853d9 100644
@@ -14,4 +14,5 @@
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Offloads State Configuration / Setting Offloads State}
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
  \item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+\item \ref{drivernormative:Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
  \end{itemize}

introduction.tex +37 -0

index 287c5fc..36b620f 100644
@@ -99,6 +99,50 @@ \section{Normative References}\label{sec:Normative References}
      Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
  	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:gre_rfc2784}\textbf{[GRE_rfc2784]} &
+    Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol.
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:gre_rfc2890}\textbf{[GRE_rfc2890]} &
+    Key and Sequence Number Extensions to GRE \ref{intro:gre_rfc2784}. This protocol describes extensions by which two fields, Key and
+    Sequence Number, can be optionally carried in the GRE Header \ref{intro:gre_rfc2784}.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2890}\\
+	\phantomsection\label{intro:gre_rfc7676}\textbf{[GRE_rfc7676]} &
+    IPv6 Support for Generic Routing Encapsulation (GRE). This protocol is specified for IPv6 and used as either the payload or
+    delivery protocol. Note that this does not change the GRE header format or any behaviors specified by RFC 2784 or RFC 2890.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7676/}\\
+	\phantomsection\label{intro:gre_in_udp_rfc8086}\textbf{[GRE-in-UDP]} &
+    GRE-in-UDP Encapsulation. This specifies a method of encapsulating network protocol packets within GRE and UDP headers.
+    This GRE-in-UDP encapsulation allows the UDP source port field to be used as an entropy field. This protocol is specified for IPv4 and IPv6,
+    and used as either the payload or delivery protocol.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc8086}\\
+	\phantomsection\label{intro:vxlan}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network.
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:vxlan-gpe}\textbf{[VXLAN-GPE]} &
+    Generic Protocol Extension for VXLAN. This protocol describes extending Virtual eXtensible Local Area Network (VXLAN) via changes to the VXLAN header.
+	\newline\url{https://www.ietf.org/archive/id/draft-ietf-nvo3-vxlan-gpe-12.txt}\\
+	\phantomsection\label{intro:geneve}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation.
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:ipip}\textbf{[IPIP]} &
+    IP Encapsulation within IP.
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:nvgre}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\phantomsection\label{intro:stt}\textbf{[STT]} &
+    Stateless Transport Tunneling. STT is particularly useful when some tunnel endpoints are in end-systems, as it utilizes the capabilities
+    of the network interface card to improve performance.
+	\newline\url{https://www.ietf.org/archive/id/draft-davie-stt-08.txt}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

device-types/net/description.tex +4 -0

+Filed \field{hash_tunnel_types} contains a bitmask of supported hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+\subparagraph{Tunnel/Encapsulated packet}

device-types/net/description.tex +11 -0

+The device MUST respond to the VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG command with VIRTIO_NET_ERR if the device
+received an unrecognized or unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ flag.
+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} to 0.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}
+
+The driver MUST have negotiated the feature VIRTIO_NET_F_HASH_TUNNEL when issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that are not supported by the device.
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}

Quoted 3 lines · Heng Qi

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

I am not sure you need symmetric hash key. Toeplitz with symmetric hashing without the symmetric key is possible too.

So just mentioning it as a 'combined with symmetric hashing' is enough.

Quoted 9 lines

  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

(quoted diff) +0 -0

@@ -198,6 +202,7 @@ \subsection{Device configuration 

layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };

Given that a set command is added via cvq, it make sense to also do symetrric work to get it via a cvq. This is also similar to the latest work for notification coalescing for VQ where get and set done using single channel = cvq.

Granted that RSS and other fields are done differently, but it was bit in the past.

With that no need to define two fields at two different places in config area and also in cvq.

Just the new opcode is needed for GET and things will be fine.

Quoted 9 lines

+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports
inner header hash and
+the driver can configure the inner header hash calculation for
encapsulated packets \ref{sec:Device Types / Network Device / Device OperatiHn / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet} +by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the
class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure
virtio_net_hash_tunnel_config.

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 1009

Quoted 5 lines

device-types/net/description.tex +5 -0

+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+

VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG_SET
and
VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG_GET

Quoted 17 lines

+Filed \field{hash_tunnel_types} contains a bitmask of supported hash
tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation
/ Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}. +
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation /
Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}

(quoted diff) +2 -0

+
+A tunnel packet is encapsulated from the original packet based on 

the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.

Quoted 6 lines

(quoted diff) +2 -0

+
+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received 

encapsulated packet's
+outer header matches one of the supported \field{hash_tunnel_types}, the hash
+of the inner header is calculated.

s/one of the supported/one of the configured/
Because support comes from GET or config space area; out of which a smaller or equal subset of tunnel types are configured.

Quoted 7 lines

+\devicenormative{\subparagraph}{Inner Header Hash}{Device Types / Network Device / Device Operation / Control Virtqueue / Inner Header Hash}

(quoted diff) +2 -0

+
+The device MUST calculate the outer header hash if the received 

encapsulated packet has an encapsulation type not in \field{supported_tunnel_hash_types}. +

Since the configured set can be smaller, a better reword is:
The device MUST calculate the hash from the outer header if the received encapsulated packet type is not matching from hash_tunnel_types.

Quoted 17 lines

+The device MUST respond to the VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG command with VIRTIO_NET_ERR if the device
+received an unrecognized or unsupported VIRTIO_NET_HASH_TUNNEL_TYPE_ flag.

device-types/net/description.tex +4 -0

+
+Upon reset, the device MUST initialize \field{hash_tunnel_type} to 0.
+
+\drivernormative{\subparagraph}{Inner Header Hash}{Device Types / 

Network Device / Device Operation / Control Virtqueue / Inner Header Hash}

(quoted diff) +2 -0

+
+The driver MUST have negotiated the feature VIRTIO_NET_F_HASH_TUNNEL 

when issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.

(quoted diff) +2 -0

+
+The driver MUST NOT set any VIRTIO_NET_HASH_TUNNEL_TYPE_ flags that 

are not supported by the device.
+

Quoted 8 lines

  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets} @@ -1308,6 +1450,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi #define VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET        0 (for automatic receive steering)
   #define VIRTIO_NET_CTRL_MQ_RSS_CONFIG          1 (for configurable receive steering)
   #define VIRTIO_NET_CTRL_MQ_HASH_CONFIG         2 (for configurable hash calculation)
+ #define VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG       3 (for configurable
inner header hash)

MQ_TUNNEL_CONFIG is not mutually exclusive with HASH or RSS config. Hence adding it here is incorrect.

See the text below.
"If more than one multiqueue mode is negotiated, the resulting device configuration is defined by the last command sent by the driver."

So MQ_TUNNEL config without RSS or HASH will be an undefined behavior, as it doesn't what kind of hashing to be done.

So please tunnel GET and SET commands outside of VIRTIO_NET_CTRL_MQ. This way it can work side by side with rss, hash and auto modes.

NOTE: We only have two modes now. The mode is not equal to the command. We can add VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG for commands, but there are still only two modes:
"
This specification defines the following modes that a device MAY implement for operation with multiple transmit/receive virtqueues:
\begin{itemize}
\item Automatic receive steering as defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode}.
  If a device supports this mode, it offers the VIRTIO_NET_F_MQ feature
bit.
\item Receive-side scaling as defined in \ref{devicenormative:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / RSS processing}.
  If a device supports this mode, it offers the VIRTIO_NET_F_RSS
feature bit.
\end{itemize}

So, it is not incorrect that we put VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG here.

1. Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. The same flow can traverse through different tunnels, resulting in the encapsulated packets being spread across multiple receive queues (refer to the figure below). However, in certain scenarios, we may need to direct these encapsulated packets of the same flow to a single receive queue. This facilitates the processing of the flow by the same CPU to improve performance (warm caches, less locking, etc.).

                client1                    client2
                   |        +-------+         |
                   +------->|tunnels|<--------+
                            +-------+
                               |  |
                               v  v
                       +-----------------+
                       | monitoring host |
                       +-----------------+

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

2. For legacy systems, they may lack entropy fields which modern protocols have in the outer header, resulting in multiple flows with the same outer header but different inner headers being directed to the same receive queue. This results in poor receive performance.

To address this limitation, inner header hash can be used to enable the device to advertise the capability to calculate the hash for the inner packet, regaining better receive performance.

Signed-off-by Heng Qi<[email protected]>

Reviewed-by Xuan Zhuo<[email protected]>

---
v11->v12:
1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
2. Refine the commit log. @Michael S . Tsirkin
3. Add some tunnel types.

v10->v11:
1. Revise commit log for clarity for readers.
2. Some modifications to avoid undefined terms. @Parav Pandit
3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
4. Add the normative statements. @Parav Pandit

v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex        | 143 ++++++++++++++++++++++++
  device-types/net/device-conformance.tex |   1 +
  device-types/net/driver-conformance.tex |   1 +
  introduction.tex                        |  44 ++++++++
  4 files changed, 189 insertions(+)

device-types/net/description.tex +31 -0

index 0500bb6..121585e 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along with VIRTIO_NET_F_RSS and/or VIRTIO_NET_F_HASH_REPORT.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

@@ -198,6 +202,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
  The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.

@@ -212,6 +217,12 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+Field \field{supported_tunnel_hash_types} only exists if the device
+supports inner header hash, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Filed \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types} for details of supported tunnel hash types.
+
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
  The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,

@@ -848,6 +859,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
  \end{itemize}

@@ -855,6 +867,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
  \end{itemize}

@@ -870,6 +883,8 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 
  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)

@@ -980,6 +995,133 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Inner Header Hash}
+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports inner header hash and
+the driver can configure the inner header hash calculation for encapsulated packets \ref{sec:Device Types / Network Device / Device OperatiHn / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure virtio_net_hash_tunnel_config.
+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+
+Filed \field{hash_tunnel_types} contains a bitmask of supported hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+\subparagraph{Tunnel/Encapsulated packet}

device-types/net/description.tex +62 -0

+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}
+\item \hyperref[intro:vxlan]{[VXLAN]}
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}
+\item \hyperref[intro:geneve]{[GENEVE]}
+\item \hyperref[intro:ipip]{[IPIP]}
+\item \hyperref[intro:nvgre]{[NVGRE]}
+\item \hyperref[intro:sit]{[STT]}
+\end{itemize}
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type is not included in
+\field{hash_tunnel_types}, the hash of the outer header is calculated for the received encapsulated packet.
+
+The hash is calculated for the received non-encapsulated packet as if VIRTIO_NET_F_HASH_TUNNEL was not negotiated.
+
+\subparagraph{Supported/enabled encapsulation hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}
+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE        (1 << 0)
+\end{lstlisting}
+
+Supported encapsulation hash types:
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan]{[VXLAN]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 5)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 6)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:geneve]{[GENEVE]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 7)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:ipip]{[IPIP]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 8)
+\end{lstlisting}
+Hash type applicable for inner payload of the \hyperref[intro:nvgre]{[NVGRE]} packet:
+\begin{lstlisting}

Quoted 18 lines · Heng Qi

1. Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. The same flow can traverse through different tunnels, resulting in the encapsulated packets being spread across multiple receive queues (refer to the figure below). However, in certain scenarios, we may need to direct these encapsulated packets of the same flow to a single receive queue. This facilitates the processing of the flow by the same CPU to improve performance (warm caches, less locking, etc.).

                client1                    client2

                   |        +-------+         |
                   +------->|tunnels|<--------+
                            +-------+
                               |  |
                               v  v
                       +-----------------+
                       | monitoring host |
                       +-----------------+

Quoted 2 lines

To achieve this, the device can calculate a suitable hash based on the inner headers of this flow, for example using the Toeplitz combined with a symmetric hash key.

Some device allows sorting n tuples before calculating hash, do we need to consider them?

Quoted 191 lines

2. For legacy systems, they may lack entropy fields which modern protocols have in the outer header, resulting in multiple flows with the same outer header but different inner headers being directed to the same receive queue. This results in poor receive performance.

To address this limitation, inner header hash can be used to enable the device to advertise the capability to calculate the hash for the inner packet, regaining better receive performance.

Signed-off-by Heng Qi<[email protected]>

Reviewed-by Xuan Zhuo<[email protected]>

---
v11->v12:
    1. Add a command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG.
    2. Refine the commit log. @Michael S . Tsirkin
    3. Add some tunnel types.

v10->v11:
    1. Revise commit log for clarity for readers.
    2. Some modifications to avoid undefined terms. @Parav Pandit
    3. Change VIRTIO_NET_F_HASH_TUNNEL dependency. @Parav Pandit
    4. Add the normative statements. @Parav Pandit

v9->v10:
    1. Removed hash_report_tunnel related information. @Parav Pandit
    2. Re-describe the limitations of QoS for tunneling.
    3. Some clarification.

v8->v9:
    1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
    2. Add tunnel security section. @Michael S . Tsirkin
    3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
    4. Fix some typos.
    5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
    1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
    2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
    3. Removed re-definition for inner packet hashing. @Parav Pandit
    4. Fix some typos. @Michael S . Tsirkin
    5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
    1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
    2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
    1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
    2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
    3. Move the links to introduction section. @Michael S. Tsirkin
    4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
    1. Clarify some paragraphs. @Cornelia Huck
    2. Fix the u8 type. @Cornelia Huck

v3->v4:
    1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
    2. Make things clearer. @Jason Wang @Michael S. Tsirkin
    3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
    4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
    1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
    2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
    1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
    2. Clarify some paragraphs. @Jason Wang
    3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex        | 143 ++++++++++++++++++++++++ device-types/net/device-conformance.tex |   1 + device-types/net/driver-conformance.tex |   1 + introduction.tex                        |  44 ++++++++

  4 files changed, 189 insertions(+)

device-types/net/description.tex +0 -0

index 0500bb6..121585e 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / 

Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel.
  +\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash +    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

device-types/net/description.tex +0 -0

@@ -139,6 +142,7 @@ \subsubsection{Feature bit 

requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ along
with VIRTIO_NET_F_RSS and/or VIRTIO_NET_F_HASH_REPORT.
  \end{description}
    \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

device-types/net/description.tex +0 -0

@@ -198,6 +202,7 @@ \subsection{Device configuration 

layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
  The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.

device-types/net/description.tex +0 -0

@@ -212,6 +217,12 @@ \subsection{Device configuration 

layout}\label{sec:Device Types / Network Device
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types. +Field \field{supported_tunnel_hash_types} only exists if the device

device-types/net/description.tex +3 -0

+supports inner header hash, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Filed \field{supported_tunnel_hash_types} contains the bitmask of 

supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation /
Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types} for details of supported tunnel hash types. +
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
    The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,

device-types/net/description.tex +0 -0

@@ -848,6 +859,7 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask. +\item The device uses \field{hash_tunnel_types} of the
virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated. \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}). \end{itemize}

device-types/net/description.tex +0 -0

@@ -855,6 +867,7 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the
virtio_net_hash_tunnel_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated. \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}). \end{itemize}

device-types/net/description.tex +0 -0

@@ -870,6 +883,8 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
    \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} +This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)

device-types/net/description.tex +0 -0

@@ -980,6 +995,133 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}). \end{itemize}
  +\paragraph{Inner Header Hash}
+\label{sec:Device Types / Network Device / Device Operation /
Processing of Incoming Packets / Inner Header Hash}

device-types/net/description.tex +2 -0

+
+If VIRTIO_NET_F_HASH_TUNNEL has been negotiated, the device supports 

inner header hash and
+the driver can configure the inner header hash calculation for
encapsulated packets \ref{sec:Device Types / Network Device / Device OperatiHn / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet} +by issuing the command VIRTIO_NET_CTRL_MQ_TUNNEL_CONFIG from the
class VIRTIO_NET_CTRL_MQ.
+The command sets \field{hash_tunnel_types} in the structure
virtio_net_hash_tunnel_config.

device-types/net/description.tex +6 -0

+
+struct virtio_net_hash_tunnel_config {
+    le32 hash_tunnel_types;
+};
+
+Filed \field{hash_tunnel_types} contains a bitmask of supported hash 

tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation
/ Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}. +
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation /
Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}

device-types/net/description.tex +2 -0

+
+A tunnel packet is encapsulated from the original packet based on 

the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device

device-types/net/description.tex +2 -0

+calculates the hash over either the inner header or the outer header.
+

Quoted 5 lines

+If VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated packet's
+outer header matches one of the supported \field{hash_tunnel_types}, the hash
+of the inner header is calculated.

GRE allows X over Y, we'd better document what is supported in X and Y

(I guess they are implied in the supported_hash_types?)

Quoted 84 lines

device-types/net/description.tex +16 -0

+
+Supported encapsulated packet types:
+\begin{itemize}
+\item \hyperref[intro:gre_rfc2784]{[GRE_rfc2784]}
+\item \hyperref[intro:gre_rfc2890]{[GRE_rfc2890]}
+\item \hyperref[intro:gre_rfc7676]{[GRE_rfc7676]}
+\item \hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]}
+\item \hyperref[intro:vxlan]{[VXLAN]}
+\item \hyperref[intro:vxlan_gpe]{[VXLAN-GPE]}
+\item \hyperref[intro:geneve]{[GENEVE]}
+\item \hyperref[intro:ipip]{[IPIP]}
+\item \hyperref[intro:nvgre]{[NVGRE]}
+\item \hyperref[intro:sit]{[STT]}
+\end{itemize}
+
+If VIRTIO_NET_HASH_TUNNEL_TYPE_NONE is set or the encapsulation type 

is not included in
+\field{hash_tunnel_types}, the hash of the outer header is calculated for the received encapsulated packet.

device-types/net/description.tex +2 -0

+
+The hash is calculated for the received non-encapsulated packet as 

if VIRTIO_NET_F_HASH_TUNNEL was not negotiated.
+
+\subparagraph{Supported/enabled encapsulation hash types}
+\label{sec:Device Types / Network Device / Device Operation /
Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled encapsulation hash types}

device-types/net/description.tex +7 -0

+
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE        (1 << 0)
+\end{lstlisting}
+
+Supported encapsulation hash types:
+Hash type applicable for inner payload of the 

\hyperref[intro:gre_rfc2784]{[GRE_rfc2784]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2784    (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:gre_rfc2890]{[GRE_rfc2890]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_2890    (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:gre_rfc7676]{[GRE_rfc7676]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_7676    (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:gre_in_udp_rfc8086]{[GRE-in-UDP]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE_UDP     (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:vxlan]{[VXLAN]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 5)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:vxlan_gpe]{[VXLAN-GPE]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN_GPE   (1 << 6)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:geneve]{[GENEVE]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 7)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:ipip]{[IPIP]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 8)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:nvgre]{[NVGRE]} packet:
+\begin{lstlisting}

device-types/net/description.tex +3 -0

+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 9)
+\end{lstlisting}
+Hash type applicable for inner payload of the 

\hyperref[intro:stt]{[STT]} packet:
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_STT         (1 << 10) +\end{lstlisting}

Quoted 5 lines

device-types/net/description.tex +3 -0

+
+\subparagraph{Tunnel QoS limitation}
+When a specific receive queue is shared by multiple tunnels to 

receive encapsulated packets,
+there is no quality of service (QoS) for these packets.

This is up to the implementation, so I'm not sure it's worthwhile for mentioning things like this and the "limitation".

Quoted 6 lines

  For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Quoted 12 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the 

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE +      to disable inner header hash for encapsulated packets. +\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive

Quoted 5 lines

buffers of the receive queues.
+\end{itemize}

device-types/net/description.tex +2 -0

+
+The limitations mentioned above exist with/without the inner header 

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Quoted 10 lines

Quoted 5 lines

  For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

Quoted 13 lines

Quoted 13 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE +      to disable inner header hash for encapsulated packets. +\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.

Quoted 8 lines

Quoted 4 lines

+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Perhaps a "QoS Advices" is better?

Thanks!

Quoted 20 lines

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Quoted 10 lines · Heng Qi

Quoted 9 lines

Quoted 5 lines

  For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

Quoted 20 lines

Quoted 20 lines

Quoted 16 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE +      to disable inner header hash for encapsulated packets. +\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

Quoted 1 line

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Quoted 1 line

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

This advice is somewhat bogus though.

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without.

This was the whole reason I asked for a security
considerations sections.

Quoted 21 lines

Thanks!

Quoted 19 lines

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Quoted 15 lines · Michael S. Tsirkin

Quoted 10 lines · Heng Qi

Quoted 9 lines

Quoted 5 lines

  For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

Quoted 8 lines

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

Sure. Will do  in the v13.

Quoted 20 lines

Quoted 20 lines

Quoted 20 lines

Quoted 16 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE +      to disable inner header hash for encapsulated packets. +\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.
+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

Quoted 1 line

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Quoted 2 lines

Quoted 1 line

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

My fault.😅

Quoted 3 lines

This advice is somewhat bogus though.

Quoted 5 lines

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without.

I don't think so, the outer headers also has corresponding entropy for different streams.

Thanks.

Quoted 29 lines

This was the whole reason I asked for a security
considerations sections.

Quoted 21 lines

Thanks!

Quoted 19 lines

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

Quoted 10 lines

Quoted 5 lines

  For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

Quoted 13 lines

Quoted 13 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE +      to disable inner header hash for encapsulated packets. +\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.

Quoted 8 lines

Quoted 4 lines

+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Perhaps a "QoS Advices" is better?

Quoted 11 lines · Heng Qi

Quoted 10 lines

Quoted 5 lines

For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Quoted 1 line

Ok. And Michael what do you think about this?

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

Quoted 23 lines

Quoted 14 lines

Quoted 14 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+ to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.

Quoted 8 lines

Quoted 4 lines

+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Quoted 1 line

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

This advice is somewhat bogus though.

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without. This was the whole reason I asked for a security considerations sections.

--
MST

Quoted 15 lines · Michael S. Tsirkin

Quoted 10 lines · Heng Qi

Quoted 9 lines

Quoted 5 lines

For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

Quoted 8 lines

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

And this is only for GRE-in-UDP? This makes me think if we should add GRE support for the outer header like:

https://docs.napatech.com/r/Feature-Set-N-ANL9/Hash-Key-Type-10-3-Tuple-GREv0

Thanks

Quoted 32 lines

Quoted 19 lines

Quoted 11 lines

Quoted 11 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+ to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive

Quoted 8 lines

Quoted 5 lines

buffers of the receive queues.
+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Quoted 1 line

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

This advice is somewhat bogus though.

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

is impossible with the feature and possible without. This was the whole reason I asked for a security considerations sections.

Quoted 3 lines

--
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

From: Heng Qi <[email protected]>
Sent: Friday, April 14, 2023 12:01 AM

Quoted 13 lines · Jason Wang

Quoted 12 lines · Michael S. Tsirkin

Quoted 10 lines · Heng Qi

Quoted 9 lines

Quoted 5 lines

For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

Quoted 8 lines

Quoted 8 lines

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is

limited

Quoted 6 lines

Quoted 3 lines

to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

And this is only for GRE-in-UDP? This makes me think if we should add GRE support for the outer header like:

Quoted 2 lines

https://docs.napatech.com/r/Feature-Set-N-ANL9/Hash-Key-Type-10-3-Tupl
e-GREv0

I think this is for tunneling protocols with specific flow fields, such as GRE:key filed, NVGRE:FLOWID filed.

This requires us to make a requirement when calculating the hash of the tunnels when F_TUNNEL_HASH is not negotiated. It's a new work.

Quoted 33 lines

From: Heng Qi <[email protected]>
Sent: Friday, April 14, 2023 12:01 AM

Quoted 11 lines · Jason Wang

Quoted 10 lines · Michael S. Tsirkin

Quoted 9 lines · Heng Qi

Quoted 8 lines

Quoted 5 lines

For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Ok. And Michael what do you think about this?

Quoted 8 lines

Quoted 8 lines

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is

limited

Quoted 8 lines

Quoted 3 lines

to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

And this is only for GRE-in-UDP? This makes me think if we should add GRE support for the outer header like:

https://docs.napatech.com/r/Feature-Set-N-ANL9/Hash-Key-Type-10-3-Tupl
e-GREv0

Quoted 6 lines

I think this is for tunneling protocols with specific flow fields, such as GRE:key filed, NVGRE:FLOWID filed.

This requires us to make a requirement when calculating the hash of the tunnels when F_TUNNEL_HASH is not negotiated. It's a new work.

Yes, I think Jason is saying the same to extend the outer header hashing for newer protocols where outer header entropy is available.

Quoted 19 lines · Michael S. Tsirkin

Quoted 11 lines · Heng Qi

Quoted 10 lines

Quoted 5 lines

For example, when the packets of certain
+tunnels are spread across multiple receive queues, these receive queues may have an unbalanced
+amount of packets. This can cause a specific receive queue to become full, resulting in packet loss.

We have many places that can lead to packet dropping. For example, the automatic steering is best effort. I tend to avoid mentioning things like this.

Quoted 1 line

Ok. And Michael what do you think about this?

I think this text did not do a great job explaining the security aspect. Here's a better, shorter explanation:

Quoted 8 lines

It is often an expectation of users that a tunnel isolates the external network from the internal one. By completely ignoring entropy in the external header and replacing it with entropy from the internal header, for hash calculations, this expectation might be violated to a certain extent, depending on how the hash is used. When the hash use is limited to RSS queue selection, the effect will likely be limited to ability of users inside the tunnel to cause packet drops in multiple queues (as opposed to a single queue without the feature).

And this is only for GRE-in-UDP?

This makes me think if we should add
GRE support for the outer header like:

https://docs.napatech.com/r/Feature-Set-N-ANL9/Hash-Key-Type-10-3-Tuple-GREv0

Thanks

Quoted 34 lines

Quoted 24 lines

Quoted 15 lines

Quoted 15 lines

device-types/net/description.tex +4 -0

+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance to keep the

receive queue from filling up.
+\item If the QoS is unavailable, the driver can set \field{hash_tunnel_types} to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE
+ to disable inner header hash for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions. +\item Perform appropriate QoS before packets consume the receive buffers of the receive queues.

Quoted 8 lines

Quoted 4 lines

+\end{itemize}

(quoted diff) +2 -0

+
+The limitations mentioned above exist with/without the inner header

hash.

This conflicts with the tile "Tunnel QoS limitation" which readers may think it happens only for tunnel.

Quoted 1 line

Perhaps a "QoS Advices" is better?

Plural of "advice" is "advice" not "advices".

This advice is somewhat bogus though.

The point I keep trying to make is that this:

Choose a hash key that can avoid queue collisions.

Quoted 7 lines

is impossible with the feature and possible without. This was the whole reason I asked for a security considerations sections.

--
MST