Skip to contentSkip to conversation

virtio-net: support inner header hash

PATCH Superseded Heng Qi 3 participants 18 messages Mar 6, 2023 – Mar 17, 2023

v10 of 21 Superseded Latest is v21 →

lore
Share & tools
lore

Heng Qi

author
#0 lore raw

PATCH v10 1/1 virtio-net: support inner header hash

Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. Multiple flows with the same outer header but different inner headers are steered to the same receive queue. This results in poor receive performance.

To address this limitation, a new feature VIRTIO_NET_F_HASH_TUNNEL has been introduced, which enables the device to advertise the capability to calculate the hash for the inner packet header. Compared with the out header hash, it regains better receive performance.

---
v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

 device-types/net/description.tex | 120 +++++++++++++++++++++++++++++--
 introduction.tex                 |  24 +++++++
 2 files changed, 140 insertions(+), 4 deletions(-)
device-types/net/description.tex +116 -4
index 0500bb6..e53d625 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,20 +202,27 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
         u8 rss_max_key_size;
         le16 rss_max_indirection_table_length;
         le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
 };
 \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
 It specifies the maximum supported length of RSS key in bytes.
 
 The following field, \field{rss_max_indirection_table_length} only exists if VIRTIO_NET_F_RSS is set.
 It specifies the maximum number of 16-bit entries in RSS indirection table.
 
 The next field, \field{supported_hash_types} only exists if the device supports hash calculation,
-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
 
 Field \field{supported_hash_types} contains the bitmask of supported hash types.
 See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+The next field, \field{supported_tunnel_hash_types} only exists if the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Field \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types} for details of supported tunnel hash types.
+
 \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
 The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,
@@ -235,7 +246,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
 negotiated.
 
 The device MUST set \field{rss_max_key_size} to at least 40, if it offers
-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.
 
 The device MUST set \field{rss_max_indirection_table_length} to at least 128, if it offers
 VIRTIO_NET_F_RSS.
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 \begin{itemize}
 \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses the hash to determine the receive virtqueue to place incoming packets.
 \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The device reports the hash value and the hash type with the packet.
+\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The device supports inner hash calculation.
 \end{itemize}
 
 If the feature VIRTIO_NET_F_RSS was negotiated:
 \begin{itemize}
 \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_rss_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
 \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
 \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
 \end{itemize}
@@ -855,6 +868,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 If the feature VIRTIO_NET_F_RSS was not negotiated:
 \begin{itemize}
 \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
 \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
 \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
 \end{itemize}
@@ -868,8 +882,26 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}.
 \end{itemize}
 
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+A tunnel packet is encapsulated from the original packet based on the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.
+
+When the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated
+packet's outer header matches one of the supported \field{hash_tunnel_types},
+the hash of the inner header is calculated. Supported encapsulation types are listed
+in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming
+Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Some encapsulated packet types: \hyperref[intro:GRE]{[GRE]}, \hyperref[intro:VXLAN]{[VXLAN]},
+\hyperref[intro:GENEVE]{[GENEVE]}, \hyperref[intro:IPIP]{[IPIP]} and \hyperref[intro:NVGRE]{[NVGRE]}.
+
 \subparagraph{Supported/enabled hash types}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
 Hash types applicable for IPv4 packets:
 \begin{lstlisting}
 #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)
@@ -889,6 +921,39 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 #define VIRTIO_NET_HASH_TYPE_UDP_EX            (1 << 8)
 \end{lstlisting}
 
+\subparagraph{Supported/enabled tunnel hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types}
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the value of
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE,
+the device calculates the hash using the outer header of the encapsulated packet.
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE         (1 << 0)
+\end{lstlisting}
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the encapsulation
+hash type below indicates that the hash is calculated over the inner header of
+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE         (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the vxlan-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the geneve-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the ip-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the nvgre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 5)
+\end{lstlisting}
+
 \subparagraph{IPv4 packets}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv4 packets}
 The device calculates the hash on IPv4 packets according to 'Enabled hash types' bitmask as follows:
@@ -980,6 +1045,44 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
 (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
 \end{itemize}
 
+\subparagraph{Inner hash calculation of an encapsulated packet}
+If the driver negotiates the VIRTIO_NET_F_HASH_TUNNEL feature, it can configure the
+hash parameters (including \field{hash_tunnel_types}) for inner hash calculation by
+sending the VIRTIO_NET_CTRL_MQ_HASH_CONFIG command. Additionally, if the VIRTIO_NET_F_RSS
+feature is also negotiated, the driver can use the VIRTIO_NET_CTRL_RSS_CONFIG command to
+configure the hash parameters. If multiple commands are sent, the device configuration
+will be defined by the last command received.
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the corresponding
+encapsulation hash type is set in \field{hash_tunnel_types}, the device calculates the
+hash on the inner header of an encapsulated packet (See \ref{sec:Device Types
+/ Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}). If the encapsulation
+type of an encapsulated packet is not included in \field{hash_tunnel_types} or the value
+of \field{hash_tunnel_types} is VIRTIO_NET_HASH_TUNNEL_TYPE_NONE, the device calculates
+the hash on the outer header.
+
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the
+unencapsulated packets.
+
+\subparagraph{Tunnel QoS limitation}
+Note that the limitation mentioned below is not only introduced by inner hash calculation,
+and the limitation of the tunnel itself, and even the driver may have only one receive queue.
+
+When a specific receive queue is shared by multiple tunnels to receive encapsulating packets,
+there is no quality of service (QoS) for these packets of multiple tunnels. For example, when the
+flooded packets of a certain tunnel are hashed to the queue, it may cause the traffic of this
+queue to be unbalanced, resulting in potential packet loss and data delay.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance such as DPDK to keep the queue from filling up.
+\item If the quality of service is unavailable, the driver can set \field{hash_tunnel_types} to
+      VIRTIO_NET_HASH_TUNNEL_TYPE_NONE to disable inner hash calculation for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Outside the device, prevent abnormal traffic from entering or switch the traffic to attack clusters.
+\end{itemize}
+
 \paragraph{Hash reporting for incoming packets}
 \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
 
@@ -1392,12 +1495,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le16 reserved[4];
     u8 hash_key_length;
     u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
 };
 \end{lstlisting}
 Field \field{hash_types} contains a bitmask of allowed hash types as
 defined in
 \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}.
-Initially the device has all hash types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
+
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Initially the device has all hash types and hash tunnel types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
 
 Field \field{reserved} MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure,
 defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS)}.
@@ -1421,6 +1529,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le16 max_tx_vq;
     u8 hash_key_length;
     u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
 };
 \end{lstlisting}
 Field \field{hash_types} contains a bitmask of allowed hash types as
@@ -1441,6 +1550,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 Fields \field{hash_key_length} and \field{hash_key_data} define the key to be used in hash calculation.
 
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
 \drivernormative{\subparagraph}{Setting RSS parameters}{Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
 
 A driver MUST NOT send the VIRTIO_NET_CTRL_MQ_RSS_CONFIG command if the feature VIRTIO_NET_F_RSS has not been negotiated.
introduction.tex +24 -0
index 287c5fc..25c9d48 100644
@@ -99,6 +99,30 @@ \section{Normative References}\label{sec:Normative References}
     Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
 	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:GRE}\textbf{[GRE]} &
+    Generic Routing Encapsulation
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:VXLAN}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:GENEVE}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation
+	\phantomsection\label{intro:IPIP}\textbf{[IPIP]} &
+    IP Encapsulation within IP
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:IPIP}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc791}\\
+	\phantomsection\label{intro:UDP}\textbf{[UDP]} &
+    User Datagram Protocol
+	\newline\url{https://www.rfc-editor.org/rfc/rfc768}\\
+	\phantomsection\label{intro:TCP}\textbf{[TCP]} &
+    TRANSMISSION CONTROL PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc793}\\
 \end{longtable}
 
 \section{Non-Normative References}
Signature · 18 lines
2.19.1.6.gb485710b


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
author
#1 lore raw

Do you have any comments, please?

Thanks. :)

Quoted 323 lines · Heng Qi

Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. Multiple flows with the same outer header but different inner headers are steered to the same receive queue. This results in poor receive performance.

To address this limitation, a new feature VIRTIO_NET_F_HASH_TUNNEL has been introduced, which enables the device to advertise the capability to calculate the hash for the inner packet header. Compared with the out header hash, it regains better receive performance.

---
v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex | 120 +++++++++++++++++++++++++++++--
  introduction.tex                 |  24 +++++++
  2 files changed, 140 insertions(+), 4 deletions(-)
device-types/net/description.tex +116 -4
index 0500bb6..e53d625 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,20 +202,27 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
  It specifies the maximum supported length of RSS key in bytes.
 
  The following field, \field{rss_max_indirection_table_length} only exists if VIRTIO_NET_F_RSS is set.
  It specifies the maximum number of 16-bit entries in RSS indirection table.
 
  The next field, \field{supported_hash_types} only exists if the device supports hash calculation,
-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
 
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+The next field, \field{supported_tunnel_hash_types} only exists if the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Field \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types} for details of supported tunnel hash types.
+
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
  The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,
@@ -235,7 +246,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
  negotiated.
 
  The device MUST set \field{rss_max_key_size} to at least 40, if it offers
-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.
 
  The device MUST set \field{rss_max_indirection_table_length} to at least 128, if it offers
  VIRTIO_NET_F_RSS.
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  \begin{itemize}
  \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses the hash to determine the receive virtqueue to place incoming packets.
  \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The device reports the hash value and the hash type with the packet.
+\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The device supports inner hash calculation.
  \end{itemize}
 
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_rss_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
  \end{itemize}
@@ -855,6 +868,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
  \end{itemize}
@@ -868,8 +882,26 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
   \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}.
  \end{itemize}
 
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+A tunnel packet is encapsulated from the original packet based on the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.
+
+When the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated
+packet's outer header matches one of the supported \field{hash_tunnel_types},
+the hash of the inner header is calculated. Supported encapsulation types are listed
+in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming
+Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Some encapsulated packet types: \hyperref[intro:GRE]{[GRE]}, \hyperref[intro:VXLAN]{[VXLAN]},
+\hyperref[intro:GENEVE]{[GENEVE]}, \hyperref[intro:IPIP]{[IPIP]} and \hyperref[intro:NVGRE]{[NVGRE]}.
+
  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)
@@ -889,6 +921,39 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  #define VIRTIO_NET_HASH_TYPE_UDP_EX            (1 << 8)
  \end{lstlisting}
 
+\subparagraph{Supported/enabled tunnel hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types}
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the value of
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE,
+the device calculates the hash using the outer header of the encapsulated packet.
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE         (1 << 0)
+\end{lstlisting}
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the encapsulation
+hash type below indicates that the hash is calculated over the inner header of
+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE         (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the vxlan-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the geneve-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the ip-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the nvgre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 5)
+\end{lstlisting}
+
  \subparagraph{IPv4 packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv4 packets}
  The device calculates the hash on IPv4 packets according to 'Enabled hash types' bitmask as follows:
@@ -980,6 +1045,44 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\subparagraph{Inner hash calculation of an encapsulated packet}
+If the driver negotiates the VIRTIO_NET_F_HASH_TUNNEL feature, it can configure the
+hash parameters (including \field{hash_tunnel_types}) for inner hash calculation by
+sending the VIRTIO_NET_CTRL_MQ_HASH_CONFIG command. Additionally, if the VIRTIO_NET_F_RSS
+feature is also negotiated, the driver can use the VIRTIO_NET_CTRL_RSS_CONFIG command to
+configure the hash parameters. If multiple commands are sent, the device configuration
+will be defined by the last command received.
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the corresponding
+encapsulation hash type is set in \field{hash_tunnel_types}, the device calculates the
+hash on the inner header of an encapsulated packet (See \ref{sec:Device Types
+/ Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}). If the encapsulation
+type of an encapsulated packet is not included in \field{hash_tunnel_types} or the value
+of \field{hash_tunnel_types} is VIRTIO_NET_HASH_TUNNEL_TYPE_NONE, the device calculates
+the hash on the outer header.
+
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the
+unencapsulated packets.
+
+\subparagraph{Tunnel QoS limitation}
+Note that the limitation mentioned below is not only introduced by inner hash calculation,
+and the limitation of the tunnel itself, and even the driver may have only one receive queue.
+
+When a specific receive queue is shared by multiple tunnels to receive encapsulating packets,
+there is no quality of service (QoS) for these packets of multiple tunnels. For example, when the
+flooded packets of a certain tunnel are hashed to the queue, it may cause the traffic of this
+queue to be unbalanced, resulting in potential packet loss and data delay.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance such as DPDK to keep the queue from filling up.
+\item If the quality of service is unavailable, the driver can set \field{hash_tunnel_types} to
+      VIRTIO_NET_HASH_TUNNEL_TYPE_NONE to disable inner hash calculation for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Outside the device, prevent abnormal traffic from entering or switch the traffic to attack clusters.
+\end{itemize}
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
 
@@ -1392,12 +1495,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 reserved[4];
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
  defined in
  \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}.
-Initially the device has all hash types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
+
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Initially the device has all hash types and hash tunnel types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
 
  Field \field{reserved} MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure,
  defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS)}.
@@ -1421,6 +1529,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 max_tx_vq;
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
@@ -1441,6 +1550,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  Fields \field{hash_key_length} and \field{hash_key_data} define the key to be used in hash calculation.
 
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
  \drivernormative{\subparagraph}{Setting RSS parameters}{Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
 
  A driver MUST NOT send the VIRTIO_NET_CTRL_MQ_RSS_CONFIG command if the feature VIRTIO_NET_F_RSS has not been negotiated.
introduction.tex +17 -0
index 287c5fc..25c9d48 100644
@@ -99,6 +99,30 @@ \section{Normative References}\label{sec:Normative References}
      Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
  	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:GRE}\textbf{[GRE]} &
+    Generic Routing Encapsulation
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:VXLAN}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:GENEVE}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation
+	\phantomsection\label{intro:IPIP}\textbf{[IPIP]} &
+    IP Encapsulation within IP
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:IPIP}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL

introduction.tex \section{Normative References}\label{sec:Normative References} · line 128

Quoted 10 lines
introduction.tex +7 -0
+	\newline\url{https://www.rfc-editor.org/rfc/rfc791}\\
+	\phantomsection\label{intro:UDP}\textbf{[UDP]} &
+    User Datagram Protocol
+	\newline\url{https://www.rfc-editor.org/rfc/rfc768}\\
+	\phantomsection\label{intro:TCP}\textbf{[TCP]} &
+    TRANSMISSION CONTROL PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc793}\\
  \end{longtable}
 
  \section{Non-Normative References}

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#2 lore raw
Quoted 5 lines · Heng Qi

Do you have any comments, please?

Thanks. :)

在 2023/3/6 下午11:48, Heng Qi 写道:

It's been just 2 partial days. Sit tight pls.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
author
#3 lore raw
In reply to Heng Qi

Hi, Parav, do you have any comments on this? :)

Thanks!

Quoted 323 lines · Heng Qi

Currently, a received encapsulated packet has an outer and an inner header, but the virtio device is unable to calculate the hash for the inner header. Multiple flows with the same outer header but different inner headers are steered to the same receive queue. This results in poor receive performance.

To address this limitation, a new feature VIRTIO_NET_F_HASH_TUNNEL has been introduced, which enables the device to advertise the capability to calculate the hash for the inner packet header. Compared with the out header hash, it regains better receive performance.

---
v9->v10:
1. Removed hash_report_tunnel related information. @Parav Pandit
2. Re-describe the limitations of QoS for tunneling.
3. Some clarification.

v8->v9:
1. Merge hash_report_tunnel_types into hash_report. @Parav Pandit
2. Add tunnel security section. @Michael S . Tsirkin
3. Add VIRTIO_NET_F_HASH_REPORT_TUNNEL.
4. Fix some typos.
5. Add more tunnel types. @Michael S . Tsirkin

v7->v8:
1. Add supported_hash_tunnel_types. @Jason Wang, @Parav Pandit
2. Change hash_report_tunnel to hash_report_tunnel_types. @Parav Pandit
3. Removed re-definition for inner packet hashing. @Parav Pandit
4. Fix some typos. @Michael S . Tsirkin
5. Clarify some sentences. @Michael S . Tsirkin

v6->v7:
1. Modify the wording of some sentences for clarity. @Michael S. Tsirkin
2. Fix some syntax issues. @Michael S. Tsirkin

v5->v6:
1. Fix some syntax and capitalization issues. @Michael S. Tsirkin
2. Use encapsulated/encaptulation uniformly. @Michael S. Tsirkin
3. Move the links to introduction section. @Michael S. Tsirkin
4. Clarify some sentences. @Michael S. Tsirkin

v4->v5:
1. Clarify some paragraphs. @Cornelia Huck
2. Fix the u8 type. @Cornelia Huck

v3->v4:
1. Rename VIRTIO_NET_F_HASH_GRE_VXLAN_GENEVE_INNER to VIRTIO_NET_F_HASH_TUNNEL. @Jason Wang
2. Make things clearer. @Jason Wang @Michael S. Tsirkin
3. Keep the possibility to use inner hash for automatic receive steering. @Jason Wang
4. Add the "Tunnel packet" paragraph to avoid repeating the GRE etc. many times. @Michael S. Tsirkin

v2->v3:
1. Add a feature bit for GRE/VXLAN/GENEVE inner hash. @Jason Wang
2. Chang \field{hash_tunnel} to \field{hash_report_tunnel}. @Jason Wang, @Michael S. Tsirkin

v1->v2:
1. Remove the patch for the bitmask fix. @Michael S. Tsirkin
2. Clarify some paragraphs. @Jason Wang
3. Add \field{hash_tunnel} and VIRTIO_NET_HASH_REPORT_GRE. @Yuri Benditovich

  device-types/net/description.tex | 120 +++++++++++++++++++++++++++++--
  introduction.tex                 |  24 +++++++
  2 files changed, 140 insertions(+), 4 deletions(-)
device-types/net/description.tex +116 -4
index 0500bb6..e53d625 100644
@@ -83,6 +83,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -198,20 +202,27 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
  It specifies the maximum supported length of RSS key in bytes.
 
  The following field, \field{rss_max_indirection_table_length} only exists if VIRTIO_NET_F_RSS is set.
  It specifies the maximum number of 16-bit entries in RSS indirection table.
 
  The next field, \field{supported_hash_types} only exists if the device supports hash calculation,
-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
 
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+The next field, \field{supported_tunnel_hash_types} only exists if the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+
+Field \field{supported_tunnel_hash_types} contains the bitmask of supported tunnel hash types.
+See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types} for details of supported tunnel hash types.
+
  \devicenormative{\subsubsection}{Device configuration layout}{Device Types / Network Device / Device configuration layout}
 
  The device MUST set \field{max_virtqueue_pairs} to between 1 and 0x8000 inclusive,
@@ -235,7 +246,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
  negotiated.
 
  The device MUST set \field{rss_max_key_size} to at least 40, if it offers
-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.
 
  The device MUST set \field{rss_max_indirection_table_length} to at least 128, if it offers
  VIRTIO_NET_F_RSS.
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  \begin{itemize}
  \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses the hash to determine the receive virtqueue to place incoming packets.
  \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The device reports the hash value and the hash type with the packet.
+\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The device supports inner hash calculation.
  \end{itemize}
 
  If the feature VIRTIO_NET_F_RSS was negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_rss_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_rss_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_rss_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) / Setting RSS parameters}).
  \end{itemize}
@@ -855,6 +868,7 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  If the feature VIRTIO_NET_F_RSS was not negotiated:
  \begin{itemize}
  \item The device uses \field{hash_types} of the virtio_net_hash_config structure as 'Enabled hash types' bitmask.
+\item The device uses \field{hash_tunnel_types} of the virtio_net_hash_config structure as 'Enabled hash tunnel types' bitmask if VIRTIO_NET_F_HASH_TUNNEL was negotiated.
  \item The device uses a key as defined in \field{hash_key_data} and \field{hash_key_length} of the virtio_net_hash_config structure (see
  \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}).
  \end{itemize}
@@ -868,8 +882,26 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
   \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Automatic receive steering in multiqueue mode / Hash calculation}.
  \end{itemize}
 
+\subparagraph{Tunnel/Encapsulated packet}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Tunnel/Encapsulated packet}
+A tunnel packet is encapsulated from the original packet based on the tunneling
+protocol (only a single level of encapsulation is currently supported). The
+encapsulated packet contains an outer header and an inner header, and the device
+calculates the hash over either the inner header or the outer header.
+
+When the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and a received encapsulated
+packet's outer header matches one of the supported \field{hash_tunnel_types},
+the hash of the inner header is calculated. Supported encapsulation types are listed
+in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming
+Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Some encapsulated packet types: \hyperref[intro:GRE]{[GRE]}, \hyperref[intro:VXLAN]{[VXLAN]},
+\hyperref[intro:GENEVE]{[GENEVE]}, \hyperref[intro:IPIP]{[IPIP]} and \hyperref[intro:NVGRE]{[NVGRE]}.
+
  \subparagraph{Supported/enabled hash types}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}
+This paragraph relies on definitions from \hyperref[intro:IP]{[IP]},
+\hyperref[intro:UDP]{[UDP]} and \hyperref[intro:TCP]{[TCP]}.
  Hash types applicable for IPv4 packets:
  \begin{lstlisting}
  #define VIRTIO_NET_HASH_TYPE_IPv4              (1 << 0)
@@ -889,6 +921,39 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  #define VIRTIO_NET_HASH_TYPE_UDP_EX            (1 << 8)
  \end{lstlisting}
 
+\subparagraph{Supported/enabled tunnel hash types}
+\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled tunnel hash types}
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the value of
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE,
+the device calculates the hash using the outer header of the encapsulated packet.
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NONE         (1 << 0)
+\end{lstlisting}
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the encapsulation
+hash type below indicates that the hash is calculated over the inner header of
+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE         (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the vxlan-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the geneve-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 3)
+\end{lstlisting}
+Hash type applicable for inner payload of the ip-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the nvgre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 5)
+\end{lstlisting}
+
  \subparagraph{IPv4 packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv4 packets}
  The device calculates the hash on IPv4 packets according to 'Enabled hash types' bitmask as follows:
@@ -980,6 +1045,44 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\subparagraph{Inner hash calculation of an encapsulated packet}
+If the driver negotiates the VIRTIO_NET_F_HASH_TUNNEL feature, it can configure the
+hash parameters (including \field{hash_tunnel_types}) for inner hash calculation by
+sending the VIRTIO_NET_CTRL_MQ_HASH_CONFIG command. Additionally, if the VIRTIO_NET_F_RSS
+feature is also negotiated, the driver can use the VIRTIO_NET_CTRL_RSS_CONFIG command to
+configure the hash parameters. If multiple commands are sent, the device configuration
+will be defined by the last command received.
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the corresponding
+encapsulation hash type is set in \field{hash_tunnel_types}, the device calculates the
+hash on the inner header of an encapsulated packet (See \ref{sec:Device Types
+/ Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}). If the encapsulation
+type of an encapsulated packet is not included in \field{hash_tunnel_types} or the value
+of \field{hash_tunnel_types} is VIRTIO_NET_HASH_TUNNEL_TYPE_NONE, the device calculates
+the hash on the outer header.
+
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the
+unencapsulated packets.
+
+\subparagraph{Tunnel QoS limitation}
+Note that the limitation mentioned below is not only introduced by inner hash calculation,
+and the limitation of the tunnel itself, and even the driver may have only one receive queue.
+
+When a specific receive queue is shared by multiple tunnels to receive encapsulating packets,
+there is no quality of service (QoS) for these packets of multiple tunnels. For example, when the
+flooded packets of a certain tunnel are hashed to the queue, it may cause the traffic of this
+queue to be unbalanced, resulting in potential packet loss and data delay.
+
+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance such as DPDK to keep the queue from filling up.
+\item If the quality of service is unavailable, the driver can set \field{hash_tunnel_types} to
+      VIRTIO_NET_HASH_TUNNEL_TYPE_NONE to disable inner hash calculation for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Outside the device, prevent abnormal traffic from entering or switch the traffic to attack clusters.
+\end{itemize}
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
 
@@ -1392,12 +1495,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 reserved[4];
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
  defined in
  \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}.
-Initially the device has all hash types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
+
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Initially the device has all hash types and hash tunnel types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
 
  Field \field{reserved} MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure,
  defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS)}.
@@ -1421,6 +1529,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 max_tx_vq;
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
@@ -1441,6 +1550,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  Fields \field{hash_key_length} and \field{hash_key_data} define the key to be used in hash calculation.
 
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
  \drivernormative{\subparagraph}{Setting RSS parameters}{Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
 
  A driver MUST NOT send the VIRTIO_NET_CTRL_MQ_RSS_CONFIG command if the feature VIRTIO_NET_F_RSS has not been negotiated.
introduction.tex +17 -0
index 287c5fc..25c9d48 100644
@@ -99,6 +99,30 @@ \section{Normative References}\label{sec:Normative References}
      Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
  	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
 
+	\phantomsection\label{intro:GRE}\textbf{[GRE]} &
+    Generic Routing Encapsulation
+	\newline\url{https://datatracker.ietf.org/doc/rfc2784/}\\
+	\phantomsection\label{intro:VXLAN}\textbf{[VXLAN]} &
+    Virtual eXtensible Local Area Network
+	\newline\url{https://datatracker.ietf.org/doc/rfc7348/}\\
+	\phantomsection\label{intro:GENEVE}\textbf{[GENEVE]} &
+    Generic Network Virtualization Encapsulation
+	\phantomsection\label{intro:IPIP}\textbf{[IPIP]} &
+    IP Encapsulation within IP
+	\newline\url{https://www.rfc-editor.org/rfc/rfc2003}\\
+	\phantomsection\label{intro:IPIP}\textbf{[NVGRE]} &
+    NVGRE: Network Virtualization Using Generic Routing Encapsulation
+	\newline\url{https://www.rfc-editor.org/rfc/rfc7637.html}\\
+	\newline\url{https://datatracker.ietf.org/doc/rfc8926/}\\
+	\phantomsection\label{intro:IP}\textbf{[IP]} &
+    INTERNET PROTOCOL

introduction.tex \section{Normative References}\label{sec:Normative References} · line 128

Quoted 10 lines
introduction.tex +7 -0
+	\newline\url{https://www.rfc-editor.org/rfc/rfc791}\\
+	\phantomsection\label{intro:UDP}\textbf{[UDP]} &
+    User Datagram Protocol
+	\newline\url{https://www.rfc-editor.org/rfc/rfc768}\\
+	\phantomsection\label{intro:TCP}\textbf{[TCP]} &
+    TRANSMISSION CONTROL PROTOCOL
+	\newline\url{https://www.rfc-editor.org/rfc/rfc793}\\
  \end{longtable}
 
  \section{Non-Normative References}

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw
In reply to Heng Qi
Quoted 2 lines · Heng Qi

+\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash

May be to say inner packet header hash..
This make its little more clear about "which header" that you explained in the commit log.

device-types/net/description.tex \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device · line 145

Quoted 10 lines
device-types/net/description.tex +3 -0
+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +142,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.

I think this should also say that HASH_TUNNEL requires either of the F_RSS or F_HASH_REPORT. Because without it HASH_TUNNEL is not useful.
Right?
if no, than my below comments are meaningless.

Quoted 4 lines
device-types/net/description.tex +0 -0
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

device-types/net/description.tex \subsection{Device configuration layout}\label{sec:Device Types / Network Device · line 210

Quoted 10 lines
device-types/net/description.tex +2 -1
@@ -198,20 +202,27 @@ \subsection{Device configuration layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;
+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
  It specifies the maximum supported length of RSS key in bytes.
 

I think rss_max_key_size field dependency should be only of the existing feature bits F_RSS and F_HASH_REPORT. This is because those are the bits really deciding to consider rss_max_key_size.

device-types/net/description.tex \subsection{Device configuration layout}\label{sec:Device Types / Network Device · line 216

Quoted 7 lines
device-types/net/description.tex +1 -1
  The following field, \field{rss_max_indirection_table_length} only exists if VIRTIO_NET_F_RSS is set.
  It specifies the maximum number of 16-bit entries in RSS indirection table.
 
  The next field, \field{supported_hash_types} only exists if the device supports hash calculation,
-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set.
 

Same as above.

device-types/net/description.tex \subsection{Device configuration layout}\label{sec:Device Types / Network Device · line 222

Quoted 6 lines
device-types/net/description.tex +3 -0
  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types.
 
+The next field, \field{supported_tunnel_hash_types} only exists if the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is set.
+

Above line "the next field .." can be just same as "Device supports inner packet header hash calculation, i.e..." This is because here, the term "header" is missed, which is present in the definition of feature bit 52.

device-types/net/description.tex \subsection{Device configuration layout}\label{sec:Device Types / Network Device · line 250

Quoted 5 lines
device-types/net/description.tex +1 -1
  The device MUST set \field{rss_max_key_size} to at least 40, if it offers
-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.
 

This needs change if the above first comment about rss_max_key_size is right.

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 859

Quoted 9 lines
device-types/net/description.tex +1 -0
  The device MUST set \field{rss_max_indirection_table_length} to at least 128, if it offers
  VIRTIO_NET_F_RSS.
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  \begin{itemize}
  \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses the hash to determine the receive virtqueue to place incoming packets.
  \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The device reports the hash value and the hash type with the packet.
+\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The device supports inner hash calculation.
  \end{itemize}
 

inner packet header hash ..

Quoted 14 lines
device-types/net/description.tex +14 -0
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the encapsulation
+hash type below indicates that the hash is calculated over the inner header of
+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE         (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the vxlan-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the geneve-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 3)

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 956

Quoted 10 lines
device-types/net/description.tex +10 -0
+\end{lstlisting}
+Hash type applicable for inner payload of the ip-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the nvgre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 5)
+\end{lstlisting}
+

Any encapsulation technology that includes UDP/L4 header likely do not prefer based on the inner header. This is because the outer header src port entropy is added based on the inner header.

I was not able to follow the discussion in v9 that you had with Michael. Did you conclude if this is needed for vxlan too?

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Quoted 21 lines

\subparagraph{IPv4 packets}
\label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv4 packets} The device calculates the hash on IPv4 packets according to 'Enabled hash types' bitmask as follows:

device-types/net/description.tex +14 -0
@@ -980,6 +1045,44 @@ \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}).
  \end{itemize}
 
+\subparagraph{Inner hash calculation of an encapsulated packet}
+If the driver negotiates the VIRTIO_NET_F_HASH_TUNNEL feature, it can configure the
+hash parameters (including \field{hash_tunnel_types}) for inner hash calculation by
+sending the VIRTIO_NET_CTRL_MQ_HASH_CONFIG command. Additionally, if the VIRTIO_NET_F_RSS
+feature is also negotiated, the driver can use the VIRTIO_NET_CTRL_RSS_CONFIG command to
+configure the hash parameters. If multiple commands are sent, the device configuration
+will be defined by the last command received.
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the corresponding
+encapsulation hash type is set in \field{hash_tunnel_types}, the device calculates the
+hash on the inner header of an encapsulated packet (See \ref{sec:Device Types
+/ Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated packet}). If the encapsulation
+type of an encapsulated packet is not included in \field{hash_tunnel_types} or the value

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 1071

Quoted 10 lines
device-types/net/description.tex +10 -0
+of \field{hash_tunnel_types} is VIRTIO_NET_HASH_TUNNEL_TYPE_NONE, the device calculates
+the hash on the outer header.
+
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the
+unencapsulated packets.
+
+\subparagraph{Tunnel QoS limitation}
+Note that the limitation mentioned below is not only introduced by inner hash calculation,
+and the limitation of the tunnel itself, and even the driver may have only one receive queue.
+

When there is a single receive queue, there is no tunnel QoS issue. Because the same queue is used one or more tunnels. So please remove mentioning single queue text here.

Also it is better to first describe the limitation and after the reader understand, it make sense to say that the limitation already exists with/without inner header hash calculation.

Review \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 1073

Quoted 2 lines
(quoted diff) +2 -0
+When a specific receive queue is shared by multiple tunnels to receive encapsulating packets,
+there is no quality of service (QoS) for these packets of multiple tunnels. 

"of multiple tunnels at the tail of sentence is not needed because you already have it at "shared by multiple tunnels".

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 1076

Quoted 4 lines

For example, when the

device-types/net/description.tex +3 -0
+flooded packets of a certain tunnel are hashed to the queue, it may cause the traffic of this
+queue to be unbalanced, resulting in potential packet loss and data delay.
+

Your example is only talking about single queue..

You probably wanted to say,

When the packets of certain tunnels results in spreading these packets to multiple receive queues, these receive queues may have unbalanced amount of packets. This can result in a specific receive queue being full resulting in packet loss.

Or something similar..

Quoted 3 lines

+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance such as DPDK to keep the queue from filling up.

I dont think it is necessary to talk about specific tools like DPDK in specification. Even if you omit "such as DPDK", the line reads fine.
so I suggest to drop "such as DPDK".

Quoted 5 lines

+\item If the quality of service is unavailable, the driver can set \field{hash_tunnel_types} to
+ VIRTIO_NET_HASH_TUNNEL_TYPE_NONE to disable inner hash calculation for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions.
+\item Outside the device, prevent abnormal traffic from entering or switch the traffic to attack clusters. +\end{itemize}

It can still enter the switch but you want to avoid entering the virtio device. Cluster is very broad term and not defined in the spec, better to avoid it.

you can rewrite it something like, Perform appropriate QoS before packets consume the receive buffers..

This is generic solution that can be done in device or egress of switch etc, which keep the spec scope upto the device.

Quoted 32 lines
device-types/net/description.tex +8 -1
+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
@@ -1392,12 +1495,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 reserved[4];
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
  defined in
  \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}.
-Initially the device has all hash types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
+
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
+Initially the device has all hash types and hash tunnel types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
 
  Field \field{reserved} MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure,
  defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS)}.
@@ -1421,6 +1529,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 max_tx_vq;
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1557

Quoted 8 lines
device-types/net/description.tex +3 -0
@@ -1441,6 +1550,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  Fields \field{hash_key_length} and \field{hash_key_data} define the key to be used in hash calculation.
 
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
  \drivernormative{\subparagraph}{Setting RSS parameters}{Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }
 

You likely need to add device and driver normative statements derived from above text and add their references into device-types/net/*-conformance.tex files.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
Quoted 2 lines · Parav Pandit

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

So what's left, GRE? GRE is actually different, in that it's not IP at all.

So if we are talking about GRE, hash is indeed not calculated at all at the moment, right? And I would say a natural first step for GRE is actually adding a hash type that will support this protocol.

How about doing that? It seems like this should be a small step and completely uncontroversial.

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
author
#6 lore raw
Quoted 5 lines · Michael S. Tsirkin
Quoted 2 lines · Parav Pandit

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

So what's left, GRE? GRE is actually different, in that it's not IP at all.

I do not think so, I mentioned that VXLAN and GENEVE need inner symmetric hashing, and we need this.

And we know inner hashing doesn't conflict with other ways of adding entropy.

Thanks.

Quoted 8 lines

So if we are talking about GRE, hash is indeed not calculated at all at the moment, right? And I would say a natural first step for GRE is actually adding a hash type that will support this protocol.

How about doing that? It seems like this should be a small step and completely uncontroversial.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#7 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 2 lines · Parav Pandit

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

So what's left, GRE? GRE is actually different, in that it's not IP at all.

Sorry, I wrongly wrote nvegre above.

IPoIP, GRE and NVGRE are left.

vxlan and geneve has the udp src entropy.

Quoted 1 line

Not sure I understand "its not IP at all".

GRE has outer IP header + GRE header with the key to identify the flow. The key is effectively the hash for the flow.

Quoted 2 lines

So if we are talking about GRE, hash is indeed not calculated at all at the moment, right?

Hash of the outer IP header of the src and dst IP can be still calculated currently for GRE when the optional key is not present.

Quoted 3 lines

And I would say a natural first step for GRE is actually adding a hash type that will support this protocol.

For GRE and NVGRE GRE_header.key as the flow/hash identifier should work without inner header hash. Older version of the GRE doesn't have key, so inner header hash is useful.

Quoted 2 lines

How about doing that? It seems like this should be a small step and completely uncontroversial.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#8 lore raw
Quoted 11 lines · Parav Pandit
Quoted 2 lines

So what's left, GRE? GRE is actually different, in that it's not IP at all.

Sorry, I wrongly wrote nvegre above.

IPoIP, GRE and NVGRE are left.

vxlan and geneve has the udp src entropy.

Quoted 1 line

Not sure I understand "its not IP at all".

GRE header is distinct from IP header and has its own flow ID.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#9 lore raw
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, March 16, 2023 3:36 AM

Quoted 6 lines · Parav Pandit
Quoted 2 lines

So what's left, GRE? GRE is actually different, in that it's not IP at all.

Sorry, I wrongly wrote nvegre above.

IPoIP, GRE and NVGRE are left.

Quoted 7 lines
Quoted 5 lines

vxlan and geneve has the udp src entropy.

Quoted 1 line

Not sure I understand "its not IP at all".

GRE header is distinct from IP header and has its own flow ID.

Yes. GRE is IP header + GRE header.
The latest one has the key in GRE header, the older one has only IP.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#10 lore raw
In reply to Parav Pandit
Quoted 22 lines · Parav Pandit
Quoted 6 lines · Michael S. Tsirkin
Quoted 2 lines · Parav Pandit

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

So what's left, GRE?  GRE is actually different, in that it's not IP at all.

Sorry, I wrongly wrote nvegre above.

IPoIP, GRE and NVGRE are left.

vxlan and geneve has the udp src entropy.

Quoted 1 line

Not sure I understand "its not IP at all".

GRE has outer IP header + GRE header with the key to identify the flow. The key is effectively the hash for the flow.

Quoted 2 lines

So if we are talking about GRE, hash is indeed not calculated at all at the moment, right?

Quoted 7 lines

Hash of the outer IP header of the src and dst IP can be still calculated currently for GRE when the optional key is not present.

Quoted 3 lines

And I would say a natural first step for GRE is actually adding a hash type that will support this protocol.

For GRE and NVGRE GRE_header.key as the flow/hash identifier should work without inner header hash. Older version of the GRE doesn't have key, so inner header hash is useful.

Yes, indeed.
The old GRE does not have keys such as flow id. Even with the new GRE, we have no chance to use optional fields, because we have connected with various operators, and the most basic fields can get the best compatibility.

Thanks.

Quoted 7 lines
Quoted 3 lines

How about doing that? It seems like this should be a small step and completely uncontroversial.

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#11 lore raw
In reply to Parav Pandit
Quoted 8 lines · Parav Pandit
Quoted 1 line · Heng Qi

  +\item[VIRTIO_NET_F_HASH_TUNNEL(52)] Device supports inner header hash

May be to say inner packet header hash..
This make its little more clear about "which header" that you explained in the commit log.

Sure, I'll add this.

Quoted 4 lines
Quoted 4 lines

+    for tunnel-encapsulated packets.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.

Quoted 8 lines
Quoted 6 lines
(quoted diff) +0 -0
@@ -139,6 +142,7 @@ \subsubsection{Feature bit 

requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.

I think this should also say that HASH_TUNNEL requires either of the F_RSS or F_HASH_REPORT. Because without it HASH_TUNNEL is not useful.

F_HASH_TUNNEL indicates that the hash should be calculated using the inner packet header, even without F_RSS or F_HASH_REPORT, we can continue to use the hash value in scenarios such as RPS or ebpf programs.

Quoted 2 lines

Right?
if no, than my below comments are meaningless.

I think it's fine to let F_HASH_TUNNEL rely on F_RSS or _F_HASH_REPORT as those are probably important scenarios where inner packet header hash is used.

Quoted 30 lines
Quoted 9 lines

  \end{description}
    \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}

(quoted diff) +0 -0
@@ -198,20 +202,27 @@ \subsection{Device configuration 

layout}\label{sec:Device Types / Network Device
          u8 rss_max_key_size;
          le16 rss_max_indirection_table_length;
          le32 supported_hash_types;

Quoted 9 lines

+        le32 supported_tunnel_hash_types;
  };
  \end{lstlisting}
-The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+The following field, \field{rss_max_key_size} only exists if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL is set. It specifies the maximum supported length of RSS key in bytes.

I think rss_max_key_size field dependency should be only of the existing feature bits F_RSS and F_HASH_REPORT. This is because those are the bits really deciding to consider rss_max_key_size.

Quoted 7 lines

  The following field, \field{rss_max_indirection_table_length} only exists if VIRTIO_NET_F_RSS is set. It specifies the maximum number of 16-bit entries in RSS indirection table.
    The next field, \field{supported_hash_types} only exists if the device supports hash calculation,

(quoted diff) +1 -1
-i.e. if VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT is set.
+i.e. if VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or 

VIRTIO_NET_F_HASH_TUNNEL is set.

Same as above.

Quoted 8 lines
Quoted 6 lines

  Field \field{supported_hash_types} contains the bitmask of supported hash types.
  See \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types} for details of supported hash types. +The next field, \field{supported_tunnel_hash_types} only exists if the device
+supports inner hash calculation, i.e. if VIRTIO_NET_F_HASH_TUNNEL is
set.
+

Above line "the next field .." can be just same as "Device supports inner packet header hash calculation, i.e..." This is because here, the term "header" is missed, which is present in the definition of feature bit 52.

I'll rephrase the description to make the whole text more consistent.

Quoted 6 lines

device-types/net/description.tex \subsection{Device configuration layout}\label{sec:Device Types / Network Device · line 249

Quoted 4 lines

    The device MUST set \field{rss_max_key_size} to at least 40, if it offers

device-types/net/description.tex +1 -1
-VIRTIO_NET_F_RSS or VIRTIO_NET_F_HASH_REPORT.
+VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT or VIRTIO_NET_F_HASH_TUNNEL.

This needs change if the above first comment about rss_max_key_size is right.

Quoted 12 lines
Quoted 11 lines

  The device MUST set \field{rss_max_indirection_table_length} to at least 128, if it offers VIRTIO_NET_F_RSS.

(quoted diff) +0 -0
@@ -843,11 +854,13 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
  \begin{itemize}
  \item The feature VIRTIO_NET_F_RSS was negotiated. The device uses the hash to determine the receive virtqueue to place incoming packets. \item The feature VIRTIO_NET_F_HASH_REPORT was negotiated. The device reports the hash value and the hash type with the packet. +\item The feature VIRTIO_NET_F_HASH_TUNNEL was negotiated. The
device supports inner hash calculation.
  \end{itemize}

inner packet header hash ..

Ok.

Quoted 30 lines
Quoted 18 lines

+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated, the encapsulation
+hash type below indicates that the hash is calculated over the inner header of

device-types/net/description.tex +10 -0
+the encapsulated packet:
+Hash type applicable for inner payload of the gre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GRE         (1 << 1)
+\end{lstlisting}
+Hash type applicable for inner payload of the vxlan-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_VXLAN       (1 << 2)
+\end{lstlisting}
+Hash type applicable for inner payload of the geneve-encapsulated 

packet
+\begin{lstlisting}

device-types/net/description.tex +1 -0
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_GENEVE      (1 << 3)

device-types/net/description.tex \subsubsection{Processing of Incoming Packets}\label{sec:Device Types / Network · line 956

Quoted 10 lines
device-types/net/description.tex +10 -0
+\end{lstlisting}
+Hash type applicable for inner payload of the ip-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_IPIP        (1 << 4)
+\end{lstlisting}
+Hash type applicable for inner payload of the nvgre-encapsulated packet
+\begin{lstlisting}
+#define VIRTIO_NET_HASH_TUNNEL_TYPE_NVGRE       (1 << 5)
+\end{lstlisting}
+

Any encapsulation technology that includes UDP/L4 header likely do not prefer based on the inner header. This is because the outer header src port entropy is added based on the inner header.

Quoted 4 lines

I was not able to follow the discussion in v9 that you had with Michael. Did you conclude if this is needed for vxlan too?

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Symmetric hashing ignores the order of the five-tuples when calculating the hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can calculate the same hash. There is a scenario that the two directions client1->client2 and client2->client1 of the same flow may pass through different tunnels.
In order to allow the data in two directions to be processed by the same CPU, we need to calculate a symmetric hash based on the inner packet header. Sorry I didn't mention this earlier just to avoid introducing the concept of symmetric hashing.

Quoted 40 lines
Quoted 40 lines

  \subparagraph{IPv4 packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv4 packets} The device calculates the hash on IPv4 packets according to 'Enabled hash types' bitmask as follows:

(quoted diff) +0 -0
@@ -980,6 +1045,44 @@ \subsubsection{Processing of Incoming 

Packets}\label{sec:Device Types / Network
  (see \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / IPv6 packets without extension header}). \end{itemize}
  +\subparagraph{Inner hash calculation of an encapsulated packet}
+If the driver negotiates the VIRTIO_NET_F_HASH_TUNNEL feature, it
can configure the
+hash parameters (including \field{hash_tunnel_types}) for inner hash
calculation by
+sending the VIRTIO_NET_CTRL_MQ_HASH_CONFIG command. Additionally, if
the VIRTIO_NET_F_RSS
+feature is also negotiated, the driver can use the
VIRTIO_NET_CTRL_RSS_CONFIG command to
+configure the hash parameters. If multiple commands are sent, the
device configuration

device-types/net/description.tex +3 -0
+will be defined by the last command received.
+
+If the feature VIRTIO_NET_F_HASH_TUNNEL is negotiated and the 

corresponding
+encapsulation hash type is set in \field{hash_tunnel_types}, the device calculates the
+hash on the inner header of an encapsulated packet (See \ref{sec:Device Types

(quoted diff) +2 -0
+/ Network Device / Device Operation / Processing of Incoming Packets /
+Hash calculation for incoming packets / Tunnel/Encapsulated 

packet}). If the encapsulation
+type of an encapsulated packet is not included in \field{hash_tunnel_types} or the value
+of \field{hash_tunnel_types} is VIRTIO_NET_HASH_TUNNEL_TYPE_NONE, the device calculates

device-types/net/description.tex +3 -0
+the hash on the outer header.
+
+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE 

by the device for the

(quoted diff) +2 -0
+unencapsulated packets.
+
Quoted 9 lines
Quoted 6 lines

+\subparagraph{Tunnel QoS limitation}
+Note that the limitation mentioned below is not only introduced by inner hash calculation,
+and the limitation of the tunnel itself, and even the driver may have only one receive queue.
+

When there is a single receive queue, there is no tunnel QoS issue. Because the same queue is used one or more tunnels. So please remove mentioning single queue text here.

Ok.

Quoted 1 line

Also it is better to first describe the limitation and after the reader understand, it make sense to say that the limitation already exists with/without inner header hash calculation.

I'll update this.

Quoted 6 lines
Quoted 5 lines

+When a specific receive queue is shared by multiple tunnels to receive encapsulating packets,
+there is no quality of service (QoS) for these packets of multiple tunnels.

"of multiple tunnels at the tail of sentence is not needed because you already have it at "shared by multiple tunnels".

Ok.

Quoted 9 lines
Quoted 7 lines

For example, when the
+flooded packets of a certain tunnel are hashed to the queue, it may cause the traffic of this
+queue to be unbalanced, resulting in potential packet loss and data delay.
+

Your example is only talking about single queue..

Quoted 3 lines

You probably wanted to say,

When the packets of certain tunnels results in spreading these packets to multiple receive queues, these receive queues may have unbalanced amount of packets. This can result in a specific receive queue being full resulting in packet loss.

Yes, I didn't describe clearly. Thanks for the example!

Quoted 10 lines

Or something similar..

Quoted 4 lines

+Possible mitigations:
+\begin{itemize}
+\item Use a tool with good forwarding performance such as DPDK to keep the queue from filling up.

I dont think it is necessary to talk about specific tools like DPDK in specification. Even if you omit "such as DPDK", the line reads fine.
so I suggest to drop "such as DPDK".

I agree.

Quoted 9 lines
Quoted 9 lines

+\item If the quality of service is unavailable, the driver can set \field{hash_tunnel_types} to
+      VIRTIO_NET_HASH_TUNNEL_TYPE_NONE to disable inner hash calculation for encapsulated packets.
+\item Choose a hash key that can avoid queue collisions. +\item Outside the device, prevent abnormal traffic from entering or switch the traffic to attack clusters.
+\end{itemize}

Quoted 6 lines

It can still enter the switch but you want to avoid entering the virtio device. Cluster is very broad term and not defined in the spec, better to avoid it.

you can rewrite it something like, Perform appropriate QoS before packets consume the receive buffers..

This is generic solution that can be done in device or egress of switch etc, which keep the spec scope upto the device.

I see. This does add a new concept "cluster" beyond the device, and I'll remove it.

Quoted 37 lines
Quoted 37 lines

+
  \paragraph{Hash reporting for incoming packets}
  \label{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash reporting for incoming packets}
  @@ -1392,12 +1495,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le16 reserved[4];
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as
  defined in
  \ref{sec:Device Types / Network Device / Device Operation / Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash types}. -Initially the device has all hash types disabled and reports only
VIRTIO_NET_HASH_REPORT_NONE.

(quoted diff) +2 -0
+
+Field \field{hash_tunnel_types} contains a bitmask of allowed hash 

tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation
/ Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.

(quoted diff) +2 -0
+
+Initially the device has all hash types and hash tunnel types 

disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.
    Field \field{reserved} MUST contain zeroes. It is defined to make the structure to match the layout of virtio_net_rss_config structure, defined in \ref{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS)}.

(quoted diff) +0 -0
@@ -1421,6 +1529,7 @@ \subsubsection{Control 

Virtqueue}\label{sec:Device Types / Network Device / Devi le16 max_tx_vq;
      u8 hash_key_length;
      u8 hash_key_data[hash_key_length];
+    le32 hash_tunnel_types;
  };
  \end{lstlisting}
  Field \field{hash_types} contains a bitmask of allowed hash types as

(quoted diff) +0 -0
@@ -1441,6 +1550,9 @@ \subsubsection{Control 

Virtqueue}\label{sec:Device Types / Network Device / Devi

Quoted 8 lines
Quoted 6 lines

    Fields \field{hash_key_length} and \field{hash_key_data} define the key to be used in hash calculation.
  +Field \field{hash_tunnel_types} contains a bitmask of allowed hash tunnel types as
+defined in \ref{sec:Device Types / Network Device / Device Operation
/ Processing of Incoming Packets / Hash calculation for incoming packets / Supported/enabled hash tunnel types}.
+
  \drivernormative{\subparagraph}{Setting RSS parameters}{Device Types / Network Device / Device Operation / Control Virtqueue / Receive-side scaling (RSS) }

You likely need to add device and driver normative statements derived from above text and add their references into device-types/net/*-conformance.tex files.

Sure, thanks for the suggestion! :)

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#12 lore raw
Quoted 11 lines · Heng Qi
Quoted 10 lines

Any encapsulation technology that includes UDP/L4 header likely do not prefer based on the inner header. This is because the outer header src port entropy is added based on the inner header.

I was not able to follow the discussion in v9 that you had with Michael. Did you conclude if this is needed for vxlan too?

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Quoted 9 lines

Symmetric hashing ignores the order of the five-tuples when calculating the hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can calculate the same hash.
There is a scenario that the two directions client1->client2 and client2->client1 of the same flow may pass through different tunnels. In order to allow the data in two directions to be processed by the same CPU, we need to calculate a symmetric hash based on the inner packet header. Sorry I didn't mention this earlier just to avoid introducing the concept of symmetric hashing.

But the hash is already there in the port. Is it then maybe just the question of ignoring the IP addresses when hashing?

Signature · 6 lines
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
author
#13 lore raw
Quoted 20 lines · Michael S. Tsirkin
Quoted 10 lines · Heng Qi
Quoted 10 lines

Any encapsulation technology that includes UDP/L4 header likely do not prefer based on the inner header. This is because the outer header src port entropy is added based on the inner header.

I was not able to follow the discussion in v9 that you had with Michael. Did you conclude if this is needed for vxlan too?

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Quoted 9 lines

Symmetric hashing ignores the order of the five-tuples when calculating the hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can calculate the same hash.
There is a scenario that the two directions client1->client2 and client2->client1 of the same flow may pass through different tunnels. In order to allow the data in two directions to be processed by the same CPU, we need to calculate a symmetric hash based on the inner packet header. Sorry I didn't mention this earlier just to avoid introducing the concept of symmetric hashing.

Quoted 2 lines

But the hash is already there in the port. Is it then maybe just the question of ignoring the IP addresses when hashing?

We do not ignore the IP address, because after the tunnel sends the packets to the processing host, the processing host will parse the outer headers, and then use the inner symmetric hash to hand over the packets of the same flow to the same cpu for processing (for the network topology, please check my latest reply thread).

Thanks.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#14 lore raw
In reply to Heng Qi
Quoted 7 lines · Heng Qi
Quoted 4 lines · Parav Pandit

On 3/6/2023 10:48 AM, Heng Qi wrote:

[..]

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.

I think this should also say that HASH_TUNNEL requires either of the F_RSS or F_HASH_REPORT. Because without it HASH_TUNNEL is not useful.

F_HASH_TUNNEL indicates that the hash should be calculated using the inner packet header, even without F_RSS or F_HASH_REPORT, we can continue to use the hash value in scenarios such as RPS or ebpf programs.

Yes.
Even for rps or ebpf programs, F_HASH_TUNNEL is fine.
When such feature arrives in future, it above line will have OR condition for RPS feature bit.

Quoted 3 lines

I think it's fine to let F_HASH_TUNNEL rely on F_RSS or _F_HASH_REPORT as those are probably important scenarios where inner packet header hash is used.

Yes.

Quoted 5 lines
Quoted 1 line

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Symmetric hashing ignores the order of the five-tuples when calculating the hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can calculate the same hash. There is a scenario that the two directions client1->client2 and client2->client1 of the same flow may pass through different tunnels.
In order to allow the data in two directions to be processed by the same CPU, we need to calculate a symmetric hash based on the inner packet header.

I am lost in two directions and two clients above.
When you say two directions, do you mean tx and rx?
and do symmetric hashing between tx and rx between two end points within single tunnel?

Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines

+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the

(quoted diff) +2 -0
+unencapsulated packets.
+

I missed this before. unencapsulated is not a term. s/unencapsulated packets/Non encapsulated packets or non tunneled packets

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#15 lore raw
Quoted 46 lines · Parav Pandit
Quoted 7 lines · Heng Qi
Quoted 4 lines · Parav Pandit

On 3/6/2023 10:48 AM, Heng Qi wrote:

[..]

Quoted 10 lines
Quoted 5 lines
Quoted 2 lines

  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_HASH_TUNNEL] Requires VIRTIO_NET_F_CTRL_VQ.

I think this should also say that HASH_TUNNEL requires either of the F_RSS or F_HASH_REPORT.
Because without it HASH_TUNNEL is not useful.

F_HASH_TUNNEL indicates that the hash should be calculated using the inner packet header, even without F_RSS or F_HASH_REPORT, we can continue to use the hash value in scenarios such as RPS or ebpf programs.

Yes.
Even for rps or ebpf programs, F_HASH_TUNNEL is fine. When such feature arrives in future, it above line will have OR condition for RPS feature bit.

Quoted 5 lines

I think it's fine to let F_HASH_TUNNEL rely on F_RSS or _F_HASH_REPORT as those are probably important scenarios where inner packet header hash is used.

Yes.

Quoted 4 lines
Quoted 3 lines

If not, for now it may be better to skip vxlan and nvegre as they inherently have unique outer header UDP src port based on the inner header.

Quoted 9 lines

Symmetric hashing ignores the order of the five-tuples when calculating the hash, that is, using (a1,a2,p1,p2) and (a2,a1,p2,p1) respectively can calculate the same hash. There is a scenario that the two directions client1->client2 and client2->client1 of the same flow may pass through different tunnels.
In order to allow the data in two directions to be processed by the same CPU, we need to calculate a symmetric hash based on the inner packet header.

Quoted 5 lines

I am lost in two directions and two clients above.
When you say two directions, do you mean tx and rx?
and do symmetric hashing between tx and rx between two end points within single tunnel?

A rough description of this scene is as follows:
Client1 sends packets to client2, and client2 sends packets to client1 respectively. This is called two directions, and they are in the same flow.
The packets in the two directions may reach the processing host through different tunnels. In this scenario, we need to hash these packets to the same queue for the same cpu to process, so inner symmetric hashing is required.

client1 client2

    |                         |
    |      __________         |
    +----->| tunnels |<-------+
           |---------|
              |  |
              |  |
              |  |
              v  v
        +-----------------+
        | processing host |
        +-----------------+
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines

+\field{hash_tunnel_types} is set to VIRTIO_NET_HASH_TUNNEL_TYPE_NONE by the device for the

(quoted diff) +2 -0
+unencapsulated packets.
+

I missed this before. unencapsulated is not a term. s/unencapsulated packets/Non encapsulated packets or non tunneled packets

Yes, you are right!

Thanks.

Quoted 7 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 8 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#16 lore raw
Quoted 14 lines

From: Heng Qi <[email protected]>
Sent: Thursday, March 16, 2023 9:39 AM
A rough description of this scene is as follows:

Client1 sends packets to client2, and client2 sends packets to client1 respectively.
This is called two directions, and they are in the same flow.
The packets in the two directions may reach the processing host through different tunnels. In this scenario, we need to hash these packets to the same queue for the same cpu to process, so inner symmetric hashing is required.

client1 client2

Quoted 12 lines
    |                         |
    |      __________         |
    +----->| tunnels |<-------+
           |---------|
              |  |
              |  |
              |  |
              v  v
        +-----------------+
        | processing host |
        +-----------------+

I understood now. The key piece is the processing host is different from clients 1 and 2, I was missing that. The key is there are two tunnels and processing data on the inner header for the different tunnels is your primary requirement. Hence inner hashing helps to serialize them.

author
#17 lore raw
Quoted 24 lines · Parav Pandit
Quoted 12 lines

From: Heng Qi <[email protected]>
Sent: Thursday, March 16, 2023 9:39 AM
A rough description of this scene is as follows:
Client1 sends packets to client2, and client2 sends packets to client1 respectively.
This is called two directions, and they are in the same flow.
The packets in the two directions may reach the processing host through different tunnels. In this scenario, we need to hash these packets to the same queue for the same cpu to process, so inner symmetric hashing is required.

client1 client2

Quoted 12 lines
     |                         |
     |      __________         |
     +----->| tunnels |<-------+
            |---------|
               |  |
               |  |
               |  |
               v  v
         +-----------------+
         | processing host |
         +-----------------+
Quoted 3 lines

I understood now. The key piece is the processing host is different from clients 1 and 2, I was missing that. The key is there are two tunnels and processing data on the inner header for the different tunnels is your primary requirement. Hence inner hashing helps to serialize them.

Yes, you are right.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/