Skip to contentSkip to conversation

virtio-net: support setting coalescing params for multiple vqs

PATCH Superseded Heng Qi 4 participants 20 messages Jan 15, 2024 – Jan 25, 2024

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Heng Qi

author
#0 lore raw

PATCH v2 1/1 virtio-net: support setting coalescing params for multiple vqs

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

 device-types/net/description.tex | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +20 -6
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 for notification coalescing.
 
 If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
 \begin{lstlisting}
 struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     struct virtio_net_ctrl_coal coal;
 };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
 \end{lstlisting}
 
 Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
       for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
 \end{itemize}
 
 The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
 \end{enumerate}
 
 The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
 The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
 The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
 Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
 to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 18 lines
1.8.3.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw
Quoted 9 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use the queue as "queue". Such driver limitation should be removed in the driver. It does not qualify as limitation.

This will enable driver to enqueue multiple cvq commands without waiting for previous one. This will also enable device to find natural coalescing done on multiple commands.

Quoted 6 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural coalescing on the commands of a given cvq.

For multiple DMAs, we need to way to send 8 bytes of data without 16 bytes of indirection via a descriptor. This is what we discussed a while back to do in txq and Stefan suggested to generalize for more queues, which is also a good idea.

Quoted 11 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

The batching done may end up modifying the given VQ's parameters multiple times.
The right test on Linux to do without rtnl lock which is anyway ugly and wrong semantic to use blocking the whole netdev stack. (in case if you used that).

Quoted 4 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

Quoted 4 lines

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only means that moderation parameters are just delayed by the driver.

Quoted 3 lines

The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 27 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give v1->the
absolute value directly. @Michael

 device-types/net/description.tex | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex
index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can -
send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
Quoted 3 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

Quoted 115 lines

\begin{lstlisting}
struct virtio_net_ctrl_coal {

device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

     struct virtio_net_ctrl_coal coal;

};

device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
 \end{lstlisting}
 
 Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is read-only for the
driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:

device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled transmit/receive virtqueue whose
index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled transmit/receive virtqueue whose
index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries} enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is \field{vq_index}.
\end{enumerate}

The device may generate notifications more or less frequently than specified
by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of an enabled
transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The driver MUST set \field{num_entries} to a non-zero value and MUST
+NOT set \field{num_entries} to a value greater than the number of enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature
when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

The driver MUST ignore the values of coalescing parameters received from
the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. +The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device MUST set
the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
--
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#2 lore raw
Quoted 10 lines · Parav Pandit
Quoted 8 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use the queue as "queue". Such driver limitation should be removed in the driver. It does not qualify as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously. The user command cannot return before the final result is obtained.
And wait is not the problem this patch solves.

Quoted 3 lines

This will enable driver to enqueue multiple cvq commands without waiting for previous one. This will also enable device to find natural coalescing done on multiple commands.

When batch user commands occur, ensuring synchronization is a concern.

Quoted 9 lines
Quoted 7 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command.
Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Quoted 3 lines

For multiple DMAs, we need to way to send 8 bytes of data without 16 bytes of indirection via a descriptor. This is what we discussed a while back to do in txq and Stefan suggested to generalize for more queues, which is also a good idea.

Yes, this sounds good.

Quoted 5 lines
Quoted 5 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

Quoted 8 lines
Quoted 7 lines

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

The batching done may end up modifying the given VQ's parameters multiple times.

In practice, we do not try to accumulate multiple parameter modifications for a specific vqn.

Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and wrong semantic to use blocking the whole netdev stack. (in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

Quoted 6 lines
Quoted 5 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

YES. Makes sense for our DPUs.

Quoted 7 lines
Quoted 5 lines

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only means that moderation parameters are just delayed by the driver.

Why is it delayed by the driver? It is not delayed by the driver, the kick still happens for every command. In theory and practice, it will not affect DIM performance, but it will significantly reduce CPU consumption caused by waiting.

Quoted 26 lines
Quoted 3 lines

The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 21 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give v1->the
absolute value directly. @Michael

  device-types/net/description.tex | 26 ++++++++++++++++++++------
  1 file changed, 20 insertions(+), 6 deletions(-)
device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex
index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

Quoted 10 lines
Quoted 9 lines

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can -
send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

I tried to extend it to the command VIRTIO_NET_F_VQ_NOTF_COAL, is it too late?

Thanks!

Quoted 116 lines
Quoted 116 lines
  \begin{lstlisting}
  struct virtio_net_ctrl_coal {
device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

      struct virtio_net_ctrl_coal coal;

};

device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
  \end{lstlisting}
 
  Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is read-only for the
driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:

device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled transmit/receive virtqueue whose
index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled transmit/receive virtqueue whose
index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries} enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is \field{vq_index}.
\end{enumerate}

The device may generate notifications more or less frequently than specified
by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of an enabled
transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The driver MUST set \field{num_entries} to a non-zero value and MUST
+NOT set \field{num_entries} to a value greater than the number of enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature
when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

The driver MUST ignore the values of coalescing parameters received from
the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. +The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device MUST set
the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
--
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 13 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw
Quoted 14 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 8 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

Quoted 8 lines

the queue as "queue".

Quoted 1 line

Such driver limitation should be removed in the driver. It does not qualify

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep for several micro to milliseconds.

Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the request can progress such as that of netdim.

Quoted 16 lines
Quoted 2 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 1 line

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a concern.

Quoted 4 lines
Quoted 4 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 3 lines
Quoted 3 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

Quoted 7 lines
Quoted 2 lines

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side, which can ignore them when they do not bother about it. Backend DMAs should be reduced by avoiding the LIFO pattern followed by the splitq driver.
Placing the descriptors contiguously like packedq reduces amount of DMA naturally.

The second predicable DMA to avoid is having 8Bytes of data inline in the descriptor, instead of 16B indirection and extra dma.

Quoted 9 lines
Quoted 3 lines

For multiple DMAs, we need to way to send 8 bytes of data without 16

bytes of indirection via a descriptor.

Quoted 1 line

This is what we discussed a while back to do in txq and Stefan suggested to

generalize for more queues, which is also a good idea.

Yes, this sounds good.

This the next item to focus as soon as flow filters are stable/merged.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

The batching done may end up modifying the given VQ's parameters

multiple times.

In practice, we do not try to accumulate multiple parameter modifications for a specific vqn.

I meant to accumulate parameters of multiple VQs to batch in single command.

Quoted 7 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Quoted 14 lines
Quoted 6 lines
Quoted 5 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

YES. Makes sense for our DPUs.

Quoted 2 lines
Quoted 2 lines

Maintainers may be concerned about whether the batch command

method

Quoted 1 line
Quoted 1 line

can optimize the above problems: accumulate multiple request

commands

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only

means that moderation parameters are just delayed by the driver.

Quoted 1 line

Why is it delayed by the driver? It is not delayed by the driver, the kick still happens for every command.

Since the kick is delayed, the driver lost the amount of time and device reaction time is driven by this wait time. The guest driver is not aware how much a shared DPU is busy.
So if it waits too much, the VQs moderation is slow.

Quoted 3 lines

In theory and practice, it will not affect DIM performance, but it will significantly reduce CPU consumption caused by waiting.

Once the guest driver sleeps after enqueuing the request, it should be fine.
The DIM calls are in highly parallel worker threads, where/if the thread sleep, a new worker thread is invoked.

But for sure I agree that vq notification moderation should happen quickly enough under < 100usec or so. May be now that we bring the aq resources, we can start migrating on it.

Quoted 36 lines
Quoted 25 lines
Quoted 3 lines

The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 20 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not v1->give the
absolute value directly. @Michael

  device-types/net/description.tex | 26 ++++++++++++++++++++------
  1 file changed, 20 insertions(+), 6 deletions(-)
device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

Quoted 10 lines
Quoted 9 lines

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver
can - send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

Quoted 3 lines

I tried to extend it to the command VIRTIO_NET_F_VQ_NOTF_COAL, is it too late?

For existing command it is not.
But there may be a device which may not support coalesced command.
Better to have the feature bit for this coalesce command, instead of random NOSUPP failure.

Quoted 146 lines

Thanks!

Quoted 60 lines
Quoted 60 lines
  \begin{lstlisting}
  struct virtio_net_ctrl_coal {
device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

      struct virtio_net_ctrl_coal coal;

};

device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
  \end{lstlisting}
 
  Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is
read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:

device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries}
+ enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is \field{vq_index}.
\end{enumerate}

The device may generate notifications more or less frequently than
specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of an
enabled transmit or receive virtqueue.

Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +3 -0
+The driver MUST set \field{num_entries} to a non-zero value and MUST
+NOT set \field{num_entries} to a value greater than the number of
+enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL

feature

Quoted 5 lines
Quoted 5 lines

when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 7 lines
Quoted 7 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

Quoted 8 lines
Quoted 8 lines

The driver MUST ignore the values of coalescing parameters received
from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to the

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 1 line
Quoted 1 line

command with VIRTIO_NET_ERR if it was not able to change the

parameters.

Quoted 1 line
Quoted 1 line

+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -

VIRTIO_NET_ERR if the

Quoted 4 lines
Quoted 4 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with

VIRTIO_NET_ERR if the

Quoted 5 lines
Quoted 5 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device
MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

command, or,

Quoted 26 lines
Quoted 12 lines

if the driver did not set any TX coalescing parameters, to 0. --
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#4 lore raw
Quoted 18 lines · Parav Pandit
Quoted 18 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 8 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 1 line

Such driver limitation should be removed in the driver. It does not qualify

as limitation.

Yes, we don't have to wait.

Quoted 8 lines
Quoted 2 lines

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep for several micro to milliseconds.

Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/[email protected]/

Quoted 14 lines
Quoted 14 lines
Quoted 2 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 1 line

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a concern.

Quoted 3 lines
Quoted 3 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 2 lines
Quoted 2 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side, which can ignore them when they do not bother about it.

Driver notifications need to be processed by the DPU, which interferes with the CPU on the DPU.

Quoted 2 lines

Backend DMAs should be reduced by avoiding the LIFO pattern followed by the splitq driver. Placing the descriptors contiguously like packedq reduces amount of DMA naturally.

splitq is widely used, migrating to packedq is not that easy, especially when there are many components and hardware involved.

Quoted 2 lines

The second predicable DMA to avoid is having 8Bytes of data inline in the descriptor, instead of 16B indirection and extra dma.

Looking forward to working inline!
But I think this does not conflict with batch work, and combining the two will be more beneficial.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

For multiple DMAs, we need to way to send 8 bytes of data without 16

bytes of indirection via a descriptor.

Quoted 1 line

This is what we discussed a while back to do in txq and Stefan suggested to

generalize for more queues, which is also a good idea.

Yes, this sounds good.

This the next item to focus as soon as flow filters are stable/merged.

Greate!

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

The batching done may end up modifying the given VQ's parameters

multiple times.

In practice, we do not try to accumulate multiple parameter modifications for a specific vqn.

I meant to accumulate parameters of multiple VQs to batch in single command.

Yes, this is what is done now, practical example: https://lore.kernel.org/all/[email protected]/
In our scenario, only benefits are seen, no negative effects.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Totally looking forward to it.

Thanks,
Heng

Quoted 66 lines
Quoted 10 lines
Quoted 6 lines
Quoted 5 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

YES. Makes sense for our DPUs.

Quoted 1 line
Quoted 1 line

Maintainers may be concerned about whether the batch command

method

Quoted 8 lines
Quoted 1 line
Quoted 1 line

can optimize the above problems: accumulate multiple request

commands

Quoted 3 lines
Quoted 2 lines

to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only

means that moderation parameters are just delayed by the driver. Why is it delayed by the driver? It is not delayed by the driver, the kick still happens for every command.

Since the kick is delayed, the driver lost the amount of time and device reaction time is driven by this wait time. The guest driver is not aware how much a shared DPU is busy.
So if it waits too much, the VQs moderation is slow.

Quoted 3 lines

In theory and practice, it will not affect DIM performance, but it will significantly reduce CPU consumption caused by waiting.

Once the guest driver sleeps after enqueuing the request, it should be fine.
The DIM calls are in highly parallel worker threads, where/if the thread sleep, a new worker thread is invoked.

But for sure I agree that vq notification moderation should happen quickly enough under < 100usec or so. May be now that we bring the aq resources, we can start migrating on it.

Quoted 35 lines
Quoted 35 lines
Quoted 3 lines

The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 26 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not v1->give the
absolute value directly. @Michael

   device-types/net/description.tex | 26 ++++++++++++++++++++------
   1 file changed, 20 insertions(+), 6 deletions(-)
device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver
can - send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
Quoted 3 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

Quoted 152 lines
Quoted 3 lines

I tried to extend it to the command VIRTIO_NET_F_VQ_NOTF_COAL, is it too late?

For existing command it is not.
But there may be a device which may not support coalesced command.
Better to have the feature bit for this coalesce command, instead of random NOSUPP failure.

Quoted 144 lines

Thanks!

Quoted 59 lines
Quoted 59 lines
   \begin{lstlisting}
   struct virtio_net_ctrl_coal {
device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

       struct virtio_net_ctrl_coal coal;

};

device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
   #define VIRTIO_NET_CTRL_NOTF_COAL 6
    #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
    #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
    #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
    #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
   \end{lstlisting}
 
   Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is
read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:

device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries}
+ enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is \field{vq_index}.
\end{enumerate}

The device may generate notifications more or less frequently than
specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of an
enabled transmit or receive virtqueue.

Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +3 -0
+The driver MUST set \field{num_entries} to a non-zero value and MUST
+NOT set \field{num_entries} to a value greater than the number of
+enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL

feature

Quoted 5 lines
Quoted 5 lines

when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 7 lines
Quoted 7 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

Quoted 8 lines
Quoted 8 lines

The driver MUST ignore the values of coalescing parameters received
from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to the

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 1 line
Quoted 1 line

command with VIRTIO_NET_ERR if it was not able to change the

parameters.

Quoted 1 line
Quoted 1 line

+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -

VIRTIO_NET_ERR if the

Quoted 4 lines
Quoted 4 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with

VIRTIO_NET_ERR if the

Quoted 5 lines
Quoted 5 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device
MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

command, or,

Quoted 25 lines
Quoted 12 lines

if the driver did not set any TX coalescing parameters, to 0. --
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
Quoted 31 lines

From: Heng Qi <[email protected]>
Sent: Monday, January 22, 2024 8:27 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 2 lines

Such driver limitation should be removed in the driver. It does not qualify

Quoted 7 lines
Quoted 6 lines

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

for several micro to milliseconds.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

Inside the virtnet_rx_dim_work() there should be rtnl lock call. A virtio_device level lock to be used for cvq. :)

Quoted 5 lines

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
[email protected]/

In the listed above driver patch the motivation "to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result."

Is not right.
Because guest is still busy waiting.
Without batching, due to rtnl lock every VQ command is serialized as one outstanding command at a time in virtnet_rx_dim_work(). Due to this device is unable to take benefit of DMA batching at large scale.

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 3 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 2 lines

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a concern.

Quoted 4 lines
Quoted 4 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 5 lines
Quoted 2 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

coalescing on the commands of a given cvq.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side,

which can ignore them when they do not bother about it.

Driver notifications need to be processed by the DPU, which interferes with the CPU on the DPU.

I was asking, if there is anyway to disable for your DPU to ignore these notifications while previous one is pending? From your above description, it seems there isn’t.

Quoted 7 lines
Quoted 1 line

Backend DMAs should be reduced by avoiding the LIFO pattern followed by

the splitq driver.

Quoted 1 line

Placing the descriptors contiguously like packedq reduces amount of DMA

naturally.

splitq is widely used, migrating to packedq is not that easy, especially when there are many components and hardware involved.

I am not suggesting to packed_VQ.
I am suggesting fixing the driver to not do LIFO on descriptors for splitq.
In other words, using contiguous set of descriptors on splitq will improve the splitq for DMA.
This will allow using splitq more efficiently for dma as short-term solution for DMA until more efficient queues are defined.

Quoted 9 lines
Quoted 3 lines

The second predicable DMA to avoid is having 8Bytes of data inline in the

descriptor, instead of 16B indirection and extra dma.

Looking forward to working inline!
But I think this does not conflict with batch work, and combining the two will be more beneficial.

It does not conflict. However, batching for large number of queues may not use the inline as the data bytes may not fit in the inline.

Quoted 30 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

For multiple DMAs, we need to way to send 8 bytes of data without 16

bytes of indirection via a descriptor.

Quoted 2 lines

This is what we discussed a while back to do in txq and Stefan suggested to

generalize for more queues, which is also a good idea.

Yes, this sounds good.

This the next item to focus as soon as flow filters are stable/merged.

Greate!

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
Quoted 5 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

Quoted 8 lines
Quoted 7 lines

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

The batching done may end up modifying the given VQ's parameters

multiple times.

In practice, we do not try to accumulate multiple parameter modifications for a specific vqn.

Quoted 8 lines
Quoted 1 line

I meant to accumulate parameters of multiple VQs to batch in single

command.

Yes, this is what is done now, practical example:
https://lore.kernel.org/all/1705410693-118895-3-git-send-email-
[email protected]/
In our scenario, only benefits are seen, no negative effects.

The right way to test is without the rtnl lock and enqueue the command to cvq without being blocked.

The new optional batch command with new feature bit may be useful for devices to which CVQ kicks are interruptions.

Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

Quoted 239 lines

Totally looking forward to it.

Thanks,
Heng

Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
Quoted 5 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

YES. Makes sense for our DPUs.

Quoted 1 line
Quoted 1 line

Maintainers may be concerned about whether the batch command

method

Quoted 1 line
Quoted 1 line

can optimize the above problems: accumulate multiple request

commands

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only

means that moderation parameters are just delayed by the driver. Why is it delayed by the driver? It is not delayed by the driver, the kick still happens for every command.

Since the kick is delayed, the driver lost the amount of time and device

reaction time is driven by this wait time.

Quoted 8 lines

The guest driver is not aware how much a shared DPU is busy. So if it waits too much, the VQs moderation is slow.

Quoted 3 lines

In theory and practice, it will not affect DIM performance, but it will significantly reduce CPU consumption caused by waiting.

Once the guest driver sleeps after enqueuing the request, it should be fine. The DIM calls are in highly parallel worker threads, where/if the thread

sleep, a new worker thread is invoked.

Quoted 2 lines

But for sure I agree that vq notification moderation should happen quickly

enough under < 100usec or so.

Quoted 3 lines

May be now that we bring the aq resources, we can start migrating on it.

Quoted 1 line
Quoted 1 line
Quoted 1 line

The batch command method is used by us to optimize the CPU

overhead

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 2 lines

of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 20 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not v1->give the
absolute value directly. @Michael

   device-types/net/description.tex | 26 ++++++++++++++++++++------
   1 file changed, 20 insertions(+), 6 deletions(-)
device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the
driver can - send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

Quoted 6 lines
Quoted 3 lines

I tried to extend it to the command VIRTIO_NET_F_VQ_NOTF_COAL, is it too late?

For existing command it is not.
But there may be a device which may not support coalesced command. Better to have the feature bit for this coalesce command, instead of

random NOSUPP failure.

Quoted 23 lines
Quoted 23 lines

Thanks!

Quoted 19 lines
Quoted 19 lines
   \begin{lstlisting}
   struct virtio_net_ctrl_coal {
device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

       struct virtio_net_ctrl_coal coal;

};

device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
   #define VIRTIO_NET_CTRL_NOTF_COAL 6
    #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
    #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
    #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
    #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
   \end{lstlisting}
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Coalescing parameters:

device-types/net/description.tex +0 -0
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the

structure

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is
read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following

commands:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

transmit/receive virtqueue whose index is \field{vq_index}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries}
+ enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is

\field{vq_index}.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

\end{enumerate}

The device may generate notifications more or less frequently
than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL

class.

Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of
an enabled transmit or receive virtqueue.

Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +3 -0
+The driver MUST set \field{num_entries} to a non-zero value and
+MUST NOT set \field{num_entries} to a value greater than the
+number of enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL

feature

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 10 lines
Quoted 10 lines

and

Quoted 9 lines
Quoted 9 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

The driver MUST ignore the values of coalescing parameters
received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to

VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

Quoted 57 lines
Quoted 57 lines

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to the

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 1 line
Quoted 1 line

command with VIRTIO_NET_ERR if it was not able to change the

parameters.

Quoted 1 line
Quoted 1 line

+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -

VIRTIO_NET_ERR if the

Quoted 4 lines
Quoted 4 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with

VIRTIO_NET_ERR if the

Quoted 5 lines
Quoted 5 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device
MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

command, or,

Quoted 27 lines
Quoted 12 lines

if the driver did not set any TX coalescing parameters, to 0. --
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

--------------------------------------------------------------------
- To unsubscribe, e-mail:
[email protected]
For additional commands, e-mail:
[email protected]

#6 lore raw
Quoted 9 lines · Parav Pandit
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level. Yes some drivers will take a command and just queue it for execution later, but this means that errors can not be propagated back at all. Imagine device with mac 0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode

If command 1 fails but 2 proceeds then packets with MAC 0xabc will be dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver or device level.

So, here's my question: what exactly is the guest behaviour that is driving this work? Is it with a linux guest? which commands does userspace issue that we need to send multiple vq coalescing commands? If all you want is to send same config to all VQs then why not just use VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#7 lore raw
Quoted 7 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, January 22, 2024 1:06 PM

Quoted 3 lines · Parav Pandit
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level.

Not for requests which are initiated by the kernel stack (non user initiated).

Quoted 8 lines

Yes some drivers will take a command
and just queue it for execution later, but this means that errors can not be propagated back at all. Imagine device with mac
0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode

User initiated commands error can be propagated when the command completes. Enqueuing command is at the different bottom level in the driver.

Quoted 6 lines

If command 1 fails but 2 proceeds then packets with MAC 0xabc will be dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver or device level.

There is no suggestion to batch arbitrary commands from the driver side. The suggestion is to batch VQs notification coalescing from the driver side.

Quoted 2 lines

So, here's my question: what exactly is the guest behaviour that is driving this work? Is it with a linux guest?

At least looks to me yes based on the partial patches which are taking rtnl lock on netdim's worker callbacks.

Quoted 2 lines

which commands does userspace issue that we
need to send multiple vq coalescing commands?

None.

Quoted 4 lines

If all you want is to send
same config to all VQs then why not just use
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Only kernel stack initiated VQ notification coalescing changes.
Since every VQ has different values, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET is not sufficient.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw
Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, January 22, 2024 1:06 PM

Quoted 4 lines · Parav Pandit
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level.

Not for requests which are initiated by the kernel stack (non user initiated).

So how is this different? Is it basically just because tweaking coalescing in unexpected ways is considered mostly harmless?

Quoted 26 lines
Quoted 8 lines

Yes some drivers will take a command
and just queue it for execution later, but this means that errors can not be propagated back at all. Imagine device with mac
0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode

User initiated commands error can be propagated when the command completes. Enqueuing command is at the different bottom level in the driver.

Quoted 6 lines

If command 1 fails but 2 proceeds then packets with MAC 0xabc will be dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver or device level.

There is no suggestion to batch arbitrary commands from the driver side. The suggestion is to batch VQs notification coalescing from the driver side.

Quoted 2 lines

So, here's my question: what exactly is the guest behaviour that is driving this work? Is it with a linux guest?

At least looks to me yes based on the partial patches which are taking rtnl lock on netdim's worker callbacks.

Quoted 2 lines

which commands does userspace issue that we
need to send multiple vq coalescing commands?

Quoted 8 lines

None.

Quoted 4 lines

If all you want is to send
same config to all VQs then why not just use
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Only kernel stack initiated VQ notification coalescing changes.
Since every VQ has different values, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET is not sufficient.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw
Quoted 18 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, January 23, 2024 12:45 PM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, January 22, 2024 1:06 PM

Quoted 7 lines · Parav Pandit
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start

with.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level.

Not for requests which are initiated by the kernel stack (non user initiated).

So how is this different? Is it basically just because tweaking coalescing in unexpected ways is considered mostly harmless?

It is different in two ways.

1. requests are unrelated to each other (unlike the promisc/mac example). For example, changing vq1.params have no effect on vq2.params.

2. Therefore they don't need to be blocked.

3. There is no user expecting a success/failure on this command.
If it fails, possibly the queues can be marked in error for recovery/re-enablement.

Quoted 30 lines
Quoted 9 lines
Quoted 8 lines

Yes some drivers will take a command and just queue it for execution later, but this means that errors can not be propagated back at all. Imagine device with mac
0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode

User initiated commands error can be propagated when the command

completes.

Quoted 8 lines

Enqueuing command is at the different bottom level in the driver.

Quoted 6 lines

If command 1 fails but 2 proceeds then packets with MAC 0xabc will be dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver or device level.

Quoted 6 lines

There is no suggestion to batch arbitrary commands from the driver side. The suggestion is to batch VQs notification coalescing from the driver side.

Quoted 2 lines

So, here's my question: what exactly is the guest behaviour that is driving this work? Is it with a linux guest?

At least looks to me yes based on the partial patches which are taking rtnl

lock on netdim's worker callbacks.

Quoted 5 lines
Quoted 3 lines

which commands does userspace issue that we need to send multiple vq coalescing commands?

None.

Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

If all you want is to send
same config to all VQs then why not just use
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Only kernel stack initiated VQ notification coalescing changes. Since every VQ has different values, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET

is not sufficient.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#10 lore raw
In reply to Michael S. Tsirkin
Quoted 11 lines · Michael S. Tsirkin
Quoted 10 lines · Parav Pandit
Quoted 10 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, January 22, 2024 1:06 PM

Quoted 6 lines · Parav Pandit
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

I feel the issue is at the conceptual level.

Not for requests which are initiated by the kernel stack (non user initiated).

So how is this different? Is it basically just because tweaking coalescing in unexpected ways is considered mostly harmless?

DIM sends configurations frequently, which is try best.

Quoted 27 lines
Quoted 27 lines
Quoted 9 lines

Yes some drivers will take a command
and just queue it for execution later, but this means that errors can not be propagated back at all. Imagine device with mac
0x123 in promisc mode. Now commands:

1- program MAC 0xabcdef
2- disable promisc mode

User initiated commands error can be propagated when the command completes. Enqueuing command is at the different bottom level in the driver.

Quoted 6 lines

If command 1 fails but 2 proceeds then packets with MAC 0xabc will be dropped.

Any attempts to batch arbitrary commands will have this issue - be it at driver or device level.

There is no suggestion to batch arbitrary commands from the driver side. The suggestion is to batch VQs notification coalescing from the driver side.

Quoted 2 lines

So, here's my question: what exactly is the guest behaviour that is driving this work? Is it with a linux guest?

At least looks to me yes based on the partial patches which are taking rtnl lock on netdim's worker callbacks.

Quoted 2 lines

which commands does userspace issue that we
need to send multiple vq coalescing commands?

Quoted 8 lines
Quoted 8 lines

None.

Quoted 4 lines

If all you want is to send
same config to all VQs then why not just use
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET as opposed to
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET ?

Only kernel stack initiated VQ notification coalescing changes.
Since every VQ has different values, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET is not sufficient.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#11 lore raw
In reply to Parav Pandit
Quoted 45 lines · Parav Pandit
Quoted 28 lines

From: Heng Qi <[email protected]>
Sent: Monday, January 22, 2024 8:27 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 2 lines

Such driver limitation should be removed in the driver. It does not qualify

Quoted 7 lines
Quoted 6 lines

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

Quoted 9 lines

for several micro to milliseconds.

Quoted 4 lines
Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

Inside the virtnet_rx_dim_work() there should be rtnl lock call. A virtio_device level lock to be used for cvq. :)

Quoted 5 lines

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
[email protected]/

Quoted 6 lines

In the listed above driver patch the motivation "to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result."

Is not right.
Because guest is still busy waiting.

There is no busy wait for guests, see get_cvq_work().

Quoted 2 lines

Without batching, due to rtnl lock every VQ command is serialized as one outstanding command at a time in virtnet_rx_dim_work(). Due to this device is unable to take benefit of DMA batching at large scale.

Adding dim commands is now asynchronous, and the device will receive batches of commands.

Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 25 lines
Quoted 3 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 2 lines

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a concern.

Quoted 4 lines
Quoted 4 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 5 lines
Quoted 2 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Driver notifications are PCI writes so it should not hamper device side,

which can ignore them when they do not bother about it.

Driver notifications need to be processed by the DPU, which interferes with the CPU on the DPU.

I was asking, if there is anyway to disable for your DPU to ignore these notifications while previous one is pending?
From your above description, it seems there isn’t.

While the device is processing the request, additional kicks are ignored.

Thanks.

Quoted 61 lines
Quoted 8 lines
Quoted 2 lines

Backend DMAs should be reduced by avoiding the LIFO pattern followed by

the splitq driver.

Quoted 1 line

Placing the descriptors contiguously like packedq reduces amount of DMA

naturally.

splitq is widely used, migrating to packedq is not that easy, especially when there are many components and hardware involved.

I am not suggesting to packed_VQ.
I am suggesting fixing the driver to not do LIFO on descriptors for splitq.
In other words, using contiguous set of descriptors on splitq will improve the splitq for DMA.
This will allow using splitq more efficiently for dma as short-term solution for DMA until more efficient queues are defined.

Quoted 7 lines
Quoted 1 line

The second predicable DMA to avoid is having 8Bytes of data inline in the

descriptor, instead of 16B indirection and extra dma.

Looking forward to working inline!
But I think this does not conflict with batch work, and combining the two will be more beneficial.

It does not conflict. However, batching for large number of queues may not use the inline as the data bytes may not fit in the inline.

Quoted 27 lines
Quoted 9 lines
Quoted 8 lines
Quoted 1 line

For multiple DMAs, we need to way to send 8 bytes of data without 16

bytes of indirection via a descriptor.

Quoted 2 lines

This is what we discussed a while back to do in txq and Stefan suggested to

generalize for more queues, which is also a good idea.

Yes, this sounds good.

This the next item to focus as soon as flow filters are stable/merged.

Greate!

Quoted 16 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines

In addition,
each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

Quoted 8 lines
Quoted 7 lines

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested

The batching done may end up modifying the given VQ's parameters

multiple times.

In practice, we do not try to accumulate multiple parameter modifications for a specific vqn.

Quoted 8 lines
Quoted 1 line

I meant to accumulate parameters of multiple VQs to batch in single

command.

Yes, this is what is done now, practical example:
https://lore.kernel.org/all/1705410693-118895-3-git-send-email-
[email protected]/
In our scenario, only benefits are seen, no negative effects.

The right way to test is without the rtnl lock and enqueue the command to cvq without being blocked.

The new optional batch command with new feature bit may be useful for devices to which CVQ kicks are interruptions.

Quoted 245 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

The right test on Linux to do without rtnl lock which is anyway ugly and

wrong semantic to use blocking the whole netdev stack.

Quoted 1 line

(in case if you used that).

Do you have any good directions and attempts to remove rtnl_lock?

I think per device lock instead of rtnl is first step that we can start with.

Wil check internally who if someone already started working on it.

Quoted 236 lines

Totally looking forward to it.

Thanks,
Heng

Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 4 lines

8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi- device and multi-queue DIM.

This is good.

YES. Makes sense for our DPUs.

Quoted 1 line
Quoted 1 line

Maintainers may be concerned about whether the batch command

method

Quoted 1 line
Quoted 1 line

can optimize the above problems: accumulate multiple request

commands

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

to kick the device once, and obtain the processing results of the corresponding commands asynchronously.

This is unlikely to improve, rather it will have negative impact as it only

means that moderation parameters are just delayed by the driver. Why is it delayed by the driver? It is not delayed by the driver, the kick still happens for every command.

Since the kick is delayed, the driver lost the amount of time and device

reaction time is driven by this wait time.

Quoted 8 lines

The guest driver is not aware how much a shared DPU is busy. So if it waits too much, the VQs moderation is slow.

Quoted 3 lines

In theory and practice, it will not affect DIM performance, but it will significantly reduce CPU consumption caused by waiting.

Once the guest driver sleeps after enqueuing the request, it should be fine. The DIM calls are in highly parallel worker threads, where/if the thread

sleep, a new worker thread is invoked.

Quoted 1 line

But for sure I agree that vq notification moderation should happen quickly

enough under < 100usec or so.

Quoted 3 lines

May be now that we bring the aq resources, we can start migrating on it.

Quoted 1 line
Quoted 1 line
Quoted 1 line

The batch command method is used by us to optimize the CPU

overhead

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 2 lines

of the DIM worker caused by the guest being busy waiting for the command response result.

In that case fixing the guest driver which is not yet written is the right fix.

Quoted 20 lines

This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not v1->give the
absolute value directly. @Michael

device-types/net/description.tex | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-)

device- renamed from device-types/net/description.tex +0 -0

types/net/description.tex index aff5e08..b3766c4 100644

device-types/net/description.tex +0 -0
@@ -1667,8 +1667,8 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi for notification coalescing.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the
driver can - send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET -for virtqueue notification coalescing.

device-types/net/description.tex +2 -0
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.

A new feature bit is needed for this extra functionality.

Quoted 6 lines
Quoted 3 lines

I tried to extend it to the command VIRTIO_NET_F_VQ_NOTF_COAL, is it too late?

For existing command it is not.
But there may be a device which may not support coalesced command. Better to have the feature bit for this coalesce command, instead of

random NOSUPP failure.

Quoted 22 lines
Quoted 22 lines

Thanks!

Quoted 19 lines
Quoted 19 lines
    \begin{lstlisting}
    struct virtio_net_ctrl_coal {
device-types/net/description.tex +0 -0
@@ -1682,11 +1682,17 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

        struct virtio_net_ctrl_coal coal;
    };
device-types/net/description.tex +5 -0
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[]; };
+
    #define VIRTIO_NET_CTRL_NOTF_COAL 6
     #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
     #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
     #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
     #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
    \end{lstlisting}
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Coalescing parameters:

device-types/net/description.tex +0 -0
@@ -1706,6 +1712,7 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the

structure

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

virtio_net_ctrl_coal_vq is write-only for the driver.
\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
\field{vq_index} and \field{reserved} are write-only
for the driver, and the structure virtio_net_ctrl_coal is
read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
\end{itemize}

The class VIRTIO_NET_CTRL_NOTF_COAL has the following

commands:

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
device-types/net/description.tex +0 -0
@@ -1716,6 +1723,9 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi
for an enabled
transmit/receive virtqueue whose index is \field{vq_index}.
\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure
virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
for an enabled

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

transmit/receive virtqueue whose index is \field{vq_index}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+ for \field{num_entries}
+ enabled transmit/receive
virtqueues. The corresponding index value
+ of each configured virtqueue is

\field{vq_index}.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

\end{enumerate}

The device may generate notifications more or less frequently
than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL

class.

Quoted 13 lines
Quoted 13 lines
Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +0 -0
@@ -1782,9 +1792,13 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The driver MUST set \field{vq_index} to the virtqueue index of
an enabled transmit or receive virtqueue.

Quoted 6 lines
Quoted 6 lines
device-types/net/description.tex +3 -0
+The driver MUST set \field{num_entries} to a non-zero value and
+MUST NOT set \field{num_entries} to a value greater than the
+number of enabled

transmit and receive virtqueues.
+
The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL

feature

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 10 lines
Quoted 10 lines

and

Quoted 9 lines
Quoted 9 lines

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature
+when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.

The driver MUST ignore the values of coalescing parameters
received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

device responds with VIRTIO_NET_ERR.

device-types/net/description.tex +0 -0
@@ -1794,10 +1808,10 @@ \subsubsection{Control

Virtqueue}\label{sec:Device Types / Network Device / Devi

The device SHOULD respond to

VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

Quoted 57 lines
Quoted 57 lines

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to the

VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

Quoted 1 line
Quoted 1 line

command with VIRTIO_NET_ERR if it was not able to change the

parameters.

Quoted 1 line
Quoted 1 line

+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with

VIRTIO_NET_ERR if it

Quoted 3 lines
Quoted 3 lines

was not able to change the parameters.

-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET

and

Quoted 1 line
Quoted 1 line

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with -

VIRTIO_NET_ERR if the

Quoted 4 lines
Quoted 4 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

device-types/net/description.tex +2 -0
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
+VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and

VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with

VIRTIO_NET_ERR if the

Quoted 5 lines
Quoted 5 lines

designated virtqueue is not an enabled transmit or receive virtqueue.

Upon disabling and re-enabling a transmit virtqueue, the device
MUST set the coalescing parameters of the virtqueue to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET

command, or,

Quoted 27 lines
Quoted 12 lines

if the driver did not set any TX coalescing parameters, to 0. --
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-
open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

--------------------------------------------------------------------
- To unsubscribe, e-mail:
[email protected]
For additional commands, e-mail:
[email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#12 lore raw
Quoted 53 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 24, 2024 6:31 PM

Quoted 47 lines · Parav Pandit
Quoted 30 lines

From: Heng Qi <[email protected]>
Sent: Monday, January 22, 2024 8:27 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 2 lines

Such driver limitation should be removed in the driver. It does not qualify

Quoted 8 lines
Quoted 6 lines

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

for several micro to milliseconds.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

Inside the virtnet_rx_dim_work() there should be rtnl lock call. A virtio_device level lock to be used for cvq. :)

Quoted 5 lines

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
[email protected]/

Quoted 9 lines
Quoted 6 lines

In the listed above driver patch the motivation "to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result."

Is not right.
Because guest is still busy waiting.

There is no busy wait for guests, see get_cvq_work().

Ok. not always busy waiting, sometimes it does. virtnet_cvq_response() should not have flag..

Who ever gets the OS global rtnl lock is calling virtnet_cvq_response() and checking and releasing. Shouldn’t be done this way with try lock etc.
Rtnl lock is not supposed to protect low level driver some ctrlvq response flag.

Quoted 7 lines
Quoted 1 line

Without batching, due to rtnl lock every VQ command is serialized as one

outstanding command at a time in virtnet_rx_dim_work().

Quoted 1 line

Due to this device is unable to take benefit of DMA batching at large scale.

Adding dim commands is now asynchronous, and the device will receive batches of commands.

I am comparing the code where there is no rtnl lock with the code you posted. The reference code of patch [1] needs to begin without OS global rtnl lock as base line.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 30 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 2 lines

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a

concern.

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 5 lines
Quoted 2 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

Quoted 7 lines
Quoted 5 lines

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side,

which can ignore them when they do not bother about it.

Quoted 9 lines
Quoted 4 lines
Quoted 3 lines

Driver notifications need to be processed by the DPU, which interferes with the CPU on the DPU.

I was asking, if there is anyway to disable for your DPU to ignore these

notifications while previous one is pending?

Quoted 1 line

From your above description, it seems there isn’t.

While the device is processing the request, additional kicks are ignored.

Ok. that is good. In that case the kicks are not interfering, right?

author
#13 lore raw
Quoted 55 lines · Parav Pandit
Quoted 55 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 24, 2024 6:31 PM

Quoted 50 lines · Parav Pandit
Quoted 29 lines

From: Heng Qi <[email protected]>
Sent: Monday, January 22, 2024 8:27 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 2 lines

Such driver limitation should be removed in the driver. It does not qualify

Quoted 8 lines
Quoted 6 lines

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

for several micro to milliseconds.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

Inside the virtnet_rx_dim_work() there should be rtnl lock call. A virtio_device level lock to be used for cvq. :)

Quoted 5 lines

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
[email protected]/

In the listed above driver patch the motivation "to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result."

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Is not right.
Because guest is still busy waiting.

There is no busy wait for guests, see get_cvq_work().

Ok. not always busy waiting, sometimes it does.

Busy waiting will only occur when the user command or dim command cannot find the available buffer for cvq.

The user command is still in polling mode for now, I have not tried to optimize this. Now it's about improving dim performance.

Quoted 1 line

virtnet_cvq_response() should not have flag..

The flag is mainly used to identify whether it is a user command. If so, the previous polling mode will still be maintained.

Quoted 4 lines

Who ever gets the OS global rtnl lock is calling virtnet_cvq_response() and checking and releasing. Shouldn’t be done this way with try lock etc.
Rtnl lock is not supposed to protect low level driver some ctrlvq response flag.

To summarize, we now want to make some improvements to cvq:
1. Reasonable timeout in busy waiting mode or interrupt-based etc.
2. Batch processing (the core problem is how to get results from user commands synchronously)
3. Remove rtnl_lock’s protection for ctrlq.

Yes, these improve the processing efficiency of cvq and avoid the problem of OS hang. I think 1, 3 need short term long term work and we need to try.
I don’t see any demand for batching user commands.

So for now, let's move forward with optimizing dim.

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Without batching, due to rtnl lock every VQ command is serialized as one

outstanding command at a time in virtnet_rx_dim_work().

Quoted 1 line

Due to this device is unable to take benefit of DMA batching at large scale.

Adding dim commands is now asynchronous, and the device will receive batches of commands.

I am comparing the code where there is no rtnl lock with the code you posted.

Okay, I can post a patch later based on the code without rtnl_lock.

Quoted 35 lines

The reference code of patch [1] needs to begin without OS global rtnl lock as base line.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 31 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

This will enable driver to enqueue multiple cvq commands without waiting

for previous one.

Quoted 2 lines

This will also enable device to find natural coalescing done on multiple

commands.

When batch user commands occur, ensuring synchronization is a

concern.

Quoted 22 lines
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines

The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times; 3. The device completes processing and replies

with a response.

Quoted 5 lines
Quoted 2 lines

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU.

When there is large devices and multiple driver notifications by a cpu that is N times faster than the device side cpu, the device may find natural

Quoted 7 lines
Quoted 5 lines

coalescing on the commands of a given cvq.

First we have to solve the ctrlq batch adding user (ethtool) command. Even if processed in a batch way on device side, the number of kicks and the number of backend DMAs has not been reduced.

Driver notifications are PCI writes so it should not hamper device side,

which can ignore them when they do not bother about it.

Driver notifications need to be processed by the DPU, which interferes with the CPU on the DPU.

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

I was asking, if there is anyway to disable for your DPU to ignore these

notifications while previous one is pending?

Quoted 1 line

From your above description, it seems there isn’t.

While the device is processing the request, additional kicks are ignored.

Ok. that is good. In that case the kicks are not interfering, right?

YES.

When a VM sends many DIM requests, the requests are serialized and processed one at a time, and the number of kicks increases.

Even when we add commands asynchronously (such as this patch https://lore.kernel.org/all/[email protected]/),
the device needs to process each command present in the ctrlq individually.
Each command requires the device CPU to wait for the DMA operation to complete before replying.
Although kicks are reduced, the device's DMA operations for batched commands are serialized, which affects efficiency.

In addition, I would like to extend the VIRTIO_NET_F_VQ_NOTF_COAL feature to support the new VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET command. I think this is ok.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#14 lore raw
Quoted 77 lines · Heng Qi
Quoted 55 lines · Parav Pandit
Quoted 55 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 24, 2024 6:31 PM

Quoted 50 lines · Parav Pandit
Quoted 29 lines

From: Heng Qi <[email protected]>
Sent: Monday, January 22, 2024 8:27 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, January 17, 2024 10:22 AM

Quoted 9 lines · Parav Pandit
Quoted 7 lines

From: [email protected] <[email protected] open.org> On Behalf Of Heng Qi
Sent: Monday, January 15, 2024 6:36 PM

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return.

It does not need to wait. This is some driver limitation that does not use

the queue as "queue".

Quoted 2 lines

Such driver limitation should be removed in the driver. It does not qualify

Quoted 8 lines
Quoted 6 lines

as limitation.

Yes, we don't have to wait.

But in general, for user commands, it is necessary to obtain the final results synchronously.

Yes. Use initiated command can enqueue the request to cvq. Go to sleep

for several micro to milliseconds.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

The user command cannot return before the final result is obtained. And wait is not the problem this patch solves.

By not holding the rtnl lock, rest of the context that needs to enqueue the

request can progress such as that of netdim.

Would like to see the using of rtnl lock changed.

Inside the virtnet_rx_dim_work() there should be rtnl lock call. A virtio_device level lock to be used for cvq. :)

Quoted 5 lines

In addition, I have made batching and asynchronousization of the netdim command, you can refer to this patch:
https://lore.kernel.org/all/1705410693-118895-4-git-send-email-
[email protected]/

In the listed above driver patch the motivation "to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result."

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Is not right.
Because guest is still busy waiting.

There is no busy wait for guests, see get_cvq_work().

Ok. not always busy waiting, sometimes it does.

Busy waiting will only occur when the user command or dim command cannot find the available buffer for cvq.

The user command is still in polling mode for now, I have not tried to optimize this. Now it's about improving dim performance.

Quoted 1 line

virtnet_cvq_response() should not have flag..

The flag is mainly used to identify whether it is a user command. If so, the previous polling mode will still be maintained.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

Who ever gets the OS global rtnl lock is calling virtnet_cvq_response() and checking and releasing. Shouldn’t be done this way with try lock etc.
Rtnl lock is not supposed to protect low level driver some ctrlvq response flag.

To summarize, we now want to make some improvements to cvq:
1. Reasonable timeout in busy waiting mode or interrupt-based etc.

Let's use interrupt to avoid tricky code.

Quoted 3 lines

2. Batch processing (the core problem is how to get results from user commands synchronously)
3. Remove rtnl_lock’s protection for ctrlq.

Virtio-comment is not the right place to discuss these. Let's move it to netdev.

Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#15 lore raw
In reply to Heng Qi
Quoted 20 lines · Heng Qi

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Quoted 6 lines

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

Hmm. Could the issue be actually that
1. you are using split vqs so processing is complex
2. Linux driver uses indirect to pass coalescing commands around so even more complex
?

Is packed an option for you at all? Guest should also be changed to avoid indirect with ctrl vq. As a quick test, disable indirect maybe that is fast enough?

Quoted 91 lines

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

 device-types/net/description.tex | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +20 -6
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 for notification coalescing.
 
 If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
 \begin{lstlisting}
 struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     struct virtio_net_ctrl_coal coal;
 };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
 \end{lstlisting}
 
 Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
       for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
 \end{itemize}
 
 The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
 \end{enumerate}
 
 The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
 The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
 The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
 Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
 to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 1 line
1.8.3.1

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#16 lore raw
Quoted 28 lines · Michael S. Tsirkin
Quoted 20 lines · Heng Qi

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Quoted 6 lines

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

Quoted 2 lines

Hmm. Could the issue be actually that
1. you are using split vqs so processing is complex

This is part of the problem.

Quoted 3 lines

2. Linux driver uses indirect to pass coalescing commands around so even more complex
?

Theoretically, when ctrlq cmd is sent,
the number of indirect DMA times will be 3 times less than
the number of chain DMA times. After the actual test, the indirect method of ctrlq is about 10% more efficient than the chain method.

Merging requests can both reduce the number of kicks and greatly reduce the number of DMAs. For example, after 8 queues requests are merged, only 1 kick + 8 DMAs are required. Without merging, 8 kicks + 64 DMAs are required.

Quoted 2 lines

Is packed an option for you at all?

On our existing hardware, migrating from splitq to packedq would take a long time.

Thanks.

Quoted 96 lines

Guest should also be changed to
avoid indirect with ctrl vq. As a quick test, disable indirect maybe that is fast enough?

Quoted 87 lines

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

  device-types/net/description.tex | 26 ++++++++++++++++++++------
  1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +20 -6
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  for notification coalescing.
 
  If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
  \begin{lstlisting}
  struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      struct virtio_net_ctrl_coal coal;
  };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
  \end{lstlisting}
 
  Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
  \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
        for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
  \end{itemize}
 
  The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
  \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
  \end{enumerate}
 
  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
  The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
  The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
Quoted 8 lines
Quoted 4 lines
device-types/net/description.tex +0 -0
  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
  to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 1 line
1.8.3.1

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#17 lore raw
In reply to Heng Qi
Quoted 103 lines · Heng Qi

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

 device-types/net/description.tex | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +17 -3
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 for notification coalescing.
 
 If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
 \begin{lstlisting}
 struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     struct virtio_net_ctrl_coal coal;
 };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
 \end{lstlisting}
 
 Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
       for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
 \end{itemize}
 
 The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
 \end{enumerate}
 
 The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
 The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
 The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1811

Quoted 5 lines
device-types/net/description.tex +1 -1
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

This one however is vague.
In fact the addition of VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET made VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET also vague we just missed this.

So, it is not clear whether if the device responds with VIRTIO_NET_ERR - it can change parameters for some VQs but not others, or does it have to either
change parameters for all VQs or none at all.

I think a better sematic is all or nothing.

So I suggest combining this statement with one of VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and adding something like
"in this case, coalescing parameters MUST remain unchanged, for all VQs".

This bug bothers me. Maybe we should fix it before release?

I also don't know why we repeat same text for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

Quoted 10 lines
device-types/net/description.tex +2 -2
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
 Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
 to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 1 line
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#18 lore raw
Quoted 129 lines · Michael S. Tsirkin
Quoted 103 lines · Heng Qi

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

 device-types/net/description.tex | 26 ++++++++++++++++++++------
 1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +17 -3
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 for notification coalescing.
 
 If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
 \begin{lstlisting}
 struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     struct virtio_net_ctrl_coal coal;
 };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
  #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
 \end{lstlisting}
 
 Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
 \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
       for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
 \end{itemize}
 
 The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
 \end{enumerate}
 
 The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
 The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
 The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1811

Quoted 5 lines
device-types/net/description.tex +1 -1
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

This one however is vague.
In fact the addition of VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET made VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET also vague we just missed this.

So, it is not clear whether if the device responds with VIRTIO_NET_ERR - it can change parameters for some VQs but not others, or does it have to either
change parameters for all VQs or none at all.

I think a better sematic is all or nothing.

So I suggest combining this statement with one of VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and adding something like
"in this case, coalescing parameters MUST remain unchanged, for all VQs".

Quoted 4 lines

This bug bothers me. Maybe we should fix it before release?

I also don't know why we repeat same text for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

I figured it out, TX/RX are SHOULD.
I will fix TX/RX and you can add the text to this one.

Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +2 -2
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
 Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
 to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 1 line
1.8.3.1

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

author
#19 lore raw
Quoted 129 lines · Michael S. Tsirkin
Quoted 128 lines · Michael S. Tsirkin
Quoted 103 lines · Heng Qi

Currently, when each time the driver attempts to update the coalescing parameters for a vq, it needs to kick the device and wait for the ctrlq response to return. The following path is observed: 1. Driver kicks the device; 2. After the device receives the kick, CPU scheduling occurs and DMA multiple buffers multiple times;
3. The device completes processing and replies with a response.

When large-queue devices issue multiple requests and kick the device frequently, this often interrupt the work of the device-side CPU. In addition, each vq request is processed separately, causing more delays for the CPU to wait for the DMA request to complete.

These interruptions and overhead will strain the CPU responsible for controlling the path of the DPU, especially in multi-device and large-queue scenarios.

To solve the above problems, we internally tried batch request, which merges requests from multiple queues and sends them at once. We conservatively tested 8 queue commands and sent them together. The DPU processing efficiency can be improved by 8 times, which greatly eases the DPU's support for multi-device and multi-queue DIM.

Maintainers may be concerned about whether the batch command method can optimize the above problems: accumulate multiple request commands to kick the device once, and obtain the processing results of the corresponding commands asynchronously. The batch command method is used by us to optimize the CPU overhead of the DIM worker caused by the guest being busy waiting for the command response result. This is a different focus than batch request to solve the problem.

---
v1->v2: Updated commit log. Due to sensitivity, sorry that can not give the
absolute value directly. @Michael

  device-types/net/description.tex | 26 ++++++++++++++++++++------
  1 file changed, 20 insertions(+), 6 deletions(-)
device-types/net/description.tex +17 -3
index aff5e08..b3766c4 100644
@@ -1667,8 +1667,8 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  for notification coalescing.
 
  If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver can
-send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
-for virtqueue notification coalescing.
+send commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET for virtqueue notification coalescing.
 
  \begin{lstlisting}
  struct virtio_net_ctrl_coal {
@@ -1682,11 +1682,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      struct virtio_net_ctrl_coal coal;
  };
 
+struct virtio_net_ctrl_mrg_coal_vq {
+        le16 num_entries; /* indicates number of valid entries */
+        struct virtio_net_ctrl_coal_vq entries[];
+};
+
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
   #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET 4
  \end{lstlisting}
 
  Coalescing parameters:
@@ -1706,6 +1712,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, the structure virtio_net_ctrl_coal_vq is write-only for the driver.
  \item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vq_index} and \field{reserved} are write-only
        for the driver, and the structure virtio_net_ctrl_coal is read-only for the driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET, the structure virtio_net_ctrl_mrg_coal_vq is write-only for the driver.
  \end{itemize}
 
  The class VIRTIO_NET_CTRL_NOTF_COAL has the following commands:
@@ -1716,6 +1723,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
  \item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: use the structure virtio_net_ctrl_coal_vq to get the \field{max_usecs} and \field{max_packets} parameters
                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET: use the structure virtio_net_ctrl_mrg_coal_vq to set the \field{max_usecs} and \field{max_packets} parameters
+                                         for \field{num_entries} enabled transmit/receive virtqueues. The corresponding index value
+                                         of each configured virtqueue is \field{vq_index}.
  \end{enumerate}
 
  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
@@ -1782,9 +1792,13 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  The driver MUST set \field{vq_index} to the virtqueue index of an enabled transmit or receive virtqueue.
 
+The driver MUST set \field{num_entries} to a non-zero value and MUST NOT set \field{num_entries} to
+a value greater than the number of enabled transmit and receive virtqueues.
+
  The driver MUST have negotiated the VIRTIO_NET_F_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
 
-The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
+The driver MUST have negotiated the VIRTIO_NET_F_VQ_NOTF_COAL feature when issuing commands
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET.
 
  The driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if the device responds with VIRTIO_NET_ERR.
 

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1811

Quoted 5 lines
device-types/net/description.tex +1 -1
@@ -1794,10 +1808,10 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  The device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
 
-The device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.

This one however is vague.
In fact the addition of VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET made VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET also vague we just missed this.

So, it is not clear whether if the device responds with VIRTIO_NET_ERR - it can change parameters for some VQs but not others, or does it have to either
change parameters for all VQs or none at all.

I think a better sematic is all or nothing.

So I suggest combining this statement with one of VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and adding something like
"in this case, coalescing parameters MUST remain unchanged, for all VQs".

Quoted 6 lines
Quoted 4 lines

This bug bothers me. Maybe we should fix it before release?

I also don't know why we repeat same text for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.

I figured it out, TX/RX are SHOULD.
I will fix TX/RX and you can add the text to this one.

Sorry Michael, it seems I missed this email yesterday.

I will add this.

Thanks,
Heng

Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
device-types/net/description.tex +2 -2
-The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with
-VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
+The device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_VQS_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET
+commands with VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
Quoted 4 lines
Quoted 4 lines
device-types/net/description.tex +0 -0
  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
  to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
Signature · 1 line
1.8.3.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 8 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/