Skip to contentSkip to conversation

virtio-net: support the virtqueue coalescing moderation

PATCH Superseded Heng Qi 3 participants 12 messages Feb 19, 2023 – Feb 21, 2023

v4 of 14 Superseded Latest is v14 →

lore
Share & tools
lore

Heng Qi

author
#0 lore raw

PATCH v4 1/1 virtio-net: support the virtqueue coalescing moderation

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

 device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +60 -4
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
 send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
 \begin{note}
 The behavior of the device in response to these commands is best-effort:
 the device may generate notifications more or less frequently than specified.
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le32 max_usecs;
 };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
 \end{lstlisting}
 
 Coalescing parameters:
 \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
 \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
 \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
 \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
 \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
 \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
 \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
 \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.
 
 \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
 A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.
 
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
#1 lore raw
Quoted 76 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

 device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
 send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
 \begin{note}
 The behavior of the device in response to these commands is best-effort:
 the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536

Quoted 9 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le32 max_usecs;
 };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 50 lines
device-types/net/description.tex +27 -3
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
 \end{lstlisting}
 
 Coalescing parameters:
 \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
 \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
 \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
 \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
 \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
 \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
 \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
 \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 6 lines
device-types/net/description.tex +2 -1
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 5 lines
device-types/net/description.tex +1 -0
 \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
 A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 4 lines
device-types/net/description.tex +1 -0
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Quoted 11 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
author
#2 lore raw
Quoted 90 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

 device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
 send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
 \begin{note}
 The behavior of the device in response to these commands is best-effort:
 the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536

Quoted 9 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le32 max_usecs;
 };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Quoted 3 lines

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Quoted 52 lines
Quoted 52 lines
device-types/net/description.tex +27 -3
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
 \end{lstlisting}
 
 Coalescing parameters:
 \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
 \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
 \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
 \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
 \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
 \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
 \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
 \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
Quoted 7 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 5 lines
device-types/net/description.tex +2 -1
 \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 3 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 4 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 10 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 6 lines
device-types/net/description.tex +1 -0
 \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
 A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 4 lines
device-types/net/description.tex +1 -0
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 5 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 5 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 2 lines
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 14 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 4 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values. "

Quoted 14 lines
Quoted 14 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
#3 lore raw
Quoted 91 lines · Heng Qi
Quoted 90 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

 device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
 send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
 \begin{note}
 The behavior of the device in response to these commands is best-effort:
 the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536

Quoted 9 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le32 max_usecs;
 };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Quoted 8 lines
Quoted 3 lines

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Quoted 11 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

Quoted 88 lines
Quoted 52 lines
Quoted 52 lines
device-types/net/description.tex +27 -3
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
 \end{lstlisting}
 
 Coalescing parameters:
 \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
 \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
 \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
 \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
 \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
 \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
 \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
 \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
Quoted 7 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 5 lines
device-types/net/description.tex +2 -1
 \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 3 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 4 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 10 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 6 lines
device-types/net/description.tex +1 -0
 \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
 A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Quoted 49 lines
Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 4 lines
device-types/net/description.tex +1 -0
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 5 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 5 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 2 lines
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 14 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 4 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Quoted 8 lines

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

ok more or less. is this in normative section? If not avoid MUST.

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 16 lines

"

Quoted 14 lines
Quoted 14 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
author
#4 lore raw
Quoted 94 lines · Michael S. Tsirkin
Quoted 93 lines · Heng Qi
Quoted 92 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

  device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
  1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
  send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
  \begin{note}
  The behavior of the device in response to these commands is best-effort:
  the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536

Quoted 9 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le32 max_usecs;
  };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 7 lines
Quoted 4 lines

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Ok, I'll modify this.

Quoted 8 lines
Quoted 8 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

Quoted 8 lines
Quoted 4 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :

"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands. \field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. \field{reserved} is a reserved field containing 0, which is ignored by a device and write-only for a driver.

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time, for a driver, coalescing parameters are read-only and \field{vqn} is write-only. "
Do you think this is a good description?

Quoted 84 lines
Quoted 77 lines
Quoted 51 lines
Quoted 51 lines
device-types/net/description.tex +27 -3
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
  \end{lstlisting}
 
  Coalescing parameters:
  \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
  \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
  \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
  \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
  \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
  \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
  \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
  \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
  The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 5 lines
device-types/net/description.tex +2 -1
  \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 2 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 5 lines
device-types/net/description.tex +1 -0
  \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
  A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 3 lines

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Ok.

Quoted 3 lines

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Ah, you mean the scope of MUST is included in SHOULD? If you think it's better to remove the *Note* above, I'm fine.

Quoted 44 lines
Quoted 44 lines
Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 7 lines
device-types/net/description.tex +1 -0
  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 4 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 3 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 12 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

Quoted 7 lines
Quoted 6 lines

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

ok more or less. is this in normative section? If not avoid MUST.

Yes. It's in normative section. Thanks for your comments!

Quoted 17 lines

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 12 lines

"

Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
  Upon reset, a device MUST initialize all coalescing parameters to 0.
 
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
device-types/net/description.tex +0 -0
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#5 lore raw
Quoted 122 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

  device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
  1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
  send control commands for dynamically changing the coalescing parameters.
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
  \begin{note}
  The behavior of the device in response to these commands is best-effort:
  the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535

Quoted 8 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le32 max_usecs;
  };
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 7 lines
Quoted 4 lines

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Ok, I'll modify this.

Quoted 8 lines
Quoted 8 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

Quoted 8 lines
Quoted 4 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :

Quoted 6 lines

"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands.

ugh just confusing. just use the struct names.

Quoted 4 lines

\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. \field{reserved} is a reserved field containing 0, which is ignored by a device and write-only for a driver.

how important is it that devices ignore it?
if yes it needs a normative statement.

Quoted 6 lines

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.

Quoted 8 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time,

at what time?

Quoted 2 lines

for a driver, coalescing parameters are read-only and \field{vqn} is write-only.

reserved also write-only.

Quoted 2 lines

"
Do you think this is a good description?

No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.

for command X fields A B are write-only for driver and fields C D are read-only for driver.

for command Y fields A B C and D are write-only for driver.

Quoted 80 lines
Quoted 76 lines
Quoted 69 lines
Quoted 43 lines
Quoted 43 lines
device-types/net/description.tex +27 -3
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
  \end{lstlisting}
  Coalescing parameters:
  \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
  \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
  \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
  \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
  \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
  \end{itemize}
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
  \end{enumerate}
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626

Quoted 7 lines
device-types/net/description.tex +2 -1
  \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
  The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 2 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 3 lines
device-types/net/description.tex +1 -0
  \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
  A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 3 lines

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 7 lines

Ok.

Quoted 3 lines

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Ah, you mean the scope of MUST is included in SHOULD?

No I mean just list commands explicitly. Do not make reader guess what does VIRTIO_NET_CTRL_NOTF_COAL mean when there is no command like this.

Quoted 50 lines

If you think it's better to remove the *Note* above, I'm fine.

Quoted 48 lines
Quoted 48 lines
Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 5 lines
device-types/net/description.tex +1 -0
  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 4 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 3 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 12 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

Quoted 3 lines
Quoted 1 line

ok more or less. is this in normative section? If not avoid MUST.

Yes. It's in normative section. Thanks for your comments!

then pls also add a non normative explanatory informal description.

Quoted 19 lines
Quoted 19 lines

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 14 lines

"

Quoted 12 lines
Quoted 12 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
  Upon reset, a device MUST initialize all coalescing parameters to 0.
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
author
#6 lore raw
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

   device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
   1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
   \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
       channel.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
   \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
   \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
   \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
   \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
   \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
   \end{description}
   \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
   send control commands for dynamically changing the coalescing parameters.
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
   \begin{note}
   The behavior of the device in response to these commands is best-effort:
   the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535

Quoted 8 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
       le32 max_usecs;
   };
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 7 lines
Quoted 4 lines

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Ok, I'll modify this.

Quoted 7 lines
Quoted 7 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

Quoted 7 lines
Quoted 4 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :

Quoted 7 lines
Quoted 6 lines

"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands.

ugh just confusing. just use the struct names.

Ok.

Quoted 7 lines
Quoted 5 lines

\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
 \field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.

how important is it that devices ignore it?
if yes it needs a normative statement.

Maybe "containing 0" is enough.

Quoted 27 lines
Quoted 6 lines

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.

Quoted 8 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time,

at what time?

Quoted 2 lines

for a driver, coalescing parameters are read-only and \field{vqn} is write-only.

reserved also write-only.

Quoted 2 lines

"
Do you think this is a good description?

No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.

Quoted 4 lines

for command X fields A B are write-only for driver and fields C D are read-only for driver.

for command Y fields A B C and D are write-only for driver.

I think it's a better proposition! I'll modify.

Quoted 87 lines
Quoted 87 lines
Quoted 82 lines
Quoted 75 lines
Quoted 49 lines
Quoted 49 lines
device-types/net/description.tex +27 -3
   #define VIRTIO_NET_CTRL_NOTF_COAL 6
    #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
    #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
   \end{lstlisting}
   Coalescing parameters:
   \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
   \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
   \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
   \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
   \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
   \end{itemize}
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
   \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
   \end{enumerate}
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626

Quoted 7 lines
device-types/net/description.tex +2 -1
   \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
   The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 2 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 3 lines
device-types/net/description.tex +1 -0
   \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
   A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 3 lines

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Ok.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Ah, you mean the scope of MUST is included in SHOULD?

No I mean just list commands explicitly. Do not make reader guess what does VIRTIO_NET_CTRL_NOTF_COAL mean when there is no command like this.

Ok, I see. I will follow through on this deeply.

Quoted 48 lines
Quoted 48 lines

If you think it's better to remove the *Note* above, I'm fine.

Quoted 45 lines
Quoted 45 lines
Quoted 5 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640

Quoted 4 lines
device-types/net/description.tex +1 -0
   A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 4 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 2 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 12 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

Quoted 3 lines
Quoted 2 lines
Quoted 1 line

ok more or less. is this in normative section? If not avoid MUST.

Yes. It's in normative section. Thanks for your comments!

then pls also add a non normative explanatory informal description.

I'll add the description in the next version.

Thanks for your comments!

Quoted 34 lines
Quoted 8 lines
Quoted 8 lines

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 3 lines

"

Quoted 1 line
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
   Upon reset, a device MUST initialize all coalescing parameters to 0.
   \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#7 lore raw
Quoted 131 lines · Heng Qi
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

   device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
   1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
   \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
       channel.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
   \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
   \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
   \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
   \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
   \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
   \end{description}
   \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
   send control commands for dynamically changing the coalescing parameters.
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
   \begin{note}
   The behavior of the device in response to these commands is best-effort:
   the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535

Quoted 8 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
       le32 max_usecs;
   };
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 7 lines
Quoted 4 lines

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Ok, I'll modify this.

Quoted 7 lines
Quoted 7 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

Quoted 7 lines
Quoted 4 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :

Quoted 7 lines
Quoted 6 lines

"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands.

ugh just confusing. just use the struct names.

Ok.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
 \field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.

how important is it that devices ignore it?
if yes it needs a normative statement.

Maybe "containing 0" is enough.

it is better to specify what's going on from driver and device perspective. If device does not need to ignore it, you can just make it a 32 bit field.

Quoted 221 lines
Quoted 27 lines
Quoted 6 lines

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.

Quoted 8 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time,

at what time?

Quoted 2 lines

for a driver, coalescing parameters are read-only and \field{vqn} is write-only.

reserved also write-only.

Quoted 2 lines

"
Do you think this is a good description?

No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.

Quoted 4 lines

for command X fields A B are write-only for driver and fields C D are read-only for driver.

for command Y fields A B C and D are write-only for driver.

I think it's a better proposition! I'll modify.

Quoted 87 lines
Quoted 87 lines
Quoted 82 lines
Quoted 75 lines
Quoted 49 lines
Quoted 49 lines
device-types/net/description.tex +27 -3
   #define VIRTIO_NET_CTRL_NOTF_COAL 6
    #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
    #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
   \end{lstlisting}
   Coalescing parameters:
   \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
   \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
   \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
   \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
   \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
   \end{itemize}
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
   \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
   \end{enumerate}
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626

Quoted 7 lines
device-types/net/description.tex +2 -1
   \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
   The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
   \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 2 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1629

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1632

Quoted 3 lines
device-types/net/description.tex +1 -0
   \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
   A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 3 lines

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Ok.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Ah, you mean the scope of MUST is included in SHOULD?

No I mean just list commands explicitly. Do not make reader guess what does VIRTIO_NET_CTRL_NOTF_COAL mean when there is no command like this.

Ok, I see. I will follow through on this deeply.

Quoted 48 lines
Quoted 48 lines

If you think it's better to remove the *Note* above, I'm fine.

Quoted 45 lines
Quoted 45 lines
Quoted 5 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1634

Quoted 4 lines
device-types/net/description.tex +1 -0
   A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 4 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1635

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 2 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 12 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1639

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

Quoted 3 lines
Quoted 2 lines
Quoted 1 line

ok more or less. is this in normative section? If not avoid MUST.

Yes. It's in normative section. Thanks for your comments!

then pls also add a non normative explanatory informal description.

I'll add the description in the next version.

Thanks for your comments!

Quoted 34 lines
Quoted 8 lines
Quoted 8 lines

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 3 lines

"

Quoted 1 line
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
   Upon reset, a device MUST initialize all coalescing parameters to 0.
   \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

author
#8 lore raw
Quoted 133 lines · Michael S. Tsirkin
Quoted 132 lines · Heng Qi
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)

device-types/net/description.tex +9 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
    \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
        channel.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
    \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
    \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
    \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
    \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
    \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
    \end{description}
    \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
    If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
    send control commands for dynamically changing the coalescing parameters.
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive virtqueue.
+
    \begin{note}
    The behavior of the device in response to these commands is best-effort:
    the device may generate notifications more or less frequently than specified.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535

Quoted 8 lines
device-types/net/description.tex +6 -0
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
        le32 max_usecs;
    };
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+

2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.

Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.

Quoted 7 lines
Quoted 4 lines

Yes, this should be made clear. How do you think of the following descriptions:

"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the

I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.

Ok, I'll modify this.

Quoted 7 lines
Quoted 7 lines

virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.

Quoted 7 lines
Quoted 4 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "

Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.

OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :

Quoted 7 lines
Quoted 6 lines

"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands.

ugh just confusing. just use the struct names.

Ok.

Quoted 4 lines
Quoted 4 lines

\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
 \field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.

Quoted 6 lines
Quoted 3 lines
Quoted 2 lines

how important is it that devices ignore it?
if yes it needs a normative statement.

Maybe "containing 0" is enough.

it is better to specify what's going on from driver and device perspective. If device does not need to ignore it, you can just make it a 32 bit field.

Ok, I see. :)
Thanks.

Quoted 214 lines
Quoted 172 lines
Quoted 26 lines
Quoted 5 lines

VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.

Quoted 8 lines

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.

The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time,

at what time?

Quoted 2 lines

for a driver, coalescing parameters are read-only and \field{vqn} is write-only.

reserved also write-only.

Quoted 2 lines

"
Do you think this is a good description?

No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.

Quoted 4 lines

for command X fields A B are write-only for driver and fields C D are read-only for driver.

for command Y fields A B C and D are write-only for driver.

I think it's a better proposition! I'll modify.

Quoted 84 lines
Quoted 84 lines
Quoted 84 lines
Quoted 74 lines
Quoted 48 lines
Quoted 48 lines
device-types/net/description.tex +27 -3
    #define VIRTIO_NET_CTRL_NOTF_COAL 6
     #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
     #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
    \end{lstlisting}
    Coalescing parameters:
    \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
    \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
    \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
    \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
    \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
    \end{itemize}
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
    \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
    \end{enumerate}
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
+\end{itemize}
+
Quoted 8 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626

Quoted 7 lines
device-types/net/description.tex +2 -1
    \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
    The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
    \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Triple negative? I think you mean "has been negotiated".

Yes. Thanks for pointing it out.

Quoted 2 lines

Generally we should consider rewriting these in a positive way but that's a subject for another patch.

Ok, I see.

Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1629

Quoted 3 lines
device-types/net/description.tex +3 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
+if the device returns with VIRTIO_NET_ERR.

device does not return. It responds to the command.

Ok.

Quoted 6 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1632

Quoted 3 lines
device-types/net/description.tex +1 -0
    \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
    A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.

Quoted 7 lines

Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.

Quoted 8 lines
Quoted 5 lines

Ok.

Quoted 2 lines

And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.

Ah, you mean the scope of MUST is included in SHOULD?

No I mean just list commands explicitly. Do not make reader guess what does VIRTIO_NET_CTRL_NOTF_COAL mean when there is no command like this.

Ok, I see. I will follow through on this deeply.

Quoted 46 lines
Quoted 46 lines

If you think it's better to remove the *Note* above, I'm fine.

Quoted 44 lines
Quoted 44 lines
Quoted 4 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1634

Quoted 3 lines
device-types/net/description.tex +1 -0
    A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:

parameters here meanong coalescing parameters? It won't be clear to

Yes, I'll write clearly.

Quoted 4 lines

readers who see the full spec not the patch.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1635

Quoted 1 line
device-types/net/description.tex +1 -0
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:

"a" does not go with the plural "values".

Ok.

Quoted 2 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 1 line
device-types/net/description.tex +1 -0
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.

this is the first you mention global.

I mentioned "global" in the first sentence, but that might not seem very clear.

Quoted 12 lines

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1639

Quoted 3 lines
device-types/net/description.tex +3 -0
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Sentence too long at this point I can't say what happens after what.

I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.

For example, we could explain that

Quoted 8 lines

VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.

Thanks for your example, I tried to regularize it, do you think it works?

"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.

Quoted 38 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line

ok more or less. is this in normative section? If not avoid MUST.

Yes. It's in normative section. Thanks for your comments!

then pls also add a non normative explanatory informal description.

I'll add the description in the next version.

Thanks for your comments!

Quoted 31 lines
Quoted 6 lines
Quoted 6 lines

And maybe add text in a normative section too. a bit of repetition there is ok.

Quoted 2 lines

"

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
    Upon reset, a device MUST initialize all coalescing parameters to 0.
    \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]

#9 lore raw
In reply to Heng Qi
Quoted 3 lines · Heng Qi
Quoted 1 line

then pls also add a non normative explanatory informal description.

I'll add the description in the next version.

Try to pass it through a grammar checker pls ;)

#10 lore raw
In reply to Heng Qi
Quoted 71 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

 device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +8 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
 \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
 \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
 
 \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
 send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive
Quoted 1 line

virtqueue.

Use an enumeration.

Quoted 40 lines
device-types/net/description.tex +16 -3
+
 \begin{note}
 The behavior of the device in response to these commands is best-effort:
 the device may generate notifications more or less frequently than specified.
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
     le32 max_usecs;
 };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+
 #define VIRTIO_NET_CTRL_NOTF_COAL 6
  #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
  #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
 \end{lstlisting}
 
 Coalescing parameters:
 \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
 \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
 \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
 \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
 \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
 \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
 \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1569

Quoted 10 lines
device-types/net/description.tex +8 -0
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
 \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece

"sequence"

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1574

Quoted 5 lines
device-types/net/description.tex +5 -0
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.

Missing space before =.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1577

Quoted 3 lines
device-types/net/description.tex +3 -0
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.

Missing space before =.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 12 lines
device-types/net/description.tex +4 -1
+\end{itemize}
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
 \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Too many negatives here. Just "have been negotiated".

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1630

Quoted 2 lines
device-types/net/description.tex +2 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,

"parameters"

Remove the comma?

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 6 lines
device-types/net/description.tex +2 -0
+if the device returns with VIRTIO_NET_ERR.
 
 \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
 A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

"an exception"

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 9 lines
device-types/net/description.tex +6 -0
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Use an enumeration.

Quoted 9 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
 Upon reset, a device MUST initialize all coalescing parameters to 0.
 
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 17 lines
We're deep in discussion, the party's on mute.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
author
#11 lore raw

Hi David, thanks for pointing out the problem, I will fix it in the next version.

Thanks very much! :)

Quoted 187 lines · David Edmondson
Quoted 71 lines · Heng Qi

Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].

When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.

[1] https://docs.kernel.org/networking/net_dim.html

---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .

v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin

v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz

  device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++--
  1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +8 -0
index e71e33b..fa8bd26 100644
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
  \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
      channel.
 
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
  \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
 
  \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
  \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
  \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
  \end{description}
 
  \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
  If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
  send control commands for dynamically changing the coalescing parameters.
 
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
+   parameters of an enabled transmit/receive virtqueue.
+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
+   returns the coalescing parameters of an enabled transmit/receive
Quoted 1 line

virtqueue.

Use an enumeration.

Quoted 40 lines
device-types/net/description.tex +16 -3
+
  \begin{note}
  The behavior of the device in response to these commands is best-effort:
  the device may generate notifications more or less frequently than specified.
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
      le32 max_usecs;
  };
 
+struct virtio_net_ctrl_coal_vq {
+    le16 vqn;
+    le16 reserved;
+    struct virtio_net_ctrl_coal coal;
+};
+
  #define VIRTIO_NET_CTRL_NOTF_COAL 6
   #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET  0
   #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
  \end{lstlisting}
 
  Coalescing parameters:
  \begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
  \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
  \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
  \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
  \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
  \end{itemize}
 
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by the device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
  \begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1569

Quoted 10 lines
device-types/net/description.tex +8 -0
+                                        virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+                                        transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
  \end{enumerate}
 
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequece

"sequence"

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1574

Quoted 5 lines
device-types/net/description.tex +5 -0
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.

Missing space before =.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1577

Quoted 3 lines
device-types/net/description.tex +3 -0
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.

Missing space before =.

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628

Quoted 12 lines
device-types/net/description.tex +4 -1
+\end{itemize}
+
  \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
  The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 
  \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.

Too many negatives here. Just "have been negotiated".

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1630

Quoted 2 lines
device-types/net/description.tex +2 -0
+
+A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,

"parameters"

Remove the comma?

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636

Quoted 6 lines
device-types/net/description.tex +2 -0
+if the device returns with VIRTIO_NET_ERR.
 
  \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
 
  A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.

"an exception"

device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645

Quoted 9 lines
device-types/net/description.tex +6 -0
  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
+If a virtqueue is disabled, the device MUST clear its parameters configuration, and:
+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:
+after the virtqueue is enabled, its parameters SHOULD be set to the global values.
+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands:
+regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the
+virtqueue is enabled, its parameters SHOULD be set to 0.

Use an enumeration.

Quoted 9 lines
device-types/net/description.tex +6 -0
+
+When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn},
+if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
  Upon reset, a device MUST initialize all coalescing parameters to 0.
 
  \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device