virtio-net: support the virtqueue coalescing moderation
PATCH v3 1/1 virtio-net: support the virtqueue coalescing moderation
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
---v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)
content.tex +126 -4
@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:
@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:
@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:
@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR. + +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.
Signature · 1 line
2.19.1.6.gb485710b
Quoted 12 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>
Quoted 9 lines
---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
Hi Michael.
I know you suggested in v2 that I split the virtqueue SET and GET commands in series,
but I have a bit of a problem trying to split because there are some statements where SET and GET are glued together. Please see the following places.
If you think the split is better, I'll split it again and send it right away.
Thanks.
Quoted 35 lines
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +4
-0@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4511
Quoted 8 lines
content.tex +3
-1@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
here
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4524
Quoted 10 lines
content.tex +2
-1+ \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
here
Quoted 133 lines
content.tex +108
-2\begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4678
Quoted 10 lines
content.tex +5
-0+to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
here
Quoted 7 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.
Quoted 6 lines
From: Heng Qi <[email protected]>
Sent: Thursday, February 16, 2023 9:49 AM
Quoted 8 lines
Hi Michael.
I know you suggested in v2 that I split the virtqueue SET and GET commands in series, but I have a bit of a problem trying to split because there are some statements where SET and GET are glued together.
Please see the following places.If you think the split is better, I'll split it again and send it right away.
I agree with Michael, better to split.
It should be possible.
First patch without next patch consideration.
Second patch can modify the previous line, this is fine.
Its common practice and better even if it touches some of the same lines.
Quoted 33 lines · Heng Qi
Quoted 12 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>Quoted 9 lines
---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. TsirkinHi Michael.
I know you suggested in v2 that I split the virtqueue SET and GET commands in series,
but I have a bit of a problem trying to split because there are some statements where SET and GET are glued together.
Please see the following places.
Quoted 4 lines
If you think the split is better, I'll split it again and send it right away.
Thanks.
I don't mind that much. Lots of tiny patches are annoying too. Parav seems to want this though.
Quoted 42 lines
Quoted 32 lines
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +4
-0@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4510
Quoted 7 lines
content.tex +3
-1@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3here
Quoted 22 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4522
Quoted 7 lines
content.tex +2
-1+ \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:here
Quoted 12 lines
content.tex +8
-0\begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue.
an enabled
Quoted 2 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4539
Quoted 2 lines
content.tex +2
-0+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
an enabled
Quoted 19 lines
Quoted 19 lines
content.tex +19
-0+ +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566
Quoted 8 lines
content.tex +8
-0+THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
has the same effect
Quoted 3 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4569
Quoted 3 lines
content.tex +3
-0+command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
has the same effect
Quoted 28 lines
Quoted 28 lines
content.tex +22
-1+command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4611
Quoted 8 lines
content.tex +8
-0+\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
use tense consistently:
elapsed -> elapse,
was -> is
one refers to packets I guess but technically could refer to usecs.
one -> packet
usecs -> microsecons , eschew abbreviation
Quoted 10 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4622
Quoted 10 lines
content.tex +9
-0+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver.
respond is not a transitive verb
responds -> reports?
Quoted 21 lines
Quoted 21 lines
content.tex +16
-0+ \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize}
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4661
Quoted 8 lines
content.tex +8
-0+ +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver.
same
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4669
Quoted 8 lines
content.tex +6
-1+ \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
either all perfect or all in simple tense
and maybe just put the formula here.
Quoted 3 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4670
Quoted 3 lines
content.tex +2
-0+ +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to,
This is really a device conformance statement not a driver one. And MAY NOT means it is not allowed.
just drop this here. or write that driver MUST NOT rely on getting same
# back.
Quoted 2 lines
for example it MAY get a value that is a
Quoted 1 line
power of 2.
this confuses more than it clarifies.
Quoted 3 lines
Quoted 3 lines
\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the
VIRTIO_NET_CTRL_NOTF_COALcommands withVIRTIO_NET_ERRif it was not able to change the parameters.
+When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
virtqueue number, not virtqueue with number.
Quoted 8 lines
Quoted 1 line
+if the virtqueue is not enabled, the device SHOULD respond to the command with
VIRTIO_NET_ERR.here
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4682
Quoted 4 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. +
same as driver. MAY NOT means not allowed. should be MAY. going overvoard with examples. keep them out of conformance sections these must be clear by themselves.
Quoted 2 lines
Quoted 2 lines
A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.
Quoted 108 lines · Michael S. Tsirkin
Quoted 33 lines · Heng Qi
Quoted 12 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>Quoted 9 lines
---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. TsirkinHi Michael.
I know you suggested in v2 that I split the virtqueue SET and GET commands in series,
but I have a bit of a problem trying to split because there are some statements where SET and GET are glued together.
Please see the following places.Quoted 4 lines
If you think the split is better, I'll split it again and send it right away.
Thanks.
I don't mind that much. Lots of tiny patches are annoying too. Parav seems to want this though.
Quoted 42 lines
Quoted 32 lines
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +4
-0@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4510
Quoted 7 lines
content.tex +3
-1@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3here
Quoted 22 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4522
Quoted 7 lines
content.tex +2
-1+ \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:here
Quoted 12 lines
content.tex +8
-0\begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue.
Quoted 2 lines
an enabled
Will fix.
Quoted 4 lines
Quoted 2 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4539
Quoted 2 lines
content.tex +2
-0+2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.an enabled
Will fix.
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
content.tex +21
-0+ +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index:
Quoted 8 lines
Quoted 6 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4566
Quoted 6 lines
content.tex +6
-0+\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SEThas the same effect
Ok.
Quoted 7 lines
Quoted 4 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4569
Quoted 4 lines
content.tex +3
-0+command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SEThas the same effect
Ok.
Quoted 42 lines
Quoted 29 lines
Quoted 29 lines
content.tex +22
-1+command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4611
Quoted 8 lines
content.tex +8
-0+\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.use tense consistently:
elapsed -> elapse,
was -> is
Quoted 5 lines
one refers to packets I guess but technically could refer to usecs.
one -> packet
usecs -> microsecons , eschew abbreviation
Thanks for pointing this out, I think this is more clear.
Quoted 15 lines
Quoted 12 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4622
Quoted 12 lines
content.tex +9
-0+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver.respond is not a transitive verb
responds -> reports?
Ahh, will fix. But how about "returns" ?
Quoted 33 lines
Quoted 22 lines
Quoted 22 lines
content.tex +16
-0+ \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize}Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4661
Quoted 8 lines
content.tex +8
-0+ +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver.same
Quoted 11 lines
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4669
Quoted 8 lines
content.tex +6
-1+ \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.either all perfect or all in simple tense
and maybe just put the formula here.
Sure.
Quoted 11 lines
Quoted 5 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4670
Quoted 5 lines
content.tex +2
-0+ +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to,This is really a device conformance statement not a driver one. And MAY NOT means it is not allowed.
just drop this here. or write that driver MUST NOT rely on getting same
Quoted 1 line
# back.
OK, I've covered this in device conformance, and it seems better to drop this here.
Quoted 11 lines
Quoted 3 lines
for example it MAY get a value that is a
Quoted 1 line
power of 2.
this confuses more than it clarifies.
Quoted 3 lines
Quoted 3 lines
\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the
VIRTIO_NET_CTRL_NOTF_COALcommands withVIRTIO_NET_ERRif it was not able to change the parameters.
+When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},virtqueue number, not virtqueue with number.
Ok.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
+if the virtqueue is not enabled, the device SHOULD respond to the command with
VIRTIO_NET_ERR.here
Quoted 8 lines
Quoted 4 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4682
Quoted 4 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. +same as driver. MAY NOT means not allowed. should be MAY. going overvoard with examples. keep them out of conformance sections these must be clear by themselves.
I see. Thanks a lot for your suggestion!
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 8 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 207 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +117
-4@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4678
Quoted 10 lines
content.tex +5
-0+to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
What if vq is disabled? do parameters reset?
What about VIRTIO_NET_CTRL_NOTF_COAL?
what if it is envoked while some vq is disabled then vq is enabled? do parameters for this vq stay 0?
Quoted 9 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.Signature · 1 line
2.19.1.6.gb485710b
Quoted 212 lines · Michael S. Tsirkin
Quoted 211 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +120
-4@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4678
Quoted 6 lines
content.tex +2
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.What if vq is disabled? do parameters reset?
Yes, max_packets and max_usecs SHOULD be reset to 0.
When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.
As described by Alvaro in "[PATCH v5] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature":
"+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
Quoted 1 line
What about
VIRTIO_NET_CTRL_NOTF_COAL?
I think it should be handled in the same way as the above VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
Quoted 2 lines
what if it is envoked while some vq is disabled then vq is enabled? do parameters for this vq stay 0?
I think there are two kinds of sequences:
#Seq1:
1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
2. VQ_SET command arrives with max_packets=10, max_usecs = 8
3. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
After #3 is done, vq params should be max_packets = 0, max_usecs = 0, because the device cannot apply parameters when vq is disabled. Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.
#Seq1:
1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
2. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
3. VQ_SET command arrives with max_packets=10, max_usecs = 8 After 3 is done, vq params should be max_packets = 10, max_usecs = 8
Thanks.
Quoted 10 lines
Quoted 10 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.Signature · 1 line
2.19.1.6.gb485710b
Quoted 222 lines · Heng Qi
Quoted 212 lines · Michael S. Tsirkin
Quoted 211 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +120
-4@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4678
Quoted 6 lines
content.tex +2
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.What if vq is disabled? do parameters reset?
Quoted 5 lines
Yes,
max_packetsandmax_usecsSHOULD be reset to 0.
When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.As described by Alvaro in "[PATCH v5] virtio-net: Fix and update
VIRTIO_NET_F_NOTF_COALfeature":
"+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
Oh. Hmm.
What Alvaro's patch does not describe is what happens when VQ is reset.
Alvaro you said you have hardware implementing this right? How does the command interact with vq reset?
Quoted 3 lines
Quoted 1 line
What about
VIRTIO_NET_CTRL_NOTF_COAL?I think it should be handled in the same way as the above
VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
sounds consistent. but the problem is, I don't think this is how we previously behaved. and RING_RESET is out in 1.2. So we need something compatible. I am sorry.
I suspect that instead we can say that existing hardware has a default set of parameters for tx and rx. And global commands change that
besides changing all enabled VQs.
That is a side effect beyond just changing all VQs.
Hmm.
Alvaro?
Quoted 7 lines
Quoted 2 lines
what if it is envoked while some vq is disabled then vq is enabled? do parameters for this vq stay 0?
I think there are two kinds of sequences:
#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0)
2.VQ_SETcommand arrives with max_packets=10,max_usecs= 8
will not this VQ_SET fail?
Quoted 25 lines
3. vq is enabled (vq params are
max_packets= 0,max_usecs= 0)
After #3 is done, vq params should bemax_packets= 0,max_usecs= 0, because the device cannot apply parameters when vq is disabled. Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0)
2. vq is enabled (vq params aremax_packets= 0,max_usecs= 0)
3.VQ_SETcommand arrives with max_packets=10,max_usecs= 8 After 3 is done, vq params should bemax_packets= 10,max_usecs= 8Thanks.
Quoted 10 lines
Quoted 10 lines
content.tex +4
-0+ +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.Signature · 1 line
2.19.1.6.gb485710b
Quoted 6 lines
Quoted 5 lines
Yes,
max_packetsandmax_usecsSHOULD be reset to 0.
When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.As described by Alvaro in "[PATCH v5] virtio-net: Fix and update
VIRTIO_NET_F_NOTF_COALfeature":
"+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
Quoted 7 lines
Oh. Hmm.
What Alvaro's patch does not describe is what happens when VQ is reset.
Alvaro you said you have hardware implementing this right? How does the command interact with vq reset?
The device doesn't offer VIRTIO_F_RING_RESET at the moment.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
What about
VIRTIO_NET_CTRL_NOTF_COAL?I think it should be handled in the same way as the above
VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.sounds consistent. but the problem is, I don't think this is how we previously behaved. and
RING_RESETis out in 1.2. So we need something compatible. I am sorry.
Quoted 9 lines
I suspect that instead we can say that existing hardware has a default set of parameters for tx and rx. And global commands change that
besides changing all enabled VQs.That is a side effect beyond just changing all VQs.
Hmm.
Alvaro?
This is indeed a good point.
We mention the device reset case, but nothing about VQ reset.
I feel that no matter how we handle this, we break something.
Having default coalescing values may collide with "Upon reset, a device MUST initialize all coalescing parameters to 0."
We can say that VQ reset doesn't affect the "global parameters" and a device reset does, but this collides with "Device Requirements:
Virtqueue Reset".
In fact, resetting coalescing values after vq reset may be derived from "Upon reset, a device MUST initialize all coalescing parameters to 0".
This is consistent with "Device Requirements: Virtqueue Reset".
I can add in my patch some clarifications.
This will break the linux virtio_net ethtool implementation a little, we'll need to fix it.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 35 lines · Alvaro Karsz
Quoted 6 lines
Quoted 5 lines
Yes,
max_packetsandmax_usecsSHOULD be reset to 0.
When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.As described by Alvaro in "[PATCH v5] virtio-net: Fix and update
VIRTIO_NET_F_NOTF_COALfeature":
"+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."Quoted 7 lines
Oh. Hmm.
What Alvaro's patch does not describe is what happens when VQ is reset.
Alvaro you said you have hardware implementing this right? How does the command interact with vq reset?
The device doesn't offer
VIRTIO_F_RING_RESETat the moment.Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
What about
VIRTIO_NET_CTRL_NOTF_COAL?I think it should be handled in the same way as the above
VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.sounds consistent. but the problem is, I don't think this is how we previously behaved. and
RING_RESETis out in 1.2. So we need something compatible. I am sorry.Quoted 9 lines
I suspect that instead we can say that existing hardware has a default set of parameters for tx and rx. And global commands change that
besides changing all enabled VQs.That is a side effect beyond just changing all VQs.
Hmm.
Alvaro?
Quoted 7 lines
This is indeed a good point.
We mention the device reset case, but nothing about VQ reset.I feel that no matter how we handle this, we break something.
Having default coalescing values may collide with "Upon reset, a device MUST initialize all coalescing parameters to 0."
No this is after device reset.
Quoted 9 lines
We can say that VQ reset doesn't affect the "global parameters" and a device reset does, but this collides with "Device Requirements:
Virtqueue Reset".In fact, resetting coalescing values after vq reset may be derived from "Upon reset, a device MUST initialize all coalescing parameters to 0".
This is consistent with "Device Requirements: Virtqueue Reset".
Quoted 4 lines
I can add in my patch some clarifications.
This will break the linux
virtio_netethtool implementation a little, we'll need to fix it.
Not good. I feel we must come up with spec that is backwards compatible. Hmm, maybe this is why Parav kept talking about modes.
I did not realize at the time, sorry Parav.
I still feel modes are not the best way to describe things so I propose this:
- in addition to per vq parameters, device that supports global TX/RX commands and ring reset maintains two sets of default parameters: for RX and TX
- existing commands change default and change all enabled vqs
of the correct type (RX/TX) to the same value
- vq reset changes a vq to the default
- device reset changes defaults to 0 and changes all vqs to 0
note how defaults are only used for ring reset. is "vq reset parameter" a better name? I feel we will repeat "reset" too many times in a sentence if we call it that though.
So fundamentally the only change is with RING_RESET, then default is not always 0, it can be set by the global command.
Signature · 1 line
MST
Quoted 12 lines
Not good. I feel we must come up with spec that is backwards compatible. Hmm, maybe this is why Parav kept talking about modes.
I did not realize at the time, sorry Parav.I still feel modes are not the best way to describe things so I propose this:
- in addition to per vq parameters, device that supports global TX/RX commands and ring reset maintains two sets of default parameters: for RX and TX
- existing commands change default and change all enabled vqs
of the correct type (RX/TX) to the same value
- vq reset changes a vq to the default
- device reset changes defaults to 0 and changes all vqs to 0
Quoted 6 lines
note how defaults are only used for ring reset. is "vq reset parameter" a better name? I feel we will repeat "reset" too many times in a sentence if we call it that though.
So fundamentally the only change is with
RING_RESET, then default is not always 0, it can be set by the global command.
I like the idea, maintaining default coalescing parameters is compatible with vq reset.
"After a queue has been reset by the driver, the device MUST NOT execute any requests from that virtqueue, or notify the driver for it. The device MUST reset any state of a virtqueue to the *default state*, including the available state and the used state."
I wonder if some of that should be included in my patch. Most of that is not relevant before introducing the per vq command. So maybe I could just mention in the device conformance that:
"Coalescing parameters MUST/SHOULD persist a virtqueue reset."
And then Heng can edit/add on top of that.
Now that the existing commands do more than just iterating through all the virtqueues, maybe we should be able to modify all virtqueues with VIRTIO_NET_F_VQ_NOTF_COAL.
I think that using a special vqn is messy, we can split "le32 reserved" to "le16 flags" and "le16 reserved".
Then we can have:
- A flag to apply parameters to all receive VQs, vqn is ignored in this case. (same as VIRTIO_NET_CTRL_NOTF_COAL_RX_SET but without changing the default parameters).
- A flag to apply parameters to all transmit VQs, vqn is ignored in this case. (same as VIRTIO_NET_CTRL_NOTF_COAL_TX_SET but without changing the default parameters).
And maybe a flag to apply to all vqs (transmit + receive), I actually am not sure how useful this is..
Quoted 9 lines
From: Alvaro Karsz <[email protected]>
Sent: Friday, February 17, 2023 7:18 AMI wonder if some of that should be included in my patch. Most of that is not relevant before introducing the per vq command. So maybe I could just mention in the device conformance that:
"Coalescing parameters MUST/SHOULD persist a virtqueue reset."
For sure coalescing parameters must not persist on a vq reset in the device.
VQ reset operation destroys or disables the VQ.
Device should not have any obligation to maintain some parameter on the disabled/destroyed object.
User likely disabled the VQ because user may want to have a fresh start for this VQ for same or different use. And hence, reconfiguring the new addresses, new vq size, new vq notifications parameters..
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Friday, February 17, 2023 6:35 AM
Quoted 13 lines
Quoted 6 lines
We mention the device reset case, but nothing about VQ reset.
I feel that no matter how we handle this, we break something.
Having default coalescing values may collide with "Upon reset, a device MUST initialize all coalescing parameters to 0."
No this is after device reset.
Quoted 4 lines
We can say that VQ reset doesn't affect the "global parameters" and a device reset does, but this collides with "Device Requirements:
Virtqueue Reset".
Not really.
When the device resets, VQ objects are destroyed in the device. So VQ's notifications parameters doesn't exists on device reset.
And so the same case with VQ reset.
When a VQ is reset (disabled), VQ's notifications configuration is removed in the device too. Just like its desc ring and other addresses are invalid.
Quoted 23 lines
Quoted 9 lines
In fact, resetting coalescing values after vq reset may be derived from "Upon reset, a device MUST initialize all coalescing parameters to 0".
This is consistent with "Device Requirements: Virtqueue Reset".I can add in my patch some clarifications.
This will break the linux
virtio_netethtool implementation a little, we'll need to fix it.Not good. I feel we must come up with spec that is backwards compatible. Hmm, maybe this is why Parav kept talking about modes.
I did not realize at the time, sorry Parav.I still feel modes are not the best way to describe things so I propose this:
- in addition to per vq parameters, device that supports global TX/RX commands and ring reset maintains two sets of default parameters: for RX and
TX
- existing commands change default and change all enabled vqs
of the correct type (RX/TX) to the same value
- vq reset changes a vq to the default
- device reset changes defaults to 0 and changes all vqs to 0
Quoted 6 lines
note how defaults are only used for ring reset. is "vq reset parameter" a better name? I feel we will repeat "reset" too many times in a sentence if we call it that though.
So fundamentally the only change is with
RING_RESET, then default is not always 0, it can be set by the global command.
True default is not zero when global are configured.
It is ok to report VQ parameters on GET command when globals are configured.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 17 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Friday, February 17, 2023 6:35 AMQuoted 13 lines
Quoted 6 lines
We mention the device reset case, but nothing about VQ reset.
I feel that no matter how we handle this, we break something.
Having default coalescing values may collide with "Upon reset, a device MUST initialize all coalescing parameters to 0."
No this is after device reset.
Quoted 4 lines
We can say that VQ reset doesn't affect the "global parameters" and a device reset does, but this collides with "Device Requirements:
Virtqueue Reset".
Quoted 7 lines
Not really.
When the device resets, VQ objects are destroyed in the device. So VQ's notifications parameters doesn't exists on device reset.And so the same case with VQ reset.
When a VQ is reset (disabled), VQ's notifications configuration is removed in the device too. Just like its desc ring and other addresses are invalid.
Yes, but there seems to be such a situation: when the device is reactivated, as the specification says, all parameters are set to 0 (use parameters as the default configuration on the device).
When CTRL_COAL_SET and CTRL_COAL_VQ_SET are sent, the configuration is updated (the parameters of each vq may be different, but the global parameter configuration may be recorded), at this time, if vq is reset,
should the parameters be 0 or a recorded global parameters after it is re-enabled?
Quoted 33 lines
Quoted 24 lines
Quoted 10 lines
In fact, resetting coalescing values after vq reset may be derived from "Upon reset, a device MUST initialize all coalescing parameters to 0".
This is consistent with "Device Requirements: Virtqueue Reset".I can add in my patch some clarifications.
This will break the linux
virtio_netethtool implementation a little, we'll need to fix it.Not good. I feel we must come up with spec that is backwards compatible. Hmm, maybe this is why Parav kept talking about modes.
I did not realize at the time, sorry Parav.I still feel modes are not the best way to describe things so I propose this:
- in addition to per vq parameters, device that supports global TX/RX commands and ring reset maintains two sets of default parameters: for RX and
TX
- existing commands change default and change all enabled vqs
of the correct type (RX/TX) to the same value
- vq reset changes a vq to the default
- device reset changes defaults to 0 and changes all vqs to 0Quoted 6 lines
note how defaults are only used for ring reset. is "vq reset parameter" a better name? I feel we will repeat "reset" too many times in a sentence if we call it that though.
So fundamentally the only change is with
RING_RESET, then default is not always 0, it can be set by the global command.True default is not zero when global are configured.
It is ok to report VQ parameters on GET command when globals are configured.
Quoted 5 lines
From: Heng Qi <[email protected]>
Sent: Friday, February 17, 2023 12:17 PM
Quoted 7 lines
Yes, but there seems to be such a situation: when the device is reactivated, as the specification says, all parameters are set to 0 (use parameters as the default configuration on the device).
WhenCTRL_COAL_SETandCTRL_COAL_VQ_SETare sent, the configuration is updated (the parameters of each vq may be different, but the global parameter configuration may be recorded), at this time, if vq is reset, should the parameters be 0 or a recorded global parameters after it is re-enabled?
Recorded global parameters should be fine to report on an enabled vq even if the COAL_VQ_SET was not done. This is because the global value reflects on the VQ.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines
From: Heng Qi <[email protected]>
Sent: Friday, February 17, 2023 12:24 AM
Quoted 9 lines
I think there are two kinds of sequences:
#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0) 2.VQ_SETcommand arrives with max_packets=10,max_usecs= 8 3. vq is enabled (vq params aremax_packets= 0,max_usecs= 0) After #3 is done, vq params should bemax_packets= 0,max_usecs= 0, because the device cannot apply parameters when vq is disabled.
Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.
From dev point of view, above is not a valid sequence.
VQ is the object whose lifecycle (create/delete/query/modify) is triggered by the driver. So step 2 cannot happen before step 3.
If driver does that, driver should get error code INVALID_VQ on step 2.
Quoted 24 lines
#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0) 2. vq is enabled (vq params aremax_packets= 0,max_usecs= 0) 3.VQ_SETcommand arrives with max_packets=10,max_usecs= 8 After 3 is done, vq params should bemax_packets= 10,max_usecs= 8Thanks.
Quoted 4 lines
Review
Quoted 4 lines
(quoted diff) +3
-0+ +A device MAY NOT set the coalescing parameter to the exact same +value as the one passed in by a driver. For example, the value of\field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
Quoted 2 lines
Quoted 2 lines
+
A device SHOULD NOT send used buffer notifications to the driver, if thenotifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
Quoted 4 lines
Quoted 4 lines
Upon reset, a device MUST initialize all coalescing parameters to 0.
--
2.19.1.6.gb485710b
Quoted 12 lines · Parav Pandit
Quoted 12 lines
From: Heng Qi <[email protected]>
Sent: Friday, February 17, 2023 12:24 AM
I think there are two kinds of sequences:
#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0) 2.VQ_SETcommand arrives with max_packets=10,max_usecs= 8 3. vq is enabled (vq params aremax_packets= 0,max_usecs= 0) After #3 is done, vq params should bemax_packets= 0,max_usecs= 0, because the device cannot apply parameters when vq is disabled.
Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.
Quoted 5 lines
From dev point of view, above is not a valid sequence.
VQ is the object whose lifecycle (create/delete/query/modify) is triggered by the driver. So step 2 cannot happen before step 3.
If driver does that, driver should get error codeINVALID_VQon step 2.
Yes, This example is not appropriate. The driver must not make any requests on vq while it is being reset,
unless the reset of vq has been completed.
Quoted 21 lines
Quoted 21 lines
#Seq1:
1. vq is disabled (vq params are reset tomax_packets= 0,max_usecs= 0) 2. vq is enabled (vq params aremax_packets= 0,max_usecs= 0) 3.VQ_SETcommand arrives with max_packets=10,max_usecs= 8 After 3 is done, vq params should bemax_packets= 10,max_usecs= 8Thanks.
Quoted 3 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ +A device MAY NOT set the coalescing parameter to the exact same +value as the one passed in by a driver. For example, the value of\field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
Quoted 2 lines
Quoted 2 lines
+
A device SHOULD NOT send used buffer notifications to the driver, if thenotifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
Quoted 3 lines
Quoted 3 lines
Upon reset, a device MUST initialize all coalescing parameters to 0.
--
2.19.1.6.gb485710b
Quoted 3 lines
From: Heng Qi <[email protected]>
Sent: Thursday, February 16, 2023 9:43 AM
[..]
Quoted 3 lines
#define
VIRTIO_NET_CTRL_NOTF_COAL6
- #defineVIRTIO_NET_CTRL_NOTF_COAL_TX_SET0
+ #defineVIRTIO_NET_CTRL_NOTF_COAL_TX_SET0
Unrelated change. Please remove this hunk.
Quoted 15 lines
#define
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET1
+ #defineVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET2 #define
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET3content.tex +2
-1+ \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and
Quoted 6 lines
\field{tx_max_packets} parameters.
\itemVIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and
\field{rx_max_packets} parameters.
+\itemVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
+ transmit/receive virtqueue whose number is \field{vqn}.
In the spec we have mix of notion as vqn or vq index. For example, recent aq patch from Michael refers to aq vq index. Mmio has vq index register too for long time.
And some places we don't refer to index vs number. (pcie queue_select register). We all understand that they refer to same.
But it is time to get consistency for new additions.
So lets agree on either one of the naming convention.
Michael,
I am inclined to name it as vq index given existings references and with your aq vq index patch. Do you have preference?
If its index, above should be changed from vqn/vq_index.
Quoted 6 lines
+\item
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
+ a enabled transmit/receive virtqueue whose number is
\field{vqn}, and then
+ responds them to the driver.
\end{enumerate}
It is confusing line of "device gets".
It is better to write it as,
The device returns max_packets and max_usecs for the enabled transmit or receive virtuque whose index is vq_index.
Quoted 22 lines
+If the
VIRTIO_NET_F_VQ_NOTF_COALfeature is negotiated:
+1. a driver can send aVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand to set the coalescing
+ parameters of a enabled transmit/receive virtqueue.
+2. a driver can send aVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand to a device, and the device
+ responds to the driver with the coalescing parameters of a enabled
transmit/receive virtqueue.content.tex +13
-0+ +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive
Quoted 1 line
virtqueue, excluding the control virtqueue.
Transmit and receive vq is self-explanatory. No need to mention control vq. The word "excluding" doesn't fit here well given it refers to only single vq at a time.
Quoted 5 lines
+\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
+\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565
Quoted 10 lines
content.tex +10
-0+ +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} +
No need to provide the math here, it is about enabled vq.
The spec somewhere else already describes what the valid index of enabled vq.
Quoted 11 lines
(quoted diff) +2
-0+The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for eachvirtqueue of receiveq1\ldots receiveqN.
(quoted diff) +3
-0+ +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for eachvirtqueue of transmitq1\ldots transmitqN.
(quoted diff) +3
-0+ +If coalescing parameters are being set, the device applies the last +coalescing parameters received for a virtqueue, regardless of the commandused to deliver the parameters. For example:
used to set the parameters.
Quoted 67 lines
+# Command sequence 1:
+1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand with $ \field{max_packets}
+= 15 $ and $ \field{max_usecs} = 10 $ 2. +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and$ \field{max_usecs} = 10 $.content.tex +3
-0+ +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $,$
+\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ 2. +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
+
\subparagraph{RX Notifications}\label{sec:Device Types / Network Device /
Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}If, for example:
(quoted diff) +0
-0@@ -4532,9 +4594,33 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
\begin{itemize}
\item The device will count received packets until it accumulates 15, or until 10
usecs elapsed since the first one was received.
-\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +11
-0+ +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because itsvirtqueue number is 0.
+\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +6
-0+ +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and +\field{max_usecs} parameters from the receiveq1 whose virtqueue number is
Quoted 2 lines
0, and then responds them to the driver.
+
The device returns the value of X and Y parameters for the supplied vq index.
Quoted 57 lines
\subparagraph{TX Notifications}\label{sec:Device Types / Network Device /
Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}If, for example:
(quoted diff) +0
-0@@ -4550,14 +4636,50 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
\end{itemize}content.tex +11
-0+Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because itsvirtqueue number is 1.
+\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +8
-0+ +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and +\field{max_usecs} parameters from the transmitq1 whose virtqueue numberis 1, and then responds them to the driver.
+
\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types /
Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}-If the
VIRTIO_NET_F_NOTF_COALfeature has not been negotiated, the driver MUST NOT issueVIRTIO_NET_CTRL_NOTF_COALcommands.
+If neither theVIRTIO_NET_F_NOTF_COALnor theVIRTIO_NET_F_VQ_NOTF_COAL
+feature has not been negotiated, the driver MUST NOT issueVIRTIO_NET_CTRL_NOTF_COALcommands.(quoted diff) +3
-0+ +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST +set \field{vqn} to a legal value, that is, \field{vqn} points to an enabledtransmit/receive virtqueue.
(quoted diff) +2
-0+ +A driver MAY NOT get exactly the same value as the coalescing parameter it
Quoted 2 lines
was set to, for example it MAY get a value that is a power of 2.
It MAY get a value that is close to power of 2 value of the previously set value.
Quoted 7 lines
\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types /
Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}A device SHOULD respond to the
VIRTIO_NET_CTRL_NOTF_COALcommands
withVIRTIO_NET_ERRif it was not able to change the parameters.
Silently dropping the command is not good, we cannot change past now.
But lets please fix the below one.
Please add the note around above text that,
Note: VQ_COAL is exception to it, which must return NET_ERR if device is unable to change the parameters.
Quoted 4 lines
(quoted diff) +2
-0+When a device receives a virtqueue command to set/get coalescing +parameters for a virtqueue with number \field{vqn}, if the virtqueue is notenabled, the device SHOULD respond to the command with
VIRTIO_NET_ERR. +
Device MUST respond with an error on the object that doesn't exist. Please change it to MUST.
Please add below text. Need some wording change for above SHOULD.
A device MUST respond to the VIRTIO_NET_CTRL_VQ_NOTF_COAL_GET and SET command with with VIRTIO_NET_ERR if it is not able to change the parameters.
Review
Quoted 2 lines
(quoted diff) +2
-0+A device MAY NOT set the coalescing parameter to the exact same value +as the one passed in by a driver.
s/as the one passed in by a driver/as the one set by the driver
For example, the value of \field{max_packets}
Quoted 11 lines
set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
+
A device SHOULD NOT send used buffer notifications to the driver, if the
notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.Upon reset, a device MUST initialize all coalescing parameters to 0.
--
2.19.1.6.gb485710b
Hi, Parav. Thanks for your suggested revisions.
Quoted 7 lines · Parav Pandit
Quoted 2 lines
From: Heng Qi <[email protected]>
Sent: Thursday, February 16, 2023 9:43 AM[..]
Quoted 3 lines
#define
VIRTIO_NET_CTRL_NOTF_COAL6
- #defineVIRTIO_NET_CTRL_NOTF_COAL_TX_SET0
+ #defineVIRTIO_NET_CTRL_NOTF_COAL_TX_SET0Unrelated change. Please remove this hunk.
Ok. Will fix.
Quoted 26 lines
Quoted 16 lines
#define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1+ #define
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET2 #define
+VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET3content.tex +2
-1+ \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} andQuoted 6 lines
\field{tx_max_packets} parameters.
\itemVIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and
\field{rx_max_packets} parameters.
+\itemVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
+ transmit/receive virtqueue whose number is \field{vqn}.In the spec we have mix of notion as vqn or vq index. For example, recent aq patch from Michael refers to aq vq index. Mmio has vq index register too for long time.
Quoted 3 lines
And some places we don't refer to index vs number. (pcie
queue_selectregister). We all understand that they refer to same.
But it is time to get consistency for new additions.
I totally agree.
I see that "\item [vqn] VQ number to be notified." is currently present in \section{Driver Notifications}, and also
"When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
The driver sends an available buffer notification to the device by writing the 16-bit virtqueue index”
These all seem to exist at the same time, and I think we should indeed unify.
Quoted 13 lines
So lets agree on either one of the naming convention.
Michael,
I am inclined to name it as vq index given existings references and with your aq vq index patch. Do you have preference?
If its index, above should be changed from vqn/vq_index.Quoted 6 lines
+\item
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
+ a enabled transmit/receive virtqueue whose number is
\field{vqn}, and then
+ responds them to the driver.
\end{enumerate}
Quoted 4 lines
It is confusing line of "device gets".
It is better to write it as,
The device returnsmax_packetsandmax_usecsfor the enabled transmit or receive virtuque whose index isvq_index.
Ok. Will modify.
Quoted 17 lines
Quoted 17 lines
+If the
VIRTIO_NET_F_VQ_NOTF_COALfeature is negotiated:
+1. a driver can send aVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand to set the coalescing
+ parameters of a enabled transmit/receive virtqueue.
+2. a driver can send aVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand to a device, and the device
+ responds to the driver with the coalescing parameters of a enabled
transmit/receive virtqueue.content.tex +8
-0+ +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +};
Quoted 8 lines
Quoted 6 lines
+\end{lstlisting}
content.tex +4
-0+ +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receivevirtqueue, excluding the control virtqueue.
Transmit and receive vq is self-explanatory. No need to mention control vq. The word "excluding" doesn't fit here well given it refers to only single vq at a time.
Ok. Will remove.
Quoted 9 lines
Quoted 9 lines
+\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
+\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize}content.tex +4
-0+ +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
Quoted 8 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4565
Quoted 6 lines
content.tex +6
-0+The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} +No need to provide the math here, it is about enabled vq.
The spec somewhere else already describes what the valid index of enabled vq.
I saw this:
"
\begin{description}
\item[0] receiveq1
\item[1] transmitq1
\item[\ldots]
\item[2(N-1)] receiveqN
\item[2(N-1)+1] transmitqN
\item[2N]controlq
\end{description}
"
, it seems that simply giving the conversion formula here is also helpful for readers to understand?
Quoted 13 lines
Quoted 11 lines
(quoted diff) +2
-0+The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for eachvirtqueue of receiveq1\ldots receiveqN.
(quoted diff) +3
-0+ +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for eachvirtqueue of transmitq1\ldots transmitqN.
(quoted diff) +3
-0+ +If coalescing parameters are being set, the device applies the last +coalescing parameters received for a virtqueue, regardless of the commandQuoted 1 line
used to deliver the parameters. For example:
used to set the parameters.
Ok.
Quoted 65 lines
Quoted 65 lines
+# Command sequence 1:
+1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETcommand with $ \field{max_packets}
+= 15 $ and $ \field{max_usecs} = 10 $ 2. +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and$ \field{max_usecs} = 10 $.content.tex +3
-0+ +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $,$
+\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ 2. +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
+
\subparagraph{RX Notifications}\label{sec:Device Types / Network Device /
Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}If, for example:
(quoted diff) +0
-0@@ -4532,9 +4594,33 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi
\begin{itemize}
\item The device will count received packets until it accumulates 15, or until 10
usecs elapsed since the first one was received.
-\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +11
-0+ +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because itsvirtqueue number is 0.
+\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +4
-0+ +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize}
Quoted 5 lines
Quoted 4 lines
+The device gets the values of the \field{max_packets} and +\field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, and then responds them to the driver.
+The device returns the value of X and Y parameters for the supplied vq index.
Ok.
Quoted 46 lines
Quoted 46 lines
\subparagraph{TX Notifications}\label{sec:Device Types / Network Device /
Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}If, for example:
(quoted diff) +0
-0@@ -4550,14 +4636,50 @@ \subsubsection{ControlVirtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
\end{itemize}content.tex +11
-0+Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because itsvirtqueue number is 1.
+\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
+\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
+ the device will not send an used buffer notification as long as the
notifications are suppressed.
+\end{itemize}content.tex +8
-0+ +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and +\field{max_usecs} parameters from the transmitq1 whose virtqueue numberis 1, and then responds them to the driver.
+
\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types /
Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
Quoted 15 lines
Quoted 12 lines
-If the
VIRTIO_NET_F_NOTF_COALfeature has not been negotiated, the driver MUST NOT issueVIRTIO_NET_CTRL_NOTF_COALcommands.
+If neither theVIRTIO_NET_F_NOTF_COALnor theVIRTIO_NET_F_VQ_NOTF_COAL
+feature has not been negotiated, the driver MUST NOT issueVIRTIO_NET_CTRL_NOTF_COALcommands.(quoted diff) +3
-0+ +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST +set \field{vqn} to a legal value, that is, \field{vqn} points to an enabledtransmit/receive virtqueue.
(quoted diff) +2
-0+ +A driver MAY NOT get exactly the same value as the coalescing parameter itQuoted 2 lines
was set to, for example it MAY get a value that is a power of 2.
It MAY get a value that is close to power of 2 value of the previously set value.
Ok. This is in line with Michael's suggestion.
Quoted 5 lines
Quoted 5 lines
\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types /
Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
Quoted 7 lines
Quoted 3 lines
A device SHOULD respond to the
VIRTIO_NET_CTRL_NOTF_COALcommands
withVIRTIO_NET_ERRif it was not able to change the parameters.Silently dropping the command is not good, we cannot change past now.
But lets please fix the below one.
Please add the note around above text that,
Note:VQ_COALis exception to it, which must returnNET_ERRif device is unable to change the parameters.
Why are the corresponding behaviors of VIRTIO_NET_CTRL_NOTF_COAL and VIRTIO_NET_CTRL_VQ_NOTF_COAL_SET inconsistent in the same scenario?
VIRTIO_NET_CTRL_NOTF_COAL --> SHOULD, but VIRTIO_NET_CTRL_VQ_NOTF_COAL_SET --> MUST?
My guess is that when VIRTIO_NET_CTRL_NOTF_COAL is set for receiveq1\ldots receiveqN, if recieveq1 is not set successfully because it is disabled, all other receiveq2\ldots receiveqN are set successfully. At this time, the device can choose not to return an error?
Quoted 11 lines
Quoted 4 lines
(quoted diff) +2
-0+When a device receives a virtqueue command to set/get coalescing +parameters for a virtqueue with number \field{vqn}, if the virtqueue is notenabled, the device SHOULD respond to the command with
VIRTIO_NET_ERR. +Device MUST respond with an error on the object that doesn't exist. Please change it to MUST.
Please add below text. Need some wording change for above SHOULD.
A device MUST respond to the
VIRTIO_NET_CTRL_VQ_NOTF_COAL_GETand SET command with withVIRTIO_NET_ERRif it is not able to change the parameters.
When the device receives _VQ_NOTF_COAL_GET, it will not try to change the parameters. And seeing here, we seem to be able to avoid splitting the patch, because it seems redundant to say this description once for GET and SET?
Quoted 5 lines
Review
Quoted 4 lines
(quoted diff) +2
-0+A device MAY NOT set the coalescing parameter to the exact same value +as the one passed in by a driver.s/as the one passed in by a driver/as the one set by the driver
Ok.
Quoted 2 lines
For example, the value of \field{max_packets}
This is how I described it in the original v3 patch, did you miss this sentence?
Thanks. :)
Quoted 8 lines
Quoted 8 lines
set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
+
A device SHOULD NOT send used buffer notifications to the driver, if the
notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
Quoted 7 lines
Quoted 3 lines
Upon reset, a device MUST initialize all coalescing parameters to 0.
--
2.19.1.6.gb485710b---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Hi Heng,
Quoted 48 lines
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
content.tex +18
-2+ notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4549
Quoted 10 lines
content.tex +10
-0+ +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} +
Maybe we can use struct virtio_net_ctrl_coal inside struct virtio_net_ctrl_coal_vq instead of repeating max_usecs and max_packets?
I'm not sure if it would be confusing, what do you think?
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same for max_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?
Quoted 30 lines
content.tex +26
-0+THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4598
Quoted 6 lines
content.tex +2
-1@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed.
Both sentences seem similar to me.
Quoted 57 lines
content.tex +47
-0+\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4672
Quoted 10 lines
content.tex +7
-1-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
Maybe we can add here that the driver must ignore the values received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, if the device responds with VIRTIO_NET_ERR?
Quoted 13 lines
content.tex +6
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR. + +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.
Quoted 57 lines · Alvaro Karsz
Hi Heng,
Quoted 55 lines
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
content.tex +25
-2+ notifications coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs;
Quoted 9 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4549
Quoted 3 lines
content.tex +3
-0+}; +\end{lstlisting} +Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?
Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
};struct virtio_net_ctrl_coal_vq {
le16 vqn;
le16 reserved;
struct virtio_net_ctrl_coal coal;
} coal_vq;#2
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
le16 vqn; // if _F_VQ_NOTF_COAL is negotiated le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
};
If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
I think we should hear from Michael and Parav.
Quoted 50 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand with VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?Quoted 31 lines
content.tex +26
-0+THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
Quoted 7 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4598
Quoted 5 lines
content.tex +2
-1\begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed.Both sentences seem similar to me.
Yes, this is an irrelevant modification, I will remove this modification.
Quoted 59 lines
Quoted 59 lines
content.tex +47
-0+\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
Quoted 13 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4672
Quoted 9 lines
content.tex +7
-1-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.Maybe we can add here that the driver must ignore the values received from
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, if the device responds withVIRTIO_NET_ERR?
Good idea.
Thanks.
Quoted 13 lines
Quoted 13 lines
content.tex +6
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR. + +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
Quoted 5 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4685
Quoted 1 line
content.tex +0
-0Upon reset, a device MUST initialize all coalescing parameters to 0.---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 30 lines
Quoted 5 lines
Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;#2
structvirtio_net_ctrl_coal{
le32max_packets;
le32max_usecs;
le16 vqn; // if_F_VQ_NOTF_COALis negotiated le16 reserved; // if_F_VQ_NOTF_COALis negotiated
};
Quoted 5 lines
If it's #1, I think the format is a bit ugly, it's not semantic to use
coal_vqto send global commands when_F_VQ_NOTF_COALis not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, usingvirtio_net_ctrl_coalas a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.I think we should hear from Michael and Parav.
I meant #1.
We can see virtio_net_ctrl_coal as a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.
Quoted 11 lines
Quoted 11 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand with
Quoted 8 lines
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?
Any thoughts on this part?
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 61 lines · Alvaro Karsz
Quoted 30 lines
Quoted 5 lines
Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;#2
structvirtio_net_ctrl_coal{
le32max_packets;
le32max_usecs;
le16 vqn; // if_F_VQ_NOTF_COALis negotiated le16 reserved; // if_F_VQ_NOTF_COALis negotiated
};Quoted 5 lines
If it's #1, I think the format is a bit ugly, it's not semantic to use
coal_vqto send global commands when_F_VQ_NOTF_COALis not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, usingvirtio_net_ctrl_coalas a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.I think we should hear from Michael and Parav.
I meant #1.
We can seevirtio_net_ctrl_coalas a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.Quoted 11 lines
Quoted 11 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand withQuoted 8 lines
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?
Quoted 1 line
Any thoughts on this part?
Good idea, and if so, is there a good way to expose vqn to the interpretation of max_packets ?
#1
\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue.
\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
#2
\item \field{max_packets}: Maximum number of packets to receive/send before a RX/TX notification.
Thanks.
Quoted 62 lines · Heng Qi
Quoted 61 lines · Alvaro Karsz
Quoted 30 lines
Quoted 5 lines
Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;#2
structvirtio_net_ctrl_coal{
le32max_packets;
le32max_usecs;
le16 vqn; // if_F_VQ_NOTF_COALis negotiated le16 reserved; // if_F_VQ_NOTF_COALis negotiated
};Quoted 5 lines
If it's #1, I think the format is a bit ugly, it's not semantic to use
coal_vqto send global commands when_F_VQ_NOTF_COALis not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, usingvirtio_net_ctrl_coalas a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.I think we should hear from Michael and Parav.
I meant #1.
We can seevirtio_net_ctrl_coalas a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.Quoted 19 lines
Quoted 19 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand with VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?
Quoted 3 lines
Quoted 1 line
Any thoughts on this part?
Good idea, and if so, is there a good way to expose vqn to the interpretation of
max_packets?
not sure what you are asking here.
Quoted 3 lines
#1
\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue.
an enabled - 1st time you mention a virtqueue.
Quoted 6 lines
\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
#2
\item \field{max_packets}: Maximum number of packets to receive/send before a RX/TX notification.Thanks.
Quoted 33 lines · Alvaro Karsz
Quoted 33 lines
Quoted 5 lines
Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;#2
structvirtio_net_ctrl_coal{
le32max_packets;
le32max_usecs;
le16 vqn; // if_F_VQ_NOTF_COALis negotiated le16 reserved; // if_F_VQ_NOTF_COALis negotiated
};If it's #1, I think the format is a bit ugly, it's not semantic to use
coal_vqto send global commands when_F_VQ_NOTF_COALis not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, usingvirtio_net_ctrl_coalas a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
Quoted 7 lines
Quoted 2 lines
I think we should hear from Michael and Parav.
I meant #1.
We can seevirtio_net_ctrl_coalas a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.
Reusing virtio_net_ctrl_coal is a nice thought. Makes it a bit clearer these have exactly the same role.
Whether to put vqn first or last does not matter imho.
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand withQuoted 8 lines
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?
Quoted 1 line
Any thoughts on this part?
I think I agree. Generally I think we should first of all describe the new VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text to that.
Then just explain that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET have the same effect
as VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET repeated for
all currently enabled tx/rx vqs.
Plus maybe a single annotated example where there's a mix of VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands. For example with 2 vq pairs:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets for vq 0 and 2, vq 1 and 3 retain reset value
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 0, vq 2 retains value from 1
3. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 1, vq 3 retains reset value
4. VIRTIO_NET_CTRL_NOTF_COAL_TX_SET overrides command 3
no need for many examples.
Signature · 1 line
MST
Quoted 26 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Friday, February 17, 2023 5:07 AMQuoted 19 lines
Quoted 6 lines
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };Quoted 8 lines
struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;I think we should hear from Michael and Parav.
I meant #1.
We can seevirtio_net_ctrl_coalas a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.
+1 for #1.
Quoted 4 lines
Reusing
virtio_net_ctrl_coalis a nice thought. Makes it a bit clearer these have exactly the same role.
Whether to put vqn first or last does not matter imho.
Putting VQN first looks better to me, to say for vqn -> below are the parameters. We may find more parameters in future too in same struct.
At that point also having vqn first reads better.
Quoted 40 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
+Virtqueue coalescing parameters:
+\begin{itemize}
+\item \field{vqn}: The virtqueue number of the enabled transmit orreceive virtqueue, excluding the control virtqueue.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+\item \field{max_packets}: The maximum number of packets
sent/received by the specified virtqueue before a TX/RX notification.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+\item \field{max_usecs}: The maximum number of TX/RX usecs that
the specified virtqueue delays a TX/RX notification.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 4 lines
+\end{itemize}
content.tex +3
-0+ +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
with VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET. ("Maximum number of packets to receive/send before a RX/TX
notification").
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?Any thoughts on this part?
I think I agree. Generally I think we should first of all describe the new
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text to that.Then just explain that
VIRTIO_NET_CTRL_NOTF_COAL_TX_SETandVIRTIO_NET_CTRL_NOTF_COAL_RX_SEThave the same effect asVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETrepeated for all currently enabled tx/rx vqs.
Plus maybe a single annotated example where there's a mix ofVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands. For example with 2 vq pairs:
Quoted 6 lines
1.
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets for vq 0 and 2, vq 1 and 3 retain reset value 2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets vq 0, vq 2 retains value from 1 3.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets vq 1, vq 3 retains reset value 4.VIRTIO_NET_CTRL_NOTF_COAL_TX_SEToverrides command 3no need for many examples.
+1.
Quoted 74 lines · Michael S. Tsirkin
Quoted 32 lines · Alvaro Karsz
Quoted 32 lines
Quoted 5 lines
Maybe we can use struct
virtio_net_ctrl_coalinside structvirtio_net_ctrl_coal_vqinstead of repeatingmax_usecsandmax_packets?
I'm not sure if it would be confusing, what do you think?Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal { le32 max_packets; le32 max_usecs; };struct virtio_net_ctrl_coal_vq { le16 vqn; le16 reserved; struct virtio_net_ctrl_coal coal; } coal_vq;#2
structvirtio_net_ctrl_coal{
le32max_packets;
le32max_usecs;
le16 vqn; // if_F_VQ_NOTF_COALis negotiated le16 reserved; // if_F_VQ_NOTF_COALis negotiated
};If it's #1, I think the format is a bit ugly, it's not semantic to use
coal_vqto send global commands when_F_VQ_NOTF_COALis not negotiated, and the presence of vqn and reserved is awkward. If it's #2, I think this is a bit like the v1 version, usingvirtio_net_ctrl_coalas a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.Quoted 6 lines
Quoted 2 lines
I think we should hear from Michael and Parav.
I meant #1.
We can seevirtio_net_ctrl_coalas a struct holding coalescing parameters, regardless of the commands.
Yes, let's wait for more comments on that.Reusing
virtio_net_ctrl_coalis a nice thought. Makes it a bit clearer these have exactly the same role.
Whether to put vqn first or last does not matter imho.Quoted 18 lines
Quoted 10 lines
Quoted 10 lines
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4558
Quoted 9 lines
content.tex +9
-0+Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. +
max_packetsis the same withVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETand withQuoted 8 lines
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification"). The fact that this is applied to all VQs or to a specific one is derived from the command.
Same formax_usecs.
Maybe we can join the coalescing parameters somehow instead of repeating the explanations?Quoted 1 line
Any thoughts on this part?
I think I agree. Generally I think we should first of all describe the new
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text to that.Then just explain that
VIRTIO_NET_CTRL_NOTF_COAL_TX_SETandVIRTIO_NET_CTRL_NOTF_COAL_RX_SEThave the same effect
asVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETrepeated for
all currently enabled tx/rx vqs.
Plus maybe a single annotated example where there's a mix ofVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands. For example with 2 vq pairs:
Quoted 6 lines
1.
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets for vq 0 and 2, vq 1 and 3 retain reset value
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets vq 0, vq 2 retains value from 1
3.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets vq 1, vq 3 retains reset value
4.VIRTIO_NET_CTRL_NOTF_COAL_TX_SEToverrides command 3no need for many examples.
Good idea. This is a clear and comprehensive example.
Thanks.
Quoted 1 line
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 38 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +0
-0
content.tex \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits · line 3088
Quoted 5 lines
content.tex +2
-0@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing.
btw no "the" here. 1st time we mention it. and notification not notifications - reduced relative here, uses the singular form.
Quoted 183 lines
content.tex +124
-4+ \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR. + +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.Signature · 1 line
2.19.1.6.gb485710b
Quoted 42 lines · Michael S. Tsirkin
Quoted 41 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszcontent.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 126 insertions(+), 4 deletions(-)content.tex +0
-0@@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel.
Quoted 5 lines
content.tex \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits · line 3088
Quoted 2 lines
content.tex +2
-0+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue + notifications coalescing.btw no "the" here. 1st time we mention it. and notification not notifications - reduced relative here, uses the singular form.
Will fix.
Thanks.
Quoted 200 lines
Quoted 184 lines
content.tex +124
-4+ \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi }; #define VIRTIO_NET_CTRL_NOTF_COAL 6 - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 + \end{lstlisting} Coalescing parameters:@@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters. \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then + responds them to the driver. \end{enumerate} +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of a enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + le32 max_packets; + le32 max_usecs; +}; +\end{lstlisting} + +Virtqueue coalescing parameters: +\begin{itemize} +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue. +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification. +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification. +\end{itemize} + +\field{reserved} is reserved and it is ignored by the device. + +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. +The conversion between \field{vqn} and transmitq/receiveq index: +\begin{itemize} +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq. +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq. +\end{itemize} + +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of receiveq1\ldots receiveqN. + +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET +command repeated for each virtqueue of transmitq1\ldots transmitqN. + +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to deliver the parameters. For example: +# Command sequence 1: +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + +# Command sequence 2: +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $ +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $. + \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications} If, for example:@@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \begin{itemize} \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received. -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of setting coalescing parameters for a receive virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 0 +\end{itemize} + +The device will operate as follows: +\begin{itemize} +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0. +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a receive virtqueue: +\begin{itemize} +\item \field{vqn} = 0. \end{itemize} +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0, +and then responds them to the driver. + \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications} If, for example:@@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed. \end{itemize} +Example of setting coalescing parameters for a transmit virtqueue: +\begin{itemize} +\item \field{max_packets} = 15 +\item \field{max_usecs} = 10 +\item \field{vqn} = 1 +\end{itemize} + +The device will operate as follows: + +\begin{itemize} +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1. +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent. +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, + the device will not send an used buffer notification as long as the notifications are suppressed. +\end{itemize} + +Example of getting coalescing parameters from a transmit virtqueue: +\begin{itemize} +\item \field{vqn} = 1. +\end{itemize} + +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1, +and then responds them to the driver. + \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn} +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue. + +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn}, +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR. + +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example, +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16. + A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired. Upon reset, a device MUST initialize all coalescing parameters to 0.Signature · 1 line
2.19.1.6.gb485710bThis publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/