virtio-net: support the virtqueue coalescing moderation
PATCH v4 1/1 virtio-net: support the virtqueue coalescing moderation
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
device-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +60 -4
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.
@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; + #define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters. A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and: +1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET: +after the virtqueue is enabled, its parameters SHOULD be set to the global values. +2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0. + +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 1 line
2.19.1.6.gb485710b
Quoted 76 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536
Quoted 9 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +
2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET some fields are output others are input. This is unusual for commands and needs to be documented very clearly.
Quoted 50 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 6 lines
device-types/net/description.tex +2
-1@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
Triple negative? I think you mean "has been negotiated".
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.
device does not return. It responds to the command.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 5 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.
I don't get it. what is exception to what and what's the difference, what does "which" refer to what return VIRTIO_NET_ERR where.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 4 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:
parameters here meanong coalescing parameters? It won't be clear to readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:
"a" does not go with the plural "values".
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.
this is the first you mention global.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.
Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.
Quoted 11 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710b
Quoted 90 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536
Quoted 9 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Quoted 3 lines
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. "
Quoted 52 lines
Quoted 52 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
Quoted 7 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 5 lines
device-types/net/description.tex +2
-1\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 3 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 4 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 10 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 6 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respond VIRTIO_NET_ERR for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 4 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 5 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 2 lines
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 14 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 4 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.
Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values. "
Quoted 14 lines
Quoted 14 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710b
Quoted 91 lines · Heng Qi
Quoted 90 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536
Quoted 9 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Quoted 8 lines
Quoted 3 lines
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use the
I think it's better to just say VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.
Quoted 11 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "
Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
Quoted 88 lines
Quoted 52 lines
Quoted 52 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / DeviQuoted 7 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 5 lines
device-types/net/description.tex +2
-1\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 3 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 4 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 10 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 6 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.
Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
So document the two types separately. Change VIRTIO_NET_CTRL_NOTF_COAL to VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence for VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
And VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET does not change anything so I have no idea why mention it at all.
Quoted 49 lines
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 4 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 5 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 2 lines
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 14 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 4 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.
Quoted 8 lines
Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.
ok more or less. is this in normative section? If not avoid MUST.
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 16 lines
"
Quoted 14 lines
Quoted 14 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710b
Quoted 94 lines · Michael S. Tsirkin
Quoted 93 lines · Heng Qi
Quoted 92 lines · Michael S. Tsirkin
Quoted 76 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1536
Quoted 9 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.
Quoted 7 lines
Quoted 4 lines
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use theI think it's better to just say
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETand
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.
Ok, I'll modify this.
Quoted 8 lines
Quoted 8 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.
Quoted 8 lines
Quoted 4 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :
"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used by VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands. \field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. \field{reserved} is a reserved field containing 0, which is ignored by a device and write-only for a driver.
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands use the virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.
The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} in virtio_net_ctrl_coal_vq. At this time, for a driver, coalescing parameters are read-only and \field{vqn} is write-only. "
Do you think this is a good description?
Quoted 84 lines
Quoted 77 lines
Quoted 51 lines
Quoted 51 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / DeviQuoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 5 lines
device-types/net/description.tex +2
-1\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 2 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 5 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
Quoted 3 lines
So document the two types separately. Change
VIRTIO_NET_CTRL_NOTF_COALtoVIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence forVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
Ok.
Quoted 3 lines
And
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETdoes not change anything so I have no idea why mention it at all.
Ah, you mean the scope of MUST is included in SHOULD? If you think it's better to remove the *Note* above, I'm fine.
Quoted 44 lines
Quoted 44 lines
Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 7 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 4 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 3 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.Thanks for your example, I tried to regularize it, do you think it works?
Quoted 7 lines
Quoted 6 lines
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.ok more or less. is this in normative section? If not avoid MUST.
Yes. It's in normative section. Thanks for your comments!
Quoted 17 lines
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 12 lines
"
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
device-types/net/description.tex +0
-0\subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710b
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 122 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535
Quoted 8 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.Quoted 7 lines
Quoted 4 lines
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use theI think it's better to just say
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETand
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.Ok, I'll modify this.
Quoted 8 lines
Quoted 8 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.Quoted 8 lines
Quoted 4 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :
Quoted 6 lines
"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used byVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETandVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommands.
ugh just confusing. just use the struct names.
Quoted 4 lines
\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn <
max_virtqueue_pairs\ast 2 $. \field{reserved} is a reserved field containing 0, which is ignored by a device and write-only for a driver.
how important is it that devices ignore it?
if yes it needs a normative statement.
Quoted 6 lines
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands use thevirtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.
Quoted 8 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. At this time,
at what time?
Quoted 2 lines
for a driver, coalescing parameters are read-only and \field{vqn} is write-only.
reserved also write-only.
Quoted 2 lines
"
Do you think this is a good description?
No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.
for command X fields A B are write-only for driver and fields C D are read-only for driver.
for command Y fields A B C and D are write-only for driver.
Quoted 80 lines
Quoted 76 lines
Quoted 69 lines
Quoted 43 lines
Quoted 43 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} +Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626
Quoted 7 lines
device-types/net/description.tex +2
-1\subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 2 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 3 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Quoted 3 lines
So document the two types separately. Change
VIRTIO_NET_CTRL_NOTF_COALtoVIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence forVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.
Quoted 7 lines
Ok.
Quoted 3 lines
And
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETdoes not change anything so I have no idea why mention it at all.Ah, you mean the scope of MUST is included in SHOULD?
No I mean just list commands explicitly. Do not make reader guess what does VIRTIO_NET_CTRL_NOTF_COAL mean when there is no command like this.
Quoted 50 lines
If you think it's better to remove the *Note* above, I'm fine.
Quoted 48 lines
Quoted 48 lines
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 5 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 4 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 3 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.
Quoted 3 lines
Quoted 1 line
ok more or less. is this in normative section? If not avoid MUST.
Yes. It's in normative section. Thanks for your comments!
then pls also add a non normative explanatory informal description.
Quoted 19 lines
Quoted 19 lines
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 14 lines
"
Quoted 12 lines
Quoted 12 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710b
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535
Quoted 8 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.Quoted 7 lines
Quoted 4 lines
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use theI think it's better to just say
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETand
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.Ok, I'll modify this.
Quoted 7 lines
Quoted 7 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.Quoted 7 lines
Quoted 4 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :
Quoted 7 lines
Quoted 6 lines
"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used byVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETandVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommands.ugh just confusing. just use the struct names.
Ok.
Quoted 7 lines
Quoted 5 lines
\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn <
max_virtqueue_pairs\ast 2 $.
\field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.how important is it that devices ignore it?
if yes it needs a normative statement.
Maybe "containing 0" is enough.
Quoted 27 lines
Quoted 6 lines
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands use thevirtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.Quoted 8 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. At this time,at what time?
Quoted 2 lines
for a driver, coalescing parameters are read-only and \field{vqn} is write-only.
reserved also write-only.
Quoted 2 lines
"
Do you think this is a good description?No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.
Quoted 4 lines
for command X fields A B are write-only for driver and fields C D are read-only for driver.
for command Y fields A B C and D are write-only for driver.
I think it's a better proposition! I'll modify.
Quoted 87 lines
Quoted 87 lines
Quoted 82 lines
Quoted 75 lines
Quoted 49 lines
Quoted 49 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} +Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626
Quoted 7 lines
device-types/net/description.tex +2
-1\subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 2 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1631
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 3 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Quoted 3 lines
So document the two types separately. Change
VIRTIO_NET_CTRL_NOTF_COALtoVIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence forVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Ok.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
And
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETdoes not change anything so I have no idea why mention it at all.Ah, you mean the scope of MUST is included in SHOULD?
No I mean just list commands explicitly. Do not make reader guess what does
VIRTIO_NET_CTRL_NOTF_COALmean when there is no command like this.
Ok, I see. I will follow through on this deeply.
Quoted 48 lines
Quoted 48 lines
If you think it's better to remove the *Note* above, I'm fine.
Quoted 45 lines
Quoted 45 lines
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1640
Quoted 4 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 4 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1641
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 2 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1642
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
ok more or less. is this in normative section? If not avoid MUST.
Yes. It's in normative section. Thanks for your comments!
then pls also add a non normative explanatory informal description.
I'll add the description in the next version.
Thanks for your comments!
Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 3 lines
"
Quoted 1 line
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710bThis publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 131 lines · Heng Qi
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535
Quoted 8 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.Quoted 7 lines
Quoted 4 lines
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use theI think it's better to just say
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETand
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.Ok, I'll modify this.
Quoted 7 lines
Quoted 7 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.Quoted 7 lines
Quoted 4 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :
Quoted 7 lines
Quoted 6 lines
"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used byVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETandVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommands.ugh just confusing. just use the struct names.
Ok.
Quoted 1 line
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn <
max_virtqueue_pairs\ast 2 $.
\field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.how important is it that devices ignore it?
if yes it needs a normative statement.Maybe "containing 0" is enough.
it is better to specify what's going on from driver and device perspective. If device does not need to ignore it, you can just make it a 32 bit field.
Quoted 221 lines
Quoted 27 lines
Quoted 6 lines
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands use thevirtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.Quoted 8 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. At this time,at what time?
Quoted 2 lines
for a driver, coalescing parameters are read-only and \field{vqn} is write-only.
reserved also write-only.
Quoted 2 lines
"
Do you think this is a good description?No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.Quoted 4 lines
for command X fields A B are write-only for driver and fields C D are read-only for driver.
for command Y fields A B C and D are write-only for driver.
I think it's a better proposition! I'll modify.
Quoted 87 lines
Quoted 87 lines
Quoted 82 lines
Quoted 75 lines
Quoted 49 lines
Quoted 49 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} +Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626
Quoted 7 lines
device-types/net/description.tex +2
-1\subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 2 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1629
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1632
Quoted 3 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Quoted 3 lines
So document the two types separately. Change
VIRTIO_NET_CTRL_NOTF_COALtoVIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence forVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Ok.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
And
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETdoes not change anything so I have no idea why mention it at all.Ah, you mean the scope of MUST is included in SHOULD?
No I mean just list commands explicitly. Do not make reader guess what does
VIRTIO_NET_CTRL_NOTF_COALmean when there is no command like this.Ok, I see. I will follow through on this deeply.
Quoted 48 lines
Quoted 48 lines
If you think it's better to remove the *Note* above, I'm fine.
Quoted 45 lines
Quoted 45 lines
Quoted 5 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1634
Quoted 4 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 4 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1635
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 2 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1639
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.Quoted 3 lines
Quoted 2 lines
Quoted 1 line
ok more or less. is this in normative section? If not avoid MUST.
Yes. It's in normative section. Thanks for your comments!
then pls also add a non normative explanatory informal description.
I'll add the description in the next version.
Thanks for your comments!
Quoted 34 lines
Quoted 8 lines
Quoted 8 lines
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 3 lines
"
Quoted 1 line
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710bThis publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 133 lines · Michael S. Tsirkin
Quoted 132 lines · Heng Qi
Quoted 118 lines · Michael S. Tsirkin
Quoted 117 lines · Heng Qi
Quoted 89 lines · Michael S. Tsirkin
Quoted 88 lines · Heng Qi
Quoted 87 lines · Michael S. Tsirkin
Quoted 72 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)
device-types/net/description.tex +9
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive virtqueue. + \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1535
Quoted 8 lines
device-types/net/description.tex +6
-0@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; +2 structures now, what are they doing? which command uses which? reader is supposed to guess and that's not good.
Also, with
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETsome fields are output others are input. This is unusual for commands and needs to be documented very clearly.Quoted 7 lines
Quoted 4 lines
Yes, this should be made clear. How do you think of the following descriptions:
"
The VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET commands use theI think it's better to just say
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETand
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET
We don't document the {} syntax anywhere.Ok, I'll modify this.
Quoted 7 lines
Quoted 7 lines
virtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.Quoted 7 lines
Quoted 4 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified
virtqueue from the device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. "Not enough imho. Document which fields are write-only for driver and which are read-only for driver. and includes reserved field not just vqn.
OK I rephrase it into following sentences (Combined with the following, \field{vqn}'s and\field{reserved}'s explanations may be a bit redundant, but I will merge them in the next version ) :
Quoted 7 lines
Quoted 6 lines
"
\field{max_usecs} and \field{max_packets} are collectively known as coalescing parameters.
\field{vqn} and \field{reserved} are only used byVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETandVIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommands.ugh just confusing. just use the struct names.
Ok.
Quoted 4 lines
Quoted 4 lines
\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn <
max_virtqueue_pairs\ast 2 $.
\field{reserved} is a reserved field containing 0, which is ignored by a
device and write-only for a driver.
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
how important is it that devices ignore it?
if yes it needs a normative statement.Maybe "containing 0" is enough.
it is better to specify what's going on from driver and device perspective. If device does not need to ignore it, you can just make it a 32 bit field.
Ok, I see. :)
Thanks.
Quoted 214 lines
Quoted 172 lines
Quoted 26 lines
Quoted 5 lines
VIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SETcommands use thevirtio_net_ctrl_coalstructure to set \field{max_usecs} and \field{max_packets} for all transmit/receive virtqueues, while coalescing parameters are write-only for a driver.Quoted 8 lines
The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand uses thevirtio_net_ctrl_coal_vqstructure to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}, while coalescing parameters and \field{vqn} are write-only for a driver.The
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETcommand gets the values of \field{max_usecs} and \field{max_packets} of the specified virtqueue from a device by only setting \field{vqn} invirtio_net_ctrl_coal_vq. At this time,at what time?
Quoted 2 lines
for a driver, coalescing parameters are read-only and \field{vqn} is write-only.
reserved also write-only.
Quoted 2 lines
"
Do you think this is a good description?No. I think you should take your previous one and explain:
One way:
commands X and Y use structure Z in command specific data.Quoted 4 lines
for command X fields A B are write-only for driver and fields C D are read-only for driver.
for command Y fields A B C and D are write-only for driver.
I think it's a better proposition! I'll modify.
Quoted 84 lines
Quoted 84 lines
Quoted 84 lines
Quoted 74 lines
Quoted 48 lines
Quoted 48 lines
device-types/net/description.tex +27
-3#define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive + virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2. +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5. +\end{itemize} +Quoted 8 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1626
Quoted 7 lines
device-types/net/description.tex +2
-1\subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Triple negative? I think you mean "has been negotiated".
Yes. Thanks for pointing it out.
Quoted 2 lines
Generally we should consider rewriting these in a positive way but that's a subject for another patch.
Ok, I see.
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1629
Quoted 3 lines
device-types/net/description.tex +3
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, +if the device returns with VIRTIO_NET_ERR.device does not return. It responds to the command.
Ok.
Quoted 6 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1632
Quoted 3 lines
device-types/net/description.tex +1
-0\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.I don't get it. what is exception to what and what's the difference, what does "which" refer to what return
VIRTIO_NET_ERRwhere.Quoted 7 lines
Sorry for not describing clearly, please see the explanation below:
When a device receives VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET respectively,
and the device was not able to change the parameters, then it behaves differently for VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} and VIRTIO_NET_CTRL_NOTF_COAL_{TX, RX}_SET, ie SHOULD and MUST. For example:
1.VIRTIO_NET_CTRL_NOTF_COAL_RX_SETsets parameters for all transmit/receive virtqueues, but if only receiveq1 fails to be set, and receiveq2\ldots receiveqN are set successfully, then the device *SHOULD* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_RX_SET.
2.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets parameters for the virtqueue whose number is \field{vqn}, if the device fails to set, the device *MUST* respondVIRTIO_NET_ERRforVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.So document the two types separately. Change
VIRTIO_NET_CTRL_NOTF_COALtoVIRTIO_NET_CTRL_NOTF_COAL_RX_SETandVIRTIO_NET_CTRL_NOTF_COAL_TX_SET, and add a new sentence forVIRTIO_NET_CTRL_NOTF_COAL_VQ_SET.Quoted 8 lines
Quoted 5 lines
Ok.
Quoted 2 lines
And
VIRTIO_NET_CTRL_NOTF_COAL_VQ_GETdoes not change anything so I have no idea why mention it at all.Ah, you mean the scope of MUST is included in SHOULD?
No I mean just list commands explicitly. Do not make reader guess what does
VIRTIO_NET_CTRL_NOTF_COALmean when there is no command like this.Ok, I see. I will follow through on this deeply.
Quoted 46 lines
Quoted 46 lines
If you think it's better to remove the *Note* above, I'm fine.
Quoted 44 lines
Quoted 44 lines
Quoted 4 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1634
Quoted 3 lines
device-types/net/description.tex +1
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and:parameters here meanong coalescing parameters? It won't be clear to
Yes, I'll write clearly.
Quoted 4 lines
readers who see the full spec not the patch.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1635
Quoted 1 line
device-types/net/description.tex +1
-0+1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET:"a" does not go with the plural "values".
Ok.
Quoted 2 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 1 line
device-types/net/description.tex +1
-0+after the virtqueue is enabled, its parameters SHOULD be set to the global values.this is the first you mention global.
I mentioned "global" in the first sentence, but that might not seem very clear.
Quoted 12 lines
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1639
Quoted 3 lines
device-types/net/description.tex +3
-0+2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Sentence too long at this point I can't say what happens after what.
I feel this enumeration of cases is confusing. Would be better to find the logic behind all this and document.
For example, we could explain that
Quoted 8 lines
VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET set
global values for coalescing parameters for RX and TX respectively and apply these global values to all VQs of the given type. Global values after device reset are 0.VIRTIO_NET_CTRL_NOTF_COAL_VQ_SETsets the coalescing parameters for a given enabled VQ without changing the global values.
Fails on a disabled VQ. Disabling and
re-enabling a VQ reverts its coalescing parameters back to global values.Thanks for your example, I tried to regularize it, do you think it works?
"
The VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands set coalescing parameters for all transmit/receive
virtqueues respectively and values of these coalescing parameters are recorded as *global values* by the device.
The device MUST set the global values of coalescing parameters to 0 after being reset.
TheVIRTIO_NET_CTRL_NOTF_COAL_VQ_SETcommand sets the coalescing parameters for a given enabled virtqueue without changing the global values. After disabling and re-enabling a virtqueue, the deivce MUST revert coalescing parameters of the virtqueue back to global values.Quoted 38 lines
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
ok more or less. is this in normative section? If not avoid MUST.
Yes. It's in normative section. Thanks for your comments!
then pls also add a non normative explanatory informal description.
I'll add the description in the next version.
Thanks for your comments!
Quoted 31 lines
Quoted 6 lines
Quoted 6 lines
And maybe add text in a normative section too. a bit of repetition there is ok.
Quoted 2 lines
"
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:DeviceSignature · 1 line
2.19.1.6.gb485710bThis publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Quoted 3 lines · Heng Qi
Quoted 1 line
then pls also add a non normative explanatory informal description.
I'll add the description in the next version.
Try to pass it through a grammar checker pls ;)
Quoted 71 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +8
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receive
Quoted 1 line
virtqueue.
Use an enumeration.
Quoted 40 lines
device-types/net/description.tex +16
-3+ \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; + #define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1569
Quoted 10 lines
device-types/net/description.tex +8
-0+ virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece
"sequence"
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1574
Quoted 5 lines
device-types/net/description.tex +5
-0+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.
Missing space before =.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1577
Quoted 3 lines
device-types/net/description.tex +3
-0+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.
Missing space before =.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 12 lines
device-types/net/description.tex +4
-1+\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
Too many negatives here. Just "have been negotiated".
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1630
Quoted 2 lines
device-types/net/description.tex +2
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,
"parameters"
Remove the comma?
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 6 lines
device-types/net/description.tex +2
-0+if the device returns with VIRTIO_NET_ERR. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters.
"an exception"
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 9 lines
device-types/net/description.tex +6
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and: +1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET: +after the virtqueue is enabled, its parameters SHOULD be set to the global values. +2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.
Use an enumeration.
Quoted 9 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
Signature · 17 lines
We're deep in discussion, the party's on mute. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
Hi David, thanks for pointing out the problem, I will fix it in the next version.
Thanks very much! :)
Quoted 187 lines · David Edmondson
Quoted 71 lines · Heng Qi
Currently coalescing parameters are grouped for all transmit and receive virtqueues. This patch supports setting or getting the parameters for a specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue is busy and another virtqueue is idle, then it will be very useful to control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by Heng Qi <[email protected]>Reviewed-by Xuan Zhuo <[email protected]>---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v3->v4:
1. Includevirtio_net_ctrl_coalin thevirtio_net_ctrl_coal_vqstructure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2.VIRTIO_NET_F_VQ_NOTF_COALno longer depends onVIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3._VQ_GETis explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust thevirtio_net_ctrl_coal_vqstructure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. RenameVIRTIO_NET_F_PERQUEUE_NOTF_COALtoVIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structurevirtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control commandVIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value0xFFFis removed becauseVIRTIO_NET_CTRL_NOTF_COALcan be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karszdevice-types/net/description.tex | 64 ++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-)device-types/net/description.tex +8
-0@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. + \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \end{description} \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}@@ -1508,6 +1511,12 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can send control commands for dynamically changing the coalescing parameters. +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing + parameters of an enabled transmit/receive virtqueue. +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device + returns the coalescing parameters of an enabled transmit/receiveQuoted 1 line
virtqueue.
Use an enumeration.
Quoted 40 lines
device-types/net/description.tex +16
-3+ \begin{note} The behavior of the device in response to these commands is best-effort: the device may generate notifications more or less frequently than specified.@@ -1519,25 +1528,55 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi le32 max_usecs; }; +struct virtio_net_ctrl_coal_vq { + le16 vqn; + le16 reserved; + struct virtio_net_ctrl_coal coal; +}; + #define VIRTIO_NET_CTRL_NOTF_COAL 6 #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 \end{lstlisting} Coalescing parameters: \begin{itemize} +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. \end{itemize} -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. + +\field{reserved} is reserved and it is ignored by the device. + +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: \begin{enumerate} -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receivedevice-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1569
Quoted 10 lines
device-types/net/description.tex +8
-0+ virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled + transmit/receive virtqueue whose number is \field{vqn}. +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. \end{enumerate} +If coalescing parameters are being set, the device applies the last coalescing parameters received for a +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: +# Command sequece"sequence"
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1574
Quoted 5 lines
device-types/net/description.tex +5
-0+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. +\begin{itemize} +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets parameters for virtqueue0, and virtqueue2 retains the values from command1. +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 0, the device returns paramters of virtqueue0 set by command2.Missing space before =.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1577
Quoted 3 lines
device-types/net/description.tex +3
-0+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets parameters for virtqueue1, and virtqueue3 retains its previous values. +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn}= 1, the device returns paramters of virtqueue1 set by command5.Missing space before =.
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1628
Quoted 12 lines
device-types/net/description.tex +4
-1+\end{itemize} + \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.@@ -1585,14 +1624,31 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.Too many negatives here. Just "have been negotiated".
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1630
Quoted 2 lines
device-types/net/description.tex +2
-0+ +A driver MUST ignore the values of coalescing paramters received from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET,"parameters"
Remove the comma?
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1636
Quoted 6 lines
device-types/net/description.tex +2
-0+if the device returns with VIRTIO_NET_ERR. \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. +Note: VIRTIO_NET_CTRL_NOTF_COAL_VQ_{SET, GET} are exception to it, which must return VIRTIO_NET_ERR if the device was unable to change the parameters."an exception"
device-types/net/description.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 1645
Quoted 9 lines
device-types/net/description.tex +6
-0A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. +If a virtqueue is disabled, the device MUST clear its parameters configuration, and: +1. if the device has received a global parameter values sent by VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET: +after the virtqueue is enabled, its parameters SHOULD be set to the global values. +2. if the device hasn't received the VIRTIO_NET_CTRL_NOTF_{RX, TX}_SET commands: +regardless of whether the device has received a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, after the +virtqueue is enabled, its parameters SHOULD be set to 0.Use an enumeration.
Quoted 9 lines
device-types/net/description.tex +6
-0+ +When a device receives a virtqueue command to set or return coalescing parameters for the supplied \field{vqn}, +if the virtqueue is not enabled, the device MUST respond to the command with VIRTIO_NET_ERR. + +A device MAY set the coalescing parameter to a value close to a power of 2 value. + Upon reset, a device MUST initialize all coalescing parameters to 0. \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device