Introduction of Virtio Network device notifications coalescing feature.
PATCH v3 1/1 Introduction of Virtio Network device notifications coalescing feature.
Control a network device notifications coalescing parameters using the control virtqueue. A new control class was added: VIRTIO_NET_CTRL_NOTF_COAL.
This class provides 2 commands:
- VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET:
Ask the network device to change the rx-usecs and tx-usecs parameters. rx-usecs - Time to delay an RX notification after packet arrival in microseconds. tx-usecs - Time to delay a TX notification after a sending a packet in microseconds.
- VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET:
Ask the network device to change the rx-max-frames and tx-max-frames parameters. rx-max-frames - Number of packets to delay an RX notification after packet arrival. tx-max-frames - Number of packets to delay a TX notification after sending a packet.
--
v2:
- Usage of notification terminology.
- Add a few lines on what device should do when driver asked to suppress notifications.
v3:
- Remove whitespaces.
- Explain with examples how the device should act.
--
--- content.tex | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+)
content.tex +68 -0
@@ -3084,6 +3084,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_NOTF_COAL(55)] Device supports notifications coalescing. + \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO (fragmenting the packet) the USO splits large UDP packet to several segments when each of these smaller packets has UDP header.
@@ -3129,6 +3131,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. \end{description}
@@ -4464,6 +4467,71 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi (necessarily when not using the legacy interface) little-endian. +\paragraph{Notifications Coalescing}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} + +If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can +send control commands for dynamically changing the coalescing parameters. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_usec { + le32 tx_usecs; + le32 rx_usecs; +}; + +struct virtio_net_ctrl_coal_frames { + le32 tx_frames_max; + le32 rx_frames_max; +}; + +#define VIRTIO_NET_CTRL_NOTF_COAL 6 + #define VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET 1 +\end{lstlisting} + +The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\begin{itemize} +\item VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET: set the rx-usecs (time to delay an RX notification after frame arrival in microseconds) and tx-usecs (time to delay a TX notification after a sending a frame in microseconds) parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET: set the rx-max-frames (number of frames to delay an RX notification after frame arrival) and tx-max-frames (number of frames to delay a TX notification after sending a frame) parameters. +\end{itemize} + +\paragraph{RX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX notifications} + +If, for example, rx_usecs = 10 and rx_frames_max = 15. + +\begin{itemize} +\item The device will count received frames until it accumulates 15 frames, or until 10 usecs elapsed since the arrival of the first frame. +\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\ +An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters. +\item The device will send a notification to the driver only if the driver hasn't suppressed notifications. +\end{itemize} + + +\paragraph{TX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX notifications} + +If, for example, tx_usecs = 10 and tx_frames_max = 15. + +\begin{itemize} +\item The device will count sent frames until it accumulates 15 frames, or until 10 usecs elapsed since first frame was sent. +\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\ +An example is if any of the VIRTIO_NET_F_HOST_TSO/UFO features are negotiated, a device could receive a big buffer, which will take more than 15 frames to send. +\item The device will reset its internal coalescing counters. +\item The device will send a notification to the driver only if the driver hasn't suppressed notifications. +\end{itemize} + +\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} + + +If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} + +A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if was not able to change the parameters.\\ \\ +A device MUST NOT accept tx_frames_max/rx_frames_max values bigger than the queue size.\\ \\ +A device SHOULD NOT send notifications to the driver, if the driver asked to suppress notifications.\\ \\ +A device SHOULD initialize all coalescing values to 0. \\ \\ + + \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device Types / Network Device / Legacy Interface: Framing Requirements}
Signature · 18 lines
2.32.0 This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: [email protected] Unsubscribe: [email protected] List help: [email protected] List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/
If there are no comments, I would like to vote on this feature.
https://github.com/oasis-tcs/virtio-spec/issues/141
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 77 lines · Alvaro Karsz
Control a network device notifications coalescing parameters using the control virtqueue. A new control class was added:
VIRTIO_NET_CTRL_NOTF_COAL.This class provides 2 commands:
-VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET:
Ask the network device to change the rx-usecs and tx-usecs parameters. rx-usecs - Time to delay an RX notification after packet arrival in microseconds. tx-usecs - Time to delay a TX notification after a sending a packet in microseconds.-
VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET:
Ask the network device to change the rx-max-frames and tx-max-frames parameters. rx-max-frames - Number of packets to delay an RX notification after packet arrival. tx-max-frames - Number of packets to delay a TX notification after sending a packet.--
v2:
- Usage of notification terminology.
- Add a few lines on what device should do when driver asked to suppress notifications.v3:
- Remove whitespaces.
- Explain with examples how the device should act.--
Signed-off-by Alvaro Karsz <[email protected]>--- content.tex | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+)content.tex +23
-0@@ -3084,6 +3084,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control channel. +\item[VIRTIO_NET_F_NOTF_COAL(55)] Device supports notifications coalescing. + \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO (fragmenting the packet) the USO splits large UDP packet to several segments when each of these smaller packets has UDP header.@@ -3129,6 +3131,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ. +\item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. \end{description}@@ -4464,6 +4467,71 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi (necessarily when not using the legacy interface) little-endian. +\paragraph{Notifications Coalescing}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} + +If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can +send control commands for dynamically changing the coalescing parameters. + +\begin{lstlisting} +struct virtio_net_ctrl_coal_usec { + le32 tx_usecs; + le32 rx_usecs; +}; + +struct virtio_net_ctrl_coal_frames { + le32 tx_frames_max; + le32 rx_frames_max; +}; + +#define VIRTIO_NET_CTRL_NOTF_COAL 6 + #define VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET 0 + #define VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET 1 +\end{lstlisting}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4499
Quoted 10 lines
content.tex +10
-0+ +The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: +\begin{itemize} +\item VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET: set the rx-usecs (time to delay an RX notification after frame arrival in microseconds) and tx-usecs (time to delay a TX notification after a sending a frame in microseconds) parameters. +\item VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET: set the rx-max-frames (number of frames to delay an RX notification after frame arrival) and tx-max-frames (number of frames to delay a TX notification after sending a frame) parameters. +\end{itemize} + +\paragraph{RX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX notifications} + +If, for example, rx_usecs = 10 and rx_frames_max = 15.
If the usecs and frames are required to be used simultaneously, we'd better tweak the command to pack them together like:
struct rx_coalesce {
le32 usecs;
le32.frames;
};content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4502
Quoted 3 lines
content.tex +3
-0+ +\begin{itemize} +\item The device will count received frames until it accumulates 15 frames, or until 10 usecs elapsed since the arrival of the first frame.
The description here looks more like GUEST_RSC instead of GUEST_GSO. The difference is GUEST_GSO doesn't coalesce packets.
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505
Quoted 3 lines
content.tex +3
-0+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\ +An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters.
So we need clarify the "packet" definition:
1) is it the packet that the device saw on the wire (before coalescing)
or
2) it's the packet that has been coalesced by the the device and put in the buffer
And when the device is expected to increase the counter
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506
Quoted 1 line
content.tex +1
-0+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications.
It's better to describe how it is expected to work with event index (probably with some examples).
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4518
Quoted 12 lines
content.tex +12
-0+\end{itemize} + + +\paragraph{TX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX notifications} + +If, for example, tx_usecs = 10 and tx_frames_max = 15. + +\begin{itemize} +\item The device will count sent frames until it accumulates 15 frames, or until 10 usecs elapsed since first frame was sent. +\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\ +An example is if any of the VIRTIO_NET_F_HOST_TSO/UFO features are negotiated, a device could receive a big buffer, which will take more than 15 frames to send. +\item The device will reset its internal coalescing counters.
So the device counts the segmented packets. I wonder what's the advantage of this over counting the unsegmented packets.
Quoted 4 lines
content.tex +4
-0+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications. +\end{itemize} + +\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4532
Quoted 10 lines
content.tex +10
-0+ + +If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. + +\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} + +A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if was not able to change the parameters.\\ \\ +A device MUST NOT accept tx_frames_max/rx_frames_max values bigger than the queue size.\\ \\ +A device SHOULD NOT send notifications to the driver, if the driver asked to suppress notifications.\\ \\ +A device SHOULD initialize all coalescing values to 0. \\ \\
Let's define the semantics of 0 value here.
Thanks
Quoted 16 lines
content.tex +2
-1+ + \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device Types / Network Device / Legacy Interface: Framing Requirements} --2.32.0
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Thank you Jason,
I will create a new version.
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
After some more thought, I'm not sure that the notifications coalescing feature can work if VIRTIO_F_EVENT_IDX is negotiated.
Quoting Virtio Spec:
Quoted 1 line
Note: For example, if
used_eventis 0, then a device usingVIRTIO_F_EVENT_IDXwould send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).
So, if the driver sets the event index to #X and doesn't change the value after receiving the notification, it is meaningless to use the interrupt coalescing parameters. since it will anyway receive a notification every 65536 packets (uint16_t overflow).
I'm thinking of adding a constraint that the driver MUST NOT accept this feature if VIRTIO_F_EVENT_IDX is offered by the device. What do you think?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 11 lines · Alvaro Karsz
After some more thought, I'm not sure that the notifications coalescing feature can work if
VIRTIO_F_EVENT_IDXis negotiated.Quoting Virtio Spec:
Quoted 1 line
Note: For example, if
used_eventis 0, then a device usingVIRTIO_F_EVENT_IDXwould send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).So, if the driver sets the event index to #X and doesn't change the value after receiving the notification, it is meaningless to use the interrupt coalescing parameters. since it will anyway receive a notification every 65536 packets (
uint16_toverflow).
It's not a conflict since we can add more conditions when the device needs to send a notification.
Quoted 4 lines
I'm thinking of adding a constraint that the driver MUST NOT accept this feature if
VIRTIO_F_EVENT_IDXis offered by the device. What do you think?
I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?
And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?
The notification is sent when any of the following condition is met:
- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to used_event
- If we hit the coalescing limit
For the driver that want to use interrupt coalescing only, it can choose:
1) avoid negotiating event index
or
2) do not publish new used_event
Thanks
Quoted 1 line
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 2 lines
The description here looks more like
GUEST_RSCinstead ofGUEST_GSO. The difference isGUEST_GSOdoesn't coalesce packets.
If I understand correctly, VIRTIO_NET_F_GUEST_TSO/UFO means that the device should coalesce ethernet frames into packets, and that's what I meant.
Something like GRO.
Which part is not clear in the patch?
Maybe the "received frames" part?
I meant received from the "outside world" and not from the host.
Quoted 9 lines
So we need clarify the "packet" definition:
1) is it the packet that the device saw on the wire (before coalescing)
or
2) it's the packet that has been coalesced by the the device and put in the buffer
And when the device is expected to increase the counter
I think that it should be based on ethernet frames (1).
So, the driver should increase the packet counter after receiving/sending an ethernet frame.
Meaning that the counter may be increased many times handling just one driver buffer (if GSO/GRO are negotiated).
This is why I added the following part:
Quoted 9 lines
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters.I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?
I think that there are some advantages:
- More flexible, It's easier to configure the device to send notifications every X packets/events.
- Allows you to set a timeout using the rx/tx-usecs parameters.
- For a HW device, it is more easy to get control commands through the control virtqueue than polling the driver area using PCI transactions.
Quoted 11 lines
And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?
The notification is sent when any of the following condition is met:
- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to
used_event
- If we hit the coalescing limitFor the driver that want to use interrupt coalescing only, it can choose:
Quoted 7 lines
1) avoid negotiating event index
or
2) do not publish new
used_eventThanks
In this case, the coalescing parameters could be set to 1 (tx/rx_frames_max) and it will neutralize the event_idx effect, since a notification will be sent for every packet.
This could be done the other way around, the driver could set event_idx to X, then increase it by 1 every notification, and it will neutralize the coalescing.
In my opinion, the user should have control over the coalescing parameters using ethtool, and the driver could suppress all notifications, but I can't see what we'll benefit from allowing both coalescing and EVENT_IDX to work in parallel.
What do you think?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 11 lines · Alvaro Karsz
Quoted 3 lines
The description here looks more like
GUEST_RSCinstead ofGUEST_GSO. The difference isGUEST_GSOdoesn't coalesce packets.If I understand correctly, VIRTIO_NET_F_GUEST_TSO/UFO means that the device should coalesce ethernet frames into packets, and that's what I meant.
Something like GRO.
Which part is not clear in the patch?
Maybe the "received frames" part?
I meant received from the "outside world" and not from the host.
So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:
We never had per frame notification in the past but per packet used buffer notification.
What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.
Quoted 7 lines
Quoted 7 lines
So we need clarify the "packet" definition:
1) is it the packet that the device saw on the wire (before coalescing)
Quoted 7 lines
Quoted 4 lines
or
2) it's the packet that has been coalesced by the the device and put in the buffer
And when the device is expected to increase the counterI think that it should be based on ethernet frames (1).
What's the advantage of counting frames here? In the case of GSO, those frames were invisible to the software, so we can't make use of the per frame notification or its coalescing.
Quoted 9 lines
So, the driver should increase the packet counter after receiving/sending an ethernet frame.
Meaning that the counter may be increased many times handling just one driver buffer (if GSO/GRO are negotiated).
This is why I added the following part:Quoted 3 lines
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters.
An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.
Quoted 42 lines
Quoted 6 lines
I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?
I think that there are some advantages:
- More flexible, It's easier to configure the device to send notifications every X packets/events.
- Allows you to set a timeout using the rx/tx-usecs parameters.
- For a HW device, it is more easy to get control commands through the control virtqueue than polling the driver area using PCI transactions.Quoted 11 lines
And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?
The notification is sent when any of the following condition is met:
- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to
used_event
- If we hit the coalescing limitFor the driver that want to use interrupt coalescing only, it can choose:
Quoted 7 lines
1) avoid negotiating event index
or
2) do not publish new
used_eventThanks
In this case, the coalescing parameters could be set to 1 (tx/rx_frames_max) and it will neutralize the
event_idxeffect, since a notification will be sent for every packet.
This could be done the other way around, the driver could setevent_idxto X, then increase it by 1 every notification, and it will neutralize the coalescing.
Quoted 7 lines
In my opinion, the user should have control over the coalescing parameters using ethtool, and the driver could suppress all notifications, but I can't see what we'll benefit from allowing both coalescing and
EVENT_IDXto work in parallel.What do you think?
So I think they don't conflict with each other. E.g some drivers like Linux can benefit from this.
If we count by frames, see above reply, we may have some conflict.
Assuming we count by used buffers (packets), at least the TX could benefit:
- under normal load, notification coalescing can help to moderate interrupt rate
- under extremely heavy load (the virtqueue is about to be full), we can use event idx to delay the tx interrupt until e.g 3/4 of the queue were drained
So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.
Thanks
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 5 lines
So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:
Quoted 8 lines
We never had per frame notification in the past but per packet used buffer notification.
What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.
You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:
Quoted 2 lines
I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.
Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.
Quoted 7 lines
An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.
You're right, that was my point in V2:
Quoted 12 lines
x-max-frames = 5
The driver wants to send a 64k packet, andVIRTIO_NET_F_HOST_TSO4is negotiated. The driver will write a 64K description on the descriptor area.
The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.
Quoted 5 lines
So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.
We have 2 approaches in implementing the device's side:
Approach 1:
The device should always count packets, even if notifications are not enabled:
Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.
Code example:
mark_desc_as_used()update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)
send_notification()
}
Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.
Code example:
mark_desc_as_used()
if (events_enabled) {
update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
send_notification()
}
} else { reset_coal_counters()
}
Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.
But, we may have a minor issue with EVENT_IDX using the first approach:
Assume:
ring current location #0EVENT_IDX #9
max-frames 10
Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer #19.
I think that we should use approach 1, and it's not a big deal to send on #10 in this example.
What do you think?
Another point with EVENT_IDX is that the device should "remember" if it already sent a notification for a given event_idx.
If the driver sets event_idx to #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.
Do you agree?
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Quoted 20 lines · Alvaro Karsz
Quoted 6 lines
So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:
Quoted 8 lines
We never had per frame notification in the past but per packet used buffer notification.
What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.
You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:
Quoted 6 lines
Quoted 2 lines
I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.
I think it's better to coalesce packets (used buffers).
Quoted 42 lines
Quoted 8 lines
An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.
You're right, that was my point in V2:
Quoted 12 lines
x-max-frames = 5
The driver wants to send a 64k packet, andVIRTIO_NET_F_HOST_TSO4is negotiated. The driver will write a 64K description on the descriptor area.
The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.Quoted 5 lines
So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.
We have 2 approaches in implementing the device's side:
Approach 1:
The device should always count packets, even if notifications are not enabled:
Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.
Quoted 7 lines
Code example:
mark_desc_as_used()update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)
I guess "events_enabled" means driver allow us to send a notification either:
1) NO_NOTIFY is clear
or
2) used idx matches used_event at least once since last notification?
Quoted 1 line
send_notification()
I think we should reset the counters only after we send a notification?
Quoted 1 line
}
Need to specify what happens if the condition is not met, I think the behaviour is to delay the notification until the condition is met.
Quoted 23 lines
Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.Code example:
mark_desc_as_used()if (events_enabled) { update_coal_counters() if (coal_cond_met) { reset_coal_counters() send_notification() } } else {
Quoted 6 lines
reset_coal_counters()
}Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.
Yes.
Quoted 9 lines
But, we may have a minor issue with
EVENT_IDXusing the first approach:
Assume:
ring current location #0EVENT_IDX#9
max-frames 10Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer #19.
I don't see how this could be useful (since it may introduce much more delay in this case).
Quoted 5 lines
I think that we should use approach 1, and it's not a big deal to send on #10 in this example.
What do you think?
I think approach 2 may lead to confusion to the user and approach 1 looks more clear:
1) coalescing working at packet level
2) event idx work at virtio ring level
Quoted 10 lines
Another point with
EVENT_IDXis that the device should "remember" if it already sent a notification for a givenevent_idx.
If the driver setsevent_idxto #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.Do you agree?
Yes, this is how it works without coalescing.
Thanks
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 9 lines
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
I will post a new version including the discussed points.
Quoted 194 lines · Jason Wang
Quoted 20 lines · Alvaro Karsz
Quoted 6 lines
So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:
Quoted 8 lines
We never had per frame notification in the past but per packet used buffer notification.
What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.
You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:
Quoted 6 lines
Quoted 2 lines
I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.
I think it's better to coalesce packets (used buffers).
Quoted 10 lines
Quoted 8 lines
An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.
Quoted 4 lines
You're right, that was my point in V2:
Quoted 3 lines
x-max-frames = 5
The driver wants to send a 64k packet, andVIRTIO_NET_F_HOST_TSO4isnegotiated.
Quoted 16 lines
Quoted 9 lines
The driver will write a 64K description on the descriptor area. The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.Quoted 5 lines
So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.
Quoted 4 lines
We have 2 approaches in implementing the device's side:
Approach 1:
The device should always count packets, even if notifications are notenabled:
Quoted 8 lines
Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.Quoted 7 lines
Code example:
mark_desc_as_used()update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)I guess "
events_enabled" means driver allow us to send a notification either:1)
NO_NOTIFYis clear
or
2) used idx matchesused_eventat least once since last notification?Quoted 1 line
send_notification()I think we should reset the counters only after we send a notification?
Quoted 1 line
}
Need to specify what happens if the condition is not met, I think the behaviour is to delay the notification until the condition is met.
Quoted 23 lines
Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.Code example:
mark_desc_as_used()if (events_enabled) { update_coal_counters() if (coal_cond_met) { reset_coal_counters() send_notification() } } else {Quoted 6 lines
reset_coal_counters()
}Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.
Yes.
Quoted 9 lines
But, we may have a minor issue with
EVENT_IDXusing the first approach:
Assume:
ring current location #0EVENT_IDX#9
max-frames 10Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer#19.
I don't see how this could be useful (since it may introduce much more delay in this case).
Quoted 5 lines
I think that we should use approach 1, and it's not a big deal to send on #10 in this example.
What do you think?
I think approach 2 may lead to confusion to the user and approach 1 looks more clear:
1) coalescing working at packet level
2) event idx work at virtio ring levelQuoted 10 lines
Another point with
EVENT_IDXis that the device should "remember" if it already sent a notification for a givenevent_idx.
If the driver setsevent_idxto #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.Do you agree?
Yes, this is how it works without coalescing.
Thanks
Quoted 8 lines
This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.
Quoted 5 lines
Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License:https://www.oasis-open.org/who/ipr/feedback_license.pdf
Quoted 1 line
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Quoted 3 lines
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Signature · 2 lines
*Alvaro Karsz, *Software +972-50-7696862| https://www.solid-run.com