Skip to contentSkip to conversation

Introduction of Virtio Network device notifications coalescing feature.

PATCH Superseded Alvaro Karsz 2 participants 11 messages May 19, 2022 – Jun 1, 2022

v3 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 Introduction of Virtio Network device notifications coalescing feature.

Control a network device notifications coalescing parameters using the control virtqueue. A new control class was added: VIRTIO_NET_CTRL_NOTF_COAL.

This class provides 2 commands:
- VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET:
Ask the network device to change the rx-usecs and tx-usecs parameters. rx-usecs - Time to delay an RX notification after packet arrival in microseconds. tx-usecs - Time to delay a TX notification after a sending a packet in microseconds.

- VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET:
Ask the network device to change the rx-max-frames and tx-max-frames parameters. rx-max-frames - Number of packets to delay an RX notification after packet arrival. tx-max-frames - Number of packets to delay a TX notification after sending a packet.

--
v2:
- Usage of notification terminology.
- Add a few lines on what device should do when driver asked to suppress notifications.

v3:
- Remove whitespaces.
- Explain with examples how the device should act.

--

---
 content.tex | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
content.tex +68 -0
index 7508dd1..b0ee98d 100644
@@ -3084,6 +3084,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_NOTF_COAL(55)] Device supports notifications coalescing.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3129,6 +3131,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4464,6 +4467,71 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 (necessarily when not using the legacy interface) little-endian.
 
 
+\paragraph{Notifications Coalescing}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+
+If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
+send control commands for dynamically changing the coalescing parameters.
+
+\begin{lstlisting}
+struct virtio_net_ctrl_coal_usec {
+    le32 tx_usecs;
+    le32 rx_usecs;
+};
+
+struct virtio_net_ctrl_coal_frames {
+    le32 tx_frames_max;
+    le32 rx_frames_max;
+};
+
+#define VIRTIO_NET_CTRL_NOTF_COAL 6
+ #define VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET  0
+ #define VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET 1
+\end{lstlisting}
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET: set the rx-usecs (time to delay an RX notification after frame arrival in microseconds) and tx-usecs (time to delay a TX notification after a sending a frame in microseconds) parameters.
+\item VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET: set the rx-max-frames (number of frames to delay an RX notification after frame arrival) and tx-max-frames (number of frames to delay a TX notification after sending a frame) parameters.
+\end{itemize}
+
+\paragraph{RX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX notifications}
+
+If, for example, rx_usecs = 10 and rx_frames_max = 15.
+
+\begin{itemize}
+\item The device will count received frames until it accumulates 15 frames, or until 10 usecs elapsed since the arrival of the first frame.
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer.
+\item The device will reset its internal coalescing counters.
+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications.
+\end{itemize}
+
+
+\paragraph{TX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX notifications}
+
+If, for example, tx_usecs = 10 and tx_frames_max = 15.
+
+\begin{itemize}
+\item The device will count sent frames until it accumulates 15 frames, or until 10 usecs elapsed since first frame was sent.
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_HOST_TSO/UFO features are negotiated, a device could receive a big buffer, which will take more than 15 frames to send.
+\item The device will reset its internal coalescing counters.
+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications.
+\end{itemize}
+
+\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+
+
+If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+
+\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+
+A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if was not able to change the parameters.\\ \\
+A device MUST NOT accept tx_frames_max/rx_frames_max values bigger than the queue size.\\ \\
+A device SHOULD NOT send notifications to the driver, if the driver asked to suppress notifications.\\ \\
+A device SHOULD initialize all coalescing values to 0. \\ \\
+
+
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
 
Signature · 18 lines
2.32.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

If there are no comments, I would like to vote on this feature.

https://github.com/oasis-tcs/virtio-spec/issues/141

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#2 lore raw
In reply to Alvaro Karsz
Quoted 77 lines · Alvaro Karsz

Control a network device notifications coalescing parameters using the control virtqueue. A new control class was added: VIRTIO_NET_CTRL_NOTF_COAL.

This class provides 2 commands:
- VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET:
Ask the network device to change the rx-usecs and tx-usecs parameters. rx-usecs - Time to delay an RX notification after packet arrival in microseconds. tx-usecs - Time to delay a TX notification after a sending a packet in microseconds.

- VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET:
Ask the network device to change the rx-max-frames and tx-max-frames parameters. rx-max-frames - Number of packets to delay an RX notification after packet arrival. tx-max-frames - Number of packets to delay a TX notification after sending a packet.

--
v2:
- Usage of notification terminology.
- Add a few lines on what device should do when driver asked to suppress notifications.

v3:
- Remove whitespaces.
- Explain with examples how the device should act.

--

---
 content.tex | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
content.tex +23 -0
index 7508dd1..b0ee98d 100644
@@ -3084,6 +3084,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
     channel.
 
+\item[VIRTIO_NET_F_NOTF_COAL(55)] Device supports notifications coalescing.
+
 \item[VIRTIO_NET_F_HOST_USO (56)] Device can receive USO packets. Unlike UFO
  (fragmenting the packet) the USO splits large UDP packet
  to several segments when each of these smaller packets has UDP header.
@@ -3129,6 +3131,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
 \item[VIRTIO_NET_F_GUEST_ANNOUNCE] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_MQ] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_CTRL_MAC_ADDR] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
 \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
 \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
 \end{description}
@@ -4464,6 +4467,71 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 (necessarily when not using the legacy interface) little-endian.
 
 
+\paragraph{Notifications Coalescing}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+
+If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
+send control commands for dynamically changing the coalescing parameters.
+
+\begin{lstlisting}
+struct virtio_net_ctrl_coal_usec {
+    le32 tx_usecs;
+    le32 rx_usecs;
+};
+
+struct virtio_net_ctrl_coal_frames {
+    le32 tx_frames_max;
+    le32 rx_frames_max;
+};
+
+#define VIRTIO_NET_CTRL_NOTF_COAL 6
+ #define VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET  0
+ #define VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET 1
+\end{lstlisting}

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4499

Quoted 10 lines
content.tex +10 -0
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\begin{itemize}
+\item VIRTIO_NET_CTRL_NOTF_COAL_USECS_SET: set the rx-usecs (time to delay an RX notification after frame arrival in microseconds) and tx-usecs (time to delay a TX notification after a sending a frame in microseconds) parameters.
+\item VIRTIO_NET_CTRL_NOTF_COAL_FRAMES_SET: set the rx-max-frames (number of frames to delay an RX notification after frame arrival) and tx-max-frames (number of frames to delay a TX notification after sending a frame) parameters.
+\end{itemize}
+
+\paragraph{RX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX notifications}
+
+If, for example, rx_usecs = 10 and rx_frames_max = 15.

If the usecs and frames are required to be used simultaneously, we'd better tweak the command to pack them together like:

struct rx_coalesce {
    le32 usecs;
    le32.frames;
};

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4502

Quoted 3 lines
content.tex +3 -0
+
+\begin{itemize}
+\item The device will count received frames until it accumulates 15 frames, or until 10 usecs elapsed since the arrival of the first frame.

The description here looks more like GUEST_RSC instead of GUEST_GSO. The difference is GUEST_GSO doesn't coalesce packets.

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4505

Quoted 3 lines
content.tex +3 -0
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer.
+\item The device will reset its internal coalescing counters.

So we need clarify the "packet" definition:

1) is it the packet that the device saw on the wire (before coalescing)

or

2) it's the packet that has been coalesced by the the device and put in the buffer

And when the device is expected to increase the counter

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4506

Quoted 1 line
content.tex +1 -0
+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications.

It's better to describe how it is expected to work with event index (probably with some examples).

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4518

Quoted 12 lines
content.tex +12 -0
+\end{itemize}
+
+
+\paragraph{TX notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX notifications}
+
+If, for example, tx_usecs = 10 and tx_frames_max = 15.
+
+\begin{itemize}
+\item The device will count sent frames until it accumulates 15 frames, or until 10 usecs elapsed since first frame was sent.
+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_HOST_TSO/UFO features are negotiated, a device could receive a big buffer, which will take more than 15 frames to send.
+\item The device will reset its internal coalescing counters.

So the device counts the segmented packets. I wonder what's the advantage of this over counting the unsegmented packets.

Quoted 4 lines
content.tex +4 -0
+\item The device will send a notification to the driver only if the driver hasn't suppressed notifications.
+\end{itemize}
+
+\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}

content.tex \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi · line 4532

Quoted 10 lines
content.tex +10 -0
+
+
+If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+
+\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
+
+A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if was not able to change the parameters.\\ \\
+A device MUST NOT accept tx_frames_max/rx_frames_max values bigger than the queue size.\\ \\
+A device SHOULD NOT send notifications to the driver, if the driver asked to suppress notifications.\\ \\
+A device SHOULD initialize all coalescing values to 0. \\ \\

Let's define the semantics of 0 value here.

Thanks

Quoted 16 lines
content.tex +2 -1
+
+
 \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 Types / Network Device / Legacy Interface: Framing Requirements}
 
--

2.32.0

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw

Thank you Jason,
I will create a new version.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw

After some more thought, I'm not sure that the notifications coalescing feature can work if VIRTIO_F_EVENT_IDX is negotiated.

Quoting Virtio Spec:

Quoted 1 line

Note: For example, if used_event is 0, then a device using VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).

So, if the driver sets the event index to #X and doesn't change the value after receiving the notification, it is meaningless to use the interrupt coalescing parameters. since it will anyway receive a notification every 65536 packets (uint16_t overflow).

I'm thinking of adding a constraint that the driver MUST NOT accept this feature if VIRTIO_F_EVENT_IDX is offered by the device. What do you think?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw
Quoted 11 lines · Alvaro Karsz

After some more thought, I'm not sure that the notifications coalescing feature can work if VIRTIO_F_EVENT_IDX is negotiated.

Quoting Virtio Spec:

Quoted 1 line

Note: For example, if used_event is 0, then a device using VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver after the first buffer is used (and again after the 65536th buffer, etc).

So, if the driver sets the event index to #X and doesn't change the value after receiving the notification, it is meaningless to use the interrupt coalescing parameters. since it will anyway receive a notification every 65536 packets (uint16_t overflow).

It's not a conflict since we can add more conditions when the device needs to send a notification.

Quoted 4 lines

I'm thinking of adding a constraint that the driver MUST NOT accept this feature if VIRTIO_F_EVENT_IDX is offered by the device. What do you think?

I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?

And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?

The notification is sent when any of the following condition is met:

- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to used_event
- If we hit the coalescing limit

For the driver that want to use interrupt coalescing only, it can choose:

1) avoid negotiating event index

or

2) do not publish new used_event

Thanks

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw
Quoted 2 lines

The description here looks more like GUEST_RSC instead of GUEST_GSO. The difference is GUEST_GSO doesn't coalesce packets.

If I understand correctly, VIRTIO_NET_F_GUEST_TSO/UFO means that the device should coalesce ethernet frames into packets, and that's what I meant.
Something like GRO.
Which part is not clear in the patch?
Maybe the "received frames" part?
I meant received from the "outside world" and not from the host.

Quoted 9 lines

So we need clarify the "packet" definition:

1) is it the packet that the device saw on the wire (before coalescing)

or
2) it's the packet that has been coalesced by the the device and put in the buffer
And when the device is expected to increase the counter

I think that it should be based on ethernet frames (1).
So, the driver should increase the packet counter after receiving/sending an ethernet frame.
Meaning that the counter may be increased many times handling just one driver buffer (if GSO/GRO are negotiated).
This is why I added the following part:

Quoted 9 lines

+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters.

I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?

I think that there are some advantages:
- More flexible, It's easier to configure the device to send notifications every X packets/events.
- Allows you to set a timeout using the rx/tx-usecs parameters.
- For a HW device, it is more easy to get control commands through the control virtqueue than polling the driver area using PCI transactions.

Quoted 11 lines

And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?

The notification is sent when any of the following condition is met:

- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to used_event
- If we hit the coalescing limit

For the driver that want to use interrupt coalescing only, it can choose:

Quoted 7 lines

1) avoid negotiating event index

or

2) do not publish new used_event

Thanks

In this case, the coalescing parameters could be set to 1 (tx/rx_frames_max) and it will neutralize the event_idx effect, since a notification will be sent for every packet.
This could be done the other way around, the driver could set event_idx to X, then increase it by 1 every notification, and it will neutralize the coalescing.

In my opinion, the user should have control over the coalescing parameters using ethtool, and the driver could suppress all notifications, but I can't see what we'll benefit from allowing both coalescing and EVENT_IDX to work in parallel.

What do you think?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#7 lore raw
Quoted 11 lines · Alvaro Karsz
Quoted 3 lines

The description here looks more like GUEST_RSC instead of GUEST_GSO. The difference is GUEST_GSO doesn't coalesce packets.

If I understand correctly, VIRTIO_NET_F_GUEST_TSO/UFO means that the device should coalesce ethernet frames into packets, and that's what I meant.
Something like GRO.
Which part is not clear in the patch?
Maybe the "received frames" part?
I meant received from the "outside world" and not from the host.

So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:

We never had per frame notification in the past but per packet used buffer notification.

What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.

Quoted 7 lines
Quoted 7 lines

So we need clarify the "packet" definition:

1) is it the packet that the device saw on the wire (before coalescing)

Quoted 7 lines
Quoted 4 lines

or
2) it's the packet that has been coalesced by the the device and put in the buffer
And when the device is expected to increase the counter

I think that it should be based on ethernet frames (1).

What's the advantage of counting frames here? In the case of GSO, those frames were invisible to the software, so we can't make use of the per frame notification or its coalescing.

Quoted 9 lines

So, the driver should increase the packet counter after receiving/sending an ethernet frame.
Meaning that the counter may be increased many times handling just one driver buffer (if GSO/GRO are negotiated).
This is why I added the following part:

Quoted 3 lines

+\item The device will check if at least one descriptor was used from the descriptor area, if not, it will continue to accumulate frames until one descriptor is used.\\
+An example is if any of the VIRTIO_NET_F_GUEST_TSO/UFO features are negotiated, a device could receive more than 15 frames, and write all in the same buffer. +\item The device will reset its internal coalescing counters.

An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.

Quoted 42 lines
Quoted 6 lines

I think we need first understand what's the advantage of using packet based interrupt coalescing over event index?

I think that there are some advantages:
- More flexible, It's easier to configure the device to send notifications every X packets/events.
- Allows you to set a timeout using the rx/tx-usecs parameters.
- For a HW device, it is more easy to get control commands through the control virtqueue than polling the driver area using PCI transactions.

Quoted 11 lines

And if the answer is yes, is it worthwhile to make coalescing and event index work in parallel?

The notification is sent when any of the following condition is met:

- If the idx field in the used ring (which determined where that descriptor index was placed) was equal to used_event
- If we hit the coalescing limit

For the driver that want to use interrupt coalescing only, it can choose:

Quoted 7 lines

1) avoid negotiating event index

or

2) do not publish new used_event

Thanks

In this case, the coalescing parameters could be set to 1 (tx/rx_frames_max) and it will neutralize the event_idx effect, since a notification will be sent for every packet.
This could be done the other way around, the driver could set event_idx to X, then increase it by 1 every notification, and it will neutralize the coalescing.

Quoted 7 lines

In my opinion, the user should have control over the coalescing parameters using ethtool, and the driver could suppress all notifications, but I can't see what we'll benefit from allowing both coalescing and EVENT_IDX to work in parallel.

What do you think?

So I think they don't conflict with each other. E.g some drivers like Linux can benefit from this.

If we count by frames, see above reply, we may have some conflict.

Assuming we count by used buffers (packets), at least the TX could benefit:

- under normal load, notification coalescing can help to moderate interrupt rate
- under extremely heavy load (the virtqueue is about to be full), we can use event idx to delay the tx interrupt until e.g 3/4 of the queue were drained

So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw
Quoted 5 lines

So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:

Quoted 8 lines

We never had per frame notification in the past but per packet used buffer notification.

What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.

You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:

Quoted 2 lines

I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.

Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.

Quoted 7 lines

An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.

You're right, that was my point in V2:

Quoted 12 lines

x-max-frames = 5
The driver wants to send a 64k packet, and VIRTIO_NET_F_HOST_TSO4 is negotiated. The driver will write a 64K description on the descriptor area.
The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.

Quoted 5 lines

So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.

We have 2 approaches in implementing the device's side:

Approach 1:
The device should always count packets, even if notifications are not enabled:
Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.

Code example:

mark_desc_as_used()
update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)
send_notification()
}

Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.

Code example:

mark_desc_as_used()

if (events_enabled) {
    update_coal_counters()
    if (coal_cond_met) {
        reset_coal_counters()
        send_notification()
    }
} else {

reset_coal_counters()
}

Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.
But, we may have a minor issue with EVENT_IDX using the first approach:
Assume:
ring current location #0
EVENT_IDX #9
max-frames 10

Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer #19.

I think that we should use approach 1, and it's not a big deal to send on #10 in this example.

What do you think?

Another point with EVENT_IDX is that the device should "remember" if it already sent a notification for a given event_idx.
If the driver sets event_idx to #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.

Do you agree?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw
Quoted 20 lines · Alvaro Karsz
Quoted 6 lines

So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:

Quoted 8 lines

We never had per frame notification in the past but per packet used buffer notification.

What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.

You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:

Quoted 6 lines
Quoted 2 lines

I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.

Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.

I think it's better to coalesce packets (used buffers).

Quoted 42 lines
Quoted 8 lines

An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.

You're right, that was my point in V2:

Quoted 12 lines

x-max-frames = 5
The driver wants to send a 64k packet, and VIRTIO_NET_F_HOST_TSO4 is negotiated. The driver will write a 64K description on the descriptor area.
The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.

Quoted 5 lines

So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.

We have 2 approaches in implementing the device's side:

Approach 1:
The device should always count packets, even if notifications are not enabled:
Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.

Quoted 7 lines

Code example:

mark_desc_as_used()
update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)

I guess "events_enabled" means driver allow us to send a notification either:

1) NO_NOTIFY is clear
or
2) used idx matches used_event at least once since last notification?

Quoted 1 line

send_notification()

I think we should reset the counters only after we send a notification?

Quoted 1 line

}

Need to specify what happens if the condition is not met, I think the behaviour is to delay the notification until the condition is met.

Quoted 23 lines

Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.

Code example:

mark_desc_as_used()

if (events_enabled) {
    update_coal_counters()
    if (coal_cond_met) {
        reset_coal_counters()
        send_notification()
    }
} else {
Quoted 6 lines

reset_coal_counters()
}

Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.

Yes.

Quoted 9 lines

But, we may have a minor issue with EVENT_IDX using the first approach:
Assume:
ring current location #0
EVENT_IDX #9
max-frames 10

Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer #19.

I don't see how this could be useful (since it may introduce much more delay in this case).

Quoted 5 lines

I think that we should use approach 1, and it's not a big deal to send on #10 in this example.

What do you think?

I think approach 2 may lead to confusion to the user and approach 1 looks more clear:

1) coalescing working at packet level
2) event idx work at virtio ring level

Quoted 10 lines

Another point with EVENT_IDX is that the device should "remember" if it already sent a notification for a given event_idx.
If the driver sets event_idx to #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.

Do you agree?

Yes, this is how it works without coalescing.

Thanks

Quoted 8 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw

I will post a new version including the discussed points.

Quoted 194 lines · Jason Wang
Quoted 20 lines · Alvaro Karsz
Quoted 6 lines

So according to the name of the feature, it tries to coalesce notification. But from what you said here, it counts by frames. This seems to be a conflict:

Quoted 8 lines

We never had per frame notification in the past but per packet used buffer notification.

What's more, looking at the spec of real hardware NIC like e810 or the ancient e1000, I don't see any per frame interrupt. They only have per TX/RX descriptor writeback interrupt which signals the completion of TX/RX pacekt.

You're right, coalescing packets was my initial intention, but I understood from the comments on V2 that is should count ethernet frames from Stefan's comment:

Quoted 6 lines
Quoted 2 lines

I think the virtio-net-level coalescing makes sense
since it counts Ethernet frames rather than virtqueue buffers.

Anyway, I think that counting packets makes more sense, and I will change it accordingly if you all are Ok with that.

I think it's better to coalesce packets (used buffers).

Quoted 10 lines
Quoted 8 lines

An example, if we tx-frames to 16, but we transmit 64K GSO on 1500 MSS. The packets were segmented to ~43 frames. If we send a notification every 16 frames, the first 2 notifications are meaningless, there's nothing that a guest driver can do. So did the receiving. I don't get why not simply coalescing the used buffer notification.

Quoted 4 lines

You're right, that was my point in V2:

Quoted 3 lines

x-max-frames = 5
The driver wants to send a 64k packet, and VIRTIO_NET_F_HOST_TSO4 is

negotiated.

Quoted 16 lines
Quoted 9 lines

The driver will write a 64K description on the descriptor area. The device will segment the payload and will send more than 40 frames (with standard MTU) in a very short time (so the timer is not a factor).
The device will send more than 8 notifications to the driver for just 1 used descriptor.

Quoted 5 lines

So I feel counting by used buffer is simpler and more useful than counting by frames. In order to make them work in parallel, we can simply say: when the coalescing condition is not met, the event notification will be delayed until we meet the coalescing condition otherwise the notification is armed and raised.

Quoted 4 lines

We have 2 approaches in implementing the device's side:

Approach 1:
The device should always count packets, even if notifications are not

enabled:

Quoted 8 lines

Something like:
- Device uses a buffer.
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
If coalescing conditions indeed met:
- Device resets the coalescing counters.
- Device sends a notification, only if notifications are enabled.

Quoted 7 lines

Code example:

mark_desc_as_used()
update_coal_counters()
if (coal_cond_met) {
reset_coal_counters()
if (events_enabled)

I guess "events_enabled" means driver allow us to send a notification either:

1) NO_NOTIFY is clear
or
2) used idx matches used_event at least once since last notification?

Quoted 1 line

send_notification()

I think we should reset the counters only after we send a notification?

Quoted 1 line

}

Need to specify what happens if the condition is not met, I think the behaviour is to delay the notification until the condition is met.

Quoted 23 lines

Approach 2:
The device counts packets only if notifications are enabled. Something like:
- Device uses a buffer.
- Device checks if notifications are enabled, if not, it just resets coalescing counters.
If notifications are enabled:
- Device increases the packet counter.
- Device checks if coalescing conditions are met.
- Device reset the coalescing counters.
- Device sends a notification.

Code example:

mark_desc_as_used()

if (events_enabled) {
    update_coal_counters()
    if (coal_cond_met) {
        reset_coal_counters()
        send_notification()
    }
} else {
Quoted 6 lines

reset_coal_counters()
}

Approach 1 is simpler, specially for a Real HW device, which needs to make a PCI transaction and wait for completion to check if notifications are enabled.

Yes.

Quoted 9 lines

But, we may have a minor issue with EVENT_IDX using the first approach:
Assume:
ring current location #0
EVENT_IDX #9
max-frames 10

Using approach 1: A notification will be sent to the driver on buffer #10, not #19.
Using approach 2: A notification will be sent to the driver on buffer

#19.

I don't see how this could be useful (since it may introduce much more delay in this case).

Quoted 5 lines

I think that we should use approach 1, and it's not a big deal to send on #10 in this example.

What do you think?

I think approach 2 may lead to confusion to the user and approach 1 looks more clear:

1) coalescing working at packet level
2) event idx work at virtio ring level

Quoted 10 lines

Another point with EVENT_IDX is that the device should "remember" if it already sent a notification for a given event_idx.
If the driver sets event_idx to #20, and never changes it, the device should send a notification after #20, then should not send any notification until the 16bit counter overflows, and it passes #20 again.

Do you agree?

Yes, this is how it works without coalescing.

Thanks

Quoted 8 lines

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 5 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License:

https://www.oasis-open.org/who/ipr/feedback_license.pdf

Quoted 1 line

List Guidelines:

https://www.oasis-open.org/policies-guidelines/mailing-lists

Quoted 3 lines

Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Signature · 2 lines
*Alvaro Karsz, *Software
+972-50-7696862| https://www.solid-run.com