Skip to contentSkip to conversation

virtio net requirements for 1.4

PATCH Superseded Parav Pandit 3 participants 30 messages Aug 15, 2023 – Aug 17, 2023

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover virtio net requirements for 1.4

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

This is live document to be updated in coming time and work towards it for its design which can result in a draft specification.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and otherwise and work towards the interface design followed by drafting the specification changes.

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

Please review mainly patch 5 at the priority.

Receive flow filters is the first item apart from counters to complete in this iteration to start drafting the design spec.
Rest of the requirements are largly untouched other than Stefan's comment.

TODO:
1. Some more refinement needed for rx low latency and header data split requirements.

---
note:
This week update is late as I was unwell for several days, but managed to cover all comments from Heng, David, Satananda.
We must close flow filters requirements this week to take to spec drafting design phase.

---
changelog:
v3->v4:
- receive flow filters requirements undergo major updates to take to spec draft level.
- Addressed comments from Xuan, Heng, David, Satananda.
- Refined wordings in rest of the requirements
v2->v3:
- addressed comments from Stefan for tx low latency and notification
- redrafted the requirements to use rearm term and avoid queue enable confusion for notification
- addressed all comments and refined receive flow filters requirements to take to design level
v1->v2:
- major update of receive flow filter requirements updated based on last two design discussions in community and offline research
- examples added
- link to use case and design goal added
- control and operation side requirements split
- more verbose
v0->v1:
- addressed comments from Heng Li
- addressed few (not all) comments from Michael
- per patch changelog

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow filters requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

 net-workstream/features-1.4.md | 375 +++++++++++++++++++++++++++++++++
 1 file changed, 375 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH v4 1/7 net-features: Add requirements document for release 1.4

Add requirements document template for the virtio net features.

Add virtio net device counters visible to driver.

---
changelog:
v3->v4:
- Addressed comment from David
- Added link to more counters that we are already discussing v0->v1:
- removed tx dropped counter
- updated requirements to mention about virtqueue interface for counters query

---
 net-workstream/features-1.4.md | 43 ++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
net-workstream/features-1.4.md new file +43 -0
new file mode 100644
index 0000000..c2b1cc8
@@ -0,0 +1,43 @@
+# 1. Introduction
+
+This document describes the overall requirements for virtio net device
+improvements for upcoming release 1.4. Some of these requirements are
+interrelated and influence the interface design, hence reviewing them
+together is desired while updating the virtio net interface.
+
+# 2. Summary
+1. Device counters visible to the driver
+
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a virtqueue directly from driver to device for
+   example using a control vq.
+2. The driver should be able to query which counters are supported using a
+   virtqueue command, for example using an existing control vq.
+3. If this device is migrated between two hosts, the driver should be able
+   get the counter values in the destination host from where it was left
+   off in the source host.
+4. If a virtio device is group member device, a group owner should be able
+   to query all the counter attributes using the administration command which
+   a virtio member device will expose via a virtqueue to the driver.
+4. If a virtio device is a group member device, a group owner should be able
+   to query all of the member device counter attributes and counters via the
+   group owner device.
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size
+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the
+    buffer in the receive queue
+3. le64 rx_gso_pkts: Packets treated as receive GSO sequence by the device
+4. le64 rx_pkts: Total packets received by the device
+
+### 3.1.2 Per transmit queue counters
+1. le64 tx_gso_pkts: Packets send as transmit GSO sequence
+2. le64 tx_pkts: Total packets send by the device
+
+### 3.1.3 More counters
+More counters discussed in [1].
+
+[1] https://lists.oasis-open.org/archives/virtio-comment/202308/msg00176.html
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

Re: [virtio-comment] [PATCH requirements v4 1/7] net-features: Add requirements document for release 1.4

Quoted 38 lines · Parav Pandit

Add requirements document template for the virtio net features.

Add virtio net device counters visible to driver.

---
changelog:
v3->v4:
- Addressed comment from David
- Added link to more counters that we are already discussing v0->v1:
- removed tx dropped counter
- updated requirements to mention about virtqueue interface for counters query

---
 net-workstream/features-1.4.md | 43 ++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
net-workstream/features-1.4.md new file +13 -0
new file mode 100644
index 0000000..c2b1cc8
@@ -0,0 +1,43 @@
+# 1. Introduction
+
+This document describes the overall requirements for virtio net device
+improvements for upcoming release 1.4. Some of these requirements are
+interrelated and influence the interface design, hence reviewing them
+together is desired while updating the virtio net interface.
+
+# 2. Summary
+1. Device counters visible to the driver
+
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for

net-workstream/features-1.4.md line 23

Quoted 10 lines
net-workstream/features-1.4.md new file +10 -0
+   debugging purpose using a virtqueue directly from driver to device for
+   example using a control vq.
+2. The driver should be able to query which counters are supported using a
+   virtqueue command, for example using an existing control vq.
+3. If this device is migrated between two hosts, the driver should be able
+   get the counter values in the destination host from where it was left
+   off in the source host.
+4. If a virtio device is group member device, a group owner should be able
+   to query all the counter attributes using the administration command which
+   a virtio member device will expose via a virtqueue to the driver.

The language seems confusing here. How about:

If a virtio device is a group member device, it must be possible to query all of the group member counter attributes via the group owner device.

net-workstream/features-1.4.md line 26

Quoted 3 lines
net-workstream/features-1.4.md new file +3 -0
+4. If a virtio device is a group member device, a group owner should be able
+   to query all of the member device counter attributes and counters via the
+   group owner device.

Similarly:

If a virtio device is a group member device, it must be possible to query all of the group member counters via the group owner device.

Quoted 17 lines
net-workstream/features-1.4.md new file +17 -0
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size
+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the
+    buffer in the receive queue
+3. le64 rx_gso_pkts: Packets treated as receive GSO sequence by the device
+4. le64 rx_pkts: Total packets received by the device
+
+### 3.1.2 Per transmit queue counters
+1. le64 tx_gso_pkts: Packets send as transmit GSO sequence
+2. le64 tx_pkts: Total packets send by the device
+
+### 3.1.3 More counters
+More counters discussed in [1].
+
+[1] https://lists.oasis-open.org/archives/virtio-comment/202308/msg00176.html
Signature · 17 lines
It's gettin', it's gettin', it's gettin' kinda hectic.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#3 lore raw

RE: [virtio-comment] [PATCH requirements v4 1/7] net-features: Add requirements document for release 1.4

Quoted 15 lines

From: David Edmondson <[email protected]>
Sent: Tuesday, August 15, 2023 2:26 PM

Quoted 2 lines

+4. If a virtio device is group member device, a group owner should be able
+ to query all the counter attributes using the administration command

which

Quoted 1 line

+ a virtio member device will expose via a virtqueue to the driver.

The language seems confusing here. How about:

If a virtio device is a group member device, it must be possible to query all of the group member counter attributes via the group owner device.

Quoted 9 lines

net-workstream/features-1.4.md line 26

Quoted 3 lines

+4. If a virtio device is a group member device, a group owner should be able

net-workstream/features-1.4.md new file +2 -0
+   to query all of the member device counter attributes and counters via the
+   group owner device.

Similarly:

If a virtio device is a group member device, it must be possible to query all of the group member counters via the group owner device.

I messed up this point 4 while addressing your comments from v3.

How about,

If a virtio device is a group member device, it must be possible to query supported counters bitmap through the group owner device. The group owner driver should be able to query such bitmap using administration command.

Notice that group owner only cares for the supported bitmap, no need to query the counters through this interface.

With this we can reuse single structure across AQ and CVQ to learn about supported bitmap in v14 patch from Xuan.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw

Re: [virtio-comment] [PATCH requirements v4 1/7] net-features: Add requirements document for release 1.4

Quoted 23 lines · Parav Pandit
Quoted 14 lines

From: David Edmondson <[email protected]>
Sent: Tuesday, August 15, 2023 2:26 PM

Quoted 2 lines

+4. If a virtio device is group member device, a group owner should be able
+ to query all the counter attributes using the administration command

which

Quoted 1 line

+ a virtio member device will expose via a virtqueue to the driver.

The language seems confusing here. How about:

If a virtio device is a group member device, it must be possible to query all of the group member counter attributes via the group owner device.

Quoted 9 lines

net-workstream/features-1.4.md line 26

Quoted 3 lines

+4. If a virtio device is a group member device, a group owner should be able

net-workstream/features-1.4.md new file +2 -0
+   to query all of the member device counter attributes and counters via the
+   group owner device.

Similarly:

If a virtio device is a group member device, it must be possible to query all of the group member counters via the group owner device.

Quoted 8 lines

I messed up this point 4 while addressing your comments from v3.

How about,

If a virtio device is a group member device, it must be possible to query supported counters bitmap through the group owner device. The group owner driver should be able to query such bitmap using administration command.

Notice that group owner only cares for the supported bitmap, no need to query the counters through this interface.

Why do we think this is the case? It would be useful for a group owner to be able to see the statistics as they are being reported to the member consumer.

Quoted 1 line

With this we can reuse single structure across AQ and CVQ to learn about supported bitmap in v14 patch from Xuan.

Makes sense.

Signature · 17 lines
I'm not living in the real world, no more, no more.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#5 lore raw

RE: [virtio-comment] [PATCH requirements v4 1/7] net-features: Add requirements document for release 1.4

Quoted 7 lines

From: David Edmondson <[email protected]>
Sent: Tuesday, August 15, 2023 2:44 PM

Quoted 2 lines

If a virtio device is a group member device, it must be possible to query

supported counters bitmap through the group owner device.

Quoted 9 lines
Quoted 1 line

The group owner driver should be able to query such bitmap using

administration command.

Quoted 2 lines

Notice that group owner only cares for the supported bitmap, no need to

query the counters through this interface.

Why do we think this is the case? It would be useful for a group owner to be able to see the statistics as they are being reported to the member consumer.

I agree, it is useful.
For LM scenario, ability to query supported bit is minimum needed, actual counters is nice to have. Usually the actual counters are located where the switch is located.

Quoted 6 lines
Quoted 1 line

With this we can reuse single structure across AQ and CVQ to learn about

supported bitmap in v14 patch from Xuan.

Makes sense.
--
I'm not living in the real world, no more, no more.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw

PATCH v4 2/7 net-features: Add low latency transmit queue requirements

In reply to Parav Pandit

Add requirements for the low latency transmit queue.

---
chagelog:
v3->v4:
- Addressed comments from David
- rewrote timestamp and completions pcie transcation requirement v1->v2:
- added generic requirement to inline the request content along with the descriptor for non virtio-net devices
- added requirement to inline the header content along with the descriptor for virtio flow filter queue as two features are similar
v0->v1:
- added design goals for which requirements are added

---
 net-workstream/features-1.4.md | 88 ++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)
net-workstream/features-1.4.md +88 -0
index c2b1cc8..40fa07f 100644
@@ -7,6 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
+2. Low latency tx virtqueue for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -41,3 +42,90 @@ together is desired while updating the virtio net interface.
 More counters discussed in [1].
 
 [1] https://lists.oasis-open.org/archives/virtio-comment/202308/msg00176.html
+
+## 3.2 Low PCI latency virtqueues
+### 3.2.1 Low PCI latency tx virtqueue
+0. Design goal
+   a. Reduce PCI access latency in packet transmit flow
+   b. Avoid O(N) descriptor parser to detect a packet stream to simplify device
+      logic
+   c. Reduce number of PCI transmit completion transactions and have unified
+      completion flow with/without transmit timestamping
+   d. Avoid partial cache line writes on transmit completions
+
+1. Packet transmit descriptor should contain data descriptors count without any
+   indirection and without any O(N) search to find the end of a packet stream.
+   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
+   subsequently) to contain a field num_next_desc for the packet stream
+   indicating that a packet is located in N data descriptors.
+
+2. Packet transmit descriptor should contain segmentation offload-related fields
+   without any indirection. For example, packet transmit descriptor to contain
+   gso_type, gso_size/mss, header length, csum placement byte offset, and
+   csum start.
+
+3. Packet transmit descriptor should be able to place a small size packet that
+   does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue memory.
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.
+
+4. Packet transmit descriptor should be able to place a full GSO header (L2 to
+   L4) after header descriptor and before data descriptors. For example, the
+   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue memory.
+   When such a GSO header is positioned adjacent to the packet transmit
+   descriptor, and when the GSO header is not aligned to 16B, the following
+   data descriptor to start on the 8B aligned boundary.
+
+5. An example of the above requirements at high level is:
+
+```
+struct virtio_packed_q_desc {
+   /* current desc for reference */
+   u64 address;
+   u32 len;
+   u16 id;
+   u16 flags;
+};
+
+/* Constant size header descriptor for tx packets */
+struct vnet_tx_hdr_desc {
+   u16 flags; /* indicate how to parse next fields */
+   u16 id; /* desc id to come back in completion */
+   u8 num_next_desc; /* indicates the number of the next 16B data desc for this
+		      * buffer.
+		      */
+   u8 gso_type;
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u8 inline_pkt_len; /* indicates the length of the inline packet after this
+		       * desc
+		       */
+   u8 reserved;
+   u8 padding[];
+};
+
+/* Example of a short packet or GSO header placed in the desc section of the vq
+ */
+struct vnet_tx_small_pkt_desc {
+   u8 raw_pkt[128];
+};
+
+/* Example of header followed by data descriptor */
+struct vnet_tx_hdr_desc hdr_desc;
+struct vnet_data_desc desc[2];
+
+```
+
+6. Ability to zero pad the transmit completion when the transmit completion is
+   shorter than the CPU cache line size.
+
+7. Ability to write per packet timestamp and also write multiple
+   transmit completions using single PCIe transcation.
+
+8. A generic feature of the virtqueue, to contain such header data inline for virtio
+   devices other than virtio-net.
+
+9. A flow filter virtqueue also similarly need the ability to inline the short flow
+   command header.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Acked-by on v4 2/7
#7 lore raw

[virtio-comment] Re: [PATCH requirements v4 2/7] net-features: Add low latency transmit queue requirements

Quoted 3 lines · Parav Pandit

Add requirements for the low latency transmit queue.

Quoted 49 lines

---
chagelog:
v3->v4:
- Addressed comments from David
- rewrote timestamp and completions pcie transcation requirement v1->v2:
- added generic requirement to inline the request content along with the descriptor for non virtio-net devices
- added requirement to inline the header content along with the descriptor for virtio flow filter queue as two features are similar
v0->v1:
- added design goals for which requirements are added

---
 net-workstream/features-1.4.md | 88 ++++++++++++++++++++++++++++++++++
 1 file changed, 88 insertions(+)
net-workstream/features-1.4.md +17 -0
index c2b1cc8..40fa07f 100644
@@ -7,6 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
+2. Low latency tx virtqueue for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -41,3 +42,90 @@ together is desired while updating the virtio net interface.
 More counters discussed in [1].
 
 [1] https://lists.oasis-open.org/archives/virtio-comment/202308/msg00176.html
+
+## 3.2 Low PCI latency virtqueues
+### 3.2.1 Low PCI latency tx virtqueue
+0. Design goal
+   a. Reduce PCI access latency in packet transmit flow
+   b. Avoid O(N) descriptor parser to detect a packet stream to simplify device
+      logic
+   c. Reduce number of PCI transmit completion transactions and have unified
+      completion flow with/without transmit timestamping
+   d. Avoid partial cache line writes on transmit completions
+
+1. Packet transmit descriptor should contain data descriptors count without any
+   indirection and without any O(N) search to find the end of a packet stream.
+   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
+   subsequently) to contain a field num_next_desc for the packet stream
+   indicating that a packet is located in N data descriptors.

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 70

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+
+2. Packet transmit descriptor should contain segmentation offload-related fields
+   without any indirection. For example, packet transmit descriptor to contain
+   gso_type, gso_size/mss, header length, csum placement byte offset, and
+   csum start.
+
+3. Packet transmit descriptor should be able to place a small size packet that
+   does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue memory.
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.

The presence of "L4 data" seems like a distraction - just give an indication of size ranges being considered.

Quoted 61 lines
net-workstream/features-1.4.md +61 -0
+
+4. Packet transmit descriptor should be able to place a full GSO header (L2 to
+   L4) after header descriptor and before data descriptors. For example, the
+   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue memory.
+   When such a GSO header is positioned adjacent to the packet transmit
+   descriptor, and when the GSO header is not aligned to 16B, the following
+   data descriptor to start on the 8B aligned boundary.
+
+5. An example of the above requirements at high level is:
+
+```
+struct virtio_packed_q_desc {
+   /* current desc for reference */
+   u64 address;
+   u32 len;
+   u16 id;
+   u16 flags;
+};
+
+/* Constant size header descriptor for tx packets */
+struct vnet_tx_hdr_desc {
+   u16 flags; /* indicate how to parse next fields */
+   u16 id; /* desc id to come back in completion */
+   u8 num_next_desc; /* indicates the number of the next 16B data desc for this
+		      * buffer.
+		      */
+   u8 gso_type;
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u8 inline_pkt_len; /* indicates the length of the inline packet after this
+		       * desc
+		       */
+   u8 reserved;
+   u8 padding[];
+};
+
+/* Example of a short packet or GSO header placed in the desc section of the vq
+ */
+struct vnet_tx_small_pkt_desc {
+   u8 raw_pkt[128];
+};
+
+/* Example of header followed by data descriptor */
+struct vnet_tx_hdr_desc hdr_desc;
+struct vnet_data_desc desc[2];
+
+```
+
+6. Ability to zero pad the transmit completion when the transmit completion is
+   shorter than the CPU cache line size.
+
+7. Ability to write per packet timestamp and also write multiple
+   transmit completions using single PCIe transcation.
+
+8. A generic feature of the virtqueue, to contain such header data inline for virtio
+   devices other than virtio-net.
+
+9. A flow filter virtqueue also similarly need the ability to inline the short flow
+   command header.
Signature · 17 lines
And you're standing here beside me, I love the passing of time.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#8 lore raw

[virtio-comment] RE: [PATCH requirements v4 2/7] net-features: Add low latency transmit queue requirements

Quoted 4 lines

From: David Edmondson <[email protected]>
Sent: Tuesday, August 15, 2023 2:15 PM

Quoted 9 lines
Quoted 2 lines

+3. Packet transmit descriptor should be able to place a small size packet that
+ does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue

memory.

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 70

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.

The presence of "L4 data" seems like a distraction - just give an indication of size ranges being considered.

Yes, its mainly the size consideration.
Will update.
Thanks.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw

PATCH v4 3/7 net-features: Add low latency receive queue requirements

In reply to Parav Pandit

Add requirements for the low latency receive queue.

---
changelog:
v0->v1:
- clarified the requirements further
- added line for the gro case
- added design goals as the motivation for the requirements

---
 net-workstream/features-1.4.md | 45 +++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)
net-workstream/features-1.4.md +44 -1
index 40fa07f..72d04bd 100644
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
-2. Low latency tx virtqueue for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -129,3 +129,46 @@ struct vnet_data_desc desc[2];
 
 9. A flow filter virtqueue also similarly need the ability to inline the short flow
    command header.
+
+### 3.2.2 Low latency rx virtqueue
+0. Design goal:
+   a. Keep packet metadata and buffer data together which is consumed by driver
+      layer and make it available in a single cache line of cpu
+   b. Instead of having per packet descriptors which is complex to scale for
+      the device, supply the page directly to the device to consume it based
+      on packet size
+1. The device should be able to write a packet receive completion that consists
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write limit
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it to
+   64B or CPU cache line size whenever possible.
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */
+struct vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any size
+   from 64B to 9K bytes.
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.
+8. The device should be able to consume multiple pages for a receive GSO stream.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#10 lore raw

[virtio-comment] Re: [PATCH requirements v4 3/7] net-features: Add low latency receive queue requirements

Quoted 27 lines · Parav Pandit

Add requirements for the low latency receive queue.

---
changelog:
v0->v1:
- clarified the requirements further
- added line for the gro case
- added design goals as the motivation for the requirements

---
 net-workstream/features-1.4.md | 45 +++++++++++++++++++++++++++++++++-
 1 file changed, 44 insertions(+), 1 deletion(-)
net-workstream/features-1.4.md +1 -1
index 40fa07f..72d04bd 100644
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
-2. Low latency tx virtqueue for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 137

Quoted 9 lines
net-workstream/features-1.4.md +6 -0
@@ -129,3 +129,46 @@ struct vnet_data_desc desc[2];
 
 9. A flow filter virtqueue also similarly need the ability to inline the short flow
    command header.
+
+### 3.2.2 Low latency rx virtqueue
+0. Design goal:
+   a. Keep packet metadata and buffer data together which is consumed by driver
+      layer and make it available in a single cache line of cpu
+   b. Instead of having per packet descriptors which is complex to scale for

This really is "per buffer" rather than "per packet".

Quoted 37 lines
net-workstream/features-1.4.md +37 -0
+      the device, supply the page directly to the device to consume it based
+      on packet size
+1. The device should be able to write a packet receive completion that consists
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write limit
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it to
+   64B or CPU cache line size whenever possible.
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */
+struct vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any size
+   from 64B to 9K bytes.
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.
+8. The device should be able to consume multiple pages for a receive GSO stream.
Signature · 17 lines
Swimming around in a plastic bag.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#11 lore raw

PATCH v4 4/7 net-features: Add notification coalescing requirements

In reply to Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
 net-workstream/features-1.4.md | 11 +++++++++++
 1 file changed, 11 insertions(+)
net-workstream/features-1.4.md +11 -0
index 72d04bd..cb72442 100644
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
 # 2. Summary
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
    which can be recycled by the driver when the packets from the completed
    page is fully consumed.
 8. The device should be able to consume multiple pages for a receive GSO stream.
+
+## 3.3 Virtqueue notification coalescing re-arming support
+0. Design goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and when such
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#12 lore raw

[virtio-comment] Re: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 36 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
  net-workstream/features-1.4.md | 11 +++++++++++
  1 file changed, 11 insertions(+)
net-workstream/features-1.4.md +1 -0
index 72d04bd..cb72442 100644
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
  # 2. Summary
  1. Device counters visible to the driver
  2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
  # 3. Requirements
  ## 3.1 Device counters
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
     which can be recycled by the driver when the packets from the completed
     page is fully consumed.
  8. The device should be able to consume multiple pages for a receive GSO stream.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+
+## 3.3 Virtqueue notification coalescing re-arming support
+0. Design goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and when such
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.

I'm wondering how this relates to the existing notification coalesing[1] and notification suppression[2]:

[1]
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}.

[2]
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
\begin{itemize}
\item The driver MUST ignore the \field{avail_event} value.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If \field{flags} is 1, the driver SHOULD NOT send a notification. \item If \field{flags} is 0, the driver MUST send a notification. \end{itemize}
\end{itemize}

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
\begin{itemize}
\item The driver MUST ignore the lower bit of \field{flags}.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If the \field{idx} field in the available ring (which determined where that descriptor index was placed) was equal to \field{avail_event}, the driver MUST send a notification. \item Otherwise the driver SHOULD NOT send a notification. \end{itemize}
\end{itemize}

Regarding notification suppression:
1.When there is VIRTIO_NET_F_EVENT_IDX, even if the notification coalesing condition is met, we need to wait for
the used_event notification condition to be met(the driver does not rearms the notification of the virtqueue now and the avail ring  is set VRING_AVAIL_F_NO_INTERRUPT in flag). 2.When there is no VIRTIO_NET_F_EVENT_IDX, if the driver turns off the notification, even if the notidication condition is met, the device cannot send the notification.

Therefore, if I'm not wrong, a device can issue a notification only if the device is not suppressed from notifying the driver. [1][2] seems to have met this condition.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#13 lore raw

[virtio-comment] RE: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 96 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 2:01 PM

Quoted 33 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
  net-workstream/features-1.4.md | 11 +++++++++++
  1 file changed, 11 insertions(+)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 72d04bd..cb72442 100644

net-workstream/features-1.4.md +1 -0
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
  # 2. Summary
  1. Device counters visible to the driver
  2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
  # 3. Requirements
  ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 175

Quoted 3 lines
net-workstream/features-1.4.md +0 -0
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
     which can be recycled by the driver when the packets from the completed
     page is fully consumed.
  8. The device should be able to consume multiple pages for a receive GSO

stream.

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+
+## 3.3 Virtqueue notification coalescing re-arming support 0. Design
+goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and when

such

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.

I'm wondering how this relates to the existing notification coalesing[1] and notification suppression[2]:

[1]
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}.

[2]
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
\begin{itemize}
\item The driver MUST ignore the \field{avail_event} value.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If \field{flags} is 1, the driver SHOULD NOT send a notification. \item If \field{flags} is 0, the driver MUST send a notification. \end{itemize}
\end{itemize}

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
\begin{itemize}
\item The driver MUST ignore the lower bit of \field{flags}.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If the \field{idx} field in the available ring (which determined where that descriptor index was placed) was equal to \field{avail_event}, the driver MUST send a notification. \item Otherwise the driver SHOULD NOT send a notification. \end{itemize}
\end{itemize}

Regarding notification suppression:
1.When there is VIRTIO_NET_F_EVENT_IDX, even if the notification coalesing condition is met, we need to wait for the used_event notification condition to be met(the driver does not rearms the notification of the virtqueue now and the avail ring  is set VRING_AVAIL_F_NO_INTERRUPT in flag).
2.When there is no VIRTIO_NET_F_EVENT_IDX, if the driver turns off the notification, even if the notidication condition is met, the device cannot send the notification.

Quoted 3 lines

Therefore, if I'm not wrong, a device can issue a notification only if the device is not suppressed from notifying the driver.
[1][2] seems to have met this condition.

Notification suppression using _EVENT_IDX for non-memory transport is just sub-optimal for two reasons.

1. It requires device to poll on the used event to learn about when to un-suppress. (arm)
2. this bit also controls driver notifications yet again demand device to arbitrarily poll on new descriptors posting

Hence, an efficient scheme is needed and device notifications to be detached from driver notification.
And now that VQ level notification coalescing is in place, which suppresses the device notifications, it is logical to combine it with VQ device notifications.

#14 lore raw

[virtio-comment] Re: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 97 lines · Parav Pandit
Quoted 94 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 2:01 PM

Quoted 33 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
   net-workstream/features-1.4.md | 11 +++++++++++
   1 file changed, 11 insertions(+)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 72d04bd..cb72442 100644

net-workstream/features-1.4.md +1 -0
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
   # 2. Summary
   1. Device counters visible to the driver
   2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
   # 3. Requirements
   ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 175

Quoted 3 lines
net-workstream/features-1.4.md +0 -0
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
      which can be recycled by the driver when the packets from the completed
      page is fully consumed.
   8. The device should be able to consume multiple pages for a receive GSO

stream.

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+
+## 3.3 Virtqueue notification coalescing re-arming support 0. Design
+goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and when

such

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.

I'm wondering how this relates to the existing notification coalesing[1] and notification suppression[2]:

[1]
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}.

[2]
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
\begin{itemize}
\item The driver MUST ignore the \field{avail_event} value.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If \field{flags} is 1, the driver SHOULD NOT send a notification. \item If \field{flags} is 0, the driver MUST send a notification. \end{itemize}
\end{itemize}

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
\begin{itemize}
\item The driver MUST ignore the lower bit of \field{flags}.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If the \field{idx} field in the available ring (which determined where that descriptor index was placed) was equal to \field{avail_event}, the driver MUST send a notification. \item Otherwise the driver SHOULD NOT send a notification. \end{itemize}
\end{itemize}

Regarding notification suppression:
1.When there is VIRTIO_NET_F_EVENT_IDX, even if the notification coalesing condition is met, we need to wait for the used_event notification condition to be met(the driver does not rearms the notification of the virtqueue now and the avail ring  is set VRING_AVAIL_F_NO_INTERRUPT in flag).
2.When there is no VIRTIO_NET_F_EVENT_IDX, if the driver turns off the notification, even if the notidication condition is met, the device cannot send the notification.

Quoted 3 lines

Therefore, if I'm not wrong, a device can issue a notification only if the device is not suppressed from notifying the driver.
[1][2] seems to have met this condition.

Quoted 8 lines

Notification suppression using _EVENT_IDX for non-memory transport is just sub-optimal for two reasons.

1. It requires device to poll on the used event to learn about when to un-suppress. (arm)
2. this bit also controls driver notifications yet again demand device to arbitrarily poll on new descriptors posting

Hence, an efficient scheme is needed and device notifications to be detached from driver notification.
And now that VQ level notification coalescing is in place, which suppresses the device notifications, it is logical to combine it with VQ device notifications.

Let me summarize:
1. When used idx notification is satisfied, but coalescing is not satisfied, the driver continues to suppress device notifications.
2. When used idx notification is not satisfied, even if coalescing is satisfied, the device still cannot notify the driver. I think that's what coalescing does, and the description below has satisfied this behavior:
"The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}."

Or we want to say that it has nothing to do with the used idx notification. When the coalescing is satisfied and the driver rearms the notification of the virtqueue, the device now send a notification.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw

[virtio-comment] RE: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 118 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 6:07 PM

Quoted 25 lines · Parav Pandit
Quoted 25 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 2:01 PM

Quoted 20 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
   net-workstream/features-1.4.md | 11 +++++++++++
   1 file changed, 11 insertions(+)
Quoted 5 lines
Quoted 5 lines

net-workstream/features-1.4.md together is desired while updating the virtio net

Quoted 5 lines
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 72d04bd..cb72442 100644

net-workstream/features-1.4.md +0 -0
@@ -8,6 +8,7 @@ together is desired while updating the virtio net

interface.

Quoted 9 lines
Quoted 9 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 173

Quoted 9 lines
net-workstream/features-1.4.md +1 -0
   # 2. Summary
   1. Device counters visible to the driver
   2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
   # 3. Requirements
   ## 3.1 Device counters
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
      which can be recycled by the driver when the packets from the

completed

Quoted 49 lines
Quoted 49 lines
Quoted 3 lines
net-workstream/features-1.4.md +0 -0
      page is fully consumed.
   8. The device should be able to consume multiple pages for a

receive GSO

stream.

net-workstream/features-1.4.md

Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+
+## 3.3 Virtqueue notification coalescing re-arming support 0.
+Design
+goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and
+when

such

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.

I'm wondering how this relates to the existing notification coalesing[1] and notification suppression[2]:

[1]
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}.

[2]
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
\begin{itemize}
\item The driver MUST ignore the \field{avail_event} value.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If \field{flags} is 1, the driver SHOULD NOT send a notification. \item If \field{flags} is 0, the driver MUST send a notification. \end{itemize}
\end{itemize}

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
\begin{itemize}
\item The driver MUST ignore the lower bit of \field{flags}.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If the \field{idx} field in the available ring (which determined where that descriptor index was placed) was equal to \field{avail_event}, the driver MUST send a notification. \item Otherwise the driver SHOULD NOT send a notification. \end{itemize}
\end{itemize}

Quoted 5 lines
Quoted 5 lines

Regarding notification suppression:
1.When there is VIRTIO_NET_F_EVENT_IDX, even if the notification coalesing condition is met, we need to wait for the used_event notification condition to be met(the driver does not rearms the notification of the virtqueue now and the avail ring  is set

VRING_AVAIL_F_NO_INTERRUPT in flag).

Quoted 8 lines
Quoted 7 lines

2.When there is no VIRTIO_NET_F_EVENT_IDX, if the driver turns off the notification, even if the notidication condition is met, the device cannot send the notification.

Therefore, if I'm not wrong, a device can issue a notification only if the device is not suppressed from notifying the driver. [1][2] seems to have met this condition.

Notification suppression using _EVENT_IDX for non-memory transport is just

sub-optimal for two reasons.

Quoted 6 lines

1. It requires device to poll on the used event to learn about when to un-suppress. (arm) 2. this bit also controls driver notifications yet again demand device to arbitrarily poll on new descriptors posting

Hence, an efficient scheme is needed and device notifications to be detached

Quoted 8 lines

from driver notification.

Quoted 1 line

And now that VQ level notification coalescing is in place, which suppresses

the device notifications, it is logical to combine it with VQ device notifications.

Quoted 1 line

Let me summarize:
1. When used idx notification is satisfied, but coalescing is not satisfied, the driver continues to suppress device notifications.

Ack.

Quoted 2 lines

2. When used idx notification is not satisfied, even if coalescing is satisfied, the device still cannot notify the driver.

Ack.

Quoted 7 lines

I think that's what coalescing does, and the description below has satisfied this behavior:
"The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}."

Ack.
the proposal here is to not use EVENT_IDX scheme, instead driver to enable/disable notification coalescing in different way, even when notification coalescing parameters are configured. And this to be done in fairly fast way (not like a cvq) command. For example like driver notifications.

Quoted 1 line

Or we want to say that it has nothing to do with the used idx notification.

When

Quoted 3 lines

the coalescing is satisfied and the driver rearms the notification of the virtqueue, the device now send a notification.

Right.
F_NOTIFION_ARM is mutually exclusive with F_EVENT_IDX. (Like packed vq is mutually exclusive with split q.)

#16 lore raw

[virtio-comment] Re: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 138 lines · Parav Pandit
Quoted 114 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 6:07 PM

Quoted 24 lines · Parav Pandit
Quoted 24 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 2:01 PM

Quoted 20 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
changelog:
v3->v4:
- no change

v1->v2:
- addressed comments from Stefan
- redrafted the requirements to use rearm term and avoid queue enable confusion
v0->v1:
- updated the description

---
    net-workstream/features-1.4.md | 11 +++++++++++
    1 file changed, 11 insertions(+)
Quoted 5 lines
Quoted 5 lines

net-workstream/features-1.4.md together is desired while updating the virtio net

Quoted 5 lines
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 72d04bd..cb72442 100644

net-workstream/features-1.4.md +0 -0
@@ -8,6 +8,7 @@ together is desired while updating the virtio net

interface.

Quoted 9 lines
Quoted 9 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 173

Quoted 9 lines
net-workstream/features-1.4.md +1 -0
    # 2. Summary
    1. Device counters visible to the driver
    2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
    # 3. Requirements
    ## 3.1 Device counters
@@ -172,3 +173,13 @@ struct vnet_rx_completion {
       which can be recycled by the driver when the packets from the

completed

Quoted 49 lines
Quoted 49 lines
Quoted 3 lines
net-workstream/features-1.4.md +0 -0
       page is fully consumed.
    8. The device should be able to consume multiple pages for a

receive GSO

stream.

net-workstream/features-1.4.md

Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+
+## 3.3 Virtqueue notification coalescing re-arming support 0.
+Design
+goal:
+   a. Avoid constant notifications from the device even in conditions when
+      the driver may not have acted on the previous pending notification.
+1. When Tx and Rx virtqueue notification coalescing is enabled, and
+when

such

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   a notification is reported by the device, the device stops sending further
+   notifications until the driver rearms the notifications of the virtqueue.
+2. When the driver rearms the notification of the virtqueue, the device
+   to notify again if notification coalescing conditions are met.

I'm wondering how this relates to the existing notification coalesing[1] and notification suppression[2]:

[1]
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}.

[2]
If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
\begin{itemize}
\item The driver MUST ignore the \field{avail_event} value.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If \field{flags} is 1, the driver SHOULD NOT send a notification. \item If \field{flags} is 0, the driver MUST send a notification. \end{itemize}
\end{itemize}

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
\begin{itemize}
\item The driver MUST ignore the lower bit of \field{flags}.
\item After the driver writes a descriptor index into the available ring:
   \begin{itemize}
         \item If the \field{idx} field in the available ring (which determined where that descriptor index was placed) was equal to \field{avail_event}, the driver MUST send a notification. \item Otherwise the driver SHOULD NOT send a notification. \end{itemize}
\end{itemize}

Quoted 5 lines
Quoted 5 lines

Regarding notification suppression:
1.When there is VIRTIO_NET_F_EVENT_IDX, even if the notification coalesing condition is met, we need to wait for the used_event notification condition to be met(the driver does not rearms the notification of the virtqueue now and the avail ring  is set

VRING_AVAIL_F_NO_INTERRUPT in flag).

Quoted 8 lines
Quoted 7 lines

2.When there is no VIRTIO_NET_F_EVENT_IDX, if the driver turns off the notification, even if the notidication condition is met, the device cannot send the notification.

Therefore, if I'm not wrong, a device can issue a notification only if the device is not suppressed from notifying the driver. [1][2] seems to have met this condition.

Notification suppression using _EVENT_IDX for non-memory transport is just

sub-optimal for two reasons.

Quoted 4 lines

1. It requires device to poll on the used event to learn about when to un-suppress. (arm) 2. this bit also controls driver notifications yet again demand device to arbitrarily poll on new descriptors posting

Quoted 7 lines
Quoted 1 line

Hence, an efficient scheme is needed and device notifications to be detached

from driver notification.

Quoted 1 line

And now that VQ level notification coalescing is in place, which suppresses

the device notifications, it is logical to combine it with VQ device notifications. Let me summarize:
1. When used idx notification is satisfied, but coalescing is not satisfied, the driver continues to suppress device notifications.

Ack.

Quoted 2 lines

2. When used idx notification is not satisfied, even if coalescing is satisfied, the device still cannot notify the driver.

Ack.

Quoted 7 lines

I think that's what coalescing does, and the description below has satisfied this behavior:
"The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Supppression}."

Ack.
the proposal here is to not use EVENT_IDX scheme, instead driver to enable/disable notification coalescing in different way, even when notification coalescing parameters are configured. And this to be done in fairly fast way (not like a cvq) command. For example like driver notifications.

Quoted 7 lines
Quoted 1 line

Or we want to say that it has nothing to do with the used idx notification.

When

Quoted 3 lines

the coalescing is satisfied and the driver rearms the notification of the virtqueue, the device now send a notification.

Right.
F_NOTIFION_ARM is mutually exclusive with F_EVENT_IDX.

OK, I think I get your point, F_NOTIFION_ARM is mutually exclusive
with VIRTQ_AVAIL_F_NO_INTERRUPT\used idx\VIRTIO_F_NOTIFY_ON_EMPT,
and it seems that F_NOTIFION_ARM has the highest priority, and it needs a new feature bit. Am I right :)?

Thanks!

Quoted 1 line

(Like packed vq is mutually exclusive with split q.)

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#17 lore raw

[virtio-comment] RE: [PATCH requirements v4 4/7] net-features: Add notification coalescing requirements

Quoted 6 lines

From: Heng Qi <[email protected]>
Sent: Thursday, August 17, 2023 10:44 AM

Quoted 7 lines
Quoted 1 line

F_NOTIFION_ARM is mutually exclusive with F_EVENT_IDX.

OK, I think I get your point, F_NOTIFION_ARM is mutually exclusive with VIRTQ_AVAIL_F_NO_INTERRUPT\used idx\VIRTIO_F_NOTIFY_ON_EMPT, and it seems that F_NOTIFION_ARM has the highest priority, and it needs a new feature bit. Am I right :)?

Yes for the new feature bit.
Since its mutually exclusive, there is no notion of priority.

Quoted 3 lines

Thanks!

Quoted 1 line

(Like packed vq is mutually exclusive with split q.)

#18 lore raw

PATCH v4 5/7 net-features: Add n-tuple receive flow filters requirements

In reply to Parav Pandit

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
 net-workstream/features-1.4.md | 151 +++++++++++++++++++++++++++++++++
 1 file changed, 151 insertions(+)
net-workstream/features-1.4.md +151 -0
index cb72442..78bb3d2 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
+4  Virtqueue receive flow filters (RFF)
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
    notifications until the driver rearms the notifications of the virtqueue.
 2. When the driver rearms the notification of the virtqueue, the device
    to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF)
+0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven receive
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for ETHTOOL_SRXCLSRLINS
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec to 1M/sec
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow filter queue
+   count and its start vq index to the driver.
+3. As each device may be operating for different performance characteristic,
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.
+5. Flow filter operations are often accelerated by device in a hardware. Ability
+   to handle them on a queue other than control vq is desired. This achieves near
+   zero modifications to existing implementations to add new operations on new
+   purpose built queues (similar to transmit and receive queue).
+   Therefore, when flow filter queues are supported, it is strongly recommended
+   to use it, when flow filter queues are not supported, if the device support
+   it using cvq, driver should be able to use over cvq.
+6. The filter masks are optional; the device should be able to expose if it
+   support filter masks.
+7. The driver may want to have priority among group of flow entries; to facilitate
+   the device support grouping flow filter entries by a notion of a flow group.
+   Each flow group defines priority in processing flow.
+8. The driver and group owner driver should be able to query supported device
+   limits for the receive flow filters.
+
+### 3.4.2 flow operations path
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   be steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also optionally include the field mask.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination is a receive virtqueue index.
+7. Receive packet processing chain is:
+   a. filters programmed using cvq commands VIRTIO_NET_CTRL_RX,
+      VIRTIO_NET_CTRL_MAC and VIRTIO_NET_CTRL_VLAN.
+   b. filters programmed using RFF functiionality.
+   c. filters programmed using RSS VIRTIO_NET_CTRL_MQ_RSS_CONFIG command.
+   Whichever filtering and steering functionality is enabled, they are applied
+   in the above order.
+9. If multiple entries are programmed which has overlapping filtering attributes
+   for a received packet, the driver to define the location/priority of the entry.
+10. The filter entries are usually short in size of few tens of bytes,
+   for example IPv6 + TCP tuple would be 36 bytes, and ops/sec rate is
+   high, hence supplying fields inside the queue descriptor is preferred for
+   up to a certain fixed size, say 96 bytes.
+11. A flow filter entry consists of (a) match criteria, (b) action,
+    (c) destination and (d) a unique 32 bit flow id, all supplied by the
+    driver.
+12. The driver should be able to query and delete flow filter entry by the
+    the device by the flow id.
+
+### 3.4.3 interface example
+
+1. Flow filter capabilities to query using a DMA interface such as cvq
+using two different commands.
+
+```
+/* command 1 */
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;
+	le32 max_flow_filters; /* max flow_id in add/del 
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;
+};
+
+/* command 2 */
+struct flow_filter_fields_support_mask {
+	le64 supported_packet_field_mask_bmap[1];
+};
+
+```
+
+2. Group add/delete cvq commands:
+```
+
+struct virtio_net_rff_group_add {
+	le16 priority;
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {
+	le16 group_id;
+
+```
+
+3. Flow filter entry add/modify, delete over flow vq:
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+
+```
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct virtio_net_rff_add_modify or 
+				     * struct virtio_net_rff_delete
+                                     */
+};
+
+```
+
+### 3.4.4 For incremental future
+a. Driver should be able to specify a specific packet byte offset, number
+   of bytes and mask as math criteria.
+b. Support RSS context, in addition to a specific RQ.
+c. If/when virtio switch object is implemented, support ingress/egress flow
+   filters at the switch port level.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#19 lore raw

[virtio-comment] RE: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Comments below from today's bi-weekly meeting to address in v5.

Quoted 81 lines

From: Parav Pandit <[email protected]>
Sent: Tuesday, August 15, 2023 1:16 PM

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
 net-workstream/features-1.4.md | 151
+++++++++++++++++++++++++++++++++
 1 file changed, 151 insertions(+)
net-workstream/features-1.4.md +0 -0
index cb72442..78bb3d2 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
 1. Device counters visible to the driver  2. Low latency tx and rx virtqueues for

PCI transport 3. Virtqueue notification coalescing re-arming support +4 Virtqueue receive flow filters (RFF)

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +6 -0
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
    notifications until the driver rearms the notifications of the virtqueue.
 2. When the driver rearms the notification of the virtqueue, the device
    to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF) 0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven receive
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for

ETHTOOL_SRXCLSRLINS

net-workstream/features-1.4.md +4 -0
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec to

1M/sec

net-workstream/features-1.4.md +3 -0
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow filter

queue

net-workstream/features-1.4.md +10 -0
+   count and its start vq index to the driver.
+3. As each device may be operating for different performance characteristic,
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.
+5. Flow filter operations are often accelerated by device in a hardware. Ability
+   to handle them on a queue other than control vq is desired. This achieves

near
+ zero modifications to existing implementations to add new operations on
new

net-workstream/features-1.4.md +2 -0
+   purpose built queues (similar to transmit and receive queue).
+   Therefore, when flow filter queues are supported, it is strongly

recommended

net-workstream/features-1.4.md struct vnet_rx_completion { · line 215

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   to use it, when flow filter queues are not supported, if the device support
+   it using cvq, driver should be able to use over cvq.

Rephase is like below.
0. Flow filter queues and flow filter commands on cvq are mutually exclusive.

1. When flow queues are supported, driver should create flow filter queues and use it. (Since cvq is not enabled for flow filters, any flow filter command coming on cvq must fail).

2. If driver wants to use flow filters over cvq, driver must explicitly enable flow filters on cvq via a command, when it is enabled on the cvq driver cannot use flow filter queues. This eliminates any synchronization needed by the device among different types of queues.

Quoted 55 lines

+6. The filter masks are optional; the device should be able to expose if it
+ support filter masks.
+7. The driver may want to have priority among group of flow entries; to facilitate

net-workstream/features-1.4.md +3 -0
+   the device support grouping flow filter entries by a notion of a flow group.
+   Each flow group defines priority in processing flow.
+8. The driver and group owner driver should be able to query supported

device

net-workstream/features-1.4.md +9 -0
+   limits for the receive flow filters.
+
+### 3.4.2 flow operations path
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   be steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as

mac

net-workstream/features-1.4.md +9 -0
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also optionally include the field mask.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination is a receive virtqueue index.
+7. Receive packet processing chain is:
+   a. filters programmed using cvq commands VIRTIO_NET_CTRL_RX,
+      VIRTIO_NET_CTRL_MAC and VIRTIO_NET_CTRL_VLAN.
+   b. filters programmed using RFF functiionality.
+   c. filters programmed using RSS VIRTIO_NET_CTRL_MQ_RSS_CONFIG

command.

net-workstream/features-1.4.md +27 -0
+   Whichever filtering and steering functionality is enabled, they are applied
+   in the above order.
+9. If multiple entries are programmed which has overlapping filtering attributes
+   for a received packet, the driver to define the location/priority of the entry.
+10. The filter entries are usually short in size of few tens of bytes,
+   for example IPv6 + TCP tuple would be 36 bytes, and ops/sec rate is
+   high, hence supplying fields inside the queue descriptor is preferred for
+   up to a certain fixed size, say 96 bytes.
+11. A flow filter entry consists of (a) match criteria, (b) action,
+    (c) destination and (d) a unique 32 bit flow id, all supplied by the
+    driver.
+12. The driver should be able to query and delete flow filter entry by the
+    the device by the flow id.
+
+### 3.4.3 interface example
+
+1. Flow filter capabilities to query using a DMA interface such as cvq
+using two different commands.
+
+```
+/* command 1 */
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;

net-workstream/features-1.4.md struct vnet_rx_completion { · line 276

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+	le32 max_flow_filters; /* max flow_id in add/del
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;
+};
+
+/* command 2 */
+struct flow_filter_fields_support_mask {
+	le64 supported_packet_field_mask_bmap[1];
+};

Explain this bitmap that it indicates well known packet field such as src mac, dest ip, etc.

Also expose it on AQ command so that live migration flow/provision flow can decide which device to use.

Quoted 61 lines
net-workstream/features-1.4.md +48 -0
+
+```
+
+2. Group add/delete cvq commands:
+```
+
+struct virtio_net_rff_group_add {
+	le16 priority;
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {
+	le16 group_id;
+
+```
+
+3. Flow filter entry add/modify, delete over flow vq:
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+
+```
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct

virtio_net_rff_add_modify or

net-workstream/features-1.4.md +11 -1
+				     * struct virtio_net_rff_delete
+                                     */ };
+
+```
+
+### 3.4.4 For incremental future
+a. Driver should be able to specify a specific packet byte offset, number
+   of bytes and mask as math criteria.
+b. Support RSS context, in addition to a specific RQ.
+c. If/when virtio switch object is implemented, support ingress/egress flow
+   filters at the switch port level.
--
Quoted 1 line

2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#20 lore raw

[virtio-comment] Re: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 1 line · Parav Pandit

Comments below from today's bi-weekly meeting to address in v5.

Thanks Parav!

Quoted 67 lines
Quoted 67 lines

From: Parav Pandit <[email protected]>
Sent: Tuesday, August 15, 2023 1:16 PM

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
  net-workstream/features-1.4.md | 151
+++++++++++++++++++++++++++++++++
  1 file changed, 151 insertions(+)
net-workstream/features-1.4.md +0 -0
index cb72442..78bb3d2 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
  1. Device counters visible to the driver  2. Low latency tx and rx virtqueues for

PCI transport 3. Virtqueue notification coalescing re-arming support +4 Virtqueue receive flow filters (RFF)

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +6 -0
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
     notifications until the driver rearms the notifications of the virtqueue.
  2. When the driver rearms the notification of the virtqueue, the device
     to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF) 0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven receive
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for

ETHTOOL_SRXCLSRLINS

net-workstream/features-1.4.md +4 -0
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec to

1M/sec

net-workstream/features-1.4.md +3 -0
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow filter

queue

Quoted 8 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 208

Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+   count and its start vq index to the driver.
+3. As each device may be operating for different performance characteristic,
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.

Regarding this description, I want to say that ARFS will be enabled at runtime. But ethtool RFF will be used at any time as long as the device is ready.

Combining what was discussed in today's meeting, flow vqs and ctrlq are mutually exclusive, so if flow vqs are supported, then ethtool RFF can use flow vq:

"
0. Flow filter queues and flow filter commands on cvq are mutually exclusive.

1. When flow queues are supported, the driver should create flow filter queues and use it. (Since cvq is not enabled for flow filters, any flow filter command coming on cvq must fail).

2. If driver wants to use flow filters over cvq, driver must explicitly enable flow filters on cvq via a command, when it is enabled on the cvq driver cannot use flow filter queues. This eliminates any synchronization needed by the device among different types of queues.
"

Well the "likely create these queues when flow filters are enabled" described here is confusing. Because if ethtool RFF is used, we need to create a flow vq in the probe stage, right?

There are several other reasons:
1. The behavior of dynamically creating flow vq will break the current virtio spec. Please see the "Device Initialization" chapter. ctrlq, as a configuration queue similar to flow vq, is also created in the probe phase. So if we support the "dynamically creating", we need to update the spec.

2. Flow vq is similar to transmit q, and does not need to fill descriptors in advance, so the consumption of resources is relatively small.

3. Dynamic creation of virtqueue seems to be a new thread of virtio spec, and it should also be applicable to rxqs and txqs. We can temporarily support creating flow vq in the probe stage, and subsequent dynamic creation can be an extension.

So, should we create the flow vqs at the initial stage of the driver probe?

Thanks!

Quoted 143 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 215

Quoted 10 lines

+5. Flow filter operations are often accelerated by device in a hardware. Ability
+ to handle them on a queue other than control vq is desired. This achieves
near
+ zero modifications to existing implementations to add new operations on
new

net-workstream/features-1.4.md +2 -0
+   purpose built queues (similar to transmit and receive queue).
+   Therefore, when flow filter queues are supported, it is strongly

recommended

net-workstream/features-1.4.md +2 -0
+   to use it, when flow filter queues are not supported, if the device support
+   it using cvq, driver should be able to use over cvq.

Rephase is like below.
0. Flow filter queues and flow filter commands on cvq are mutually exclusive.

1. When flow queues are supported, driver should create flow filter queues and use it. (Since cvq is not enabled for flow filters, any flow filter command coming on cvq must fail).

2. If driver wants to use flow filters over cvq, driver must explicitly enable flow filters on cvq via a command, when it is enabled on the cvq driver cannot use flow filter queues. This eliminates any synchronization needed by the device among different types of queues.

Quoted 55 lines

+6. The filter masks are optional; the device should be able to expose if it
+ support filter masks.
+7. The driver may want to have priority among group of flow entries; to facilitate

net-workstream/features-1.4.md +3 -0
+   the device support grouping flow filter entries by a notion of a flow group.
+   Each flow group defines priority in processing flow.
+8. The driver and group owner driver should be able to query supported

device

net-workstream/features-1.4.md +9 -0
+   limits for the receive flow filters.
+
+### 3.4.2 flow operations path
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   be steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as

mac

net-workstream/features-1.4.md +9 -0
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also optionally include the field mask.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination is a receive virtqueue index.
+7. Receive packet processing chain is:
+   a. filters programmed using cvq commands VIRTIO_NET_CTRL_RX,
+      VIRTIO_NET_CTRL_MAC and VIRTIO_NET_CTRL_VLAN.
+   b. filters programmed using RFF functiionality.
+   c. filters programmed using RSS VIRTIO_NET_CTRL_MQ_RSS_CONFIG

command.

net-workstream/features-1.4.md +27 -0
+   Whichever filtering and steering functionality is enabled, they are applied
+   in the above order.
+9. If multiple entries are programmed which has overlapping filtering attributes
+   for a received packet, the driver to define the location/priority of the entry.
+10. The filter entries are usually short in size of few tens of bytes,
+   for example IPv6 + TCP tuple would be 36 bytes, and ops/sec rate is
+   high, hence supplying fields inside the queue descriptor is preferred for
+   up to a certain fixed size, say 96 bytes.
+11. A flow filter entry consists of (a) match criteria, (b) action,
+    (c) destination and (d) a unique 32 bit flow id, all supplied by the
+    driver.
+12. The driver should be able to query and delete flow filter entry by the
+    the device by the flow id.
+
+### 3.4.3 interface example
+
+1. Flow filter capabilities to query using a DMA interface such as cvq
+using two different commands.
+
+```
+/* command 1 */
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;

net-workstream/features-1.4.md struct vnet_rx_completion { · line 276

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+	le32 max_flow_filters; /* max flow_id in add/del
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;
+};
+
+/* command 2 */
+struct flow_filter_fields_support_mask {
+	le64 supported_packet_field_mask_bmap[1];
+};

Explain this bitmap that it indicates well known packet field such as src mac, dest ip, etc.

Also expose it on AQ command so that live migration flow/provision flow can decide which device to use.

Quoted 54 lines
net-workstream/features-1.4.md +48 -0
+
+```
+
+2. Group add/delete cvq commands:
+```
+
+struct virtio_net_rff_group_add {
+	le16 priority;
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {
+	le16 group_id;
+
+```
+
+3. Flow filter entry add/modify, delete over flow vq:
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+
+```
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct

virtio_net_rff_add_modify or

net-workstream/features-1.4.md +5 -0
+				     * struct virtio_net_rff_delete
+                                     */ };
+
+```
+
Quoted 8 lines
Quoted 8 lines
net-workstream/features-1.4.md +6 -1
+### 3.4.4 For incremental future
+a. Driver should be able to specify a specific packet byte offset, number
+   of bytes and mask as math criteria.
+b. Support RSS context, in addition to a specific RQ.
+c. If/when virtio switch object is implemented, support ingress/egress flow
+   filters at the switch port level.
--

2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#21 lore raw

[virtio-comment] RE: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 85 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 1:08 PM

Quoted 45 lines
Quoted 45 lines

From: Parav Pandit <[email protected]>
Sent: Tuesday, August 15, 2023 1:16 PM

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
  net-workstream/features-1.4.md | 151
+++++++++++++++++++++++++++++++++
  1 file changed, 151 insertions(+)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index cb72442..78bb3d2 100644

net-workstream/features-1.4.md +0 -0
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
  1. Device counters visible to the driver  2. Low latency tx and rx

virtqueues for PCI transport 3. Virtqueue notification coalescing re-arming support
+4 Virtqueue receive flow filters (RFF)

Quoted 10 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 190

Quoted 10 lines

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +4 -0
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
     notifications until the driver rearms the notifications of the virtqueue.
  2. When the driver rearms the notification of the virtqueue, the device
     to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF) 0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven

receive

Quoted 8 lines
Quoted 8 lines
net-workstream/features-1.4.md +2 -0
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for

ETHTOOL_SRXCLSRLINS

net-workstream/features-1.4.md +5 -0
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec
+to
Quoted 8 lines
Quoted 8 lines

1M/sec

net-workstream/features-1.4.md +4 -0
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow
+filter

queue
+ count and its start vq index to the driver.
+3. As each device may be operating for different performance

characteristic,

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 208

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.
Quoted 3 lines

Regarding this description, I want to say that ARFS will be enabled at runtime. But ethtool RFF will be used at any time as long as the device is ready.

Yes, but ethool RFS is blocking callback in which slow task such as q creation can be done, only when one wants to add flows. ARFS is anyway controlled using set_features() callback.

Quoted 12 lines

Combining what was discussed in today's meeting, flow vqs and ctrlq are mutually exclusive, so if flow vqs are supported, then ethtool RFF can use flow vq:

"
0. Flow filter queues and flow filter commands on cvq are mutually exclusive.

1. When flow queues are supported, the driver should create flow filter queues and use it.
(Since cvq is not enabled for flow filters, any flow filter command coming on cvq must fail).

Quoted 7 lines

2. If driver wants to use flow filters over cvq, driver must explicitly enable flow filters on cvq via a command, when it is enabled on the cvq driver cannot use flow filter queues.
This eliminates any synchronization needed by the device among different types of queues.
"

Ack.

Quoted 5 lines

Well the "likely create these queues when flow filters are enabled" described here is confusing.
Because if ethtool RFF is used, we need to create a flow vq in the probe stage, right?

Current spec wording limits one to create queues before DRIVER_OK.
But with introduction of _RESET bit one can create an empty queue and disable it (reset it! What a grand name).

And re-enable it during ethtool callbacks.
This would be workaround to dynamically create the queue.

Quoted 8 lines

There are several other reasons:
1. The behavior of dynamically creating flow vq will break the current virtio spec.
   Please see the "Device Initialization" chapter. ctrlq, as a configuration queue similar to flow vq, is also created in the probe phase. So if we support the
"dynamically creating",
   we need to update the spec.

Quoted 4 lines

2. Flow vq is similar to transmit q, and does not need to fill descriptors in advance,
   so the consumption of resources is relatively small.

Only the queue descriptors memory is consumed, which is not a lot.
But concept of creating resource without consuming is just bad.
We learnt the lesson from mlx5 driver that dynamic creation is efficient.
Many part of Linux kernel also moving in this direction, all the way upto dynamically individual msix vector.

So we should strive to enable them dynamically and improve the virtio spec.

It should an orthogonal feature, sadly how the RING_RESET feature is done. :(

Quoted 7 lines

3. Dynamic creation of virtqueue seems to be a new thread of virtio spec, and it should also be
   applicable to rxqs and txqs. We can temporarily support creating flow vq in
the probe stage,
   and subsequent dynamic creation can be an extension.

So, should we create the flow vqs at the initial stage of the driver probe?

One option is to follow the above workaround.
Second option is to add feature bit to feature bit to indicate dynamic Q creation.

#22 lore raw

[virtio-comment] Re: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 114 lines · Parav Pandit
Quoted 76 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 1:08 PM

Quoted 45 lines
Quoted 45 lines

From: Parav Pandit <[email protected]>
Sent: Tuesday, August 15, 2023 1:16 PM

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
   net-workstream/features-1.4.md | 151
+++++++++++++++++++++++++++++++++
   1 file changed, 151 insertions(+)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index cb72442..78bb3d2 100644

net-workstream/features-1.4.md +0 -0
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
   1. Device counters visible to the driver  2. Low latency tx and rx

virtqueues for PCI transport 3. Virtqueue notification coalescing re-arming support
+4 Virtqueue receive flow filters (RFF)

Quoted 10 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 190

Quoted 10 lines

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +4 -0
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
      notifications until the driver rearms the notifications of the virtqueue.
   2. When the driver rearms the notification of the virtqueue, the device
      to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF) 0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven

receive

Quoted 8 lines
Quoted 8 lines
net-workstream/features-1.4.md +2 -0
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for

ETHTOOL_SRXCLSRLINS

net-workstream/features-1.4.md +5 -0
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec
+to
Quoted 8 lines
Quoted 8 lines

1M/sec

net-workstream/features-1.4.md +4 -0
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow
+filter

queue
+ count and its start vq index to the driver.
+3. As each device may be operating for different performance

characteristic,

Quoted 9 lines
Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 208

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.

Regarding this description, I want to say that ARFS will be enabled at runtime. But ethtool RFF will be used at any time as long as the device is ready.

Yes, but ethool RFS is blocking callback in which slow task such as q creation can be done, only when one wants to add flows. ARFS is anyway controlled using set_features() callback.

Quoted 12 lines

Combining what was discussed in today's meeting, flow vqs and ctrlq are mutually exclusive, so if flow vqs are supported, then ethtool RFF can use flow vq:

"
0. Flow filter queues and flow filter commands on cvq are mutually exclusive.

1. When flow queues are supported, the driver should create flow filter queues and use it.
(Since cvq is not enabled for flow filters, any flow filter command coming on cvq must fail).

Quoted 7 lines

2. If driver wants to use flow filters over cvq, driver must explicitly enable flow filters on cvq via a command, when it is enabled on the cvq driver cannot use flow filter queues.
This eliminates any synchronization needed by the device among different types of queues.
"

Ack.

Quoted 5 lines

Well the "likely create these queues when flow filters are enabled" described here is confusing.
Because if ethtool RFF is used, we need to create a flow vq in the probe stage, right?

Quoted 5 lines

Current spec wording limits one to create queues before DRIVER_OK.
But with introduction of _RESET bit one can create an empty queue and disable it (reset it! What a grand name).

And re-enable it during ethtool callbacks.
This would be workaround to dynamically create the queue.

Yes, this is workaround, we can just set the number of flow vqs for the device, but not allocate resources nor enable. But this is not exhaustive, because xdp may also require dynamic q creation/destruction.

Quoted 9 lines
Quoted 9 lines

There are several other reasons:
1. The behavior of dynamically creating flow vq will break the current virtio spec.
   Please see the "Device Initialization" chapter. ctrlq, as a configuration queue similar to flow vq, is also created in the probe phase. So if we support the
"dynamically creating",
   we need to update the spec.

Quoted 8 lines
Quoted 4 lines

2. Flow vq is similar to transmit q, and does not need to fill descriptors in advance,
   so the consumption of resources is relatively small.

Only the queue descriptors memory is consumed, which is not a lot.
But concept of creating resource without consuming is just bad.
We learnt the lesson from mlx5 driver that dynamic creation is efficient.
Many part of Linux kernel also moving in this direction, all the way upto dynamically individual msix vector.

Ok. I got it.

Quoted 4 lines

So we should strive to enable them dynamically and improve the virtio spec.

It should an orthogonal feature, sadly how the RING_RESET feature is done. :(

RING_RESET is performed without changing the number of queues. But what you said above is a workaround.

Quoted 10 lines
Quoted 8 lines

3. Dynamic creation of virtqueue seems to be a new thread of virtio spec, and it should also be
   applicable to rxqs and txqs. We can temporarily support creating flow vq in
the probe stage,
   and subsequent dynamic creation can be an extension.

So, should we create the flow vqs at the initial stage of the driver probe?

One option is to follow the above workaround.
Second option is to add feature bit to feature bit to indicate dynamic Q creation.

I'm leaning towards the second option, which makes the work orthogonal and also works in the case of XDP.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#23 lore raw

[virtio-comment] RE: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 11 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 4:41 PM

Quoted 2 lines

One option is to follow the above workaround.
Second option is to add feature bit to feature bit to indicate dynamic Q

creation.

I'm leaning towards the second option, which makes the work orthogonal and also works in the case of XDP.

Yes. lets add the bit. I will send patch as part of this work.

#24 lore raw

[virtio-comment] Re: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

In reply to Parav Pandit

Hi, Parav.

There are some minor updates!

Quoted 120 lines · Parav Pandit

Add virtio net device requirements for receive flow filters.

---
changelog:
v3->v4:
- Addressed comments from Satananda, Heng, David
- removed context specific wording, replaced with destination
- added group create/delete examples and updated requirements
- added optional support to use cvq for flor filter commands
- added example of transporting flow filter commands over cvq
- made group size to be 16-bit
- added concept of 0->n max flow filter entries based on max count
- added concept of 0->n max flow group based on max count
- split field bitmask to separate command from other filter capabilities
- rewrote rx filter processing chain order with respect to existing filter commands and rss
- made flow_id flat across all groups
v1->v2:
- split setup and operations requirements
- added design goal
- worded requirements more precisely
v0->v1:
- fixed comments from Heng Li
- renamed receive flow steering to receive flow filters
- clarified byte offset in match criteria

---
  net-workstream/features-1.4.md | 151 +++++++++++++++++++++++++++++++++
  1 file changed, 151 insertions(+)
net-workstream/features-1.4.md +75 -0
index cb72442..78bb3d2 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
  1. Device counters visible to the driver
  2. Low latency tx and rx virtqueues for PCI transport
  3. Virtqueue notification coalescing re-arming support
+4  Virtqueue receive flow filters (RFF)
 
  # 3. Requirements
  ## 3.1 Device counters
@@ -183,3 +184,153 @@ struct vnet_rx_completion {
     notifications until the driver rearms the notifications of the virtqueue.
  2. When the driver rearms the notification of the virtqueue, the device
     to notify again if notification coalescing conditions are met.
+
+## 3.4 Virtqueue receive flow filters (RFF)
+0. Design goal:
+   To filter and/or to steer packet based on specific pattern match to a
+   specific destination to support application/networking stack driven receive
+   processing.
+1. Two use cases are: to support Linux netdev set_rxnfc() for ETHTOOL_SRXCLSRLINS
+   and to support netdev feature NETIF_F_NTUPLE aka ARFS.
+
+### 3.4.1 control path
+1. The number of flow filter operations/sec can range from 100k/sec to 1M/sec
+   or even more. Hence flow filter operations must be done over a queueing
+   interface using one or more queues.
+2. The device should be able to expose one or more supported flow filter queue
+   count and its start vq index to the driver.
+3. As each device may be operating for different performance characteristic,
+   start vq index and count may be different for each device. Secondly, it is
+   inefficient for device to provide flow filters capabilities via a config space
+   region. Hence, the device should be able to share these attributes using
+   dma interface, instead of transport registers.
+4. Since flow filters are enabled much later in the driver life cycle, driver
+   will likely create these queues when flow filters are enabled.
+5. Flow filter operations are often accelerated by device in a hardware. Ability
+   to handle them on a queue other than control vq is desired. This achieves near
+   zero modifications to existing implementations to add new operations on new
+   purpose built queues (similar to transmit and receive queue).
+   Therefore, when flow filter queues are supported, it is strongly recommended
+   to use it, when flow filter queues are not supported, if the device support
+   it using cvq, driver should be able to use over cvq.
+6. The filter masks are optional; the device should be able to expose if it
+   support filter masks.
+7. The driver may want to have priority among group of flow entries; to facilitate
+   the device support grouping flow filter entries by a notion of a flow group.
+   Each flow group defines priority in processing flow.
+8. The driver and group owner driver should be able to query supported device
+   limits for the receive flow filters.
+
+### 3.4.2 flow operations path
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   be steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also optionally include the field mask.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination is a receive virtqueue index.
+7. Receive packet processing chain is:
+   a. filters programmed using cvq commands VIRTIO_NET_CTRL_RX,
+      VIRTIO_NET_CTRL_MAC and VIRTIO_NET_CTRL_VLAN.
+   b. filters programmed using RFF functiionality.
+   c. filters programmed using RSS VIRTIO_NET_CTRL_MQ_RSS_CONFIG command.
+   Whichever filtering and steering functionality is enabled, they are applied
+   in the above order.
+9. If multiple entries are programmed which has overlapping filtering attributes
+   for a received packet, the driver to define the location/priority of the entry.
+10. The filter entries are usually short in size of few tens of bytes,
+   for example IPv6 + TCP tuple would be 36 bytes, and ops/sec rate is
+   high, hence supplying fields inside the queue descriptor is preferred for
+   up to a certain fixed size, say 96 bytes.
+11. A flow filter entry consists of (a) match criteria, (b) action,
+    (c) destination and (d) a unique 32 bit flow id, all supplied by the
+    driver.
+12. The driver should be able to query and delete flow filter entry by the
+    the device by the flow id.
+
+### 3.4.3 interface example
+
+1. Flow filter capabilities to query using a DMA interface such as cvq
+using two different commands.
+
+```
+/* command 1 */

net-workstream/features-1.4.md struct vnet_rx_completion { · line 270

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;
+	le32 max_flow_filters; /* max flow_id in add/del
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;

+ u8 padding[3];

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+};
+
+/* command 2 */
+struct flow_filter_fields_support_mask {

net-workstream/features-1.4.md struct vnet_rx_completion { · line 284

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+	le64 supported_packet_field_mask_bmap[1];
+};
+
+```
+
+2. Group add/delete cvq commands:
+```
+
+struct virtio_net_rff_group_add {
+	le16 priority;

Please explicitly explain the relationship between the number and the priority, for example, the smaller the number, the higher the priority :)

Quoted 5 lines
net-workstream/features-1.4.md +5 -0
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {

net-workstream/features-1.4.md struct vnet_rx_completion { · line 299

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+	le16 group_id;
+
+```
+
+3. Flow filter entry add/modify, delete over flow vq:
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;

s/padding/priority

Each rule needs a priority.

Quoted 16 lines
net-workstream/features-1.4.md +16 -0
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+
+```

net-workstream/features-1.4.md struct vnet_rx_completion { · line 325

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct virtio_net_rff_add_modify or
+				     * struct virtio_net_rff_delete

For flow vq, we no longer distinguish operations by command, but by flow_op.
But for ctrlq, this field will be carried. We should make it clear that when ctrlq is delivered based on cmd, the flow_op field is ignored.

Thanks!

net-workstream/features-1.4.md struct vnet_rx_completion { · line 336

Quoted 11 lines
net-workstream/features-1.4.md +11 -0
+                                     */
+};
+
+```
+
+### 3.4.4 For incremental future
+a. Driver should be able to specify a specific packet byte offset, number
+   of bytes and mask as math criteria.
+b. Support RSS context, in addition to a specific RQ.
+c. If/when virtio switch object is implemented, support ingress/egress flow
+   filters at the switch port level.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#25 lore raw

[virtio-comment] RE: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 4 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 5:12 PM

Quoted 14 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 270

Quoted 11 lines
net-workstream/features-1.4.md +11 -0
+/* command 1 */
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;
+	le32 max_flow_filters; /* max flow_id in add/del
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;

+ u8 padding[3];

Ack.

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 284

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+struct virtio_net_rff_group_add {
+	le16 priority;

Please explicitly explain the relationship between the number and the priority, for example, the smaller the number, the higher the priority :)

Right. Will do.
I was thinking of higher the value higher the priority, so that one doesnt need to invert this in brain every time seeing the priority field. :)

Quoted 9 lines
Quoted 9 lines
net-workstream/features-1.4.md +9 -0
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {
+	le16 group_id;
+
+```
+
Quoted 9 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 299

Quoted 6 lines

+3. Flow filter entry add/modify, delete over flow vq:

net-workstream/features-1.4.md +5 -0
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;

s/padding/priority

Ack.

Quoted 28 lines

Each rule needs a priority.

Quoted 15 lines
net-workstream/features-1.4.md +15 -0
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+```
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct

virtio_net_rff_add_modify or

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 325

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+				     * struct virtio_net_rff_delete

For flow vq, we no longer distinguish operations by command, but by flow_op. But for ctrlq, this field will be carried. We should make it clear that when ctrlq is delivered based on cmd, the flow_op field is ignored.

Since cvq is only the communication medium for delivering of command, it is better use the flow_op as is, and cvq commands field to be ignored. This way, software layers are more organized cvq or flow vq.

#26 lore raw

[virtio-comment] Re: [PATCH requirements v4 5/7] net-features: Add n-tuple receive flow filters requirements

Quoted 18 lines · Parav Pandit
Quoted 16 lines

From: Heng Qi <[email protected]>
Sent: Wednesday, August 16, 2023 5:12 PM

net-workstream/features-1.4.md struct vnet_rx_completion { · line 270

Quoted 11 lines
net-workstream/features-1.4.md +11 -0
+/* command 1 */
+struct flow_filter_capabilities {
+	le16 start_vq_index;
+	le16 num_flow_filter_vqs;
+	le16 max_flow_groups;
+	le16 max_group_priorities; /* max priorities of the group */
+	le32 max_flow_filters_per_group;
+	le32 max_flow_filters; /* max flow_id in add/del
+				* is equal = max_flow_filters - 1.
+				*/
+	u8 max_priorities_per_group;

+ u8 padding[3];

Ack.

Quoted 8 lines
Quoted 5 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 284

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+struct virtio_net_rff_group_add {
+	le16 priority;

Please explicitly explain the relationship between the number and the priority, for example, the smaller the number, the higher the priority :)

Right. Will do.
I was thinking of higher the value higher the priority, so that one doesnt need to invert this in brain every time seeing the priority field. :)

It's ok :)

Quoted 47 lines
Quoted 8 lines
Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+	le16 group_id;
+};
+
+
+struct virtio_net_rff_group_delete {
+	le16 group_id;
+
+```
Quoted 9 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 299

Quoted 7 lines
net-workstream/features-1.4.md +7 -0
+
+3. Flow filter entry add/modify, delete over flow vq:
+
+```
+struct virtio_net_rff_add_modify {
+	u8 flow_op;
+	u8 padding;

s/padding/priority

Ack.

Quoted 28 lines

Each rule needs a priority.

Quoted 15 lines
net-workstream/features-1.4.md +15 -0
+	u16 group_id;
+	le32 flow_id;
+	struct match_criteria mc;
+	struct destination dest;
+	struct action action;
+
+	struct match_criteria mask;	/* optional */
+};
+
+struct virtio_net_rff_delete {
+	u8 flow_op;
+	u8 padding[3];
+	le32 flow_id;
+};
+

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+```
+
+4. Flow filter commands over cvq:
+
+```
+
+struct virtio_net_rff_cmd {
+	u8 class; /* RFF class */
+	u8 commands; /* RFF cmd = A */
+	u8 command-specific-data[]; /* contains struct

virtio_net_rff_add_modify or

Quoted 7 lines
Quoted 5 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 325

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+				     * struct virtio_net_rff_delete

For flow vq, we no longer distinguish operations by command, but by flow_op. But for ctrlq, this field will be carried. We should make it clear that when ctrlq is delivered based on cmd, the flow_op field is ignored.

Since cvq is only the communication medium for delivering of command, it is better use the flow_op as is, and cvq commands field to be ignored. This way, software layers are more organized cvq or flow vq.

Agree.

Thanks!

Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#27 lore raw

PATCH v4 6/7 net-features: Add packet timestamp requirements

In reply to Parav Pandit

Add tx and rx packet timestamp requirements.

---
changelog:
v3->v4:
- no change

---
 net-workstream/features-1.4.md | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
net-workstream/features-1.4.md +26 -0
index 78bb3d2..82b907a 100644
@@ -10,6 +10,7 @@ together is desired while updating the virtio net interface.
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
 4  Virtqueue receive flow filters (RFF)
+5. Device timestamp for tx and rx packets
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -334,3 +335,28 @@ a. Driver should be able to specify a specific packet byte offset, number
 b. Support RSS context, in addition to a specific RQ.
 c. If/when virtio switch object is implemented, support ingress/egress flow
    filters at the switch port level.
+
+## 3.5 Packet timestamp
+1. Device should provide transmit timestamp and receive timestamp of the packets
+   at per packet level when the device is enabled.
+2. Device should provide the current free running clock in the least latency
+   possible using an MMIO register read of 64-bit to have the least jitter.
+3. Device should provide the current frequency and the frequency unit for the
+   software to synchronize the reference point of software and the device using
+   a control vq command.
+
+### 3.5.1 Transmit timestamp
+1. Transmit completion must contain a packet transmission timestamp when the
+   device is enabled for it.
+2. The device should record the packet transmit timestamp in the completion at
+   the farthest egress point towards the network.
+3. The device must provide a transmit packet timestamp in a single DMA
+   transaction along with the rest of the transmit completion fields.
+
+### 3.5.2 Receive timestamp
+1. Receive completion must contain a packet reception timestamp when the device
+   is enabled for it.
+2. The device should record the received packet timestamp at the closet ingress
+   point of reception from the network.
+3. The device should provide a receive packet timestamp in a single DMA
+   transaction along with the rest of the receive completion fields.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#28 lore raw

PATCH v4 7/7 net-features: Add header data split requirements

In reply to Parav Pandit

Add header data split requirements for the receive packets.

---
 net-workstream/features-1.4.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)
net-workstream/features-1.4.md +13 -0
index 82b907a..5e359b6 100644
@@ -11,6 +11,7 @@ together is desired while updating the virtio net interface.
 3. Virtqueue notification coalescing re-arming support
 4  Virtqueue receive flow filters (RFF)
 5. Device timestamp for tx and rx packets
+6. Header data split for the receive virtqueue
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -360,3 +361,15 @@ c. If/when virtio switch object is implemented, support ingress/egress flow
    point of reception from the network.
 3. The device should provide a receive packet timestamp in a single DMA
    transaction along with the rest of the receive completion fields.
+
+## 3.6 Header data split for the receive virtqueue
+1. The device should be able to DMA the packet header and data to two different
+   memory locations, this enables driver and networking stack to perform zero
+   copy to application buffer(s).
+2. The driver should be able to configure maximum header buffer size per
+   virtqueue.
+3. The header buffer to be in a physically contiguous memory per virtqueue
+4. The device should be able to indicate header data split in the receive
+   completion.
+5. The device should be able to zero pad the header buffer when the received
+   header is shorter than cpu cache line size.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
Acked-by on v4 7/7
#29 lore raw

[virtio-comment] Re: [PATCH requirements v4 7/7] net-features: Add header data split requirements

Quoted 3 lines · Parav Pandit

Add header data split requirements for the receive packets.

Quoted 32 lines
---
 net-workstream/features-1.4.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)
net-workstream/features-1.4.md +13 -0
index 82b907a..5e359b6 100644
@@ -11,6 +11,7 @@ together is desired while updating the virtio net interface.
 3. Virtqueue notification coalescing re-arming support
 4  Virtqueue receive flow filters (RFF)
 5. Device timestamp for tx and rx packets
+6. Header data split for the receive virtqueue
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -360,3 +361,15 @@ c. If/when virtio switch object is implemented, support ingress/egress flow
    point of reception from the network.
 3. The device should provide a receive packet timestamp in a single DMA
    transaction along with the rest of the receive completion fields.
+
+## 3.6 Header data split for the receive virtqueue
+1. The device should be able to DMA the packet header and data to two different
+   memory locations, this enables driver and networking stack to perform zero
+   copy to application buffer(s).
+2. The driver should be able to configure maximum header buffer size per
+   virtqueue.
+3. The header buffer to be in a physically contiguous memory per virtqueue
+4. The device should be able to indicate header data split in the receive
+   completion.
+5. The device should be able to zero pad the header buffer when the received
+   header is shorter than cpu cache line size.
Signature · 17 lines
I used to worry, thought I was goin' mad in a hurry.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/