Skip to contentSkip to conversation

virtio net new features requirements

PATCH In review Parav Pandit 6 participants 58 messages Jun 1, 2023 – Jun 14, 2023

lore
Share & tools
lore
#0 lore raw

cover virtio net new features requirements

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

I would like to capture if there are any other requirements linked to the features captured in this document to consider in the interface design.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and work towards the interface design followed by drafting the specification changes.

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

This is not a specification document. It is a pre-work document that helps to understand the draft specification addressing these requirements.

Please review.

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow steering requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

 net-workstream/features-1.4.md | 224 +++++++++++++++++++++++++++++++++
 1 file changed, 224 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH 1/7 net-features: Add requirements document for release 1.4

Add requirements document template for the virtio net features.

Add virtio net device counters visible to driver.

---
 net-workstream/features-1.4.md | 36 ++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
net-workstream/features-1.4.md new file +36 -0
new file mode 100644
index 0000000..03b4eb3
@@ -0,0 +1,36 @@
+# 1. Introduction
+
+This document describes the overall requirements for virtio net device
+improvements for upcoming release 1.4. Some of these requirements are
+interrelated and influence the interface design, hence reviewing them
+together is desired while updating the virtio net interface.
+
+# 2. Summary
+1. Device counters visible to the driver
+
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a control vq command.
+2. The driver should be able to query which counters are supported using a
+   control vq command.
+3. If this device is migrated between two hosts, the driver should be able
+   get the counter values in the destination host from where it was left
+   off in the source host.
+4. If a virtio device is group member device, a group owner should be able
+   to query all the counter attributes using the admin queue command which
+   a virtio device will expose via a control vq to the driver.
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size
+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the
+    buffer in the receive queue
+3. le64 rx_gro_pkts: Packets treated as guest GSO sequence by the device
+4. le64 rx_pkts: Total packets received by the device
+
+### 3.1.2 Per transmit queue counters
+1. le64 tx_bad_desc_errors: Descriptors dropped by the device due to errors in
+   descriptors
+2. le64 tx_gso_pkts: Packets send as host GSO sequence
+3. le64 tx_pkts: Total packets send by the device
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 22 lines · Parav Pandit

Add requirements document template for the virtio net features.

Add virtio net device counters visible to driver.

---
 net-workstream/features-1.4.md | 36 ++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
net-workstream/features-1.4.md new file +6 -0
new file mode 100644
index 0000000..03b4eb3
@@ -0,0 +1,36 @@
+# 1. Introduction
+
+This document describes the overall requirements for virtio net device
+improvements for upcoming release 1.4. Some of these requirements are
+interrelated and influence the interface design, hence reviewing them
+together is desired while updating the virtio net interface.

net-workstream/features-1.4.md line 16

Quoted 10 lines
net-workstream/features-1.4.md new file +10 -0
+
+# 2. Summary
+1. Device counters visible to the driver
+
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a control vq command.
+2. The driver should be able to query which counters are supported using a
+   control vq command.

why does it matter for requirements whether it's a control vq?

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

net-workstream/features-1.4.md line 19

Quoted 3 lines
net-workstream/features-1.4.md new file +3 -0
+3. If this device is migrated between two hosts, the driver should be able
+   get the counter values in the destination host from where it was left
+   off in the source host.

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?
so maybe it means there needs to be a way to set counters? so there's no need to mention migration - just that it should be possible to move counters between devices.

net-workstream/features-1.4.md line 22

Quoted 3 lines
net-workstream/features-1.4.md new file +3 -0
+4. If a virtio device is group member device, a group owner should be able
+   to query all the counter attributes using the admin queue command which
+   a virtio device will expose via a control vq to the driver.

this seems weirdly specific.
what is the actual requirement?

net-workstream/features-1.4.md line 26

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size

with mergeable buffers how does this differ from 2?

net-workstream/features-1.4.md line 29

Quoted 3 lines
net-workstream/features-1.4.md new file +3 -0
+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the
+    buffer in the receive queue
+3. le64 rx_gro_pkts: Packets treated as guest GSO sequence by the device

what does this mean exactly? packets before or after they are combined?

pls stick to device/driver terminology not guest/host

net-workstream/features-1.4.md line 30

Quoted 1 line
net-workstream/features-1.4.md new file +1 -0
+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters
+1. le64 tx_bad_desc_errors: Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

net-workstream/features-1.4.md line 35

Quoted 1 line
net-workstream/features-1.4.md new file +1 -0
+2. le64 tx_gso_pkts: Packets send as host GSO sequence

same questions as gro

net-workstream/features-1.4.md line 36

Quoted 1 line
net-workstream/features-1.4.md new file +1 -0
+3. le64 tx_pkts: Total packets send by the device

sent

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#3 lore raw

RE: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:15 PM

Quoted 9 lines

net-workstream/features-1.4.md line 16

Quoted 6 lines
net-workstream/features-1.4.md new file +6 -0
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a control vq command.
+2. The driver should be able to query which counters are supported using a
+   control vq command.

why does it matter for requirements whether it's a control vq?

It matters for requirements, so we produce design that addresses it.
We don't want to add config space every growing bit map which may be different between different devices.

Quoted 3 lines

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

They don't have to be.
I don't see how it can be every synchronized.

Quoted 7 lines

net-workstream/features-1.4.md line 19

Quoted 4 lines

+3. If this device is migrated between two hosts, the driver should be able

net-workstream/features-1.4.md new file +2 -0
+   get the counter values in the destination host from where it was left
+   off in the source host.

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?

True, but design need to keep this in mind if it has some touch points like of #4 so when migration arrives, it has the building block in place.

Quoted 4 lines

so maybe it means there needs to be a way to set counters?
so there's no need to mention migration - just that it should be possible to move counters between devices.

Quoted 8 lines

net-workstream/features-1.4.md line 22

Quoted 3 lines

+4. If a virtio device is group member device, a group owner should be able

net-workstream/features-1.4.md new file +2 -0
+   to query all the counter attributes using the admin queue command which
+   a virtio device will expose via a control vq to the driver.

this seems weirdly specific.
what is the actual requirement?

I don't follow the question.
When a device migrates from src to dst, group owner needs to know if both side underlying member device has same counter attributes or not.

Quoted 8 lines

net-workstream/features-1.4.md line 26

Quoted 5 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size

with mergeable buffers how does this differ from 2?

Quoted 7 lines

Review

Quoted 4 lines

+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the

(quoted diff) +3 -0
+    buffer in the receive queue
+3. le64 rx_gro_pkts: Packets treated as guest GSO sequence by the
+device

what does this mean exactly? packets before or after they are combined?

Before.

Quoted 2 lines

pls stick to device/driver terminology not guest/host

Yes, will change the GUEST surfaced from the current F_GUEST terminology of the net device.

Quoted 4 lines
Quoted 1 line

+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

Not including. Will add this clarification further in v1.

Quoted 8 lines

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters 1. le64 tx_bad_desc_errors:
+Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

It is left to the device to implement on what to do on bad desc.

Quoted 8 lines
Quoted 1 line

+2. le64 tx_gso_pkts: Packets send as host GSO sequence

same questions as gro

Quoted 1 line

+3. le64 tx_pkts: Total packets send by the device

sent

Ack.

Quoted 1 line
Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#4 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 14 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:15 PM

Quoted 9 lines

net-workstream/features-1.4.md line 16

Quoted 6 lines
net-workstream/features-1.4.md new file +6 -0
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a control vq command.
+2. The driver should be able to query which counters are supported using a
+   control vq command.

why does it matter for requirements whether it's a control vq?

Quoted 2 lines

It matters for requirements, so we produce design that addresses it.
We don't want to add config space every growing bit map which may be different between different devices.

then say that you want to conserve config space, that is the requirement. not cvq specifically.

Quoted 4 lines
Quoted 3 lines

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

They don't have to be.

Then say that.

Quoted 1 line

I don't see how it can be every synchronized.

you could program them through the vq itself.

Quoted 8 lines
Quoted 7 lines

net-workstream/features-1.4.md line 19

Quoted 4 lines

+3. If this device is migrated between two hosts, the driver should be able

net-workstream/features-1.4.md new file +2 -0
+   get the counter values in the destination host from where it was left
+   off in the source host.

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?

True, but design need to keep this in mind if it has some touch points like of #4 so when migration arrives, it has the building block in place.

so again, let's make sure we capture the actual spec requirement.

Quoted 9 lines
Quoted 9 lines

so maybe it means there needs to be a way to set counters?
so there's no need to mention migration - just that it should be possible to move counters between devices.

Quoted 3 lines

+4. If a virtio device is group member device, a group owner should be able

net-workstream/features-1.4.md new file +2 -0
+   to query all the counter attributes using the admin queue command which
+   a virtio device will expose via a control vq to the driver.
Quoted 5 lines
Quoted 3 lines

this seems weirdly specific.
what is the actual requirement?

I don't follow the question.
When a device migrates from src to dst, group owner needs to know if both side underlying member device has same counter attributes or not.

whether it's through a command or not is not a requirement and I still do not know what the requirement is. what does "same counter attributes" mean? you never mentioned attributes before.

Quoted 13 lines
Quoted 13 lines

net-workstream/features-1.4.md line 26

Quoted 5 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size

with mergeable buffers how does this differ from 2?

Quoted 4 lines

+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the

(quoted diff) +3 -0
+    buffer in the receive queue
+3. le64 rx_gro_pkts: Packets treated as guest GSO sequence by the
+device
Quoted 7 lines
Quoted 2 lines

what does this mean exactly? packets before or after they are combined?

Before.

Quoted 2 lines

pls stick to device/driver terminology not guest/host

Yes, will change the GUEST surfaced from the current F_GUEST terminology of the net device.

So? this predates 1.x spec we never bothered changing them.

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

Not including. Will add this clarification further in v1.

Quoted 8 lines

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters 1. le64 tx_bad_desc_errors:
+Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

Quoted 1 line

It is left to the device to implement on what to do on bad desc.

then how can you count drops? why does this even matter? and why on tx specifically?
I feel addressing descriptor errors is a completely separate project.

Quoted 10 lines
Quoted 8 lines
Quoted 1 line

+2. le64 tx_gso_pkts: Packets send as host GSO sequence

same questions as gro

Quoted 1 line

+3. le64 tx_pkts: Total packets send by the device

sent

Ack.

Quoted 1 line
Quoted 1 line

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw

RE: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 6 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: Tuesday, June 6, 2023 6:57 PM

Quoted 7 lines
Quoted 2 lines

It matters for requirements, so we produce design that addresses it. We don't want to add config space every growing bit map which may be

different between different devices.

then say that you want to conserve config space, that is the requirement. not cvq specifically.

Well in one meeting you specifically told that requirements and design to be combined together, so it is drafted this way. Instead of very abstract like "conserve config space".

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

They don't have to be.

Then say that.

I thought this is very obvious as querying counter is hundred-time slower operation than packet processing.

Quoted 4 lines
Quoted 1 line

I don't see how it can be every synchronized.

you could program them through the vq itself.

Do you mean in the packet transmit and receive completions itself? It would be too heavy to do so to mix the control fields in the data path.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?

True, but design need to keep this in mind if it has some touch points like of

#4 so when migration arrives, it has the building block in place.

so again, let's make sure we capture the actual spec requirement.

It is an actual spec requirement to be done and drafted in the spec. We may not do everything in the first phase, this are the broad requirements. And in design, we will say, requirement #4 is phase 2.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines

so maybe it means there needs to be a way to set counters? so there's no need to mention migration - just that it should be possible to move counters between devices.

Quoted 1 line

+4. If a virtio device is group member device, a group owner should be

able

Quoted 1 line
Quoted 1 line
Quoted 1 line

+ to query all the counter attributes using the admin queue command

which

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

+ a virtio device will expose via a control vq to the driver.

this seems weirdly specific.
what is the actual requirement?

Quoted 9 lines
Quoted 2 lines

I don't follow the question.
When a device migrates from src to dst, group owner needs to know if both

side underlying member device has same counter attributes or not.

whether it's through a command or not is not a requirement and I still do not know what the requirement is.
what does "same counter attributes" mean? you never mentioned attributes before.

I will refine this further and drop the word "attribute".
If device support X counters, group owner should be able to know this bitmap.

Quoted 5 lines
Quoted 1 line

Yes, will change the GUEST surfaced from the current F_GUEST terminology of

the net device.

So? this predates 1.x spec we never bothered changing them.

Will remove the guest wording and will change to transmit and receive.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 1 line

+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

Not including. Will add this clarification further in v1.

Quoted 8 lines

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters 1. le64 tx_bad_desc_errors:
+Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

Quoted 3 lines
Quoted 1 line

It is left to the device to implement on what to do on bad desc.

then how can you count drops?

A device may count the drops based on errors.
It is counting drops based on the error it got.

Quoted 1 line

why does this even matter?

To debug.

Quoted 1 line

and why on tx specifically?

Missed the rx, will add.

Quoted 1 line

I feel addressing descriptor errors is a completely separate project.

Not sure if it is that big project.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 10 lines · Parav Pandit
Quoted 10 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: Tuesday, June 6, 2023 6:57 PM

Quoted 2 lines

It matters for requirements, so we produce design that addresses it. We don't want to add config space every growing bit map which may be

different between different devices.

Quoted 8 lines
Quoted 3 lines

then say that you want to conserve config space, that is the requirement. not cvq specifically.

Well in one meeting you specifically told that requirements and design to be combined together, so it is drafted this way. Instead of very abstract like "conserve config space".

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

just don't keep listing this in each feature. my plan is to work on cfgvq so we can keep using config space without these limitations.

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

They don't have to be.

Then say that.

Quoted 8 lines

I thought this is very obvious as querying counter is hundred-time slower operation than packet processing.

Quoted 4 lines
Quoted 1 line

I don't see how it can be every synchronized.

you could program them through the vq itself.

Do you mean in the packet transmit and receive completions itself? It would be too heavy to do so to mix the control fields in the data path.

maybe. but notice how you have a specific design in mind so you are jumping ahead. you asked how we could make these synch, i told you how.
the actual point is you want to say "we don't need this synchronous", design idea: use cvq

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?

True, but design need to keep this in mind if it has some touch points like of

#4 so when migration arrives, it has the building block in place.

so again, let's make sure we capture the actual spec requirement.

It is an actual spec requirement to be done and drafted in the spec. We may not do everything in the first phase, this are the broad requirements. And in design, we will say, requirement #4 is phase 2.

yes but this one I don't know what it means, spec wise. some kind of block? what for?

Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines

so maybe it means there needs to be a way to set counters? so there's no need to mention migration - just that it should be possible to move counters between devices.

Quoted 1 line

+4. If a virtio device is group member device, a group owner should be

able

Quoted 1 line
Quoted 1 line
Quoted 1 line

+ to query all the counter attributes using the admin queue command

which

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+ a virtio device will expose via a control vq to the driver.

this seems weirdly specific.
what is the actual requirement?

I don't follow the question.
When a device migrates from src to dst, group owner needs to know if both

side underlying member device has same counter attributes or not.

Quoted 7 lines
Quoted 5 lines

whether it's through a command or not is not a requirement and I still do not know what the requirement is.
what does "same counter attributes" mean? you never mentioned attributes before.

I will refine this further and drop the word "attribute".
If device support X counters, group owner should be able to know this bitmap.

i guess device might only support part of counters then?

Quoted 22 lines
Quoted 5 lines
Quoted 1 line

Yes, will change the GUEST surfaced from the current F_GUEST terminology of

the net device.

So? this predates 1.x spec we never bothered changing them.

Will remove the guest wording and will change to transmit and receive.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 1 line

+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

Not including. Will add this clarification further in v1.

Quoted 8 lines

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters 1. le64 tx_bad_desc_errors:
+Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

It is left to the device to implement on what to do on bad desc.

then how can you count drops?

A device may count the drops based on errors.
It is counting drops based on the error it got.

Quoted 1 line

why does this even matter?

To debug.

for debugging it's easiest if you just stop the vq, instead of drops.

Quoted 6 lines
Quoted 1 line

and why on tx specifically?

Missed the rx, will add.

Quoted 1 line

I feel addressing descriptor errors is a completely separate project.

Not sure if it is that big project.

I would have a separate debug section.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#7 lore raw

[virtio-comment] Re: [virtio] Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 17 lines · Michael S. Tsirkin
Quoted 16 lines · Parav Pandit
Quoted 6 lines

From: [email protected] <[email protected] open.org> On Behalf Of Michael S. Tsirkin
Sent: Tuesday, June 6, 2023 6:57 PM

Quoted 7 lines
Quoted 2 lines

It matters for requirements, so we produce design that addresses it. We don't want to add config space every growing bit map which may be

different between different devices.

then say that you want to conserve config space, that is the requirement. not cvq specifically.

Well in one meeting you specifically told that requirements and design to be combined together, so it is drafted this way. Instead of very abstract like "conserve config space".

Quoted 6 lines
Quoted 2 lines

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

just don't keep listing this in each feature. my plan is to work on cfgvq so we can keep using config space without these limitations.

cfgvq?

Did I miss someting?

Thanks.

Quoted 108 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

what matters is whether they have to be synchronized with a given queue - I get it they don't have to.

They don't have to be.

Then say that.

Quoted 8 lines

I thought this is very obvious as querying counter is hundred-time slower operation than packet processing.

Quoted 4 lines
Quoted 1 line

I don't see how it can be every synchronized.

you could program them through the vq itself.

Do you mean in the packet transmit and receive completions itself? It would be too heavy to do so to mix the control fields in the data path.

maybe. but notice how you have a specific design in mind so you are jumping ahead. you asked how we could make these synch, i told you how.
the actual point is you want to say "we don't need this synchronous", design idea: use cvq

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

we don't cover migration currently don't see how this is a spec rquirement. unless maybe it's justification for 4?

True, but design need to keep this in mind if it has some touch points like of

#4 so when migration arrives, it has the building block in place.

so again, let's make sure we capture the actual spec requirement.

It is an actual spec requirement to be done and drafted in the spec. We may not do everything in the first phase, this are the broad requirements. And in design, we will say, requirement #4 is phase 2.

yes but this one I don't know what it means, spec wise. some kind of block? what for?

Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines

so maybe it means there needs to be a way to set counters? so there's no need to mention migration - just that it should be possible to move counters between devices.

Quoted 1 line

+4. If a virtio device is group member device, a group owner should be

able

Quoted 1 line
Quoted 1 line
Quoted 1 line

+ to query all the counter attributes using the admin queue command

which

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

+ a virtio device will expose via a control vq to the driver.

this seems weirdly specific.
what is the actual requirement?

I don't follow the question.
When a device migrates from src to dst, group owner needs to know if both

side underlying member device has same counter attributes or not.

Quoted 7 lines
Quoted 5 lines

whether it's through a command or not is not a requirement and I still do not know what the requirement is.
what does "same counter attributes" mean? you never mentioned attributes before.

I will refine this further and drop the word "attribute".
If device support X counters, group owner should be able to know this bitmap.

i guess device might only support part of counters then?

Quoted 22 lines
Quoted 5 lines
Quoted 1 line

Yes, will change the GUEST surfaced from the current F_GUEST terminology of

the net device.

So? this predates 1.x spec we never bothered changing them.

Will remove the guest wording and will change to transmit and receive.

Quoted 14 lines
Quoted 14 lines
Quoted 4 lines
Quoted 1 line

+4. le64 rx_pkts: Total packets received by the device

including dropped ones or not?

Not including. Will add this clarification further in v1.

Quoted 8 lines

net-workstream/features-1.4.md line 34

Quoted 4 lines
net-workstream/features-1.4.md new file +4 -0
+
+### 3.1.2 Per transmit queue counters 1. le64 tx_bad_desc_errors:
+Descriptors dropped by the device due to errors in
+   descriptors

since when do we drop packets on error in descriptor? we just as likely stall ...

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

It is left to the device to implement on what to do on bad desc.

then how can you count drops?

A device may count the drops based on errors.
It is counting drops based on the error it got.

Quoted 1 line

why does this even matter?

To debug.

for debugging it's easiest if you just stop the vq, instead of drops.

Quoted 6 lines
Quoted 1 line

and why on tx specifically?

Missed the rx, will add.

Quoted 1 line

I feel addressing descriptor errors is a completely separate project.

Not sure if it is that big project.

I would have a separate debug section.

Quoted 9 lines

--
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#8 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

In reply to Parav Pandit
Quoted 2 lines · Parav Pandit

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

Just to stress, cvq has a bunch of drawbacks. For example it is hard to access from the hypervisor since it's DMA from VF. This is why I think our direction should be to add a vq transport that does all config accesses over admin commands. Or maybe I'm wrong.

But, let's distinguish between requirements, and between requirements and design. Getting supported counters is a requirement. Doing this over cvq is a design. Reducing MMIO registers is a requirement. OK?

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#9 lore raw

RE: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, June 7, 2023 4:36 PM

Quoted 8 lines
Quoted 2 lines · Parav Pandit

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

Just to stress, cvq has a bunch of drawbacks. For example it is hard to access from the hypervisor since it's DMA from VF.
This is why I think our direction should be to add a vq transport that does all config accesses over admin commands.

We debate this before, so will write is below.

Quoted 6 lines

Or maybe I'm wrong.

But, let's distinguish between requirements, and between requirements and design. Getting supported counters is a requirement.
Doing this over cvq is a design. Reducing MMIO registers is a requirement. OK?

Yes, I would write it as reducing MMIO and accessing it over vq of the own device is a requirement.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 14 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, June 7, 2023 4:36 PM

Quoted 8 lines
Quoted 2 lines · Parav Pandit

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

Just to stress, cvq has a bunch of drawbacks. For example it is hard to access from the hypervisor since it's DMA from VF.
This is why I think our direction should be to add a vq transport that does all config accesses over admin commands.

We debate this before, so will write is below.

Quoted 8 lines
Quoted 6 lines

Or maybe I'm wrong.

But, let's distinguish between requirements, and between requirements and design. Getting supported counters is a requirement.
Doing this over cvq is a design. Reducing MMIO registers is a requirement. OK?

Yes, I would write it as reducing MMIO and accessing it over vq of the own device is a requirement.

reducing mmio is a requirement (unrelated to counters btw, it's general). access over vq is a design point I think.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#11 lore raw

RE: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

Quoted 25 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, June 7, 2023 4:50 PM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Wednesday, June 7, 2023 4:36 PM

On Tue, Jun 06, 2023 at 11:08:12PM +0000, Parav Pandit wrote:

Quoted 14 lines
Quoted 7 lines
Quoted 2 lines

We can debate and change from cvq to new cfgvq as part of the journey. It is fine.

Just to stress, cvq has a bunch of drawbacks. For example it is hard to access from the hypervisor since it's DMA from VF. This is why I think our direction should be to add a vq transport that does all config accesses over admin commands.

We debate this before, so will write is below.

Quoted 6 lines

Or maybe I'm wrong.

But, let's distinguish between requirements, and between requirements and design. Getting supported counters is a requirement. Doing this over cvq is a design. Reducing MMIO registers is a requirement.

Quoted 7 lines

OK?

Quoted 2 lines

Yes, I would write it as reducing MMIO and accessing it over vq of the own

device is a requirement.

reducing mmio is a requirement (unrelated to counters btw, it's general). access over vq is a design point I think.

Didn't you ask to combine requirements and high-level design together in the first draft? What changed?

Should we create a new queue, reuse cvq, generalize over cfg q, etc are detailed design discussions.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#12 lore raw

Re: [virtio-comment] [PATCH requirements 1/7] net-features: Add requirements document for release 1.4

In reply to Parav Pandit
Quoted 42 lines · Parav Pandit

Add requirements document template for the virtio net features.

Add virtio net device counters visible to driver.

---
 net-workstream/features-1.4.md | 36 ++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
net-workstream/features-1.4.md new file +26 -0
new file mode 100644
index 0000000..03b4eb3
@@ -0,0 +1,36 @@
+# 1. Introduction
+
+This document describes the overall requirements for virtio net device
+improvements for upcoming release 1.4. Some of these requirements are
+interrelated and influence the interface design, hence reviewing them
+together is desired while updating the virtio net interface.
+
+# 2. Summary
+1. Device counters visible to the driver
+
+# 3. Requirements
+## 3.1 Device counters
+1. The driver should be able to query the device and/or per vq counters for
+   debugging purpose using a control vq command.
+2. The driver should be able to query which counters are supported using a
+   control vq command.
+3. If this device is migrated between two hosts, the driver should be able
+   get the counter values in the destination host from where it was left
+   off in the source host.
+4. If a virtio device is group member device, a group owner should be able
+   to query all the counter attributes using the admin queue command which
+   a virtio device will expose via a control vq to the driver.
+
+### 3.1.1 Per receive queue counters
+1. le64 rx_oversize_pkt_errors: Packet dropped due to receive packet being
+    oversize than the buffer size

net-workstream/features-1.4.md line 36

Quoted 10 lines
net-workstream/features-1.4.md new file +10 -0
+2. le64 rx_no_buffer_pkt_errors: Packet dropped due to unavailability of the
+    buffer in the receive queue
+3. le64 rx_gro_pkts: Packets treated as guest GSO sequence by the device
+4. le64 rx_pkts: Total packets received by the device
+
+### 3.1.2 Per transmit queue counters
+1. le64 tx_bad_desc_errors: Descriptors dropped by the device due to errors in
+   descriptors
+2. le64 tx_gso_pkts: Packets send as host GSO sequence
+3. le64 tx_pkts: Total packets send by the device

We hope to support device custom counter. That is, virtio spec provides a channel for driver and device, and both key and value are provided by device.

We discussed this issue earlier, and after some internal practice, I think it is still necessary to discuss this again.

It is very important, each cloud vendor will always have some special counters, these counters may not exist in another vendor. At the same time, if we have to discuss it in the spec every time we add a counter, or add a feature, I think it is very inconvenient. Manufacturers may add some new counters at any time based on some new requirements. Some counters may also be removed at any time.

Of course I know that doing this might hurt migration. But what I want to say is, why does it affect live migration? These counters we plan to give to users through ethtool, and some changes have taken place in the ethtool counters output by users. Does this have any practical impact? Or do we directly use some other output in a way, we can clearly tell the user that these counters may change during the migration process. For example, the driver is migrated to some new devices. These devices support some new counters. I think users should be able to see these new counters. These new counters may be the purpose of the migration.

We don't need to support live migration at this point.

Thanks.

Quoted 11 lines

--
2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#13 lore raw

PATCH 2/7 net-features: Add low latency transmit queue requirements

In reply to Parav Pandit

Add requirements for the low latency transmit queue.

---
 net-workstream/features-1.4.md | 73 ++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
net-workstream/features-1.4.md +73 -0
index 03b4eb3..55f1b1f 100644
@@ -7,6 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
+2. Low latency tx virtqueue for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -34,3 +35,75 @@ together is desired while updating the virtio net interface.
    descriptors
 2. le64 tx_gso_pkts: Packets send as host GSO sequence
 3. le64 tx_pkts: Total packets send by the device
+
+## 3.2 Low PCI latency virtqueues
+### 3.2.1 Low PCI latency tx virtqueue
+1. Packet transmit descriptor should contain data descriptors count without any
+   indirection and without any O(N) search to find the end of a packet stream.
+   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
+   subsequently) to contain a field num_next_desc for the packet stream
+   indicating that a packet is located N data descriptors.
+
+2. Packet transmit descriptor should contain segmentation offload-related fields
+   without any indirection. For example, packet transmit descriptor to contain
+   gso_type, gso_size/mss, header length, csum placement byte offset, and
+   csum start.
+
+3. Packet transmit descriptor should be able to place a small size packet that
+   does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue memory.
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.
+
+4. Packet transmit descriptor should be able to place a full GSO header (L2 to
+   L4) after header descriptor and before data descriptors. For example, the
+   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue memory.
+   When such a GSO header is positioned adjacent to the packet transmit
+   descriptor, and when the GSO header is not aligned to 16B, the following
+   data descriptor to start on the 8B aligned boundary.
+
+5. An example of the above requirements at high level is:
+
+```
+struct vitio_packed_q_desc {
+   /* current desc for reference */
+   u64 address;
+   u32 len;
+   u16 id;
+   u16 flags;
+};
+
+/* Constant size header descriptor for tx packets */
+struct vnet_tx_hdr_desc {
+   u16 flags; /* indicate how to parse next fields */
+   u16 id; /* desc id to come back in completion */
+   u8 num_next_desc; /* indicates the number of the next 16B data desc for this
+		      * buffer.
+		      */
+   u8 gso_type;
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u8 inline_pkt_len; /* indicates the length of the inline packet after this
+		       * desc
+		       */
+   u8 reserved;
+   u8 padding[];
+};
+
+/* Example of a short packet or GSO header placed in the desc section of the vq
+ */
+struct vnet_tx_small_pkt_desc {
+   u8 raw_pkt[128];
+};
+
+/* Example of header followed by data descriptor */
+struct vnet_tx_hdr_desc hdr_desc;
+struct vnet_data_desc desc[2];
+
+```
+6. Ability to zero pad the transmit completion when the transmit completion is
+ shorter than the CPU cache line size.
+
+7. Ability to place all transmit completion together with it per packet stream
+   transmit timestamp using single PCIe transcation.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#14 lore raw

Re: [virtio-comment] [PATCH requirements 2/7] net-features: Add low latency transmit queue requirements

Quoted 31 lines · Parav Pandit

Add requirements for the low latency transmit queue.

---
 net-workstream/features-1.4.md | 73 ++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
net-workstream/features-1.4.md +9 -0
index 03b4eb3..55f1b1f 100644
@@ -7,6 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
+2. Low latency tx virtqueue for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -34,3 +35,75 @@ together is desired while updating the virtio net interface.
    descriptors
 2. le64 tx_gso_pkts: Packets send as host GSO sequence
 3. le64 tx_pkts: Total packets send by the device
+
+## 3.2 Low PCI latency virtqueues
+### 3.2.1 Low PCI latency tx virtqueue
+1. Packet transmit descriptor should contain data descriptors count without any
+   indirection and without any O(N) search to find the end of a packet stream.
+   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
+   subsequently) to contain a field num_next_desc for the packet stream
+   indicating that a packet is located N data descriptors.

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 55

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+
+2. Packet transmit descriptor should contain segmentation offload-related fields
+   without any indirection. For example, packet transmit descriptor to contain
+   gso_type, gso_size/mss, header length, csum placement byte offset, and
+   csum start.
+
+3. Packet transmit descriptor should be able to place a small size packet that
+   does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue memory.
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.

For IPv6? With all the crazy options? Are you sure?

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 63

Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+
+4. Packet transmit descriptor should be able to place a full GSO header (L2 to
+   L4) after header descriptor and before data descriptors. For example, the
+   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue memory.
+   When such a GSO header is positioned adjacent to the packet transmit
+   descriptor, and when the GSO header is not aligned to 16B, the following
+   data descriptor to start on the 8B aligned boundary.
+

These alignments are weirdly specific.

Are these achievable with existing VQ designs? If not this specific part is not going to be a network specific thing. which is ok but maybe mention this.

Quoted 31 lines
net-workstream/features-1.4.md +31 -0
+5. An example of the above requirements at high level is:
+
+```
+struct vitio_packed_q_desc {
+   /* current desc for reference */
+   u64 address;
+   u32 len;
+   u16 id;
+   u16 flags;
+};
+
+/* Constant size header descriptor for tx packets */
+struct vnet_tx_hdr_desc {
+   u16 flags; /* indicate how to parse next fields */
+   u16 id; /* desc id to come back in completion */
+   u8 num_next_desc; /* indicates the number of the next 16B data desc for this
+		      * buffer.
+		      */
+   u8 gso_type;
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u8 inline_pkt_len; /* indicates the length of the inline packet after this
+		       * desc
+		       */
+   u8 reserved;
+   u8 padding[];
+};
+
+/* Example of a short packet or GSO header placed in the desc section of the vq

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 104

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+ */
+struct vnet_tx_small_pkt_desc {
+   u8 raw_pkt[128];
+};
+
+/* Example of header followed by data descriptor */
+struct vnet_tx_hdr_desc hdr_desc;
+struct vnet_data_desc desc[2];
+
+```

Seems to assume huge descriptors, apparently of variable size. Are fixed size descriptors valuable?

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 109

Quoted 5 lines
net-workstream/features-1.4.md +5 -0
+6. Ability to zero pad the transmit completion when the transmit completion is
+ shorter than the CPU cache line size.
+
+7. Ability to place all transmit completion together with it per packet stream
+   transmit timestamp using single PCIe transcation.

I can't decipher the last two ones. Part of it is using non virtio terminology, part referring to PCI/CPU unnecessarily, part weird grammar.

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw

RE: [virtio-comment] [PATCH requirements 2/7] net-features: Add low latency transmit queue requirements

Quoted 21 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:26 PM

net-workstream/features-1.4.md together is desired while updating the virtio net interface.

Quoted 9 lines
net-workstream/features-1.4.md +9 -0
+## 3.2 Low PCI latency virtqueues
+### 3.2.1 Low PCI latency tx virtqueue 1. Packet transmit descriptor
+should contain data descriptors count without any
+   indirection and without any O(N) search to find the end of a packet stream.
+   For example, a packet transmit descriptor (called vnet_tx_hdr_desc
+   subsequently) to contain a field num_next_desc for the packet stream
+   indicating that a packet is located N data descriptors.
+
+2. Packet transmit descriptor should contain segmentation offload-related

fields

net-workstream/features-1.4.md together is desired while updating the virtio net interface.

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+   without any indirection. For example, packet transmit descriptor to contain
+   gso_type, gso_size/mss, header length, csum placement byte offset, and
+   csum start.
+
+3. Packet transmit descriptor should be able to place a small size packet that
+   does not have any L4 data after the vnet_tx_hdr_desc in the virtqueue
Quoted 6 lines

memory.

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 55

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   For example a TCP ack only packet can fit in a descriptor memory which
+   otherwise consume more than 25% of metadata to describe the packet.

For IPv6? With all the crazy options? Are you sure?

IPV6 may not use all option, for example #3 is not useful for IPV6.

Quoted 4 lines

net-workstream/features-1.4.md together is desired while updating the virtio net interface.

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+
+4. Packet transmit descriptor should be able to place a full GSO header (L2 to
+   L4) after header descriptor and before data descriptors. For example, the
+   GSO header is placed after struct vnet_tx_hdr_desc in the virtqueue
Quoted 8 lines

memory.

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 63

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   When such a GSO header is positioned adjacent to the packet transmit
+   descriptor, and when the GSO header is not aligned to 16B, the following
+   data descriptor to start on the 8B aligned boundary.
+

These alignments are weirdly specific.

Not sure what it means,
these alignments are mostly specific on the host side to have aligned writes.

Quoted 4 lines

Are these achievable with existing VQ designs? If not this specific part is not going to be a network specific thing. which is ok but maybe mention this.

It will be probably a VQ with a type/attributes.

Quoted 35 lines
Quoted 6 lines

+5. An example of the above requirements at high level is:

net-workstream/features-1.4.md +5 -0
+
+```
+struct vitio_packed_q_desc {
+   /* current desc for reference */
+   u64 address;

net-workstream/features-1.4.md together is desired while updating the virtio net interface.

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+   u32 len;
+   u16 id;
+   u16 flags;
+};
+
+/* Constant size header descriptor for tx packets */ struct
+vnet_tx_hdr_desc {
+   u16 flags; /* indicate how to parse next fields */
+   u16 id; /* desc id to come back in completion */
+   u8 num_next_desc; /* indicates the number of the next 16B data desc for

this

Quoted 18 lines
net-workstream/features-1.4.md +18 -0
+		      * buffer.
+		      */
+   u8 gso_type;
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u8 inline_pkt_len; /* indicates the length of the inline packet after this
+		       * desc
+		       */
+   u8 reserved;
+   u8 padding[];
+};
+
+/* Example of a short packet or GSO header placed in the desc section
+of the vq  */ struct vnet_tx_small_pkt_desc {
+   u8 raw_pkt[128];
+};
Quoted 9 lines

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 66

Quoted 5 lines
net-workstream/features-1.4.md +5 -0
+
+/* Example of header followed by data descriptor */ struct
+vnet_tx_hdr_desc hdr_desc; struct vnet_data_desc desc[2];
+
+```

Seems to assume huge descriptors, apparently of variable size. Are fixed size descriptors valuable?

Its variable size but size is known upfront. So in a way fixed size.

Quoted 11 lines

net-workstream/features-1.4.md together is desired while updating the virtio net interface. · line 109

Quoted 7 lines

+6. Ability to zero pad the transmit completion when the transmit

net-workstream/features-1.4.md +4 -0
+completion is  shorter than the CPU cache line size.
+
+7. Ability to place all transmit completion together with it per packet stream
+   transmit timestamp using single PCIe transcation.

I can't decipher the last two ones. Part of it is using non virtio terminology, part referring to PCI/CPU unnecessarily, part weird grammar.

Yes, I will rewrite this part. It got messed up.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#16 lore raw

PATCH 3/7 net-features: Add low latency receive queue requirements

In reply to Parav Pandit

Add requirements for the low latency receive queue.

---
 net-workstream/features-1.4.md | 38 +++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
net-workstream/features-1.4.md +37 -1
index 55f1b1f..054f951 100644
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
-2. Low latency tx virtqueue for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -107,3 +107,39 @@ struct vnet_data_desc desc[2];
 
 7. Ability to place all transmit completion together with it per packet stream
    transmit timestamp using single PCIe transcation.
+
+### 3.2.2 Low latency rx virtqueue
+1. The device should be able to write a packet receive completion that consists
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write limit
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it to
+   64B or CPU cache line size whenever possible.
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */
+struct vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any size
+   from 64B to 9K bytes.
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#17 lore raw

Re: [virtio-comment] [PATCH requirements 3/7] net-features: Add low latency receive queue requirements

Quoted 21 lines · Parav Pandit

Add requirements for the low latency receive queue.

---
 net-workstream/features-1.4.md | 38 +++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
net-workstream/features-1.4.md +1 -1
index 55f1b1f..054f951 100644
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver
-2. Low latency tx virtqueue for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport
 
 # 3. Requirements
 ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 114

Quoted 8 lines
net-workstream/features-1.4.md +5 -0
@@ -107,3 +107,39 @@ struct vnet_data_desc desc[2];
 
 7. Ability to place all transmit completion together with it per packet stream
    transmit timestamp using single PCIe transcation.
+
+### 3.2.2 Low latency rx virtqueue
+1. The device should be able to write a packet receive completion that consists
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.

why? what is wrong with it being linear with packet instead?

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 119

Quoted 5 lines
net-workstream/features-1.4.md +5 -0
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write limit
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it to
+   64B or CPU cache line size whenever possible.

assuming completion is used buffer, these are eactly 64 bytes with packed vq, and they are linear so can be written in one transaction. if so why list requirements which are already met?
if you want them for completeness mention this.

Quoted 11 lines
net-workstream/features-1.4.md +11 -0
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */
+struct vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;
+   le16 gso_size;

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 140

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any size
+   from 64B to 9K bytes.

possible with mrg buffers

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 142

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.

the bigger question is not communicating to device. that is trivial. the bigger question is that linux IP stack seems to benefit from variable sized packets because buffers waste precious kernel memory.
is this for non IP stack such as xdp? non-linux guests? dpdk perhaps?

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 145

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.

no idea what this means.

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#18 lore raw

RE: [virtio-comment] [PATCH requirements 3/7] net-features: Add low latency receive queue requirements

Quoted 35 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:33 PM

Quoted 22 lines · Parav Pandit

Add requirements for the low latency receive queue.

---
 net-workstream/features-1.4.md | 38
+++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 55f1b1f..054f951 100644

net-workstream/features-1.4.md +0 -0
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver -2. Low latency tx virtqueue

for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 112

Quoted 6 lines
net-workstream/features-1.4.md +3 -0
@@ -107,3 +107,39 @@ struct vnet_data_desc desc[2];
 
 7. Ability to place all transmit completion together with it per packet stream
    transmit timestamp using single PCIe transcation.
+
+### 3.2.2 Low latency rx virtqueue
+1. The device should be able to write a packet receive completion that

consists

Quoted 5 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 114

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.

why? what is wrong with it being linear with packet instead?

It prohibits header data split, it requires multiple DMAs for the metadata consumed by single driver layer. Data processed by one layer is placed in two different locations.
This hurts performance.

Quoted 10 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 116

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write

limit

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 118

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it

to

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 119

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+   64B or CPU cache line size whenever possible.

assuming completion is used buffer, these are eactly 64 bytes with packed vq, and they are linear so can be written in one transaction.

When a packet spans multiple buffers, they cannot be written in contiguous manner.

Quoted 24 lines

if so why list requirements which are already met? if you want them for completeness mention this.

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */ struct
+vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;

net-workstream/features-1.4.md struct vnet_data_desc desc[2];

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any

size

Quoted 4 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 140

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+   from 64B to 9K bytes.

possible with mrg buffers

It doesn't scale to post constant size 64B buffer.

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 142

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.

the bigger question is not communicating to device. that is trivial. the bigger question is that linux IP stack seems to benefit from variable sized packets because buffers waste precious kernel memory.

Want to avoid the wastage and still achieve constant size posting.

Quoted 2 lines

is this for non IP stack such as xdp? non-linux guests? dpdk perhaps?

For Linux kernel guest.

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 145

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.

no idea what this means.

Instead of driver nearly allocating a page and splitting into buffer nearly of same size, an approach to post the page and let device to consume it based on packet size.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#19 lore raw

Re: [virtio-comment] [PATCH requirements 3/7] net-features: Add low latency receive queue requirements

Quoted 38 lines · Parav Pandit
Quoted 38 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:33 PM

Quoted 22 lines · Parav Pandit

Add requirements for the low latency receive queue.

---
 net-workstream/features-1.4.md | 38
+++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index 55f1b1f..054f951 100644

net-workstream/features-1.4.md +0 -0
@@ -7,7 +7,7 @@ together is desired while updating the virtio net interface.
 
 # 2. Summary
 1. Device counters visible to the driver -2. Low latency tx virtqueue

for PCI transport
+2. Low latency tx and rx virtqueues for PCI transport

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 112

Quoted 6 lines
net-workstream/features-1.4.md +3 -0
@@ -107,3 +107,39 @@ struct vnet_data_desc desc[2];
 
 7. Ability to place all transmit completion together with it per packet stream
    transmit timestamp using single PCIe transcation.
+
+### 3.2.2 Low latency rx virtqueue
+1. The device should be able to write a packet receive completion that

consists

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   of struct virtio_net_hdr (or similar) and a buffer id using a single DMA write
+   PCIe TLP.
Quoted 6 lines
Quoted 2 lines

why? what is wrong with it being linear with packet instead?

It prohibits header data split,
it requires multiple DMAs for the metadata consumed by single driver layer. Data processed by one layer is placed in two different locations. This hurts performance.

only with split yes? refer to that then.

maybe combine with the header split part?
these two seem intervined.

Quoted 4 lines
Quoted 4 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 116

Quoted 3 lines
net-workstream/features-1.4.md +2 -0
+2. The device should be able to perform DMA writes of multiple packets
+   completions in a single DMA transaction up to the PCIe maximum write

limit

Quoted 8 lines
Quoted 7 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 118

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+   in a transaction.
+3. The device should be able to zero pad packet write completion to align it

to

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 119

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+   64B or CPU cache line size whenever possible.

assuming completion is used buffer, these are eactly 64 bytes with packed vq, and they are linear so can be written in one transaction.

When a packet spans multiple buffers, they cannot be written in contiguous manner.

oh you mean when you manage to stick packet itself in the queue? this idea is only mentioned down the road.

Quoted 24 lines
Quoted 24 lines

if so why list requirements which are already met? if you want them for completeness mention this.

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+4. An example of the above DMA completion structure:
+
+```
+/* Constant size receive packet completion */ struct
+vnet_rx_completion {
+   u16 flags;
+   u16 id; /* buffer id */
+   u8 gso_type;
+   u8 reserved[3];
+   le16 gso_hdr_len;

net-workstream/features-1.4.md struct vnet_data_desc desc[2];

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+   le16 gso_size;
+   le16 csum_start;
+   le16 csum_offset;
+   u16 reserved2;
+   u64 timestamp; /* explained later */
+   u8 padding[];
+};
+```
+5. The driver should be able to post constant-size buffer pages on a receive
+   queue which can be consumed by the device for an incoming packet of any

size

Quoted 5 lines
Quoted 4 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 140

Quoted 1 line
net-workstream/features-1.4.md +1 -0
+   from 64B to 9K bytes.

possible with mrg buffers

It doesn't scale to post constant size 64B buffer.

maybe - figure out the actual requirement then.

Quoted 7 lines
Quoted 6 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 142

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+6. The device should be able to know the constant buffer size at receive
+   virtqueue level instead of per buffer level.

the bigger question is not communicating to device. that is trivial. the bigger question is that linux IP stack seems to benefit from variable sized packets because buffers waste precious kernel memory.

Want to avoid the wastage and still achieve constant size posting.

ok mention this maybe.

Quoted 3 lines
Quoted 2 lines

is this for non IP stack such as xdp? non-linux guests? dpdk perhaps?

For Linux kernel guest.

constant size buffers will inevitably waste memory i don't see a way around that.

Quoted 7 lines
Quoted 6 lines

net-workstream/features-1.4.md struct vnet_data_desc desc[2]; · line 145

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+7. The device should be able to indicate when a full page buffer is consumed,
+   which can be recycled by the driver when the packets from the completed
+   page is fully consumed.

no idea what this means.

Instead of driver nearly allocating a page and splitting into buffer nearly of same size, an approach to post the page and let device to consume it based on packet size.

oh I see. interesting. not easy as this does not fit the buffer abstraction.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#20 lore raw

PATCH 4/7 net-features: Add notification coalescing requirements

In reply to Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
 net-workstream/features-1.4.md | 8 ++++++++
 1 file changed, 8 insertions(+)
net-workstream/features-1.4.md +8 -0
index 054f951..fc36f31 100644
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
 # 2. Summary
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -143,3 +144,10 @@ struct vnet_rx_completion {
 7. The device should be able to indicate when a full page buffer is consumed,
    which can be recycled by the driver when the packets from the completed
    page is fully consumed.
+
+## 3.3 Virtqueue notification coalescing re-enable support
+1. Tx and Rx virtqueue notification coalescing should auto-disable on
+   notification reporting to the driver. The driver should be able to enable
+   coalescing after processing the packets per VQ. This ensures that when
+   networking stack decides to poll, no new notifications are generated when
+   per VQ notification coalescing is used.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#21 lore raw

Re: [virtio-comment] [PATCH requirements 4/7] net-features: Add notification coalescing requirements

Quoted 20 lines · Parav Pandit

Add virtio net device notification coalescing improvements requirements.

---
 net-workstream/features-1.4.md | 8 ++++++++
 1 file changed, 8 insertions(+)
net-workstream/features-1.4.md +1 -0
index 054f951..fc36f31 100644
@@ -8,6 +8,7 @@ together is desired while updating the virtio net interface.
 # 2. Summary
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
+3. Virtqueue notification coalescing re-arming support
 
 # 3. Requirements
 ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 153

Quoted 10 lines
net-workstream/features-1.4.md +7 -0
@@ -143,3 +144,10 @@ struct vnet_rx_completion {
 7. The device should be able to indicate when a full page buffer is consumed,
    which can be recycled by the driver when the packets from the completed
    page is fully consumed.
+
+## 3.3 Virtqueue notification coalescing re-enable support
+1. Tx and Rx virtqueue notification coalescing should auto-disable on
+   notification reporting to the driver. The driver should be able to enable
+   coalescing after processing the packets per VQ. This ensures that when
+   networking stack decides to poll, no new notifications are generated when
+   per VQ notification coalescing is used.

Weirdly specific.
I don't know what does auto-disable and enable coalescing mean. Does this refer to VIRTIO_NET_F_NOTF_COAL?
what is the problem this is trying to solve?

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#22 lore raw

RE: [virtio-comment] [PATCH requirements 4/7] net-features: Add notification coalescing requirements

Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:37 PM

net-workstream/features-1.4.md struct vnet_rx_completion { 7. The device should · line 153

Quoted 13 lines

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +7 -0
@@ -143,3 +144,10 @@ struct vnet_rx_completion {  7. The device should
 be able to indicate when a full page buffer is consumed,
    which can be recycled by the driver when the packets from the completed
    page is fully consumed.
+
+## 3.3 Virtqueue notification coalescing re-enable support 1. Tx and
+Rx virtqueue notification coalescing should auto-disable on
+   notification reporting to the driver. The driver should be able to enable
+   coalescing after processing the packets per VQ. This ensures that when
+   networking stack decides to poll, no new notifications are generated when
+   per VQ notification coalescing is used.

I don't know what does auto-disable and enable coalescing mean. Does this refer to VIRTIO_NET_F_NOTF_COAL?

Quoted 2 lines

what is the problem this is trying to solve?

Current driver when uses F_VQ_NOTF_COAL, and polling, it still receives the event (interrupt) from the device, because the time is always running. (event is not auto disabled on generation of the event).
So number of interrupts are not fully controlled yet by the driver.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#23 lore raw

Re: [virtio-comment] [PATCH requirements 4/7] net-features: Add notification coalescing requirements

Quoted 22 lines · Parav Pandit
Quoted 21 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:37 PM

net-workstream/features-1.4.md struct vnet_rx_completion { 7. The device should · line 153

Quoted 13 lines

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +7 -0
@@ -143,3 +144,10 @@ struct vnet_rx_completion {  7. The device should
 be able to indicate when a full page buffer is consumed,
    which can be recycled by the driver when the packets from the completed
    page is fully consumed.
+
+## 3.3 Virtqueue notification coalescing re-enable support 1. Tx and
+Rx virtqueue notification coalescing should auto-disable on
+   notification reporting to the driver. The driver should be able to enable
+   coalescing after processing the packets per VQ. This ensures that when
+   networking stack decides to poll, no new notifications are generated when
+   per VQ notification coalescing is used.

I don't know what does auto-disable and enable coalescing mean. Does this refer to VIRTIO_NET_F_NOTF_COAL?

Quoted 5 lines
Quoted 2 lines

what is the problem this is trying to solve?

Current driver when uses F_VQ_NOTF_COAL, and polling, it still receives the event (interrupt) from the device, because the time is always running. (event is not auto disabled on generation of the event).
So number of interrupts are not fully controlled yet by the driver.

could we please speak virtio terms? you are not new in this area.. so this is asking for a way to disable used buffer notifications? we actually have this with packet vq always and with split without EVENT_IDX. what is missing?

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#24 lore raw

PATCH 5/7 net-features: Add n-tuple receive flow steering requirements

In reply to Parav Pandit

Add virtio net device requirements for receive flow steering.

---
 net-workstream/features-1.4.md | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
net-workstream/features-1.4.md +33 -0
index fc36f31..41242b4 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
+4. Receive virtqueue n-tuple steering
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -151,3 +152,35 @@ struct vnet_rx_completion {
    coalescing after processing the packets per VQ. This ensures that when
    networking stack decides to poll, no new notifications are generated when
    per VQ notification coalescing is used.
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.
+4. The match criteria may optionally also include specific packet data offset,
+   length, and matching pattern, which may not be defined in the standard RFC.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context.
+7. The device should process RFS rules before RSS rules, i.e., when there is a
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries programmed in
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique id.
+13. The driver should be able to add and delete the steering rules in out of
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the flow
+    steering that device supports.
+15. The driver and group owner driver should be able to query supported device
+    limits for the steering entries.
+ 
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#25 lore raw

Re: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Hi, Parav.

I also did some work on this feature, and I think it would be very useful if we could work on this interface design and VirtIO Spec together!

Quoted 20 lines · Parav Pandit

Add virtio net device requirements for receive flow steering.

---
  net-workstream/features-1.4.md | 33 +++++++++++++++++++++++++++++++++
  1 file changed, 33 insertions(+)
net-workstream/features-1.4.md +1 -0
index fc36f31..41242b4 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
  1. Device counters visible to the driver
  2. Low latency tx and rx virtqueues for PCI transport
  3. Virtqueue notification coalescing re-arming support
+4. Receive virtqueue n-tuple steering
 
  # 3. Requirements
  ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 156

Quoted 5 lines
net-workstream/features-1.4.md +2 -0
@@ -151,3 +152,35 @@ struct vnet_rx_completion {
     coalescing after processing the packets per VQ. This ensures that when
     networking stack decides to poll, no new notifications are generated when
     per VQ notification coalescing is used.
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)

I think this abbreviation can be replaced with eg Receive virtqueue n-tuple flow director, RFD? or something else? This allows us to distinguish more clearly from (Accelerated) Recieve Flow Steering, (A)RFS.

ARFS can *automatically* issue RFS rules based on n-tuple rules, and we can still issue n-tuple rules *manually*. So RFD seems more reasonable than RFS abbreviation, what do you think?:)

net-workstream/features-1.4.md struct vnet_rx_completion { · line 164

Quoted 8 lines
net-workstream/features-1.4.md +8 -0
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.

The mask structure can be considered to reuse the same structure as the key structure.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 168

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+4. The match criteria may optionally also include specific packet data offset,
+   length, and matching pattern, which may not be defined in the standard RFC.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context.

We may have an additional requirement internally, such as specifying the queue id of a certain VF, but we are still aligning this requirement internally, and it may be more reasonable to implement the work involving device group based on admin vq.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 174

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+7. The device should process RFS rules before RSS rules, i.e., when there is a
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.

Considering the capabilities similar to some current tools such as ethtool, if it is a simple implementation, the first match principle can be applied.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 185

Quoted 11 lines
net-workstream/features-1.4.md +11 -0
+10. The driver should be able to query flow steering table entries programmed in
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique id.
+13. The driver should be able to add and delete the steering rules in out of
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the flow
+    steering that device supports.
+15. The driver and group owner driver should be able to query supported device
+    limits for the steering entries.

Do we want to support some user-defined fields, so that manufacturers can customize the interpretation of some fields.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#26 lore raw

RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Hi Heng,

Quoted 8 lines

From: Heng Qi <[email protected]>
Sent: Thursday, June 1, 2023 11:35 PM

Hi, Parav.

I also did some work on this feature, and I think it would be very useful if we could work on this interface design and VirtIO Spec together!

Sure, happy to bring this alive together.
More below.

Quoted 5 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 156

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)

I think this abbreviation can be replaced with eg Receive virtqueue n-tuple flow director, RFD? or something else?

One hw vendor product name has director in it, so will probably want to stay away picking such TM.

May be we can just say flow filters.
A filter may have an action and a destination.
So it can be generic enough.

Quoted 3 lines

This allows us to distinguish more clearly from (Accelerated) Recieve Flow Steering, (A)RFS.

The idea was for ARFS and RFS both to reuse the same underlying infra, just the OS hooks are different. (stack vs user ethtool respectively).

Quoted 4 lines

ARFS can *automatically* issue RFS rules based on n-tuple rules, and we can still issue n-tuple rules *manually*.
So RFD seems more reasonable than RFS abbreviation, what do you think?:)

"Receive flow filters" is probably more neutral name. :)

Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 6 lines

+1. The driver should be able to define a receive packet match criteria, an

net-workstream/features-1.4.md +5 -0
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as
Quoted 6 lines

mac

Quoted 2 lines

+ address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.

The mask structure can be considered to reuse the same structure as the key structure.

Yes, mask will probably have the bit mask definition.

Quoted 12 lines
Quoted 3 lines

+4. The match criteria may optionally also include specific packet data offset,
+ length, and matching pattern, which may not be defined in the standard

RFC.

Quoted 2 lines

+5. Action includes (a) dropping or (b) forwarding the packet. +6. Destination location is a receive virtqueue index or rss context.

We may have an additional requirement internally, such as specifying the queue id of a certain VF, but we are still aligning this requirement internally, and it may be more reasonable to implement the work involving device group based on admin vq.

The switch side APIs bit more complex.
We were first considering using the flow filters/steering for the guest driver and extend further for switching side.

Quoted 10 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 174

Quoted 6 lines

+7. The device should process RFS rules before RSS rules, i.e., when there is a

net-workstream/features-1.4.md +5 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.

Considering the capabilities similar to some current tools such as ethtool, if it is a simple implementation, the first match principle can be applied.

Yes, this likely needs refinement, I just wanted to get the seed for priority to be afloat when I wrote.
I am thinking of defining the priority at interface level, and driver can choose first match in the programming model based on the use.

Quoted 10 lines
Quoted 1 line

+10. The driver should be able to query flow steering table entries

programmed in

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

Quoted 3 lines

+13. The driver should be able to add and delete the steering rules in out of
+ order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

Quoted 9 lines

flow

Quoted 2 lines

+ steering that device supports.
+15. The driver and group owner driver should be able to query supported

device

Quoted 1 line

+ limits for the steering entries.

Do we want to support some user-defined fields, so that manufacturers can customize the interpretation of some fields.

Yes, we can phase out the common case of ipv{4,6} and {udp,tcp} as first phase and more customs field as second phase. Those use define fields can be useful in custom filters.

#27 lore raw

[virtio-comment] Re: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 17 lines · Parav Pandit

Hi Heng,

Quoted 8 lines

From: Heng Qi <[email protected]>
Sent: Thursday, June 1, 2023 11:35 PM

Hi, Parav.

I also did some work on this feature, and I think it would be very useful if we could work on this interface design and VirtIO Spec together!

Sure, happy to bring this alive together.
More below.

Quoted 4 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 156

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)

I think this abbreviation can be replaced with eg Receive virtqueue n-tuple flow director, RFD? or something else?

Quoted 5 lines

One hw vendor product name has director in it, so will probably want to stay away picking such TM.

May be we can just say flow filters.
A filter may have an action and a destination.
So it can be generic enough.

Yes, it's a good generic name.

Quoted 7 lines
Quoted 5 lines

This allows us to distinguish more clearly from (Accelerated) Recieve Flow Steering, (A)RFS.

The idea was for ARFS and RFS both to reuse the same underlying infra, just the OS hooks are different. (stack vs user ethtool respectively).

Yes, so I want it to have a distinguishing name from RFS, 'recieve flow filters' below as the name of the underlying infra is enough.:)

Quoted 24 lines
Quoted 4 lines

ARFS can *automatically* issue RFS rules based on n-tuple rules, and we can still issue n-tuple rules *manually*.
So RFD seems more reasonable than RFS abbreviation, what do you think?:)

"Receive flow filters" is probably more neutral name. :)

Quoted 11 lines

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 6 lines

+1. The driver should be able to define a receive packet match criteria, an

net-workstream/features-1.4.md +5 -0
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as

mac

Quoted 2 lines

+ address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.

The mask structure can be considered to reuse the same structure as the key structure.

Yes, mask will probably have the bit mask definition.

Quoted 5 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet data offset,
+ length, and matching pattern, which may not be defined in the standard

RFC.

Quoted 2 lines

+5. Action includes (a) dropping or (b) forwarding the packet. +6. Destination location is a receive virtqueue index or rss context.

Quoted 7 lines
Quoted 5 lines

We may have an additional requirement internally, such as specifying the queue id of a certain VF, but we are still aligning this requirement internally, and it may be more reasonable to implement the work involving device group based on admin vq.

The switch side APIs bit more complex.
We were first considering using the flow filters/steering for the guest driver and extend further for switching side.

Yes, but it's not that hard to just extend a VF id. In general, we will get back to you next week after aligning because of any difficulties we need this.:)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+7. The device should process RFS rules before RSS rules, i.e., when there is a

net-workstream/features-1.4.md +5 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
Quoted 6 lines
Quoted 3 lines

Considering the capabilities similar to some current tools such as ethtool, if it is a simple implementation, the first match principle can be applied.

Yes, this likely needs refinement, I just wanted to get the seed for priority to be afloat when I wrote.
I am thinking of defining the priority at interface level, and driver can choose first match in the programming model based on the use.

Yes, in general we really need priorities, and then maybe a matching mode field to let the driver choose which matching principle to use.

Quoted 13 lines
Quoted 13 lines
Quoted 1 line

+10. The driver should be able to query flow steering table entries

programmed in

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

Quoted 3 lines

+13. The driver should be able to add and delete the steering rules in out of
+ order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

flow

Quoted 2 lines

+ steering that device supports.
+15. The driver and group owner driver should be able to query supported

Quoted 7 lines
Quoted 5 lines

device

Quoted 1 line

+ limits for the steering entries.

Do we want to support some user-defined fields, so that manufacturers can customize the interpretation of some fields.

Yes, we can phase out the common case of ipv{4,6} and {udp,tcp} as first phase and more customs field as second phase. Those use define fields can be useful in custom filters.

Yes, for example some simple tunnel matching is possible, but the full, exact tunnel matching is probably not the first phase of this new feature.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#28 lore raw

[virtio-comment] Re: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 67 lines · Heng Qi
Quoted 17 lines · Parav Pandit

Hi Heng,

Quoted 8 lines

From: Heng Qi <[email protected]>
Sent: Thursday, June 1, 2023 11:35 PM

Hi, Parav.

I also did some work on this feature, and I think it would be very useful if we could work on this interface design and VirtIO Spec together!

Sure, happy to bring this alive together.
More below.

Quoted 4 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 156

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)

I think this abbreviation can be replaced with eg Receive virtqueue n-tuple flow director, RFD? or something else?

Quoted 5 lines

One hw vendor product name has director in it, so will probably want to stay away picking such TM.

May be we can just say flow filters.
A filter may have an action and a destination.
So it can be generic enough.

Yes, it's a good generic name.

Quoted 6 lines
Quoted 5 lines

This allows us to distinguish more clearly from (Accelerated) Recieve Flow Steering, (A)RFS.

The idea was for ARFS and RFS both to reuse the same underlying infra, just the OS hooks are different. (stack vs user ethtool respectively).

Yes, so I want it to have a distinguishing name from RFS, 'recieve flow filters' below as the name of the underlying infra is enough.:)

Quoted 29 lines
Quoted 4 lines

ARFS can *automatically* issue RFS rules based on n-tuple rules, and we can still issue n-tuple rules *manually*.
So RFD seems more reasonable than RFS abbreviation, what do you think?:)

"Receive flow filters" is probably more neutral name. :)

Quoted 11 lines

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 6 lines

+1. The driver should be able to define a receive packet match criteria, an

net-workstream/features-1.4.md +5 -0
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as

mac

Quoted 2 lines

+ address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.

The mask structure can be considered to reuse the same structure as the key structure.

Yes, mask will probably have the bit mask definition.

Quoted 10 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet data offset,
+ length, and matching pattern, which may not be defined in the standard

RFC.

Quoted 2 lines

+5. Action includes (a) dropping or (b) forwarding the packet. +6. Destination location is a receive virtqueue index or rss context.

We may have an additional requirement internally, such as specifying the queue id of a certain VF, but we are still aligning this requirement internally, and it may be more reasonable to implement the work involving device group based on admin vq.

Quoted 6 lines
Quoted 2 lines

The switch side APIs bit more complex.
We were first considering using the flow filters/steering for the guest driver and extend further for switching side.

Yes, but it's not that hard to just extend a VF id. In general, we will get back to you next week after aligning because of any difficulties we need this.:)

Hi, all.

We have already aligned the requirements internally. There is a common requirement in the cloud scenario: a virtual nic with SR-IOV capability. The user of a vm splits it into one PF + multiple VFs, and each VF takes some of the queues from the PF. Each VF can get its own IP address and mac from the PF.

However, the forwarding component (here it can be understood as an eswitch) connected to the virtual nic cannot see VFs, but can only see the PF (when creating the SR-IOV virtual nic, the PF is bound to a forwarding component). This behavior does not affect users (such as users in containers) using the VF, that is, the user sees a real virtual nic. However, the data incoming and outgoing this VF nic needs to be distinguished in the forwarding component:

1. When the packet comes out of the VF nic (from the perspective of the forwarding component, it comes out of a queue of the PF), the source ip of the packet is still the ip of the VF, and the forwarding component forwards it directly;

2. When packets arrive at the forwarding component, the IPs seen by the forwarding component (the IPs of all VFs) are the IPs of the PF. If the forwarding component wants to direct packets to the corresponding VF, it should use the flow director rules with the VF id issued by the PF.

Perhaps the admin queue can achieve this capability, but the flow director rule carrying the VF id has become the ethtool and kernel standard, so we can try to include it in the flow director's ability first (this means that the flow director itself can have this behavior, especially for those PFs that intend to limit the capabilities of the VF, and this behavior can effectively support our scenario).

Thanks.

Quoted 43 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+7. The device should process RFS rules before RSS rules, i.e., when there is a

net-workstream/features-1.4.md +5 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
Quoted 6 lines
Quoted 3 lines

Considering the capabilities similar to some current tools such as ethtool, if it is a simple implementation, the first match principle can be applied.

Yes, this likely needs refinement, I just wanted to get the seed for priority to be afloat when I wrote.
I am thinking of defining the priority at interface level, and driver can choose first match in the programming model based on the use.

Yes, in general we really need priorities, and then maybe a matching mode field to let the driver choose which matching principle to use.

Quoted 13 lines
Quoted 13 lines
Quoted 1 line

+10. The driver should be able to query flow steering table entries

programmed in

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

Quoted 3 lines

+13. The driver should be able to add and delete the steering rules in out of
+ order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

flow

Quoted 2 lines

+ steering that device supports.
+15. The driver and group owner driver should be able to query supported

Quoted 7 lines
Quoted 5 lines

device

Quoted 1 line

+ limits for the steering entries.

Do we want to support some user-defined fields, so that manufacturers can customize the interpretation of some fields.

Yes, we can phase out the common case of ipv{4,6} and {udp,tcp} as first phase and more customs field as second phase. Those use define fields can be useful in custom filters.

Yes, for example some simple tunnel matching is possible, but the full, exact tunnel matching is probably not the first phase of this new feature.

Quoted 8 lines

Thanks!

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail. Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#29 lore raw

[virtio-comment] RE: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 23 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 6, 2023 8:09 AM

We have already aligned the requirements internally. There is a common requirement in the cloud scenario: a virtual nic with SR-IOV capability. The user of a vm splits it into one PF + multiple VFs, and each VF takes some of the queues from the PF. Each VF can get its own IP address and mac from the PF.

However, the forwarding component (here it can be understood as an eswitch) connected to the virtual nic cannot see VFs, but can only see the PF (when creating the SR-IOV virtual nic, the PF is bound to a forwarding component). This behavior does not affect users (such as users in containers) using the VF, that is, the user sees a real virtual nic. However, the data incoming and outgoing this VF nic needs to be distinguished in the forwarding component:

1. When the packet comes out of the VF nic (from the perspective of the forwarding component, it comes out of a queue of the PF), the source ip of the packet is still the ip of the VF, and the forwarding component forwards it directly;

Quoted 7 lines

2. When packets arrive at the forwarding component, the IPs seen by the forwarding component (the IPs of all VFs) are the IPs of the PF. If the forwarding component wants to direct packets to the corresponding VF, it should use the flow director rules with the VF id issued by the PF.

Perhaps the admin queue can achieve this capability, but the flow director rule carrying the VF id has become the ethtool and kernel standard,

To my knowledge the eswitch model is not part of the ethtool.
It is part of the tc flower offloads and it covers far more than just _forwarding_ rules.

so we should be able to steer to a vf_id with a abstraction of say switch_port_id as far as spec is concerned.

The main difference is the type of rules eswitch vs the guest driver adds are very different. But yes, both should be able to use some of the common infrastructure.

We should consider this requirement in mind and craft the "destination" to cover this.

Since the flow insertion/removal queue should be present in one case PF and other case on VFs, we likely need a different queue than the AQ or expand the scope.

We should cover these in the design/requirements further.

I will update this details in the v1.

Quoted 4 lines

so we can try to
include it in the flow director's ability first (this means that the flow director itself can have this behavior, especially for those PFs that intend to limit the capabilities of the VF, and this behavior can effectively support our scenario).

#30 lore raw

[virtio-comment] Re: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 35 lines · Parav Pandit
Quoted 20 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 6, 2023 8:09 AM
We have already aligned the requirements internally. There is a common requirement in the cloud scenario: a virtual nic with SR-IOV capability. The user of a vm splits it into one PF + multiple VFs, and each VF takes some of the queues from the PF. Each VF can get its own IP address and mac from the PF.

However, the forwarding component (here it can be understood as an eswitch) connected to the virtual nic cannot see VFs, but can only see the PF (when creating the SR-IOV virtual nic, the PF is bound to a forwarding component). This behavior does not affect users (such as users in containers) using the VF, that is, the user sees a real virtual nic. However, the data incoming and outgoing this VF nic needs to be distinguished in the forwarding component:

1. When the packet comes out of the VF nic (from the perspective of the forwarding component, it comes out of a queue of the PF), the source ip of the packet is still the ip of the VF, and the forwarding component forwards it directly;

Quoted 7 lines

2. When packets arrive at the forwarding component, the IPs seen by the forwarding component (the IPs of all VFs) are the IPs of the PF. If the forwarding component wants to direct packets to the corresponding VF, it should use the flow director rules with the VF id issued by the PF.

Perhaps the admin queue can achieve this capability, but the flow director rule carrying the VF id has become the ethtool and kernel standard,

To my knowledge the eswitch model is not part of the ethtool.
It is part of the tc flower offloads and it covers far more than just _forwarding_ rules.

so we should be able to steer to a vf_id with a abstraction of say switch_port_id as far as spec is concerned.

The main difference is the type of rules eswitch vs the guest driver adds are very different. But yes, both should be able to use some of the common infrastructure.

Quoted 7 lines

We should consider this requirement in mind and craft the "destination" to cover this.

Since the flow insertion/removal queue should be present in one case PF and other case on VFs, we likely need a different queue than the AQ or expand the scope.

We should cover these in the design/requirements further.

I will update this details in the v1.

Hi, Parav.

Do we have a work schedule for the 1.4 features?
For example, the flow director we are very concerned about recently, because different scenarios have requirements for it.

We'd like to work out with you a rough timeline so we can discuss and collaborate better!

I hope I'm not disrupting your schedule :), just for us to better promote the work of virtio!

Grateful!

Quoted 5 lines
Quoted 5 lines

so we can try to
include it in the flow director's ability first (this means that the flow director itself can have this behavior, especially for those PFs that intend to limit the capabilities of the VF, and this behavior can effectively support our scenario).

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#31 lore raw

[virtio-comment] RE: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 9 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:35 AM

Quoted 1 line

I will update this details in the v1.

Hi, Parav.

Do we have a work schedule for the 1.4 features?

Since n-tuple requirements are straightforward, below is timeline in mind,
1. Requirements agree and drafted by June-30
2. Design and its review to finish by July-30
3. Spec to review by Aug-30

Quoted 6 lines

For example, the flow director we are very concerned about recently, because different scenarios have requirements for it.

We'd like to work out with you a rough timeline so we can discuss and collaborate better!

Will above timeline workable for you to work together on it? We should work on finite requirements that we can converge and deliver. First draft was this v0 one.

Otherwise, it can be never ending list that doesn't reach anywhere as steering technology keeps improving.

Slightly unrelated to question, let's call this feature as: "receive flow filters", instead of "flow director".

#32 lore raw

Re: [virtio-comment] RE: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 10 lines · Parav Pandit
Quoted 6 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:35 AM

Quoted 1 line

I will update this details in the v1.

Hi, Parav.

Do we have a work schedule for the 1.4 features?

Since n-tuple requirements are straightforward, below is timeline in mind,
1. Requirements agree and drafted by June-30
2. Design and its review to finish by July-30
3. Spec to review by Aug-30

OK, is this timeline for all 1.4 features or individual high priority 1.4 features?
Do we have a prioritization of 1.4 features? This also facilitates time investment for community maintainers.

Quoted 7 lines
Quoted 6 lines

For example, the flow director we are very concerned about recently, because different scenarios have requirements for it.

We'd like to work out with you a rough timeline so we can discuss and collaborate better!

Will above timeline workable for you to work together on it?

This timeline is ok for us. In addition, if it is difficult for the community to carry the VF id or there is an unclear demarcation problem with the design of  the group owner management, maybe we can skip this first (In any case, it depends on the opinion of the virtio community).

Quoted 1 line

We should work on finite requirements that we can converge and deliver.

Yes. Agree.

Quoted 5 lines

First draft was this v0 one.

Otherwise, it can be never ending list that doesn't reach anywhere as steering technology keeps improving.

Slightly unrelated to question, let's call this feature as: "receive flow filters", instead of "flow director".

OK, here's our previous agreement.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#33 lore raw

[virtio-comment] Re: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

In reply to Parav Pandit
Quoted 4 lines · Parav Pandit
Quoted 3 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:35 AM

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I will update this details in the v1.

Hi, Parav.

Do we have a work schedule for the 1.4 features?

Since n-tuple requirements are straightforward, below is timeline in mind,
1. Requirements agree and drafted by June-30

Pls note a bunch of people are traveling to kvm forum, I'd add a couple of weeks.

Quoted 2 lines

2. Design and its review to finish by July-30
3. Spec to review by Aug-30

Wait a second are we talking about 1.4 or 1.3? For 1.3 indeed we need to wrap development up by end of summer - our charter is to produce releases every 12 to 16 months, previous one was July 1st, 2022.

Quoted 6 lines
Quoted 6 lines

For example, the flow director we are very concerned about recently, because different scenarios have requirements for it.

We'd like to work out with you a rough timeline so we can discuss and collaborate better!

Quoted 8 lines

Will above timeline workable for you to work together on it? We should work on finite requirements that we can converge and deliver. First draft was this v0 one.

Otherwise, it can be never ending list that doesn't reach anywhere as steering technology keeps improving.

Slightly unrelated to question, let's call this feature as: "receive flow filters", instead of "flow director".

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#34 lore raw

Re: [virtio-comment] Re: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 17 lines · Michael S. Tsirkin
Quoted 4 lines · Parav Pandit
Quoted 3 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:35 AM

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I will update this details in the v1.

Hi, Parav.

Do we have a work schedule for the 1.4 features?

Since n-tuple requirements are straightforward, below is timeline in mind,
1. Requirements agree and drafted by June-30

Pls note a bunch of people are traveling to kvm forum, I'd add a couple of weeks.

Quoted 6 lines
Quoted 2 lines

2. Design and its review to finish by July-30
3. Spec to review by Aug-30

Wait a second are we talking about 1.4 or 1.3? For 1.3 indeed we need to wrap development up by end of summer - our charter is to produce releases every 12 to 16 months, previous one was July 1st, 2022.

For the 1.3 schedule, please also see
https://lore.kernel.org/virtio-comment/[email protected]/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#35 lore raw

[virtio-comment] RE: [virtio] RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

In reply to Michael S. Tsirkin
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 13, 2023 4:58 AM

Quoted 2 lines

2. Design and its review to finish by July-30 3. Spec to review by Aug-30

Wait a second are we talking about 1.4 or 1.3? For 1.3 indeed we need to wrap development up by end of summer - our charter is to produce releases every 12 to 16 months, previous one was July 1st, 2022.

1.4.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#36 lore raw

Re: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

In reply to Parav Pandit
Quoted 2 lines · Parav Pandit

Add virtio net device requirements for receive flow steering.

While at this, I would go with a well defined RX frame processing pipeline like a modern NIC instead of adding individual features like:

- filter
- hashing
- queue selection

etc.

Thanks

Quoted 65 lines
---
 net-workstream/features-1.4.md | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
net-workstream/features-1.4.md +33 -1
index fc36f31..41242b4 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
 1. Device counters visible to the driver
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
+4. Receive virtqueue n-tuple steering
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -151,3 +152,35 @@ struct vnet_rx_completion {
    coalescing after processing the packets per VQ. This ensures that when
    networking stack decides to poll, no new notifications are generated when
    per VQ notification coalescing is used.
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.
+4. The match criteria may optionally also include specific packet data offset,
+   length, and matching pattern, which may not be defined in the standard RFC.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context.
+7. The device should process RFS rules before RSS rules, i.e., when there is a
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries programmed in
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique id.
+13. The driver should be able to add and delete the steering rules in out of
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the flow
+    steering that device supports.
+15. The driver and group owner driver should be able to query supported device
+    limits for the steering entries.
+
--

2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#37 lore raw

RE: [virtio-comment] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 7 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, June 6, 2023 10:48 PM

Quoted 1 line · Parav Pandit
Quoted 9 lines
Quoted 1 line

Add virtio net device requirements for receive flow steering.

While at this, I would go with a well defined RX frame processing pipeline like a modern NIC instead of adding individual features like:

- filter
- hashing
- queue selection

Can you explain this from requirements point of view?
Individual features are something that already exists such as queue and hashing.

We are inserting filtering to this pipe.
Do you mean to define the order in the spec? if so, yes, very lightly I described the relation of filtering with RSS.

If more, please explain.

Quoted 87 lines

etc.

Thanks

Quoted 4 lines
---
 net-workstream/features-1.4.md | 33

+++++++++++++++++++++++++++++++++

Quoted 16 lines
 1 file changed, 33 insertions(+)
(quoted diff) +0 -0

b/net-workstream/features-1.4.md index fc36f31..41242b4 100644

net-workstream/features-1.4.md +0 -0
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
 1. Device counters visible to the driver  2. Low latency tx and rx

virtqueues for PCI transport 3. Virtqueue notification coalescing re-arming support
+4. Receive virtqueue n-tuple steering

# 3. Requirements
## 3.1 Device counters

net-workstream/features-1.4.md +0 -0
@@ -151,3 +152,35 @@ struct vnet_rx_completion {
    coalescing after processing the packets per VQ. This ensures that when

net-workstream/features-1.4.md struct vnet_rx_completion { · line 162

Quoted 10 lines
net-workstream/features-1.4.md +8 -0
    networking stack decides to poll, no new notifications are generated when
    per VQ notification coalescing is used.
+
+## 3.4 Receive virtqueue n-tuple flow steering (RFS) 1. The driver
+should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as

mac

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.
+4. The match criteria may optionally also include specific packet data offset,
+   length, and matching pattern, which may not be defined in the standard

RFC.

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 9 lines
net-workstream/features-1.4.md +9 -0
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context.
+7. The device should process RFS rules before RSS rules, i.e., when there is a
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries

programmed in

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+    the device by the flow id.
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

net-workstream/features-1.4.md struct vnet_rx_completion {

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+13. The driver should be able to add and delete the steering rules in out of
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

flow

net-workstream/features-1.4.md struct vnet_rx_completion { · line 184

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+    steering that device supports.
+15. The driver and group owner driver should be able to query supported

device

Quoted 24 lines
net-workstream/features-1.4.md +2 -1
+    limits for the steering entries.
+
--

2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#38 lore raw

[virtio-comment] Re: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

In reply to Parav Pandit
Quoted 24 lines · Parav Pandit

Add virtio net device requirements for receive flow steering.

---
  net-workstream/features-1.4.md | 33 +++++++++++++++++++++++++++++++++
  1 file changed, 33 insertions(+)
net-workstream/features-1.4.md +2 -0
index fc36f31..41242b4 100644
@@ -9,6 +9,7 @@ together is desired while updating the virtio net interface.
  1. Device counters visible to the driver
  2. Low latency tx and rx virtqueues for PCI transport
  3. Virtqueue notification coalescing re-arming support
+4. Receive virtqueue n-tuple steering
 
  # 3. Requirements
  ## 3.1 Device counters
@@ -151,3 +152,35 @@ struct vnet_rx_completion {
     coalescing after processing the packets per VQ. This ensures that when
     networking stack decides to poll, no new notifications are generated when
     per VQ notification coalescing is used.
+

net-workstream/features-1.4.md struct vnet_rx_completion { · line 165

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+## 3.4 Receive virtqueue n-tuple flow steering (RFS)
+1. The driver should be able to define a receive packet match criteria, an
+   action and a destination for a packet. For example, an ipv4 packet with a
+   multicast address to be steered to the receive vq 0. The second example is
+   ipv4, tcp packet matching a specified IP address and tcp port tuple to
+   steered to receive vq 10.
+2. The match criteria should include exact tuple fields well-defined such as mac
+   address, IP addresses, tcp/udp ports, etc.
+3. The match criteria should also include the field mask.
+4. The match criteria may optionally also include specific packet data offset,

Is this the offset of the packet payload (not including the packet header)? If yes, what might be the usage scenarios?

net-workstream/features-1.4.md struct vnet_rx_completion { · line 172

Quoted 7 lines
net-workstream/features-1.4.md +7 -0
+   length, and matching pattern, which may not be defined in the standard RFC.
+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context.
+7. The device should process RFS rules before RSS rules, i.e., when there is a
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.

Why do we need to add this limit for the device please? I think that different devices may handle ctrq at different speeds.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 176

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+9. If multiple rules are programmed which has overlapping attributes for a
+   received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries programmed in
+    the device by the flow id.

Combining points 9 and 10, can I deduce that the attributes of a flow rule include identifier (id), position (index of the rule entry) and priority? According to point 11, the driver needs to provide a unique id, if the location and priority are not provided, they should have predefined default values.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 183

Quoted 7 lines
net-workstream/features-1.4.md +7 -0
+11. The driver should be able to add the entry for attributes (a) match
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique id.
+13. The driver should be able to add and delete the steering rules in out of
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the flow
+    steering that device supports.

Does a group member driver support insertion rules?  The group owner driver and a group member driver should have the ability to add/remove rules for themselves.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 186

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+15. The driver and group owner driver should be able to query supported device
+    limits for the steering entries.
+

Does this include the number of rules supported by the device? Are there other limits?

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#39 lore raw

[virtio-comment] RE: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 4 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:57 PM

[..]

Quoted 6 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet +data offset,

Is this the offset of the packet payload (not including the packet header)? If yes, what might be the usage scenarios?

Some packet header fields may not be defined initially by us. For example some protocol FOO type which has some header after UDP header. And wants to filter and steer to a specific RQ.

A virtio may not have defined such protocol at the beginning. So a generic extension allows to steer packet. At that point line between header and data is blurry. :)

Quoted 9 lines
Quoted 1 line

+ length, and matching pattern, which may not be defined in the standard

RFC.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 172

Quoted 6 lines

+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context. +7. The device should process RFS rules before RSS rules, i.e., when there is a

net-workstream/features-1.4.md +3 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
Quoted 3 lines

Why do we need to add this limit for the device please? I think that different devices may handle ctrq at different speeds.

ARFS rules insertion is fast and tcp streams should be able to converge quicker on the desired cpu.
So rule insertion should be fast enough.
Many of the device can/may be able to do this as some semi-fast path operation unlike CVQ which is typically a device level control operation, often done as slow process.

So driver-device interface to be able to scale up to low latency and high throughput flow insert/delete ops.
It is ok one device may choose to not support such high rate, but interface definition wise I imagine to be on its own dedicated queue.

Quoted 8 lines
Quoted 3 lines

+9. If multiple rules are programmed which has overlapping attributes for a
+ received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries

programmed in

Quoted 1 line

+ the device by the flow id.

Combining points 9 and 10, can I deduce that the attributes of a flow rule include identifier (id), position (index of the rule entry) and priority?

Yes, I also think that way.

Quoted 3 lines

According to point 11, the driver needs to provide a unique id, if the location and priority are not provided, they should have predefined default values.

Yes.
I am not fully sure of it as it brings some uncertainty in behavior. So having second thought as driver to always provide it.

Quoted 7 lines
Quoted 3 lines

+11. The driver should be able to add the entry for attributes (a) match
+ criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

Quoted 3 lines

+13. The driver should be able to add and delete the steering rules in out of
+ order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

Quoted 7 lines

flow

Quoted 1 line

+ steering that device supports.

Does a group member driver support insertion rules?  The group owner driver and a group member driver should have the ability to add/remove rules for themselves.

Yes, Group owner adds rule for the self, not on behalf. Same as group member.
No difference between them as they operate on self device.

In future, a group owner may want to steer the packet towards a group member, like eswitch.

Quoted 7 lines
Quoted 1 line

+15. The driver and group owner driver should be able to query supported

device

net-workstream/features-1.4.md struct vnet_rx_completion { · line 186

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+    limits for the steering entries.
+

Does this include the number of rules supported by the device? Are there other limits?

Lets list down if you have some in mind.

#40 lore raw

[virtio-comment] Re: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 4 lines · Parav Pandit
Quoted 3 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:57 PM

[..]

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet +data offset,

Is this the offset of the packet payload (not including the packet header)? If yes, what might be the usage scenarios?

Some packet header fields may not be defined initially by us. For example some protocol FOO type which has some header after UDP header. And wants to filter and steer to a specific RQ.

Ok, I got it.

Quoted 3 lines

A virtio may not have defined such protocol at the beginning. So a generic extension allows to steer packet. At that point line between header and data is blurry. :)

Then I think it's just an *offset* from the start of the packet so we can use this capability for the packet header as well.

Quoted 9 lines
Quoted 9 lines
Quoted 2 lines

+ length, and matching pattern, which may not be defined in the standard

RFC.

Quoted 6 lines

+5. Action includes (a) dropping or (b) forwarding the packet.
+6. Destination location is a receive virtqueue index or rss context. +7. The device should process RFS rules before RSS rules, i.e., when there is a

net-workstream/features-1.4.md +3 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.
Quoted 6 lines
Quoted 3 lines

Why do we need to add this limit for the device please? I think that different devices may handle ctrq at different speeds.

ARFS rules insertion is fast and tcp streams should be able to converge quicker on the desired cpu.
So rule insertion should be fast enough.
Many of the device can/may be able to do this as some semi-fast path operation unlike CVQ which is typically a device level control operation, often done as slow process.

We need to consider ARFS for receive flow filters, but receive flow filters are more infrastructure, do we need to offer a sub-feature to distinguish whether a device supports this dedicated queue or not. This becomes easier for users who only use receive flow filters without ARFS enabled. At this point, ARFS and receive flow filters become orthogonal.

Quoted 2 lines

So driver-device interface to be able to scale up to low latency and high throughput flow insert/delete ops.

Ok.

Quoted 9 lines

It is ok one device may choose to not support such high rate, but interface definition wise I imagine to be on its own dedicated queue.

Quoted 7 lines
Quoted 3 lines

+9. If multiple rules are programmed which has overlapping attributes for a
+ received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries

programmed in

Quoted 1 line

+ the device by the flow id.

Combining points 9 and 10, can I deduce that the attributes of a flow rule include identifier (id), position (index of the rule entry) and priority?

Quoted 8 lines

Yes, I also think that way.

Quoted 3 lines

According to point 11, the driver needs to provide a unique id, if the location and priority are not provided, they should have predefined default values.

Yes.
I am not fully sure of it as it brings some uncertainty in behavior. So having second thought as driver to always provide it.

Yes, and this point is specific, then we can continue to confirm it in the future spec version.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

+11. The driver should be able to add the entry for attributes (a) match
+ criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via a unique

id.

Quoted 3 lines

+13. The driver should be able to add and delete the steering rules in out of
+ order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of the

flow

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ steering that device supports.

Does a group member driver support insertion rules?  The group owner driver and a group member driver should have the ability to add/remove rules for themselves.

Yes, Group owner adds rule for the self, not on behalf. Same as group member.
No difference between them as they operate on self device.

Ok.

Quoted 2 lines

In future, a group owner may want to steer the packet towards a group member, like eswitch.

Yes.
And what I want to say is that our cloud architecture is similar to two-layer virtualization, but we only have one layer of eswitch,
so we need VF id to let eswitch provide simple flow steering for the second layer of virtualization, and do not need complicated eswitch. This is the purpose of carrying VF id, but now we can not consider this (VF id) in the feature of receive flow filters, so as not to hinder the progress of this feature.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

+15. The driver and group owner driver should be able to query supported

device

net-workstream/features-1.4.md struct vnet_rx_completion { · line 186

Quoted 2 lines
net-workstream/features-1.4.md +2 -0
+    limits for the steering entries.
+

Does this include the number of rules supported by the device? Are there other limits?

Lets list down if you have some in mind.

Not yet.

Thanks!

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#41 lore raw

[virtio-comment] RE: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 24 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 13, 2023 1:04 AM

Quoted 11 lines · Parav Pandit
Quoted 3 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:57 PM

[..]

Quoted 5 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet +data offset,

Is this the offset of the packet payload (not including the packet header)? If yes, what might be the usage scenarios?

Some packet header fields may not be defined initially by us. For example some protocol FOO type which has some header after UDP

header.

Quoted 1 line

And wants to filter and steer to a specific RQ.

Ok, I got it.

Quoted 1 line
Quoted 7 lines
Quoted 1 line

A virtio may not have defined such protocol at the beginning. So a generic

extension allows to steer packet.

Quoted 1 line

At that point line between header and data is blurry. :)

Then I think it's just an *offset* from the start of the packet so we can use this capability for the packet header as well.

Yes, will change it. "packet data" was misleading. It is just the packet offset.

Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

+ length, and matching pattern, which may not be defined in the
+ standard

RFC.

Quoted 3 lines

+5. Action includes (a) dropping or (b) forwarding the packet. +6. Destination location is a receive virtqueue index or rss context. +7. The device should process RFS rules before RSS rules, i.e., when there is

a

Quoted 7 lines
Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 172

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.

Why do we need to add this limit for the device please? I think that different devices may handle ctrq at different speeds.

ARFS rules insertion is fast and tcp streams should be able to converge quicker

on the desired cpu.

Quoted 2 lines

So rule insertion should be fast enough.
Many of the device can/may be able to do this as some semi-fast path

operation unlike CVQ which is typically a device level control operation, often done as slow process.

Quoted 7 lines

We need to consider ARFS for receive flow filters, but receive flow filters are more infrastructure, do we need to offer a sub-feature to distinguish whether a device supports this dedicated queue or not.
This becomes easier for users who only use receive flow filters without ARFS enabled.
At this point, ARFS and receive flow filters become orthogonal.

If it’s a "flowvq" dedicated queue device both the requirements are addressed. Device can choose to implement in its preferred way.

Quoted 20 lines
Quoted 2 lines

So driver-device interface to be able to scale up to low latency and high

throughput flow insert/delete ops.

Ok.

Quoted 1 line

It is ok one device may choose to not support such high rate, but interface

definition wise I imagine to be on its own dedicated queue.

Quoted 12 lines
Quoted 8 lines
Quoted 4 lines

+9. If multiple rules are programmed which has overlapping attributes for a
+ received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries

programmed in

Quoted 1 line

+ the device by the flow id.

Combining points 9 and 10, can I deduce that the attributes of a flow rule include identifier (id), position (index of the rule entry) and priority?

Yes, I also think that way.

Quoted 2 lines

According to point 11, the driver needs to provide a unique id, if the location and priority are not provided, they should have predefined

Quoted 9 lines

default values.

Quoted 4 lines
Quoted 1 line

Yes.
I am not fully sure of it as it brings some uncertainty in behavior. So having second thought as driver to always provide it.

Yes, and this point is specific, then we can continue to confirm it in the future spec version.

Ok. I will revise it in v1.

Quoted 26 lines
Quoted 11 lines
Quoted 11 lines

Review

Quoted 5 lines

+11. The driver should be able to add the entry for attributes (a) match

(quoted diff) +3 -0
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via
+a unique

id.

Review

Quoted 4 lines

+13. The driver should be able to add and delete the steering rules in out of

(quoted diff) +3 -0
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of
+the

flow

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ steering that device supports.

Does a group member driver support insertion rules?  The group owner driver and a group member driver should have the ability to add/remove rules for themselves.

Yes, Group owner adds rule for the self, not on behalf. Same as group member.
No difference between them as they operate on self device.

Ok.

Quoted 2 lines

In future, a group owner may want to steer the packet towards a group

member, like eswitch.

Quoted 9 lines

Yes.
And what I want to say is that our cloud architecture is similar to two-layer virtualization, but we only have one layer of eswitch, so we need VF id to let eswitch provide simple flow steering for the second layer of virtualization, and do not need complicated eswitch.
This is the purpose of carrying VF id, but now we can not consider this (VF id) in the feature of receive flow filters, so as not to hinder the progress of this feature.

Ok. Lets keep this in notes and mind to have the abstraction and don’t jump to bake vf id in the spec in the initial version.

#42 lore raw

[virtio-comment] Re: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 55 lines · Parav Pandit
Quoted 17 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 13, 2023 1:04 AM

Quoted 10 lines · Parav Pandit
Quoted 2 lines

From: Heng Qi <[email protected]>
Sent: Monday, June 12, 2023 10:57 PM

[..]

Quoted 5 lines
Quoted 2 lines

+4. The match criteria may optionally also include specific packet +data offset,

Is this the offset of the packet payload (not including the packet header)? If yes, what might be the usage scenarios?

Some packet header fields may not be defined initially by us. For example some protocol FOO type which has some header after UDP

header.

Quoted 9 lines
Quoted 1 line

And wants to filter and steer to a specific RQ.

Ok, I got it.

Quoted 1 line

A virtio may not have defined such protocol at the beginning. So a generic

extension allows to steer packet.

Quoted 1 line

At that point line between header and data is blurry. :)

Then I think it's just an *offset* from the start of the packet so we can use this capability for the packet header as well.

Yes, will change it. "packet data" was misleading. It is just the packet offset.

Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

+ length, and matching pattern, which may not be defined in the
+ standard

RFC.

Quoted 3 lines

+5. Action includes (a) dropping or (b) forwarding the packet. +6. Destination location is a receive virtqueue index or rss context. +7. The device should process RFS rules before RSS rules, i.e., when there is

a

Quoted 7 lines
Quoted 6 lines

net-workstream/features-1.4.md struct vnet_rx_completion { · line 172

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   miss on the RFS rule RSS rule applies.
+8. The device should be able to add/remove these rules at a rate of 100K
+   rules/sec.

Why do we need to add this limit for the device please? I think that different devices may handle ctrq at different speeds.

ARFS rules insertion is fast and tcp streams should be able to converge quicker

on the desired cpu.

Quoted 2 lines

So rule insertion should be fast enough.
Many of the device can/may be able to do this as some semi-fast path

operation unlike CVQ which is typically a device level control operation, often done as slow process.

Quoted 7 lines

We need to consider ARFS for receive flow filters, but receive flow filters are more infrastructure, do we need to offer a sub-feature to distinguish whether a device supports this dedicated queue or not.
This becomes easier for users who only use receive flow filters without ARFS enabled.
At this point, ARFS and receive flow filters become orthogonal.

Quoted 2 lines

If it’s a "flowvq" dedicated queue device both the requirements are addressed. Device can choose to implement in its preferred way.

I think this is a migration issue:
Devices A and B each offser the VIRTIO_NET_F_RECEIVE_FLOW_FILTER (this is a temporary name) feature.
And device A implements flowvq, device B does not implement flowvq, they are both using receive flow filters but ARFS is disabled. The migration should fail at this time. Does this meet user expectations?

Or do we warn in the spec that using flow vq might cause the migration issue?

Thanks!

Quoted 60 lines
Quoted 17 lines
Quoted 2 lines

So driver-device interface to be able to scale up to low latency and high

throughput flow insert/delete ops.

Ok.

Quoted 1 line

It is ok one device may choose to not support such high rate, but interface

definition wise I imagine to be on its own dedicated queue.

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

+9. If multiple rules are programmed which has overlapping attributes for a
+ received packet, the driver to define the location/priority of the rule.
+10. The driver should be able to query flow steering table entries

programmed in

Quoted 1 line

+ the device by the flow id.

Combining points 9 and 10, can I deduce that the attributes of a flow rule include identifier (id), position (index of the rule entry) and priority?

Yes, I also think that way.

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

According to point 11, the driver needs to provide a unique id, if the location and priority are not provided, they should have predefined

default values.

Quoted 3 lines

Yes.
I am not fully sure of it as it brings some uncertainty in behavior. So having second thought as driver to always provide it.

Yes, and this point is specific, then we can continue to confirm it in the future spec version.

Ok. I will revise it in v1.

Quoted 23 lines
Quoted 10 lines
Quoted 10 lines

Review

Quoted 4 lines

+11. The driver should be able to add the entry for attributes (a) match

(quoted diff) +3 -0
+    criteria, (b) action, (c) destination and (d) assign a unique id of 32 bits.
+12. The driver should be able to delete the steering rule entry via
+a unique

id.

Review

Quoted 4 lines

+13. The driver should be able to add and delete the steering rules in out of

(quoted diff) +3 -0
+    order manner without depending on previous commands.
+14. A group member device should be able to query the attributes of
+the

flow

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

+ steering that device supports.

Does a group member driver support insertion rules?  The group owner driver and a group member driver should have the ability to add/remove rules for themselves.

Yes, Group owner adds rule for the self, not on behalf. Same as group member.
No difference between them as they operate on self device.

Ok.

Quoted 1 line

In future, a group owner may want to steer the packet towards a group

member, like eswitch.

Quoted 9 lines

Yes.
And what I want to say is that our cloud architecture is similar to two-layer virtualization, but we only have one layer of eswitch, so we need VF id to let eswitch provide simple flow steering for the second layer of virtualization, and do not need complicated eswitch.
This is the purpose of carrying VF id, but now we can not consider this (VF id) in the feature of receive flow filters, so as not to hinder the progress of this feature.

Quoted 1 line

Ok. Lets keep this in notes and mind to have the abstraction and don’t jump to bake vf id in the spec in the initial version.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#43 lore raw

[virtio-comment] RE: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 6 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 13, 2023 11:43 PM

Quoted 1 line

If it’s a "flowvq" dedicated queue device both the requirements are

addressed.

Quoted 9 lines
Quoted 1 line

Device can choose to implement in its preferred way.

I think this is a migration issue:
Devices A and B each offser the VIRTIO_NET_F_RECEIVE_FLOW_FILTER (this is a temporary name) feature.
And device A implements flowvq, device B does not implement flowvq, they are both using receive flow filters but ARFS is disabled.
The migration should fail at this time. Does this meet user expectations?

If F_RECEIVE_FLOW_FILTER is offered, it means a device supports flowvq on src and dst. So migration wont fail.

It is no different than any other net _F feature. For example notification coalescing or _MQ or _RSS.

What I meant in above "preferred way" is, device can choose to operate flow table entries as 5 flow/sec. Or it can choose to do 100K flows/sec.

The basic requirement is that flow rule entries should be a low latency interface.

Quoted 2 lines

Or do we warn in the spec that using flow vq might cause the migration issue?

It should not cause migration once the device features and associated attributes can be queries by the group owner.

#44 lore raw

Re: [virtio-comment] RE: [virtio] [PATCH requirements 5/7] net-features: Add n-tuple receive flow steering requirements

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Heng Qi <[email protected]>
Sent: Tuesday, June 13, 2023 11:43 PM

Quoted 1 line

If it’s a "flowvq" dedicated queue device both the requirements are

addressed.

Quoted 1 line

Device can choose to implement in its preferred way.

I think this is a migration issue:
Devices A and B each offser the VIRTIO_NET_F_RECEIVE_FLOW_FILTER (this is a temporary name) feature.
And device A implements flowvq, device B does not implement flowvq, they are both using receive flow filters but ARFS is disabled.
The migration should fail at this time. Does this meet user expectations?

Quoted 8 lines

If F_RECEIVE_FLOW_FILTER is offered, it means a device supports flowvq on src and dst. So migration wont fail.

It is no different than any other net _F feature. For example notification coalescing or _MQ or _RSS.

What I meant in above "preferred way" is, device can choose to operate flow table entries as 5 flow/sec. Or it can choose to do 100K flows/sec.

Ok, I see. I misunderstood the "preferred way" before :)

Quoted 7 lines

The basic requirement is that flow rule entries should be a low latency interface.

Quoted 2 lines

Or do we warn in the spec that using flow vq might cause the migration issue?

It should not cause migration once the device features and associated attributes can be queries by the group owner.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#45 lore raw

PATCH 6/7 net-features: Add packet timestamp requirements

In reply to Parav Pandit

Add tx and rx packet timestamp requirements.

---
 net-workstream/features-1.4.md | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
net-workstream/features-1.4.md +25 -0
index 41242b4..f94848e 100644
@@ -10,6 +10,7 @@ together is desired while updating the virtio net interface.
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
 4. Receive virtqueue n-tuple steering
+5. Device timestamp for tx and rx packets
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -184,3 +185,27 @@ struct vnet_rx_completion {
 15. The driver and group owner driver should be able to query supported device
     limits for the steering entries.
  
+## 3.5 Packet timestamp
+1. Device should provide transmit timestamp and receive timestamp of the packets
+   at per packet level when the device is enabled.
+2. Device should provide the current free running clock in the least latency
+   possible using an MMIO register read of 64-bit to have the least jitter.
+3. Device should provide the current frequency and the frequency unit for the
+   software to synchronize the reference point of software and the device using
+   a control vq command.
+
+### 3.5.1 Transmit timestamp
+1. Transmit completion must contain a packet transmission timestamp when the
+   device is enabled for it.
+2. The device should record the packet transmit timestamp in the completion at
+   the farthest egress point towards the network.
+3. The device must provide a transmit packet timestamp in a single DMA
+   transaction along with the rest of the transmit completion fields.
+
+### 3.5.2 Receive timestamp
+1. Receive completion must contain a packet reception timestamp when the device
+   is enabled for it.
+2. The device should record the received packet timestamp at the closet ingress
+   point of reception from the network.
+3. The device should provide a receive packet timestamp in a single DMA
+   transaction along with the rest of the receive completion fields.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#46 lore raw

Re: [virtio-comment] [PATCH requirements 6/7] net-features: Add packet timestamp requirements

Quoted 20 lines · Parav Pandit

Add tx and rx packet timestamp requirements.

---
 net-workstream/features-1.4.md | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
net-workstream/features-1.4.md +1 -0
index 41242b4..f94848e 100644
@@ -10,6 +10,7 @@ together is desired while updating the virtio net interface.
 2. Low latency tx and rx virtqueues for PCI transport
 3. Virtqueue notification coalescing re-arming support
 4. Receive virtqueue n-tuple steering
+5. Device timestamp for tx and rx packets
 
 # 3. Requirements
 ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 192

Quoted 8 lines
net-workstream/features-1.4.md +5 -0
@@ -184,3 +185,27 @@ struct vnet_rx_completion {
 15. The driver and group owner driver should be able to query supported device
     limits for the steering entries.
 
+## 3.5 Packet timestamp
+1. Device should provide transmit timestamp and receive timestamp of the packets
+   at per packet level when the device is enabled.
+2. Device should provide the current free running clock in the least latency
+   possible using an MMIO register read of 64-bit to have the least jitter.

wow these reads are expensive. what is the actual requirement?

net-workstream/features-1.4.md struct vnet_rx_completion { · line 195

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+3. Device should provide the current frequency and the frequency unit for the
+   software to synchronize the reference point of software and the device using
+   a control vq command.

let's leave mechanism out of it. I think you are trying to say this is async to data vqs? and cvq is fine.

Quoted 6 lines
net-workstream/features-1.4.md +6 -0
+
+### 3.5.1 Transmit timestamp
+1. Transmit completion must contain a packet transmission timestamp when the
+   device is enabled for it.
+2. The device should record the packet transmit timestamp in the completion at
+   the farthest egress point towards the network.

net-workstream/features-1.4.md struct vnet_rx_completion { · line 211

Quoted 10 lines
net-workstream/features-1.4.md +10 -0
+3. The device must provide a transmit packet timestamp in a single DMA
+   transaction along with the rest of the transmit completion fields.
+
+### 3.5.2 Receive timestamp
+1. Receive completion must contain a packet reception timestamp when the device
+   is enabled for it.
+2. The device should record the received packet timestamp at the closet ingress
+   point of reception from the network.
+3. The device should provide a receive packet timestamp in a single DMA
+   transaction along with the rest of the receive completion fields.

how large do these need to be? ok for timer to overflow?

why do you mention migration in counters but not in timers?

is it ok for time to go back?

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#47 lore raw

RE: [virtio-comment] [PATCH requirements 6/7] net-features: Add packet timestamp requirements

Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:40 PM

Quoted 9 lines
Quoted 2 lines

+## 3.5 Packet timestamp
+1. Device should provide transmit timestamp and receive timestamp of the

packets

net-workstream/features-1.4.md struct vnet_rx_completion { · line 192

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   at per packet level when the device is enabled.
+2. Device should provide the current free running clock in the least latency
+   possible using an MMIO register read of 64-bit to have the least jitter.

wow these reads are expensive. what is the actual requirement?

I don't understand the question.

Quoted 9 lines
Quoted 1 line

+3. Device should provide the current frequency and the frequency unit for

the

Quoted 1 line

+ software to synchronize the reference point of software and the device

using

Quoted 1 line

+ a control vq command.

let's leave mechanism out of it. I think you are trying to say this is async to data vqs? and cvq is fine.

There are two clocks running, one of cpu where driver is running, other one is in the device. And timestamps are done by the device based on its clock.

So driver can convert the device timestamp to driver cpu timestamp. This is like one time query to know conversion mechanics.

Quoted 12 lines
Quoted 2 lines

+### 3.5.2 Receive timestamp
+1. Receive completion must contain a packet reception timestamp when the

device

Quoted 2 lines

+ is enabled for it.
+2. The device should record the received packet timestamp at the closet

ingress

net-workstream/features-1.4.md struct vnet_rx_completion { · line 211

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   point of reception from the network.
+3. The device should provide a receive packet timestamp in a single DMA
+   transaction along with the rest of the receive completion fields.

how large do these need to be? ok for timer to overflow?

Will add it. Usually 8B.
But overflow after 30+ years or so is common norm.

Quoted 4 lines

why do you mention migration in counters but not in timers?

is it ok for time to go back?

Should add it for timer too.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#48 lore raw

Re: [virtio-comment] [PATCH requirements 6/7] net-features: Add packet timestamp requirements

Quoted 7 lines · Parav Pandit
Quoted 7 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:40 PM

Quoted 2 lines

+## 3.5 Packet timestamp
+1. Device should provide transmit timestamp and receive timestamp of the

Quoted 8 lines
Quoted 7 lines

packets

net-workstream/features-1.4.md struct vnet_rx_completion { · line 192

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   at per packet level when the device is enabled.
+2. Device should provide the current free running clock in the least latency
+   possible using an MMIO register read of 64-bit to have the least jitter.

wow these reads are expensive. what is the actual requirement?

I don't understand the question.

placing something in a register is a solution. what is the requirement?

Quoted 32 lines
Quoted 9 lines
Quoted 1 line

+3. Device should provide the current frequency and the frequency unit for

the

Quoted 1 line

+ software to synchronize the reference point of software and the device

using

Quoted 1 line

+ a control vq command.

let's leave mechanism out of it. I think you are trying to say this is async to data vqs? and cvq is fine.

There are two clocks running, one of cpu where driver is running, other one is in the device. And timestamps are done by the device based on its clock.

So driver can convert the device timestamp to driver cpu timestamp. This is like one time query to know conversion mechanics.

Quoted 12 lines
Quoted 2 lines

+### 3.5.2 Receive timestamp
+1. Receive completion must contain a packet reception timestamp when the

device

Quoted 2 lines

+ is enabled for it.
+2. The device should record the received packet timestamp at the closet

ingress

net-workstream/features-1.4.md struct vnet_rx_completion { · line 211

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+   point of reception from the network.
+3. The device should provide a receive packet timestamp in a single DMA
+   transaction along with the rest of the receive completion fields.

how large do these need to be? ok for timer to overflow?

Will add it. Usually 8B.
But overflow after 30+ years or so is common norm.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

why do you mention migration in counters but not in timers?

is it ok for time to go back?

Should add it for timer too.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#49 lore raw

PATCH 7/7 net-features: Add header data split requirements

In reply to Parav Pandit

Add header data split requirements for the receive packets.

---
 net-workstream/features-1.4.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)
net-workstream/features-1.4.md +13 -0
index f94848e..31680b7 100644
@@ -11,6 +11,7 @@ together is desired while updating the virtio net interface.
 3. Virtqueue notification coalescing re-arming support
 4. Receive virtqueue n-tuple steering
 5. Device timestamp for tx and rx packets
+6. Header data split for the receive virtqueue
 
 # 3. Requirements
 ## 3.1 Device counters
@@ -209,3 +210,15 @@ struct vnet_rx_completion {
    point of reception from the network.
 3. The device should provide a receive packet timestamp in a single DMA
    transaction along with the rest of the receive completion fields.
+
+## 3.6 Header data split for the receive virtqueue
+1. The device should be able to DMA the packet header and data to two different
+   memory locations, this enables driver and networking stack to perform zero
+   copy to application buffer(s).
+2. The driver should be able to configure maximum header buffer size per
+   virtqueue.
+3. The header buffer to be in a physically contiguous memory per virtqueue
+4. The device should be able to indicate header data split in the receive
+   completion.
+5. The device should be able to zero pad the header buffer when the received
+   header is shorter than cpu cache line size.
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#50 lore raw

Re: [virtio-comment] [PATCH requirements 7/7] net-features: Add header data split requirements

Quoted 20 lines · Parav Pandit

Add header data split requirements for the receive packets.

---
 net-workstream/features-1.4.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)
net-workstream/features-1.4.md +1 -0
index f94848e..31680b7 100644
@@ -11,6 +11,7 @@ together is desired while updating the virtio net interface.
 3. Virtqueue notification coalescing re-arming support
 4. Receive virtqueue n-tuple steering
 5. Device timestamp for tx and rx packets
+6. Header data split for the receive virtqueue
 
 # 3. Requirements
 ## 3.1 Device counters

net-workstream/features-1.4.md struct vnet_rx_completion { · line 217

Quoted 8 lines
net-workstream/features-1.4.md +5 -0
@@ -209,3 +210,15 @@ struct vnet_rx_completion {
    point of reception from the network.
 3. The device should provide a receive packet timestamp in a single DMA
    transaction along with the rest of the receive completion fields.
+
+## 3.6 Header data split for the receive virtqueue
+1. The device should be able to DMA the packet header and data to two different
+   memory locations, this enables driver and networking stack to perform zero
+   copy to application buffer(s).

what is header here? which layer?

net-workstream/features-1.4.md struct vnet_rx_completion { · line 220

Quoted 3 lines
net-workstream/features-1.4.md +3 -0
+2. The driver should be able to configure maximum header buffer size per
+   virtqueue.
+3. The header buffer to be in a physically contiguous memory per virtqueue

this is a requirement why?

net-workstream/features-1.4.md struct vnet_rx_completion { · line 224

Quoted 4 lines
net-workstream/features-1.4.md +4 -0
+4. The device should be able to indicate header data split in the receive
+   completion.
+5. The device should be able to zero pad the header buffer when the received
+   header is shorter than cpu cache line size.

weirdly specific.
how does device even know the cache line?

Quoted 20 lines
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#51 lore raw

RE: [virtio-comment] [PATCH requirements 7/7] net-features: Add header data split requirements

Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:42 PM

Quoted 8 lines

Review

Quoted 3 lines
(quoted diff) +3 -0
+## 3.6 Header data split for the receive virtqueue 1. The device
+should be able to DMA the packet header and data to two different
+   memory locations, this enables driver and networking stack to perform

zero

Quoted 1 line

+ copy to application buffer(s).

what is header here? which layer?

Usually L2 to L4. To be defined further.

Quoted 7 lines

Review

Quoted 4 lines

+2. The driver should be able to configure maximum header buffer size per

(quoted diff) +3 -0
+   virtqueue.
+3. The header buffer to be in a physically contiguous memory per
+virtqueue

this is a requirement why?

Because fetching individual buffer memory pointer is sub-optimal.

Quoted 9 lines
Quoted 3 lines

+4. The device should be able to indicate header data split in the receive
+ completion.
+5. The device should be able to zero pad the header buffer when the

received

Quoted 1 line

+ header is shorter than cpu cache line size.

weirdly specific.
how does device even know the cache line?

A driver can tell how much padding to be done by keeping the cache line specific details in the driver, but only telling about padding size.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#52 lore raw
In reply to Parav Pandit
Quoted 22 lines · Parav Pandit

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

I would like to capture if there are any other requirements linked to the features captured in this document to consider in the interface design.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and work towards the interface design followed by drafting the specification changes.

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

This is not a specification document. It is a pre-work document that helps to understand the draft specification addressing these requirements.

Please review.

Quoted 8 lines

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow steering requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

Hi, Parav.

This 1.4 job listing is pretty cool! And I believe that these work plans overlap a lot with our/your needs, it would be interesting if we could work together to push these new features to make them actually available on VirtIO!

Thanks!

Quoted 5 lines
  net-workstream/features-1.4.md | 224 +++++++++++++++++++++++++++++++++
  1 file changed, 224 insertions(+)

create mode 100644 net-workstream/features-1.4.md

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#53 lore raw
In reply to Parav Pandit
Quoted 13 lines · Parav Pandit

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

I would like to capture if there are any other requirements linked to the features captured in this document to consider in the interface design.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and work towards the interface design followed by drafting the specification changes.

Quoted 6 lines

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

This is not a specification document. It is a pre-work document that helps to understand the draft specification addressing these requirements.

thanks for write up! great start.

some things look a bit more like a specific design that you have in mind. we need to take a step back and include the actual requirement too. it's ok to list a design idea too (you call them examples here).

splitting different areas in separate files might be a good idea.

some things will require virtio core extensions.

I have a generic comment about limitations such as alignment physically contiguous etc. It's natural for hardware vendors to try and push complexity to software, but in the end this might just shift overhead to driver/TCP stack. Please try to add motivation with each requirement. I pointed out some of such cases as "weirdly specific".

Quoted 11 lines

Please review.

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow steering requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

Quoted 24 lines
 net-workstream/features-1.4.md | 224 +++++++++++++++++++++++++++++++++
 1 file changed, 224 insertions(+)
 create mode 100644 net-workstream/features-1.4.md
Signature · 18 lines
2.26.2


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#54 lore raw
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:50 PM

Quoted 8 lines

thanks for write up! great start.

some things look a bit more like a specific design that you have in mind. we need to take a step back and include the actual requirement too. it's ok to list a design idea too (you call them examples here).

splitting different areas in separate files might be a good idea.

Yes, as it grows, will likely do it, when the design part takes shape and gets extended as section. At present with 10 to 20 lines in a file is overhead.

Quoted 5 lines

some things will require virtio core extensions.

I have a generic comment about limitations such as alignment physically contiguous etc. It's natural for hardware vendors to try and push complexity to software, but in the end this might just shift overhead to driver/TCP stack.

Mostly for sure not the TCP stack.
Some part may be to software.
Overall software cost might be same or probably slightly higher but can have overall system improvement.

Quoted 4 lines

Please try to add motivation with each requirement.

I pointed out some of such cases as "weirdly specific".

Yes. I had the rationale section, but I dropped it in the initial version to add on need basis to cut out the obvious. Will insert some of it in v1.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#55 lore raw
Quoted 16 lines · Parav Pandit
Quoted 5 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 6:50 PM

Quoted 8 lines

thanks for write up! great start.

some things look a bit more like a specific design that you have in mind. we need to take a step back and include the actual requirement too. it's ok to list a design idea too (you call them examples here).

splitting different areas in separate files might be a good idea.

Yes, as it grows, will likely do it, when the design part takes shape and gets extended as section. At present with 10 to 20 lines in a file is overhead.

Quoted 8 lines
Quoted 5 lines

some things will require virtio core extensions.

I have a generic comment about limitations such as alignment physically contiguous etc. It's natural for hardware vendors to try and push complexity to software, but in the end this might just shift overhead to driver/TCP stack.

Mostly for sure not the TCP stack.
Some part may be to software.
Overall software cost might be same or probably slightly higher but can have overall system improvement.

Then pls spend some time for each requirement documenting drawbacks too. Almost anything has tradeoffs, listing them makes you actually think whether things are worth it and shows others you are being objective.

Quoted 6 lines
Quoted 4 lines

Please try to add motivation with each requirement.

I pointed out some of such cases as "weirdly specific".

Yes. I had the rationale section, but I dropped it in the initial version to add on need basis to cut out the obvious. Will insert some of it in v1.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#56 lore raw
In reply to Parav Pandit
Quoted 32 lines · Parav Pandit

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

I would like to capture if there are any other requirements linked to the features captured in this document to consider in the interface design.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and work towards the interface design followed by drafting the specification changes.

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

This is not a specification document. It is a pre-work document that helps to understand the draft specification addressing these requirements.

Please review.

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow steering requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

Quoted 1 line
 net-workstream/features-1.4.md | 224 +++++++++++++++++++++++++++++++++

I feel that some patches go into too many details of the proposed solution which makes it more like RFC instead of a requirement.

It would be easier to start with to describe why a feature is needed and why it can't work well in current spec.

Thanks

Quoted 14 lines
 1 file changed, 224 insertions(+)
 create mode 100644 net-workstream/features-1.4.md

--
2.26.2

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#57 lore raw
Quoted 38 lines

From: Jason Wang <[email protected]>
Sent: Tuesday, June 6, 2023 10:50 PM

Quoted 32 lines · Parav Pandit

Hi All,

This document captures the virtio net device requirements for the upcoming release 1.4 that some of us are currently working on.

I would like to capture if there are any other requirements linked to the features captured in this document to consider in the interface design.

The objectives are:
1. to consider these requirements in introducing new features listed in the document and work towards the interface design followed by drafting the specification changes.

2. Define practical list of requirements that can be achieved in 1.4 timeframe incrementally and also have the ability to implement them.

This is not a specification document. It is a pre-work document that helps to understand the draft specification addressing these requirements.

Please review.

Parav Pandit (7):
net-features: Add requirements document for release 1.4
net-features: Add low latency transmit queue requirements
net-features: Add low latency receive queue requirements
net-features: Add notification coalescing requirements
net-features: Add n-tuple receive flow steering requirements
net-features: Add packet timestamp requirements
net-features: Add header data split requirements

Quoted 8 lines
Quoted 2 lines
 net-workstream/features-1.4.md | 224

+++++++++++++++++++++++++++++++++

I feel that some patches go into too many details of the proposed solution which makes it more like RFC instead of a requirement.

It would be easier to start with to describe why a feature is needed and why it can't work well in current spec.

It is worded differently as features/requirements. This probably only applies to tx and rx q. Rest all are the new features so its not about "cannot work well", because it just doesnt exist.

Instead of explaining negatively "why just yet add new descriptor for transmit timestamp", it is written with optimism in mind, how it should be done in one dma ...

But I agree that small portion can have rationale section, if its unclear.