Skip to contentSkip to conversation

Selective queue enabling

PATCH In review Eugenio Pérez 6 participants 76 messages Jun 6, 2023 – Jul 11, 2023

lore
Share & tools
lore
#0 lore raw

cover Selective queue enabling

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)
Signature · 19 lines
2.31.1



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#1 lore raw

PATCH 1/2 virtio: introduce selective queue enabling

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +13 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 Every driver and device supports either the Packed or the Split
 Virtqueue format, or both.
 
+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can enable
+individual virtqueues both before and after the DRIVER\_OK. The way to enable a
+virtqueue is transport specific, but it mimics the re-enabling after an
+hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}
 
 When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates
+  that the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 18 lines
2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#2 lore raw

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Hi Eugenio,

Quoted 60 lines

From: Eugenio Pérez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +3 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
Quoted 2 lines

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

#3 lore raw

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 59 lines · Parav Pandit

Hi Eugenio,

Quoted 57 lines

From: Eugenio Pérez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +0 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
Quoted 9 lines
Quoted 5 lines
content.tex +3 -0
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#4 lore raw

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 3:20 PM

Quoted 4 lines

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

Quoted 7 lines

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.

Quoted 2 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

This backend implementation can be improved without introducing new bits on the guest side. Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.

And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#5 lore raw

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 3:20 PM

Quoted 4 lines

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

Quoted 6 lines
Quoted 3 lines

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.

I also read it this way to be honest. From a POV, to allow it if a device does not handle it could be considered a fix, rather than an addition.

I'm assuming we need to modify the standard just because it was not explicitly allowed. To explicitly allow it before DRIVER_OK was not needed because it was already widespread.

It would be great for me if we agree the device must support it, but maybe it is just too late for that.

Thanks!

Quoted 15 lines
Quoted 2 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

This backend implementation can be improved without introducing new bits on the guest side. Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.

And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#6 lore raw

RE: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 10 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Thursday, June 8, 2023 7:54 AM

Quoted 2 lines

Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done

after DRIVER_OK.

Quoted 1 line
Quoted 7 lines

I also read it this way to be honest. From a POV, to allow it if a device does not handle it could be considered a fix, rather than an addition.

I'm assuming we need to modify the standard just because it was not explicitly allowed. To explicitly allow it before DRIVER_OK was not needed because it was already widespread.

It was explicit enough that queues can be enabled after DRIVER_OK.
And queue reset is the perfect example showed that.
Who ever implements the device should be ready to handle queue_enable writes after DRIVER_OK stage, it is really that simple. There is no need of new feature bit.

Quoted 2 lines

It would be great for me if we agree the device must support it, but maybe it is just too late for that.

Device must support it. There is nothing late about it.
And if you are considering LM flow, then for sure it is not late because LM is just not fully baked.

#7 lore raw

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 10 lines · Parav Pandit
Quoted 10 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Thursday, June 8, 2023 7:54 AM

Quoted 2 lines

Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done

after DRIVER_OK.

Quoted 1 line
Quoted 8 lines
Quoted 7 lines

I also read it this way to be honest. From a POV, to allow it if a device does not handle it could be considered a fix, rather than an addition.

I'm assuming we need to modify the standard just because it was not explicitly allowed. To explicitly allow it before DRIVER_OK was not needed because it was already widespread.

It was explicit enough that queues can be enabled after DRIVER_OK.

Weird, there's spec text saying they can't.

Quoted 3 lines

And queue reset is the perfect example showed that.
Who ever implements the device should be ready to handle queue_enable writes after DRIVER_OK stage, it is really that simple. There is no need of new feature bit.

Hard to change the past, we didn't ask people to do it.

Quoted 4 lines
Quoted 2 lines

It would be great for me if we agree the device must support it, but maybe it is just too late for that.

Device must support it. There is nothing late about it.
And if you are considering LM flow, then for sure it is not late because LM is just not fully baked.

Considering regular use, no LM.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#8 lore raw

RE: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 4 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Thursday, June 8, 2023 11:06 AM

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

I'm assuming we need to modify the standard just because it was not explicitly allowed. To explicitly allow it before DRIVER_OK was not needed because it was already widespread.

It was explicit enough that queues can be enabled after DRIVER_OK.

Weird, there's spec text saying they can't.

So far no one cited that text here. :)
You pointed to the general initialization section but it doesnt specify.

Quoted 7 lines
Quoted 2 lines

And queue reset is the perfect example showed that.
Who ever implements the device should be ready to handle queue_enable

writes after DRIVER_OK stage, it is really that simple.

Quoted 1 line

There is no need of new feature bit.

Hard to change the past, we didn't ask people to do it.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

It would be great for me if we agree the device must support it, but maybe it is just too late for that.

Device must support it. There is nothing late about it.
And if you are considering LM flow, then for sure it is not late because LM is

just not fully baked.

Considering regular use, no LM.

_RESET is the existing feature bit that allows one to enable queues after DRIVER_OK.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#9 lore raw

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Eugenio Perez Martin
Quoted 14 lines · Eugenio Perez Martin
Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 6, 2023 3:20 PM

Quoted 4 lines

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.

I also read it this way to be honest.

You have been doing spec lawyering for too long, take a step back, take a walk, then look at it from common sense POV.

Well spec says: you MUST follow this order:
- A
- B
- C

can one conclude that if C triggers then B already happened?

I would conclude yes.

and just because it does not say *and do not do it in any other order* you are saying
- A
- C
- B

is also allowed?

MUST follow this rule to me means "and do not violate this rule".

Quoted 3 lines

From a POV, to allow it if a
device does not handle it could be considered a fix, rather than an addition.

I think vhost user at least is started at DRIVER_OK and at that point it looks at # of enabled Qs and allocates a bunch of arrays to match.

Quoted 4 lines

I'm assuming we need to modify the standard just because it was not explicitly allowed. To explicitly allow it before DRIVER_OK was not needed because it was already widespread.

Quoted 29 lines

It would be great for me if we agree the device must support it, but maybe it is just too late for that.

Thanks!

Quoted 24 lines
Quoted 2 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

This backend implementation can be improved without introducing new bits on the guest side. Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.

And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#10 lore raw

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Michael S. Tsirkin
Quoted 71 lines · Michael S. Tsirkin
Quoted 59 lines · Parav Pandit

Hi Eugenio,

Quoted 57 lines

From: Eugenio P閞ez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +0 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
Quoted 9 lines
Quoted 5 lines
content.tex +3 -0
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

Quoted 5 lines

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Although there is no "all queues" here, if the queue is not ready, the device may use all the queues after "Device OK". So I also think it might be dangerous.

But I also ran into this problem. We want the device to support 100 queues, but we also want to use only 16 queues by default. The user can then manually scale up to 100, if the user has a large memory.

So I think, the point is that "num_queues" is the max and default value. Can we change "num_queues" to default, then we introduced a new value as max.

Does this way solve your problem? @Eugenio

Thanks.

Quoted 7 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#11 lore raw

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 9 lines

From: Xuan Zhuo <[email protected]>
Sent: Thursday, June 8, 2023 8:12 AM

Although there is no "all queues" here, if the queue is not ready, the device may use all the queues after "Device OK". So I also think it might be dangerous.

Device can only used enabled queues at any stage before/after DRIVER_OK. If the queue is not enabled, there is simply no rx packet or anything on that queue.

Quoted 4 lines

But I also ran into this problem. We want the device to support 100 queues, but we also want to use only 16 queues by default. The user can then manually scale up to 100, if the user has a large memory.

So, one config sequence :
a. driver can enable 16 queues before driver ok b. driver does driver ok
c. few hours later driver enables remaining 84 queues

in second config sequence:
a. driver can enable 0 queues before driver ok b. driver does driver ok
c. few hours later driver enables remaining 16+84 queues

#12 lore raw

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Xuan Zhuo
Quoted 82 lines · Xuan Zhuo
Quoted 71 lines · Michael S. Tsirkin
Quoted 59 lines · Parav Pandit

Hi Eugenio,

Quoted 57 lines

From: Eugenio P閞ez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +0 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
  \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
  handling features reserved for future use.
 
Quoted 9 lines
Quoted 5 lines
content.tex +3 -0
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

Quoted 5 lines

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Although there is no "all queues" here, if the queue is not ready, the device may use all the queues after "Device OK". So I also think it might be dangerous.

Quoted 9 lines

But I also ran into this problem. We want the device to support 100 queues, but we also want to use only 16 queues by default. The user can then manually scale up to 100, if the user has a large memory.

So I think, the point is that "num_queues" is the max and default value. Can we change "num_queues" to default, then we introduced a new value as max.

Does this way solve your problem? @Eugenio

It would need a tweak, but it gives me an idea.

I think you're talking about what the device offers, isn't it? I would need to modify it from the driver.

But it has an interesting question, how to allocate "less vrings" than virtio_pci_common_cfg.num_queues? I guess it is ok to set all vq addresses to a vring that never increments avail_idx? The driver still needs to allocate memory for that one and lose start time configuring them.

Maybe for net is valid to leave them unconfigured? it should only use one vq pair and CVQ by default.

My proposal could be done at the virtio-net level, allowing max_virtqueue_pairs to be 0 from the beginning. For example, allowing the driver to write it before DRIVER_OK. In that case, no data vq is enabled and only CVQ process requests after DRIVER_OK. After all the config is written, the driver changes the number of active queues with VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

I think it is better solved in a generic way, but I'm ok with this -net specific too.

Thanks!

Quoted 19 lines

Thanks.

Quoted 7 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#13 lore raw

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 93 lines · Eugenio Perez Martin
Quoted 82 lines · Xuan Zhuo
Quoted 71 lines · Michael S. Tsirkin
Quoted 59 lines · Parav Pandit

Hi Eugenio,

Quoted 57 lines

From: Eugenio P閞ez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +0 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
  \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
  handling features reserved for future use.
 
Quoted 9 lines
Quoted 5 lines
content.tex +3 -0
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

Quoted 5 lines

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Although there is no "all queues" here, if the queue is not ready, the device may use all the queues after "Device OK". So I also think it might be dangerous.

Quoted 9 lines

But I also ran into this problem. We want the device to support 100 queues, but we also want to use only 16 queues by default. The user can then manually scale up to 100, if the user has a large memory.

So I think, the point is that "num_queues" is the max and default value. Can we change "num_queues" to default, then we introduced a new value as max.

Does this way solve your problem? @Eugenio

Quoted 7 lines

It would need a tweak, but it gives me an idea.

I think you're talking about what the device offers, isn't it? I would need to modify it from the driver.

But it has an interesting question, how to allocate "less vrings" than virtio_pci_common_cfg.num_queues?

Just allocate less, don't configure what you do not use.

Quoted 7 lines

I guess it is ok to set all vq
addresses to a vring that never increments avail_idx? The driver still needs to allocate memory for that one and lose start time configuring them.

Maybe for net is valid to leave them unconfigured? it should only use one vq pair and CVQ by default.

It's ok for all devices, and this is how it always worked.

Quoted 7 lines

My proposal could be done at the virtio-net level, allowing max_virtqueue_pairs to be 0 from the beginning. For example, allowing the driver to write it before DRIVER_OK. In that case, no data vq is enabled and only CVQ process requests after DRIVER_OK. After all the config is written, the driver changes the number of active queues with VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

Quoted 33 lines

I think it is better solved in a generic way, but I'm ok with this -net specific too.

Thanks!

Quoted 28 lines

Thanks.

Quoted 7 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#14 lore raw

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Eugenio Perez Martin
Quoted 90 lines · Eugenio Perez Martin
Quoted 89 lines · Xuan Zhuo
Quoted 71 lines · Michael S. Tsirkin
Quoted 59 lines · Parav Pandit

Hi Eugenio,

Quoted 57 lines

From: Eugenio P閞ez <[email protected]>
Sent: Tuesday, June 6, 2023 1:55 PM

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
 Virtio Device / Feature B  \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future

extensions.
\end{description}

\begin{note}

content.tex +0 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a

Virtio Device / Virtqueues} Every driver and device supports either the Packed or the Split Virtqueue format, or both.

+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of

content.tex +7 -0
+a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
+enable individual virtqueues both before and after the DRIVER\_OK. The
+way to enable a virtqueue is transport specific, but it mimics the
+re-enabling after an hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /

Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +0 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
 Feature Bits}
  \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
  handling features reserved for future use.
 
Quoted 9 lines
Quoted 5 lines
content.tex +3 -0
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
+ the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective

Virtqueue Enable}.
+

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

Quoted 5 lines

This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think assuming it's safe to enable queues later is dangerous.

Although there is no "all queues" here, if the queue is not ready, the device may use all the queues after "Device OK". So I also think it might be dangerous.

But I also ran into this problem. We want the device to support 100 queues, but we also want to use only 16 queues by default. The user can then manually scale up to 100, if the user has a large memory.

So I think, the point is that "num_queues" is the max and default value. Can we change "num_queues" to default, then we introduced a new value as max.

Quoted 6 lines
Quoted 2 lines

Does this way solve your problem? @Eugenio

It would need a tweak, but it gives me an idea.

I think you're talking about what the device offers, isn't it?

Yes, I want the driver offers a default-num-queues and a max-num-queues.

Thanks

Quoted 19 lines

I would
need to modify it from the driver.

But it has an interesting question, how to allocate "less vrings" than virtio_pci_common_cfg.num_queues? I guess it is ok to set all vq addresses to a vring that never increments avail_idx? The driver still needs to allocate memory for that one and lose start time configuring them.

Maybe for net is valid to leave them unconfigured? it should only use one vq pair and CVQ by default.

My proposal could be done at the virtio-net level, allowing max_virtqueue_pairs to be 0 from the beginning. For example, allowing the driver to write it before DRIVER_OK. In that case, no data vq is enabled and only CVQ process requests after DRIVER_OK. After all the config is written, the driver changes the number of active queues with VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

Quoted 34 lines

I think it is better solved in a generic way, but I'm ok with this -net specific too.

Thanks!

Quoted 28 lines

Thanks.

Quoted 7 lines

E.g. vhost user backends tend to assume no new queues appear after DRIVER_OK.

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#15 lore raw

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Eugenio Pérez
Quoted 33 lines · Eugenio Pérez

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
content.tex +2 -2
index d2ab9eb..ea4a6d8 100644
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 403

Quoted 6 lines
content.tex +3 -0
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 Every driver and device supports either the Packed or the Split
 Virtqueue format, or both.
 
+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME,

No need for \_, we build with _ not being a special character.

content.tex \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues} · line 407

Quoted 5 lines

it can enable

content.tex +4 -0
+individual virtqueues both before and after the DRIVER\_OK. The way to enable a
+virtqueue is transport specific, but it mimics the re-enabling after an
+hypothetical reset.
+

This is not a good way to do it: spec describes a certain order of initialization in several places. Please find all of them and fix, don't just add a separate chapter as a kind of kludge.

Quoted 16 lines

\subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}

When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue

content.tex +4 -0
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates
+  that the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
Signature · 1 line
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#16 lore raw

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

In reply to Eugenio Pérez
Quoted 11 lines · Eugenio Pérez

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

I realized I never asked. *Why* enable CVQ so early? I think it's so that commands take effect before any buffers are processed? So maybe it's a net specific issue?

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#17 lore raw

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

Quoted 12 lines · Michael S. Tsirkin
Quoted 10 lines · Eugenio Pérez

This patch allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Quoted 8 lines
Quoted 1 line

I realized I never asked. *Why* enable CVQ so early? I think it's so that commands take effect before any buffers are processed? So maybe it's a net specific issue?

Yes, that's the reason. Can I add something to the patch message to make it clearer?

It can be solved using a net specific solution for sure, but I think we can make it general if other devices with CVQ have such needs.

Thanks.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#18 lore raw

PATCH 2/2 virtio: pci support virtqueue selective enabling

In reply to Eugenio Pérez

Reusing virtqueue reset method to enabling a vq.

---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)
transport-pci.tex +4 -0
index a5c6719..04c0429 100644
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.
 
 \item[\field{queue_notify_off}]
Signature · 18 lines
2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#19 lore raw

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

Quoted 12 lines · Eugenio Pérez

Reusing virtqueue reset method to enabling a vq.

---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)
transport-pci.tex +0 -0
index a5c6719..04c0429 100644

transport-pci.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 382

Quoted 8 lines
transport-pci.tex +4 -0
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.

No, this is always used to enable queues even with no features.

Quoted 3 lines
transport-pci.tex +0 -0
 \item[\field{queue_notify_off}]
Signature · 1 line
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#20 lore raw

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Pérez

Reusing virtqueue reset method to enabling a vq.

---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)
transport-pci.tex +0 -0
index a5c6719..04c0429 100644
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
Quoted 10 lines

transport-pci.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 382

Quoted 7 lines
transport-pci.tex +4 -0
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.

No, this is always used to enable queues even with no features.

Yes. Now I see it is confusing if reading the spec and not the diff, but my intention was to not modify the behavior without the feature flags. Maybe there is a better way to achieve it.

Quoted 4 lines
Quoted 3 lines
transport-pci.tex +0 -1
 \item[\field{queue_notify_off}]
--

2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#21 lore raw

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

Quoted 27 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Pérez

Reusing virtqueue reset method to enabling a vq.

---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)
transport-pci.tex +0 -0
index a5c6719..04c0429 100644
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
Quoted 10 lines

transport-pci.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 382

Quoted 7 lines
transport-pci.tex +4 -0
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.

No, this is always used to enable queues even with no features.

Quoted 3 lines

Yes. Now I see it is confusing if reading the spec and not the diff, but my intention was to not modify the behavior without the feature flags. Maybe there is a better way to achieve it.

This is bad:

Driver always does X.
+If Y then driver does Z.

This is good:

-Driver always does X.
-If Y, then driver does Z, otherwise X.

even though it's a bigger patch, but patch review is secondary to a readable spec.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
transport-pci.tex +0 -1
 \item[\field{queue_notify_off}]
--

2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#22 lore raw

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

Quoted 44 lines · Michael S. Tsirkin
Quoted 27 lines · Eugenio Perez Martin
Quoted 15 lines · Michael S. Tsirkin
Quoted 13 lines · Eugenio Pérez

Reusing virtqueue reset method to enabling a vq.

---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)
transport-pci.tex +0 -0
index a5c6719..04c0429 100644
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
Quoted 10 lines

transport-pci.tex \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport · line 382

Quoted 7 lines
transport-pci.tex +4 -0
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.

No, this is always used to enable queues even with no features.

Quoted 3 lines

Yes. Now I see it is confusing if reading the spec and not the diff, but my intention was to not modify the behavior without the feature flags. Maybe there is a better way to achieve it.

This is bad:

Driver always does X.
+If Y then driver does Z.

This is good:

(quoted diff) +0 -2
-Driver always does X.
-If Y, then driver does Z, otherwise X.
Quoted 3 lines

even though it's a bigger patch, but patch review is secondary to a readable spec.

I agree, I'll modify it in that direction if we agree on using PCI config space this way.

Thanks!

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
transport-pci.tex +0 -1
 \item[\field{queue_notify_off}]
--

2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#23 lore raw
In reply to Eugenio Pérez
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)
Signature · 1 line
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#24 lore raw
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Quoted 7 lines

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there. Apart from that, the back and forth introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

Thanks!

Quoted 4 lines
Quoted 4 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

Quoted 8 lines
Quoted 7 lines
 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#25 lore raw
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 6 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#26 lore raw
Quoted 23 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Quoted 9 lines
Quoted 6 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#27 lore raw
Quoted 28 lines · Eugenio Perez Martin
Quoted 27 lines · Xuan Zhuo
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

Quoted 42 lines
Quoted 2 lines

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#28 lore raw
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 30 lines · Xuan Zhuo
Quoted 27 lines · Eugenio Perez Martin
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Quoted 7 lines

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#29 lore raw
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues? How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#30 lore raw
Quoted 42 lines · Michael S. Tsirkin
Quoted 40 lines · Eugenio Perez Martin
Quoted 39 lines · Michael S. Tsirkin
Quoted 36 lines · Eugenio Perez Martin
Quoted 23 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#31 lore raw
Quoted 46 lines · Jason Wang
Quoted 45 lines · Michael S. Tsirkin
Quoted 43 lines · Eugenio Perez Martin
Quoted 41 lines · Michael S. Tsirkin
Quoted 28 lines · Eugenio Perez Martin
Quoted 27 lines · Xuan Zhuo
Quoted 25 lines · Eugenio Perez Martin
Quoted 23 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

Quoted 60 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#32 lore raw
Quoted 51 lines · Michael S. Tsirkin
Quoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Eugenio Perez Martin
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 30 lines · Xuan Zhuo
Quoted 27 lines · Eugenio Perez Martin
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Quoted 7 lines

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Thanks!

Quoted 51 lines
Quoted 51 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#33 lore raw
Quoted 68 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Quoted 6 lines

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used. Maybe we could use e.g. bit 1 in queue_enable to signal that?

Quoted 51 lines
Quoted 51 lines
Quoted 51 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#34 lore raw
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 4 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

Is there a spec section that guarantees the above behaviour? (I guess not).

Thanks

Quoted 72 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used. Maybe we could use e.g. bit 1 in queue_enable to signal that?

Quoted 63 lines
Quoted 63 lines
Quoted 62 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#35 lore raw
Quoted 76 lines · Jason Wang
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

Is there a spec section that guarantees the above behaviour? (I guess not).

Thanks

yes, the initialization section. driver MUST first setup vqs then set DRIVER_OK.

Quoted 59 lines
Quoted 59 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used. Maybe we could use e.g. bit 1 in queue_enable to signal that?

Quoted 51 lines
Quoted 51 lines
Quoted 51 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

content.tex | 15 +++++++++++++--
transport-pci.tex | 4 ++++
2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#36 lore raw
In reply to Michael S. Tsirkin
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 5 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

The idea is interesting, but we lose the net configuration in a device reset, so we need to send it again.

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in vIOMMU case, we can send all the CVQ commands before allowing the guest to modify mappings. The guest has no way to write on that page through the device as, well, dataplane is not initialized. It's the way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

I don't like the complexity of it but I like it does require even less changes to the device / spec.

Quoted 4 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

Is it even valid to not enable some of the queues?

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Quoted 2 lines

Maybe we could use e.g. bit 1 in queue_enable to signal that?

I'm totally ok to go in that direction.

Thanks!

Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#37 lore raw
Quoted 8 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Thursday, June 8, 2023 4:36 AM

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

num_queues is maximum supported count, not enabled count.

_F_RESET was introduced to enable/disable the queue after DRIVER_OK, which implies that we don’t need yet another feature bit.

Quoted 3 lines

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Queue_reset was born after queue_enable register. So before queue_reset, queue_enable cannot be redundant.

#38 lore raw
In reply to Eugenio Perez Martin
Quoted 76 lines · Eugenio Perez Martin
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

The idea is interesting, but we lose the net configuration in a device reset, so we need to send it again.

Ring reset, not device reset.

Quoted 11 lines

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in vIOMMU case, we can send all the CVQ commands before allowing the guest to modify mappings. The guest has no way to write on that page through the device as, well, dataplane is not initialized. It's the way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

I don't like the complexity of it but I like it does require even less changes to the device / spec.

This is exactly what I meant.

Quoted 8 lines
Quoted 4 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

This is max, not how many there are in practice.

Quoted 1 line

Is it even valid to not enable some of the queues?

Yes and linux will do that if max_virtqueue_pairs > #CPUs.

Quoted 75 lines

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Quoted 2 lines

Maybe we could use e.g. bit 1 in queue_enable to signal that?

I'm totally ok to go in that direction.

Thanks!

Quoted 65 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

content.tex | 15 +++++++++++++--
transport-pci.tex | 4 ++++
2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#39 lore raw
Quoted 98 lines · Michael S. Tsirkin
Quoted 76 lines · Eugenio Perez Martin
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 21 lines · Eugenio Perez Martin
Quoted 20 lines · Michael S. Tsirkin

On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently
devices tend to deduce how many queues are needed by looking
at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then
doing a reset right after DRIVER_OK? You can even allocate
memory for just one queue (zeroing it out).

Quoted 6 lines
Quoted 3 lines

Granted this looks kind of ugly but side-steps this problem with
no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

The idea is interesting, but we lose the net configuration in a device reset, so we need to send it again.

Ring reset, not device reset.

Quoted 9 lines

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in vIOMMU case, we can send all the CVQ commands before allowing the guest to modify mappings. The guest has no way to write on that page through the device as, well, dataplane is not initialized. It's the way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

Quoted 5 lines
Quoted 2 lines

I don't like the complexity of it but I like it does require even less changes to the device / spec.

This is exactly what I meant.

Ok, sorry for the misunderstanding.

Another drawback of that is the long time a device can spend to reconfigure its memory maps. taking that into account, does it make sense to send a new version with the bit 1 on enable?

Thanks!

Quoted 9 lines
Quoted 8 lines
Quoted 4 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

Quoted 82 lines

This is max, not how many there are in practice.

Quoted 1 line

Is it even valid to not enable some of the queues?

Yes and linux will do that if max_virtqueue_pairs > #CPUs.

Quoted 75 lines

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Quoted 2 lines

Maybe we could use e.g. bit 1 in queue_enable to signal that?

I'm totally ok to go in that direction.

Thanks!

Quoted 65 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

content.tex | 15 +++++++++++++--
transport-pci.tex | 4 ++++
2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#40 lore raw
Quoted 102 lines · Eugenio Perez Martin
Quoted 101 lines · Michael S. Tsirkin
Quoted 76 lines · Eugenio Perez Martin
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo
Quoted 27 lines · Eugenio Perez Martin

On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <[email protected]> wrote:

On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently
devices tend to deduce how many queues are needed by looking
at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then
doing a reset right after DRIVER_OK? You can even allocate
memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with
no need for spec changes.

The problem is that the rx queues can start receiving, as the guest
already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

The idea is interesting, but we lose the net configuration in a device reset, so we need to send it again.

Ring reset, not device reset.

Quoted 11 lines

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in vIOMMU case, we can send all the CVQ commands before allowing the guest to modify mappings. The guest has no way to write on that page through the device as, well, dataplane is not initialized. It's the way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

I don't like the complexity of it but I like it does require even less changes to the device / spec.

Quoted 7 lines
Quoted 2 lines

This is exactly what I meant.

Ok, sorry for the misunderstanding.

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take much more with this.

Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Quoted 77 lines
Quoted 77 lines
Quoted 8 lines
Quoted 4 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

This is max, not how many there are in practice.

Quoted 1 line

Is it even valid to not enable some of the queues?

Yes and linux will do that if max_virtqueue_pairs > #CPUs.

Quoted 62 lines

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Quoted 2 lines

Maybe we could use e.g. bit 1 in queue_enable to signal that?

I'm totally ok to go in that direction.

Thanks!

Quoted 52 lines
Quoted 52 lines
Quoted 52 lines
Quoted 52 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset,
write 1 just to CVQ, configure the device, and then write 1 to all
dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

content.tex | 15 +++++++++++++--
transport-pci.tex | 4 ++++
2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf

Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#41 lore raw
Quoted 107 lines · Michael S. Tsirkin
Quoted 105 lines · Eugenio Perez Martin
Quoted 103 lines · Michael S. Tsirkin
Quoted 76 lines · Eugenio Perez Martin
Quoted 75 lines · Michael S. Tsirkin
Quoted 73 lines · Eugenio Perez Martin
Quoted 54 lines · Michael S. Tsirkin
Quoted 52 lines · Jason Wang
Quoted 50 lines · Michael S. Tsirkin
Quoted 37 lines · Eugenio Perez Martin
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eugenio Perez Martin
Quoted 32 lines · Xuan Zhuo

On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <[email protected]> wrote:
On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <[email protected]> wrote:

On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently
devices tend to deduce how many queues are needed by looking
at the state at DRIVER_OK time.

Question: what is wrong with enabling queues initially and then
doing a reset right after DRIVER_OK? You can even allocate
memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with
no need for spec changes.

The problem is that the rx queues can start receiving, as the guest
already has buffers there.

Can we reset the vq before filling buffers to it?

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

Quoted 6 lines
Quoted 1 line

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

From the device POV, yes, the CVQ vring is in hypervisor memory, not

in the guest's one. That allows the hypervisor to send the CVQ commands to restore the device state in the destination, without the guest intervention.

Data vqs, on the other hand, are passthrough. The device talks directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then translating the commands in the way that suits better the vdpa device, using its vendor vdpa driver in the host. Other methods like PASID are possible too.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in hypervisor memory initially. Later reset and map to guest.

The idea is interesting, but we lose the net configuration in a device reset, so we need to send it again.

Ring reset, not device reset.

Quoted 11 lines

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in vIOMMU case, we can send all the CVQ commands before allowing the guest to modify mappings. The guest has no way to write on that page through the device as, well, dataplane is not initialized. It's the way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

I don't like the complexity of it but I like it does require even less changes to the device / spec.

This is exactly what I meant.

Quoted 8 lines
Quoted 4 lines

Ok, sorry for the misunderstanding.

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Quoted 99 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Quoted 91 lines
Quoted 91 lines
Quoted 8 lines
Quoted 4 lines

If that does not work, then I am not sure this proposal is enough since I think devices want to have a specific point in time where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk also has num_queues.

This is max, not how many there are in practice.

Quoted 1 line

Is it even valid to not enable some of the queues?

Yes and linux will do that if max_virtqueue_pairs > #CPUs.

Quoted 75 lines

I've always felt that queue_enable has been redundant before queue_reset for this reason actually.

Quoted 2 lines

Maybe we could use e.g. bit 1 in queue_enable to signal that?

I'm totally ok to go in that direction.

Thanks!

Quoted 65 lines
Quoted 64 lines
Quoted 64 lines
Quoted 63 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 55 lines

How do you do this - they are DMA from same VF no?

Quoted 51 lines
Quoted 51 lines
Quoted 50 lines
Quoted 10 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset,
write 1 just to CVQ, configure the device, and then write 1 to all
dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 35 lines

Thanks.

Thanks!

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

content.tex | 15 +++++++++++++--
transport-pci.tex | 4 ++++
2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#42 lore raw
Quoted 9 lines · Eugenio Perez Martin
Quoted 6 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#43 lore raw
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

No?

Quoted 2 lines
Signature · 1 line
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#44 lore raw
Quoted 25 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

Quoted 9 lines

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 13 lines
Quoted 4 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#45 lore raw
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

Quoted 13 lines
Quoted 13 lines
Quoted 4 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#46 lore raw
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#47 lore raw
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#48 lore raw
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#49 lore raw
Quoted 77 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

Quoted 4 lines

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#50 lore raw
Quoted 85 lines · Michael S. Tsirkin
Quoted 77 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

Quoted 4 lines

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 6 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#51 lore raw
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

Quoted 4 lines
Quoted 1 line

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 38 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 27 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#52 lore raw
Quoted 95 lines · Jason Wang
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines
Quoted 2 lines

Another drawback of that is the long time a device can spend to
reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 4 lines

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

This kind of guarantee however sounds generally useful to control which queues are accessed when.

This is only needed after restore, which is ATM a vdpa thing, so can just be a vdpa flag, but I'm ok with it as a virtio flag too.

Quoted 47 lines
Quoted 4 lines
Quoted 1 line

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 38 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 27 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#53 lore raw
Quoted 102 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin
Quoted 3 lines

Another drawback of that is the long time a device can spend to
reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines
Quoted 3 lines

The map is already used for cvq though, I don't see why it would take
much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines
Quoted 6 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 4 lines

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

Quoted 6 lines

This kind of guarantee however sounds generally useful to control which queues are accessed when.

This is only needed after restore, which is ATM a vdpa thing, so can just be a vdpa flag, but I'm ok with it as a virtio flag too.

Great. Having a virtio flag may help to reduce the extra and duplicated effort spent in vdpa.

Thanks

Quoted 35 lines
Quoted 35 lines
Quoted 5 lines
Quoted 2 lines

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 25 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines

List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#54 lore raw
Quoted 107 lines · Jason Wang
Quoted 105 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 4 lines · Michael S. Tsirkin
Quoted 3 lines · Eugenio Perez Martin

Another drawback of that is the long time a device can spend to
reconfigure its memory maps.

Quoted 15 lines
Quoted 6 lines

The map is already used for cvq though, I don't see why it would take
much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or
platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

Quoted 6 lines

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 4 lines

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

This kind of guarantee however sounds generally useful to control which queues are accessed when.

Quoted 7 lines
Quoted 3 lines

This is only needed after restore, which is ATM a vdpa thing, so can just be a vdpa flag, but I'm ok with it as a virtio flag too.

Great. Having a virtio flag may help to reduce the extra and duplicated effort spent in vdpa.

So the flag should be something in the line of:
VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after the first kick. Before that kick, it will not process any request.

?

Thanks!

Quoted 52 lines

Thanks

Quoted 49 lines
Quoted 48 lines
Quoted 5 lines
Quoted 2 lines

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 38 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 27 lines
Quoted 26 lines
Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#55 lore raw
Quoted 116 lines · Eugenio Perez Martin
Quoted 107 lines · Jason Wang
Quoted 105 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin
Quoted 19 lines · Michael S. Tsirkin

On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
Another drawback of that is the long time a device can spend to
reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take
much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or
platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 4 lines

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

This kind of guarantee however sounds generally useful to control which queues are accessed when.

Quoted 7 lines
Quoted 3 lines

This is only needed after restore, which is ATM a vdpa thing, so can just be a vdpa flag, but I'm ok with it as a virtio flag too.

Great. Having a virtio flag may help to reduce the extra and duplicated effort spent in vdpa.

Quoted 7 lines

So the flag should be something in the line of:
VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after the first kick. Before that kick, it will not process any request.

?

Thanks!

That's not bad. I am not sure adding this to spec is worth
the effort but at least it's kind of reasonable.
Are there uses such as hardening where something like this could be useful?

Quoted 37 lines
Quoted 37 lines

Thanks

Quoted 35 lines
Quoted 35 lines
Quoted 5 lines
Quoted 2 lines

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 25 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 6 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists

Quoted 9 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#56 lore raw
Quoted 122 lines · Michael S. Tsirkin
Quoted 120 lines · Eugenio Perez Martin
Quoted 107 lines · Jason Wang
Quoted 105 lines · Michael S. Tsirkin
Quoted 95 lines · Jason Wang
Quoted 87 lines · Eugenio Perez Martin
Quoted 85 lines · Michael S. Tsirkin
Quoted 81 lines · Eugenio Perez Martin
Quoted 67 lines · Michael S. Tsirkin
Quoted 57 lines · Eugenio Perez Martin
Quoted 43 lines · Michael S. Tsirkin
Quoted 33 lines · Eugenio Perez Martin
Quoted 32 lines · Michael S. Tsirkin

On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
Another drawback of that is the long time a device can spend to
reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take
much more with this.

In the case of vDPA HW, CVQ maps may not need to reach the device's or
platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

Current hardware won't support the extra bit though.

Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

Quoted 5 lines

Well then Jason said:
we should make svq work without RING_RESET
but now apparently we went full circle and propose a feature bit after all?

My understanding is that RING_RESET covers more than just delaying the enabling of net data vqs, like to change the address or properties of vqs after DRIVER_OK. While the late enabling may be easy to implement in current hw, vq property changes may be more challenging. There are current HW that only support late enabling but not reconfigure the queues. Ideally, just the subset of late enabling is needed.

Quoted 7 lines

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET. If this feature is negotiated, all vqs start as if the vqs has been reset before DRIVER_OK. This feature does not allow resetting a queue once it is live.

Thanks!

Quoted 8 lines

Maybe. No idea - I am not a hardware designer. Can some hardware vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case. And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in question are not enabled packets will be dropped.
So - fix some setups, break others?

I don't think it breaks, they are already dropped before setting DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in the future, the device is supposed to drop packets meanwhile anyway, isn't it?

Quoted 5 lines
Quoted 5 lines

I just thought of an idea. Devices generally don't poke at rings by themselves. How about at init time we cause kicks on data queues to vmexit? Then on 1st kick we map hardware so following ones will go straight to hardware. This will make sure cvq is processed first. Hmm?

Quoted 4 lines

This is not the case at least for vhost-net. It doesn't need to be kicked in order to work. And it's an example that we need clearly define the behaviour instead of depending on any existing implementation details.

This kind of guarantee however sounds generally useful to control which queues are accessed when.

Quoted 7 lines
Quoted 3 lines

This is only needed after restore, which is ATM a vdpa thing, so can just be a vdpa flag, but I'm ok with it as a virtio flag too.

Great. Having a virtio flag may help to reduce the extra and duplicated effort spent in vdpa.

So the flag should be something in the line of:
VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after the first kick. Before that kick, it will not process any request.

Quoted 7 lines
Quoted 3 lines

?

Thanks!

That's not bad. I am not sure adding this to spec is worth
the effort but at least it's kind of reasonable.
Are there uses such as hardening where something like this could be useful?

I think so, the driver can simply ignore any device notifications before the first kick. But it doesn't solve all the problems, we still need to take care about the device without this feature.

Thanks

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines

Thanks

Quoted 35 lines
Quoted 35 lines
Quoted 5 lines
Quoted 2 lines

That would be possible, but nothing prevents the hw to start processing requests even without that first kick, isn't it?

Yes.

Thanks

Quoted 25 lines

I think that was discussed also in the past, as previous qemu patches gave a spurious kick to start queues just in case. I think the conclusion was that at least the vDPA device will start processing requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

--
MST

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#57 lore raw
In reply to Eugenio Perez Martin
Quoted 11 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Tuesday, June 13, 2023 6:13 AM

Quoted 7 lines

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

Quoted 7 lines
Quoted 2 lines

No?

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at https://lists.oasis-open.org/archives/virtio-
dev/202305/msg00329.html
.

Before DRIVER_OK stage, queue_enable is already available so why to use queue reset?
They have duplicate functionality, hence there is no motivation to support queue_reset before DRIVER_OK stage?

It is not explicitly mentioned in the spec that one can setup the queue using queue reset instead of queue enable. As Jason mentioned it is implementation specific, one device supports it and one doesn't. Hence, it will break on those devices which doesnt support it.

Therefore, I would like to add it to the spec.
"queue_reset register MUST be accessed by the driver only after device has reached the DRIVER_OK stage."

Quoted 25 lines
Quoted 25 lines
Quoted 5 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#58 lore raw
Quoted 14 lines · Parav Pandit
Quoted 14 lines

From: Eugenio Perez Martin <[email protected]>
Sent: Tuesday, June 13, 2023 6:13 AM

Quoted 9 lines

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues

No?

Quoted 7 lines
Quoted 4 lines

If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for that in the past at https://lists.oasis-open.org/archives/virtio-
dev/202305/msg00329.html
.

Before DRIVER_OK stage, queue_enable is already available so why to use queue reset?
They have duplicate functionality, hence there is no motivation to support queue_reset before DRIVER_OK stage?

I do not get how. Let's make sure we are talking about the same thing. This is what I am saying:

Start operating:

- program queue address, size
- queue_enable

Now we change our mind about queue size or address. So:

- queue_reset
- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK

I don't see how it's possible with just queue_enable.

Quoted 5 lines

It is not explicitly mentioned in the spec that one can setup the queue using queue reset instead of queue enable. As Jason mentioned it is implementation specific, one device supports it and one doesn't. Hence, it will break on those devices which doesnt support it.

Setup? No, and spec explicitly says to setup one has to use queue_enable. But this is not what we discussed with Stefano here.

Quoted 2 lines

Therefore, I would like to add it to the spec.
"queue_reset register MUST be accessed by the driver only after device has reached the DRIVER_OK stage."

I don't think we can add MUST requirements after the proposal is in the released spec. If device has stricter requirements than the spec then it's not compliant.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 5 lines

--
MST

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines

Subscribe: [email protected]
Unsubscribe: [email protected] List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License:
https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines:
https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#59 lore raw
Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 13, 2023 3:55 PM

I do not get how. Let's make sure we are talking about the same thing. This is what I am saying:

Quoted 8 lines

Start operating:

- program queue address, size
- queue_enable

Now we change our mind about queue size or address. So:

- queue_reset

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given Q before DRIVER_OK.

Quoted 12 lines

- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK

I don't see how it's possible with just queue_enable.

Quoted 2 lines

It is not explicitly mentioned in the spec that one can setup the queue using

Quoted 9 lines

queue reset instead of queue enable.

Quoted 1 line

As Jason mentioned it is implementation specific, one device supports it and

one doesn't.

Quoted 2 lines

Hence, it will break on those devices which doesnt support it.

Setup? No, and spec explicitly says to setup one has to use queue_enable. But this is not what we discussed with Stefano here.

The above sequence you described is not well documented.

Quoted 8 lines
Quoted 2 lines

Therefore, I would like to add it to the spec.
"queue_reset register MUST be accessed by the driver only after device has

reached the DRIVER_OK stage."

I don't think we can add MUST requirements after the proposal is in the released spec. If device has stricter requirements than the spec then it's not compliant.

Lets resolve above question first about how many times one can use queue_enable before DRIVER_OK for a specific VQ. Without this either way driver and device combination is broken when queue_reset is used before DRIVER_OK.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#60 lore raw
Quoted 9 lines

From: [email protected] <[email protected] open.org> On Behalf Of Parav Pandit
Sent: Tuesday, June 13, 2023 5:09 PM

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given Q before DRIVER_OK.

I remember now and I found the wording which was the source of adding queue_reset register.

Spec: "The driver MUST NOT write a 0 to queue_enable."

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#61 lore raw
In reply to Parav Pandit
Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 13, 2023 3:55 PM

I do not get how. Let's make sure we are talking about the same thing. This is what I am saying:

Start operating:

- program queue address, size
- queue_enable

Quoted 6 lines
Quoted 3 lines

Now we change our mind about queue size or address. So:

- queue_reset

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given Q before DRIVER_OK.

I see you found it.

Quoted 14 lines
Quoted 14 lines

- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK

I don't see how it's possible with just queue_enable.

Quoted 2 lines

It is not explicitly mentioned in the spec that one can setup the queue using

queue reset instead of queue enable.

Quoted 1 line

As Jason mentioned it is implementation specific, one device supports it and

Quoted 8 lines
Quoted 7 lines

one doesn't.

Quoted 2 lines

Hence, it will break on those devices which doesnt support it.

Setup? No, and spec explicitly says to setup one has to use queue_enable. But this is not what we discussed with Stefano here.

The above sequence you described is not well documented.

It's the standard sequence for queue reset, is it not?

For example:

Virtqueue reset is divided into two parts. The driver first resets a queue and can afterwards optionally re-enable it.

it seems clear that queue has to be enabled before being reset.

Quoted 10 lines
Quoted 8 lines
Quoted 2 lines

Therefore, I would like to add it to the spec.
"queue_reset register MUST be accessed by the driver only after device has

reached the DRIVER_OK stage."

I don't think we can add MUST requirements after the proposal is in the released spec. If device has stricter requirements than the spec then it's not compliant.

Lets resolve above question first about how many times one can use queue_enable before DRIVER_OK for a specific VQ. Without this either way driver and device combination is broken when queue_reset is used before DRIVER_OK.

I'm not sure what's broken.

If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to \field{queue_reset} to reset the queue, the driver MUST NOT consider queue reset to be complete until it reads back 0 in \field{queue_reset}. The driver MAY re-enable the queue by writing 1 to \field{queue_enable} after ensuring that other virtqueue fields have been set up correctly. The driver MAY set driver-writeable queue configuration values to different values than those that were used before the queue reset.

So the limitation is that it happens after FEATURES_OK.

Signature · 18 lines
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
#62 lore raw
Quoted 38 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 13, 2023 5:48 PM

Quoted 2 lines

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given

Q before DRIVER_OK.

I see you found it.

Quoted 16 lines
Quoted 16 lines

- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK

I don't see how it's possible with just queue_enable.

Quoted 3 lines

It is not explicitly mentioned in the spec that one can setup the queue using

queue reset instead of queue enable.

Quoted 2 lines

As Jason mentioned it is implementation specific, one device supports it and

Quoted 8 lines
Quoted 7 lines

one doesn't.

Quoted 2 lines

Hence, it will break on those devices which doesnt support it.

Setup? No, and spec explicitly says to setup one has to use queue_enable. But this is not what we discussed with Stefano here.

The above sequence you described is not well documented.

It's the standard sequence for queue reset, is it not?

Quoted 9 lines

For example:

Virtqueue reset is divided into two parts. The driver first resets a queue
and
can afterwards optionally re-enable it.

it seems clear that queue has to be enabled before being reset.

Yes, it is clear.

Quoted 2 lines

I'm not sure what's broken.

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the fix.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#63 lore raw
Quoted 40 lines · Parav Pandit
Quoted 40 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, June 13, 2023 5:48 PM

Quoted 2 lines

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given

Q before DRIVER_OK.

I see you found it.

Quoted 15 lines
Quoted 15 lines

- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK

I don't see how it's possible with just queue_enable.

Quoted 2 lines

It is not explicitly mentioned in the spec that one can setup the queue using

queue reset instead of queue enable.

Quoted 2 lines

As Jason mentioned it is implementation specific, one device supports it and

Quoted 7 lines
Quoted 6 lines

one doesn't.

Quoted 2 lines

Hence, it will break on those devices which doesnt support it.

Setup? No, and spec explicitly says to setup one has to use queue_enable. But this is not what we discussed with Stefano here.

The above sequence you described is not well documented.

It's the standard sequence for queue reset, is it not?

For example:

Virtqueue reset is divided into two parts. The driver first resets a queue
and
can afterwards optionally re-enable it.

Quoted 8 lines
Quoted 2 lines

it seems clear that queue has to be enabled before being reset.

Yes, it is clear.

Quoted 2 lines

I'm not sure what's broken.

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the fix.

I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue
- after DRIVER_OK, the driver has to reset the queue to re-config the queue. Because a queue is considered alive when both DRIVER_OK && queue_enable, and resetting a queue also set queue_enable to zero.

At the end of the migration, hypervisor should reset the queues and config the queues with the guest configurations and enable the queues, right?

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#64 lore raw
Quoted 5 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 12:27 AM

Quoted 8 lines
Quoted 2 lines

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the

fix.
I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write queue_enable = 0. This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

Quoted 6 lines

- after DRIVER_OK, the driver has to reset the queue to re-config the queue. Because a queue is considered alive when both DRIVER_OK && queue_enable, and resetting a queue also set queue_enable to zero.

At the end of the migration, hypervisor should reset the queues and config the queues with the guest configurations and enable the queues, right?

The migration for non vdpa world would not require any of the queue enable/disable flow because device is migrated (paused) with its current state. And it can resume from where it left off.

#65 lore raw
Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 12:27 AM

Quoted 2 lines

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the

fix.
I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue

Quoted 3 lines

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write queue_enable = 0. This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

The spec requires the driver config all other fields before set queue_enable. I am not sure whether there are real occurring cases that re-config the vq fields between queue_enable and DRIVER_OK in the initialization.

But I agree maybe we should allow set queue_enable = 0.

This reset may happen after DRIVER_OK.

Quoted 9 lines
Quoted 7 lines

- after DRIVER_OK, the driver has to reset the queue to re-config the queue. Because a queue is considered alive when both DRIVER_OK && queue_enable, and resetting a queue also set queue_enable to zero.

At the end of the migration, hypervisor should reset the queues and config the queues with the guest configurations and enable the queues, right?

The migration for non vdpa world would not require any of the queue enable/disable flow because device is migrated (paused) with its current state. And it can resume from where it left off.

For non-vdpa cases, it still require to migrate the states like MQ which is controlled by cvq.

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#66 lore raw
Quoted 18 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 2:11 AM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 12:27 AM

Quoted 3 lines

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the

fix.
I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue

Quoted 8 lines
Quoted 1 line

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write

queue_enable = 0.

Quoted 2 lines

This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

The spec requires the driver config all other fields before set queue_enable. I am not sure whether there are real occurring cases that re-config the vq fields between queue_enable and DRIVER_OK in the initialization.

Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.

Quoted 2 lines

But I agree maybe we should allow set queue_enable = 0.

This functionality of queue_enable = 0, is supported using queue_reset register.

Quoted 1 line

This reset may happen after DRIVER_OK.

Yes, the queue_reset after DRIVER_OK is not ambiguous.
Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.

For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.

#67 lore raw
Quoted 26 lines · Parav Pandit
Quoted 14 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 2:11 AM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 12:27 AM

Quoted 3 lines

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the

fix.
I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue

Quoted 8 lines
Quoted 1 line

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write

queue_enable = 0.

Quoted 2 lines

This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

The spec requires the driver config all other fields before set queue_enable. I am not sure whether there are real occurring cases that re-config the vq fields between queue_enable and DRIVER_OK in the initialization.

Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.

Quoted 2 lines

But I agree maybe we should allow set queue_enable = 0.

Quoted 7 lines

This functionality of queue_enable = 0, is supported using queue_reset register.

Quoted 1 line

This reset may happen after DRIVER_OK.

Yes, the queue_reset after DRIVER_OK is not ambiguous.
Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.

For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.

I guess the intention may be:
Step 1: the hypervisor owns all queues, it enables all queues and cvq, set queue fields with phony host values. Then set DRIVER_OK, config the device and send commands through cvq.
But don't do link announce. So the device would not consume the descriptors and external network will not send any data to it.

Step 2: Reset the queues, set the fields of the queues with guest config, selectively enable the queues, so the device works for the guest.

Does this sounds good?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#68 lore raw
Quoted 36 lines · Lingshan
Quoted 32 lines · Parav Pandit
Quoted 14 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 2:11 AM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Zhu, Lingshan <[email protected]>
Sent: Wednesday, June 14, 2023 12:27 AM

Quoted 3 lines

If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live", Such device needs the

fix.
I think queue_reset just resets the queue and its fields including the queue_enable. DRIVER_OK is not a must for performing a queue_reset, My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't need a reset_queue

Quoted 8 lines
Quoted 1 line

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write

queue_enable = 0.

Quoted 2 lines

This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

The spec requires the driver config all other fields before set queue_enable. I am not sure whether there are real occurring cases that re-config the vq fields between queue_enable and DRIVER_OK in the initialization.

Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.

Quoted 2 lines

But I agree maybe we should allow set queue_enable = 0.

This functionality of queue_enable = 0, is supported using queue_reset register.

Quoted 1 line

This reset may happen after DRIVER_OK.

Yes, the queue_reset after DRIVER_OK is not ambiguous.
Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.

Quoted 9 lines
Quoted 1 line

For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.

I guess the intention may be:
Step 1: the hypervisor owns all queues, it enables all queues and cvq, set queue fields with phony host values.
Then set DRIVER_OK, config the device and send commands through cvq. But don't do link announce. So the device would not consume the descriptors and external network will not send
any data to it.

Even without a link announcement, the device can process descriptors. Some examples are receiving broadcast packets or just sending any packet (which effectively acts as a link announcement for the network topology).

Thanks!

Quoted 8 lines

Step 2: Reset the queues, set the fields of the queues with guest config, selectively enable the queues,
so the device works for the guest.

Does this sounds good?

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#69 lore raw
In reply to Michael S. Tsirkin
Quoted 51 lines · Michael S. Tsirkin
Quoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Eugenio Perez Martin
Quoted 33 lines · Michael S. Tsirkin
Quoted 31 lines · Eugenio Perez Martin
Quoted 30 lines · Xuan Zhuo
Quoted 27 lines · Eugenio Perez Martin
Quoted 16 lines · Michael S. Tsirkin
Quoted 9 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently devices tend to deduce how many queues are needed by looking at the state at DRIVER_OK time.

Quoted 7 lines

Question: what is wrong with enabling queues initially and then doing a reset right after DRIVER_OK? You can even allocate memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with no need for spec changes.

The problem is that the rx queues can start receiving, as the guest already has buffers there.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can we reset the vq before filling buffers to it?

They are passthrough from the guest so there is a window where the device can process rx descriptors.

Not if there are no descriptors there.

But the migration is driven by the hypervisor, and it cannot control that. The guest will likely have rx descriptors available.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?

Yes.

Thanks

Quoted 51 lines
Quoted 51 lines

So in this case if RX queues are enabled at the same time, the device may try to queue packets to queue 0.

Thanks

Quoted 44 lines

How do you do this - they are DMA from same VF no?

Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
Quoted 10 lines
Quoted 7 lines

Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset, write 1 just to CVQ, configure the device, and then write 1 to all dataplane vqs?

write to what?

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

Quoted 26 lines

Thanks.

Quoted 16 lines

Thanks!

Quoted 12 lines
Quoted 11 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#70 lore raw
In reply to Eugenio Pérez
Quoted 11 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Adding Yan to the thread, who mentioned that he is looking for something similar in the virtio-networking upstream meeting.

Quoted 20 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#71 lore raw
Quoted 14 lines · Eugenio Perez Martin
Quoted 11 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Quoted 2 lines

Adding Yan to the thread, who mentioned that he is looking for something similar in the virtio-networking upstream meeting.

Would there be a V2?

Thanks

Quoted 21 lines
Quoted 21 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#72 lore raw
Quoted 15 lines · Jason Wang
Quoted 14 lines · Eugenio Perez Martin
Quoted 11 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Quoted 6 lines
Quoted 2 lines

Adding Yan to the thread, who mentioned that he is looking for something similar in the virtio-networking upstream meeting.

Would there be a V2?

Thanks

I'm not sure we need to bother at this point. In linux we seem to have agreed to use a mix of RING_RESET and a vdpa specific thing. Non-vdpa solutions don't seem to need this. Right Parav?

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#73 lore raw
Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 10, 2023 1:49 AM

I'm not sure we need to bother at this point. In linux we seem to have agreed to use a mix of RING_RESET and a vdpa specific thing. Non-vdpa solutions don't seem to need this. Right Parav?

Couldn’t follow all the emails, but yes, I agree with Michael that F_RING_RESET seems enough which is serving the purpose of dynamically adding the q.

#74 lore raw
Quoted 11 lines · Parav Pandit
Quoted 9 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 10, 2023 1:49 AM

I'm not sure we need to bother at this point. In linux we seem to have agreed to use a mix of RING_RESET and a vdpa specific thing. Non-vdpa solutions don't seem to need this. Right Parav?

Couldn’t follow all the emails, but yes, I agree with Michael that F_RING_RESET seems enough which is serving the purpose of dynamically adding the q.

Not adding but enabling :)

For example, to be used for migration.

Thanks

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

#75 lore raw
In reply to Michael S. Tsirkin
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · Jason Wang
Quoted 16 lines · Eugenio Perez Martin
Quoted 11 lines · Eugenio Pérez

This series allows the driver to start the device (as set DRIVER_OK) with only some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Adding Yan to the thread, who mentioned that he is looking for something similar in the virtio-networking upstream meeting.

Quoted 7 lines
Quoted 3 lines

Would there be a V2?

Thanks

I'm not sure we need to bother at this point. In linux we seem to have agreed to use a mix of RING_RESET and a vdpa specific thing.

The question is:

1) This proposal is pretty compact and self contained
2) Some vDPA device and vhost devices already behave like this
3) Using RING_RESET requires a lot of codes and tricks

This makes me wonder if this proposal is still the better way to go.

Thanks

Quoted 36 lines

Non-vdpa solutions don't seem to
need this.
Right Parav?

Quoted 31 lines
Quoted 31 lines
Quoted 30 lines

Eugenio Pérez (2):
virtio: introduce selective queue enabling
virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

--
2.31.1

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting.

Subscribe: [email protected]
Unsubscribe: [email protected]
List help: [email protected]
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/