vhost-user.rst: Clarify enabling/disabling vrings
PATCH 1/1 vhost-user.rst: Clarify enabling/disabling vrings
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when VHOST_USER_F_PROTOCOL_FEATURES is negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.
At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every SET_FEATURES call would disable all rings, effectively halting the device. This is problematic because the VHOST_F_LOG_ALL feature is also set or cleared this way, which happens during migration. Doing so should not halt the device.
Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent SET_FEATURES call that does not set VHOST_USER_F_PROTOCOL_FEATURES will enable all of them. Here, SET_FEATURES will never disable any ring.
This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
docs/interop/vhost-user.rst +17 -6
@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled. + +If ``VHOST_USER_SET_FEATURES`` does not negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring, when started, will be +enabled immediately. + +If ``VHOST_USER_SET_FEATURES`` does negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring will remain in the disabled +state until ``VHOST_USER_SET_VRING_ENABLE`` enables it with parameter 1. + +Back-end implementations that support ``VHOST_USER_F_PROTOCOL_FEATURES`` +should implement this by initializing each ring in a disabled state, and +enabling them when ``VHOST_USER_SET_FEATURES`` is used without +negotiating ``VHOST_USER_F_PROTOCOL_FEATURES``. Other than that, rings +should only be enabled and disabled through +``VHOST_USER_SET_VRING_ENABLE``. While processing the rings (whether they are enabled or not), the back-end must support changing some configuration aspects on the fly.
Signature · 1 line
2.41.0
Quoted 13 lines · Hanna Czenczek
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when
VHOST_USER_F_PROTOCOL_FEATURESis negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every
SET_FEATUREScall would disable all rings, effectively halting the device. This is problematic because theVHOST_F_LOG_ALLfeature is also set or cleared this way, which happens during migration. Doing so should not halt the device.
Quoted 4 lines
Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent
SET_FEATUREScall that does not setVHOST_USER_F_PROTOCOL_FEATURESwill enable all of them. Here,SET_FEATURESwill never disable any ring.
Huh. I don't know why we don't set VHOST_USER_F_PROTOCOL_FEATURES on all calls though. I think it's a bug. Let's fix that first of all? Then we can still document behaviour of existing buggy QEMU.
Quoted 47 lines
This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
Signed-off-by Hanna Czenczek <[email protected]>--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)docs/interop/vhost-user.rst +17
-6@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled. + +If ``VHOST_USER_SET_FEATURES`` does not negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring, when started, will be +enabled immediately. + +If ``VHOST_USER_SET_FEATURES`` does negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring will remain in the disabled +state until ``VHOST_USER_SET_VRING_ENABLE`` enables it with parameter 1. + +Back-end implementations that support ``VHOST_USER_F_PROTOCOL_FEATURES`` +should implement this by initializing each ring in a disabled state, and +enabling them when ``VHOST_USER_SET_FEATURES`` is used without +negotiating ``VHOST_USER_F_PROTOCOL_FEATURES``. Other than that, rings +should only be enabled and disabled through +``VHOST_USER_SET_VRING_ENABLE``. While processing the rings (whether they are enabled or not), the back-end must support changing some configuration aspects on the fly.Signature · 1 line
2.41.0
Quoted 14 lines · Michael S. Tsirkin
Quoted 13 lines · Hanna Czenczek
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when
VHOST_USER_F_PROTOCOL_FEATURESis negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every
SET_FEATUREScall would disable all rings, effectively halting the device. This is problematic because theVHOST_F_LOG_ALLfeature is also set or cleared this way, which happens during migration. Doing so should not halt the device.
Quoted 7 lines
Quoted 4 lines
Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent
SET_FEATUREScall that does not setVHOST_USER_F_PROTOCOL_FEATURESwill enable all of them. Here,SET_FEATURESwill never disable any ring.Huh. I don't know why we don't set
VHOST_USER_F_PROTOCOL_FEATURESon all calls though. I think it's a bug. Let's fix that first of all? Then we can still document behaviour of existing buggy QEMU.
To my knowledge we (i.e. qemu) do. I think we’d only not set it if the back-end just doesn’t support it.
In the above paragraph, I just meant to describe how back-end implementations other than the Rust one behave (when they support F_PROTOCOL_FEATURES): They disable all vrings from the start. If SET_FEATURES is called without F_PROTOCOL_FEATURES (which qemu won’t do), they’ll enable them. But outside of SET_VRING_ENABLE, they’ll never disable them after initialization.
I.e. the case where a back-end supports F_PROTOCOL_FEATURES but the front-end doesn’t set it is just hypothetical, and not meant to describe the behavior of any current qemu version.
Hanna
Quoted 47 lines
Quoted 47 lines
This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
Signed-off-by Hanna Czenczek <[email protected]>--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)docs/interop/vhost-user.rst +17
-6@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled. + +If ``VHOST_USER_SET_FEATURES`` does not negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring, when started, will be +enabled immediately. + +If ``VHOST_USER_SET_FEATURES`` does negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring will remain in the disabled +state until ``VHOST_USER_SET_VRING_ENABLE`` enables it with parameter 1. + +Back-end implementations that support ``VHOST_USER_F_PROTOCOL_FEATURES`` +should implement this by initializing each ring in a disabled state, and +enabling them when ``VHOST_USER_SET_FEATURES`` is used without +negotiating ``VHOST_USER_F_PROTOCOL_FEATURES``. Other than that, rings +should only be enabled and disabled through +``VHOST_USER_SET_VRING_ENABLE``. While processing the rings (whether they are enabled or not), the back-end must support changing some configuration aspects on the fly.Signature · 1 line
2.41.0
Quoted 35 lines · Hanna Czenczek
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when
VHOST_USER_F_PROTOCOL_FEATURESis negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every
SET_FEATUREScall would disable all rings, effectively halting the device. This is problematic because theVHOST_F_LOG_ALLfeature is also set or cleared this way, which happens during migration. Doing so should not halt the device.Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent
SET_FEATUREScall that does not setVHOST_USER_F_PROTOCOL_FEATURESwill enable all of them. Here,SET_FEATURESwill never disable any ring.This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
Signed-off-by Hanna Czenczek <[email protected]>--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)docs/interop/vhost-user.rst +0
-0@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``.
docs/interop/vhost-user.rst and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. · line 387
Quoted 10 lines
docs/interop/vhost-user.rst +2
-6Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``. -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled.
What is the purpose of this statement? Rings cannot be used before feature negotiation (with the possible exception of legacy devices that allowed this to accomodate buggy drivers).
To me this statement complicates things and raises more questions than it answers.
docs/interop/vhost-user.rst and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. · line 391
Quoted 4 lines
docs/interop/vhost-user.rst +4
-0+ +If ``VHOST_USER_SET_FEATURES`` does not negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring, when started, will be +enabled immediately.
This sentence can be simplified a little:
"each ring, when started, will be enabled immediately" -> "rings are enabled immediately when started"
Quoted 17 lines
docs/interop/vhost-user.rst +11
-0+ +If ``VHOST_USER_SET_FEATURES`` does negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring will remain in the disabled +state until ``VHOST_USER_SET_VRING_ENABLE`` enables it with parameter 1. + +Back-end implementations that support ``VHOST_USER_F_PROTOCOL_FEATURES`` +should implement this by initializing each ring in a disabled state, and +enabling them when ``VHOST_USER_SET_FEATURES`` is used without +negotiating ``VHOST_USER_F_PROTOCOL_FEATURES``. Other than that, rings +should only be enabled and disabled through +``VHOST_USER_SET_VRING_ENABLE``. While processing the rings (whether they are enabled or not), the back-end must support changing some configuration aspects on the fly.Signature · 1 line
2.41.0
Quoted 38 lines · Stefan Hajnoczi
Quoted 37 lines · Hanna Czenczek
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when
VHOST_USER_F_PROTOCOL_FEATURESis negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every
SET_FEATUREScall would disable all rings, effectively halting the device. This is problematic because theVHOST_F_LOG_ALLfeature is also set or cleared this way, which happens during migration. Doing so should not halt the device.Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent
SET_FEATUREScall that does not setVHOST_USER_F_PROTOCOL_FEATURESwill enable all of them. Here,SET_FEATURESwill never disable any ring.This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
Signed-off-by Hanna Czenczek <[email protected]>--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)docs/interop/vhost-user.rst +0
-0@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.
Quoted 11 lines
docs/interop/vhost-user.rst and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. · line 387
Quoted 8 lines
docs/interop/vhost-user.rst +2
-6-If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled.What is the purpose of this statement? Rings cannot be used before feature negotiation (with the possible exception of legacy devices that allowed this to accomodate buggy drivers).
Perfect :)
Quoted 2 lines
To me this statement complicates things and raises more questions than it answers.
OK. The context for the statement is as follows: When the back-end supports F_PROTOCOL_FEATURES, it is supposed to initialize all vrings in a disabled state, so that when the flag is indeed negotiated, that will be the state they’re in. In contrast, older back-ends that don’t support that flag will initialize them in an enabled state (because they won’t have support for disabled vrings).
The statement was intended to make it clear that this difference in behavior is OK, and that the front-end must not rely on either of the two. Only after SET_FEATURES will and must the state be well-defined.
But if you find it just confusing because enabled/disabled has no meaning before a virt queue is started anyway, and they mustn’t be started before negotiating features, I’m happy to drop it without replacement.
Quoted 7 lines
docs/interop/vhost-user.rst and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. · line 391
Quoted 4 lines
docs/interop/vhost-user.rst +4
-0+ +If ``VHOST_USER_SET_FEATURES`` does not negotiate +``VHOST_USER_F_PROTOCOL_FEATURES``, each ring, when started, will be +enabled immediately.This sentence can be simplified a little:
"each ring, when started, will be enabled immediately" -> "rings are enabled immediately when started"
Sure!
Hanna
Quoted 64 lines · Hanna Czenczek
Quoted 38 lines · Stefan Hajnoczi
Quoted 37 lines · Hanna Czenczek
Currently, the vhost-user documentation says that rings are to be initialized in a disabled state when
VHOST_USER_F_PROTOCOL_FEATURESis negotiated. However, by the time of feature negotiation, all rings have already been initialized, so it is not entirely clear what this means.At least the vhost-user-backend Rust crate's implementation interpreted it to mean that whenever this feature is negotiated, all rings are to be put into a disabled state, which means that every
SET_FEATUREScall would disable all rings, effectively halting the device. This is problematic because theVHOST_F_LOG_ALLfeature is also set or cleared this way, which happens during migration. Doing so should not halt the device.Other implementations have interpreted this to mean that the device is to be initialized with all rings disabled, and a subsequent
SET_FEATUREScall that does not setVHOST_USER_F_PROTOCOL_FEATURESwill enable all of them. Here,SET_FEATURESwill never disable any ring.This other interpretation does not suffer the problem of unintentionally halting the device whenever features are set or cleared, so it seems better and more reasonable.
We should clarify this in the documentation.
Signed-off-by Hanna Czenczek <[email protected]>--- docs/interop/vhost-user.rst | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)docs/interop/vhost-user.rst +0
-0@@ -383,12 +383,23 @@ and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. Rings can be enabled or disabled by ``VHOST_USER_SET_VRING_ENABLE``.Quoted 11 lines
docs/interop/vhost-user.rst and stop ring upon receiving ``VHOST_USER_GET_VRING_BASE``. · line 387
Quoted 8 lines
docs/interop/vhost-user.rst +2
-6-If ``VHOST_USER_F_PROTOCOL_FEATURES`` has not been negotiated, the -ring starts directly in the enabled state. - -If ``VHOST_USER_F_PROTOCOL_FEATURES`` has been negotiated, the ring is -initialized in a disabled state and is enabled by -``VHOST_USER_SET_VRING_ENABLE`` with parameter 1. +Between initialization and the first ``VHOST_USER_SET_FEATURES`` call, it +is implementation-defined whether each ring is enabled or disabled.What is the purpose of this statement? Rings cannot be used before feature negotiation (with the possible exception of legacy devices that allowed this to accomodate buggy drivers).
Perfect :)
Quoted 2 lines
To me this statement complicates things and raises more questions than it answers.
OK. The context for the statement is as follows: When the back-end supports
F_PROTOCOL_FEATURES, it is supposed to initialize all vrings in a disabled state, so that when the flag is indeed negotiated, that will be the state they’re in. In contrast, older back-ends that don’t support that flag will initialize them in an enabled state (because they won’t have support for disabled vrings).
Quoted 8 lines
The statement was intended to make it clear that this difference in behavior is OK, and that the front-end must not rely on either of the two. Only after
SET_FEATURESwill and must the state be well-defined.But if you find it just confusing because enabled/disabled has no meaning before a virt queue is started anyway, and they mustn’t be started before negotiating features, I’m happy to drop it without replacement.
Yes, dropping this statement sounds good.
Stefan