Skip to contentSkip to conversation

vhost-user: update spec description

PATCH Superseded Michael S. Tsirkin 2 participants 3 messages Nov 15, 2015 – Nov 16, 2015

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vhost-user: update spec description

Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.

---
 docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 53 insertions(+), 9 deletions(-)
docs/specs/vhost-user.txt +53 -9
index 26dde2e..6a0ed8f 100644
@@ -87,6 +87,14 @@ Depending on the request type, payload can be:
    User address: a 64-bit user address
    mmap offset: 64-bit offset where region starts in the mapped memory
 
+* Log description
+   ---------------------------
+   | log size | log offset |
+   ---------------------------
+   log size: size of area used for logging
+   log offset: offset from start of supplied file descriptor
+       where logging starts (i.e. where guest address 0 would be logged)
+
 In QEMU the vhost-user message is implemented with the following struct:
 
 typedef struct VhostUserMsg {
@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features,
 a feature bit was dedicated for this purpose:
 #define VHOST_USER_F_PROTOCOL_FEATURES 30
 
+Starting and stopping rings
+----------------------
+Each ring is initialized in a stopped state, client must not process it until
+ring is enabled.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and
+stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters
+1 and 0 respoectively.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start
+ring processing upon receiving a kick (that is, detecting that file descriptor
+is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop
+ring processing upon receiving VHOST_USER_GET_VRING_BASE.
+
+While rings are running, client must support changing some configuration
+aspects on the fly.
+
 Multiple queue support
 ----------------------
 
@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark
 the dirty pages in a log. Once it complies to this logging, it may
 declare the VHOST_F_LOG_ALL vhost feature.
 
+To start/stop logging of data/used ring writes, server may send messages
+VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
+VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
+
 All the modifications to memory pointed by vring "descriptor" should
 be marked. Modifications to "used" vring should be marked if
-VHOST_VRING_F_LOG is part of ring's features.
+VHOST_VRING_F_LOG is part of ring's flags.
 
 Dirty pages are of size:
 #define VHOST_LOG_PAGE 0x1000
@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of
 VHOST_USER_SET_LOG_BASE message when the slave has
 VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
 
-The size of the log may be computed by using all the known guest
-addresses. The log covers from address 0 to the maximum of guest
+The size of the log is supplied as part of VhostUserMsg
+which should be large enough to cover all known guest
+addresses. Log starts at the supplied offset in the
+supplied file descriptor.
+The log covers from address 0 to the maximum of guest
 regions. In pseudo-code, to mark page at "addr" as dirty:
 
 page = addr / VHOST_LOG_PAGE
 log[page / 8] |= 1 << page % 8
 
+Where addr is the guest physical address.
+
 Use atomic operations, as the log may be concurrently manipulated.
 
+Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
+is set for this ring), log_guest_addr should be used to calculate the log
+offset: the write to first byte of the used ring is logged at this offset from
+log start. Also note that this value might be outside the legal guest physical
+address range (i.e. does not have to be covered by the VhostUserMemory table),
+but the bit offset of the last byte of the ring must fall within
+the size supplied by VhostUserLog.
+
 VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
 ancillary data, it may be used to inform the master that the log has
 been modified.
 
-Once the source has finished migration, VHOST_USER_RESET_OWNER message
-will be sent by the source. No further update must be done before the
-destination takes over with new regions & rings.
+Once the source has finished migration, rings will be stopped by
+the source. No further update must be done before rings are
+restarted.
 
 Protocol features
 -----------------
@@ -259,11 +301,11 @@ Message types
  * VHOST_USER_RESET_OWNER
 
       Id: 4
-      Equivalent ioctl: VHOST_RESET_OWNER
       Master payload: N/A
 
-      Issued when a new connection is about to be closed. The Master will no
-      longer own this connection (and will usually close it).
+      This is no longer used. Used to be sent to request stopping
+      all rings, but some clients interpreted it to also discard
+      connection state (this interpretation would lead to bugs).
 
  * VHOST_USER_SET_MEM_TABLE
 
@@ -388,6 +430,8 @@ Message types
       Master payload: vring state description
 
       Signal slave to enable or disable corresponding vring.
+      This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
+      has been negotiated.
 
  * VHOST_USER_SEND_RARP
 
Signature · 1 line
MST
#1 lore raw
Quoted 109 lines · Michael S. Tsirkin

Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.

---
 docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 53 insertions(+), 9 deletions(-)
docs/specs/vhost-user.txt +48 -6
index 26dde2e..6a0ed8f 100644
@@ -87,6 +87,14 @@ Depending on the request type, payload can be:
    User address: a 64-bit user address
    mmap offset: 64-bit offset where region starts in the mapped memory
 
+* Log description
+   ---------------------------
+   | log size | log offset |
+   ---------------------------
+   log size: size of area used for logging
+   log offset: offset from start of supplied file descriptor
+       where logging starts (i.e. where guest address 0 would be logged)
+
 In QEMU the vhost-user message is implemented with the following struct:
 
 typedef struct VhostUserMsg {
@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features,
 a feature bit was dedicated for this purpose:
 #define VHOST_USER_F_PROTOCOL_FEATURES 30
 
+Starting and stopping rings
+----------------------
+Each ring is initialized in a stopped state, client must not process it until
+ring is enabled.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and
+stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters
+1 and 0 respoectively.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start
+ring processing upon receiving a kick (that is, detecting that file descriptor
+is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop
+ring processing upon receiving VHOST_USER_GET_VRING_BASE.
+
+While rings are running, client must support changing some configuration
+aspects on the fly.
+
 Multiple queue support
 ----------------------
 
@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark
 the dirty pages in a log. Once it complies to this logging, it may
 declare the VHOST_F_LOG_ALL vhost feature.
 
+To start/stop logging of data/used ring writes, server may send messages
+VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
+VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
+
 All the modifications to memory pointed by vring "descriptor" should
 be marked. Modifications to "used" vring should be marked if
-VHOST_VRING_F_LOG is part of ring's features.
+VHOST_VRING_F_LOG is part of ring's flags.
 
 Dirty pages are of size:
 #define VHOST_LOG_PAGE 0x1000
@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of
 VHOST_USER_SET_LOG_BASE message when the slave has
 VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
 
-The size of the log may be computed by using all the known guest
-addresses. The log covers from address 0 to the maximum of guest
+The size of the log is supplied as part of VhostUserMsg
+which should be large enough to cover all known guest
+addresses. Log starts at the supplied offset in the
+supplied file descriptor.
+The log covers from address 0 to the maximum of guest
 regions. In pseudo-code, to mark page at "addr" as dirty:
 
 page = addr / VHOST_LOG_PAGE
 log[page / 8] |= 1 << page % 8
 
+Where addr is the guest physical address.
+
 Use atomic operations, as the log may be concurrently manipulated.
 
+Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
+is set for this ring), log_guest_addr should be used to calculate the log
+offset: the write to first byte of the used ring is logged at this offset from
+log start. Also note that this value might be outside the legal guest physical
+address range (i.e. does not have to be covered by the VhostUserMemory table),
+but the bit offset of the last byte of the ring must fall within
+the size supplied by VhostUserLog.
+
 VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
 ancillary data, it may be used to inform the master that the log has
 been modified.
 
-Once the source has finished migration, VHOST_USER_RESET_OWNER message
-will be sent by the source. No further update must be done before the
-destination takes over with new regions & rings.
+Once the source has finished migration, rings will be stopped by
+the source. No further update must be done before rings are
+restarted.
 
 Protocol features
 -----------------
@@ -259,11 +301,11 @@ Message types
  * VHOST_USER_RESET_OWNER

docs/specs/vhost-user.txt Message types · line 308

Quoted 10 lines
docs/specs/vhost-user.txt +3 -3
 
       Id: 4
-      Equivalent ioctl: VHOST_RESET_OWNER
       Master payload: N/A
 
-      Issued when a new connection is about to be closed. The Master will no
-      longer own this connection (and will usually close it).
+      This is no longer used. Used to be sent to request stopping
+      all rings, but some clients interpreted it to also discard
+      connection state (this interpretation would lead to bugs).

The new code uses VHOST_USER_RESET_DEVICE as the name for request Id 4. Also it is not clear whether it is recommended just to ignore the reset device request now?

Quoted 14 lines
docs/specs/vhost-user.txt +2 -0
  * VHOST_USER_SET_MEM_TABLE
 
@@ -388,6 +430,8 @@ Message types
       Master payload: vring state description
 
       Signal slave to enable or disable corresponding vring.
+      This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
+      has been negotiated.
 
  * VHOST_USER_SEND_RARP
 
Signature · 1 line
MST

--Victor

#2 lore raw
Quoted 115 lines · Victor Kaplansky
Quoted 114 lines · Michael S. Tsirkin

Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.

---
 docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 53 insertions(+), 9 deletions(-)
docs/specs/vhost-user.txt +48 -7
index 26dde2e..6a0ed8f 100644
@@ -87,6 +87,14 @@ Depending on the request type, payload can be:
    User address: a 64-bit user address
    mmap offset: 64-bit offset where region starts in the mapped memory
 
+* Log description
+   ---------------------------
+   | log size | log offset |
+   ---------------------------
+   log size: size of area used for logging
+   log offset: offset from start of supplied file descriptor
+       where logging starts (i.e. where guest address 0 would be logged)
+
 In QEMU the vhost-user message is implemented with the following struct:
 
 typedef struct VhostUserMsg {
@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features,
 a feature bit was dedicated for this purpose:
 #define VHOST_USER_F_PROTOCOL_FEATURES 30
 
+Starting and stopping rings
+----------------------
+Each ring is initialized in a stopped state, client must not process it until
+ring is enabled.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and
+stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters
+1 and 0 respoectively.
+
+If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start
+ring processing upon receiving a kick (that is, detecting that file descriptor
+is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop
+ring processing upon receiving VHOST_USER_GET_VRING_BASE.
+
+While rings are running, client must support changing some configuration
+aspects on the fly.
+
 Multiple queue support
 ----------------------
 
@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark
 the dirty pages in a log. Once it complies to this logging, it may
 declare the VHOST_F_LOG_ALL vhost feature.
 
+To start/stop logging of data/used ring writes, server may send messages
+VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with
+VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively.
+
 All the modifications to memory pointed by vring "descriptor" should
 be marked. Modifications to "used" vring should be marked if
-VHOST_VRING_F_LOG is part of ring's features.
+VHOST_VRING_F_LOG is part of ring's flags.
 
 Dirty pages are of size:
 #define VHOST_LOG_PAGE 0x1000
@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of
 VHOST_USER_SET_LOG_BASE message when the slave has
 VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature.
 
-The size of the log may be computed by using all the known guest
-addresses. The log covers from address 0 to the maximum of guest
+The size of the log is supplied as part of VhostUserMsg
+which should be large enough to cover all known guest
+addresses. Log starts at the supplied offset in the
+supplied file descriptor.
+The log covers from address 0 to the maximum of guest
 regions. In pseudo-code, to mark page at "addr" as dirty:
 
 page = addr / VHOST_LOG_PAGE
 log[page / 8] |= 1 << page % 8
 
+Where addr is the guest physical address.
+
 Use atomic operations, as the log may be concurrently manipulated.
 
+Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG
+is set for this ring), log_guest_addr should be used to calculate the log
+offset: the write to first byte of the used ring is logged at this offset from
+log start. Also note that this value might be outside the legal guest physical
+address range (i.e. does not have to be covered by the VhostUserMemory table),
+but the bit offset of the last byte of the ring must fall within
+the size supplied by VhostUserLog.
+
 VHOST_USER_SET_LOG_FD is an optional message with an eventfd in
 ancillary data, it may be used to inform the master that the log has
 been modified.
 
-Once the source has finished migration, VHOST_USER_RESET_OWNER message
-will be sent by the source. No further update must be done before the
-destination takes over with new regions & rings.
+Once the source has finished migration, rings will be stopped by
+the source. No further update must be done before rings are
+restarted.
 
 Protocol features
 -----------------
@@ -259,11 +301,11 @@ Message types
  * VHOST_USER_RESET_OWNER
 
       Id: 4
-      Equivalent ioctl: VHOST_RESET_OWNER
       Master payload: N/A
 
Quoted 8 lines

docs/specs/vhost-user.txt Message types · line 308

Quoted 5 lines
docs/specs/vhost-user.txt +3 -2
-      Issued when a new connection is about to be closed. The Master will no
-      longer own this connection (and will usually close it).
+      This is no longer used. Used to be sent to request stopping
+      all rings, but some clients interpreted it to also discard
+      connection state (this interpretation would lead to bugs).

The new code uses VHOST_USER_RESET_DEVICE as the name for request Id 4.

This was reverted before rc0.

Quoted 2 lines

Also it is not clear whether it is recommended just to
ignore the reset device request now?

Since it's never sent, does it matter?

Quoted 15 lines
Quoted 14 lines
docs/specs/vhost-user.txt +2 -0
  * VHOST_USER_SET_MEM_TABLE
 
@@ -388,6 +430,8 @@ Message types
       Master payload: vring state description
 
       Signal slave to enable or disable corresponding vring.
+      This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES
+      has been negotiated.
 
  * VHOST_USER_SEND_RARP
 
Signature · 1 line
MST

--Victor