vhost-user: update spec description
PATCH 1/1 vhost-user: update spec description
Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.
--- docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 9 deletions(-)
docs/specs/vhost-user.txt +53 -9
@@ -87,6 +87,14 @@ Depending on the request type, payload can be: User address: a 64-bit user address mmap offset: 64-bit offset where region starts in the mapped memory +* Log description + --------------------------- + | log size | log offset | + --------------------------- + log size: size of area used for logging + log offset: offset from start of supplied file descriptor + where logging starts (i.e. where guest address 0 would be logged) + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg {
@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Starting and stopping rings +---------------------- +Each ring is initialized in a stopped state, client must not process it until +ring is enabled. + +If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and +stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters +1 and 0 respoectively. + +If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start +ring processing upon receiving a kick (that is, detecting that file descriptor +is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop +ring processing upon receiving VHOST_USER_GET_VRING_BASE. + +While rings are running, client must support changing some configuration +aspects on the fly. + Multiple queue support ----------------------
@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark the dirty pages in a log. Once it complies to this logging, it may declare the VHOST_F_LOG_ALL vhost feature. +To start/stop logging of data/used ring writes, server may send messages +VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with +VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively. + All the modifications to memory pointed by vring "descriptor" should be marked. Modifications to "used" vring should be marked if -VHOST_VRING_F_LOG is part of ring's features. +VHOST_VRING_F_LOG is part of ring's flags. Dirty pages are of size: #define VHOST_LOG_PAGE 0x1000
@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of VHOST_USER_SET_LOG_BASE message when the slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature. -The size of the log may be computed by using all the known guest -addresses. The log covers from address 0 to the maximum of guest +The size of the log is supplied as part of VhostUserMsg +which should be large enough to cover all known guest +addresses. Log starts at the supplied offset in the +supplied file descriptor. +The log covers from address 0 to the maximum of guest regions. In pseudo-code, to mark page at "addr" as dirty: page = addr / VHOST_LOG_PAGE log[page / 8] |= 1 << page % 8 +Where addr is the guest physical address. + Use atomic operations, as the log may be concurrently manipulated. +Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG +is set for this ring), log_guest_addr should be used to calculate the log +offset: the write to first byte of the used ring is logged at this offset from +log start. Also note that this value might be outside the legal guest physical +address range (i.e. does not have to be covered by the VhostUserMemory table), +but the bit offset of the last byte of the ring must fall within +the size supplied by VhostUserLog. + VHOST_USER_SET_LOG_FD is an optional message with an eventfd in ancillary data, it may be used to inform the master that the log has been modified. -Once the source has finished migration, VHOST_USER_RESET_OWNER message -will be sent by the source. No further update must be done before the -destination takes over with new regions & rings. +Once the source has finished migration, rings will be stopped by +the source. No further update must be done before rings are +restarted. Protocol features -----------------
@@ -259,11 +301,11 @@ Message types * VHOST_USER_RESET_OWNER Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER Master payload: N/A - Issued when a new connection is about to be closed. The Master will no - longer own this connection (and will usually close it). + This is no longer used. Used to be sent to request stopping + all rings, but some clients interpreted it to also discard + connection state (this interpretation would lead to bugs). * VHOST_USER_SET_MEM_TABLE
@@ -388,6 +430,8 @@ Message types Master payload: vring state description Signal slave to enable or disable corresponding vring. + This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES + has been negotiated. * VHOST_USER_SEND_RARP
Signature · 1 line
MST
Quoted 109 lines · Michael S. Tsirkin
Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.
Signed-off-by Michael S. Tsirkin <[email protected]>--- docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 9 deletions(-)docs/specs/vhost-user.txt +48
-6@@ -87,6 +87,14 @@ Depending on the request type, payload can be: User address: a 64-bit user address mmap offset: 64-bit offset where region starts in the mapped memory +* Log description + --------------------------- + | log size | log offset | + --------------------------- + log size: size of area used for logging + log offset: offset from start of supplied file descriptor + where logging starts (i.e. where guest address 0 would be logged) + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg {@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Starting and stopping rings +---------------------- +Each ring is initialized in a stopped state, client must not process it until +ring is enabled. + +If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and +stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters +1 and 0 respoectively. + +If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start +ring processing upon receiving a kick (that is, detecting that file descriptor +is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop +ring processing upon receiving VHOST_USER_GET_VRING_BASE. + +While rings are running, client must support changing some configuration +aspects on the fly. + Multiple queue support ----------------------@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark the dirty pages in a log. Once it complies to this logging, it may declare the VHOST_F_LOG_ALL vhost feature. +To start/stop logging of data/used ring writes, server may send messages +VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with +VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively. + All the modifications to memory pointed by vring "descriptor" should be marked. Modifications to "used" vring should be marked if -VHOST_VRING_F_LOG is part of ring's features. +VHOST_VRING_F_LOG is part of ring's flags. Dirty pages are of size: #define VHOST_LOG_PAGE 0x1000@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of VHOST_USER_SET_LOG_BASE message when the slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature. -The size of the log may be computed by using all the known guest -addresses. The log covers from address 0 to the maximum of guest +The size of the log is supplied as part of VhostUserMsg +which should be large enough to cover all known guest +addresses. Log starts at the supplied offset in the +supplied file descriptor. +The log covers from address 0 to the maximum of guest regions. In pseudo-code, to mark page at "addr" as dirty: page = addr / VHOST_LOG_PAGE log[page / 8] |= 1 << page % 8 +Where addr is the guest physical address. + Use atomic operations, as the log may be concurrently manipulated. +Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG +is set for this ring), log_guest_addr should be used to calculate the log +offset: the write to first byte of the used ring is logged at this offset from +log start. Also note that this value might be outside the legal guest physical +address range (i.e. does not have to be covered by the VhostUserMemory table), +but the bit offset of the last byte of the ring must fall within +the size supplied by VhostUserLog. + VHOST_USER_SET_LOG_FD is an optional message with an eventfd in ancillary data, it may be used to inform the master that the log has been modified. -Once the source has finished migration, VHOST_USER_RESET_OWNER message -will be sent by the source. No further update must be done before the -destination takes over with new regions & rings. +Once the source has finished migration, rings will be stopped by +the source. No further update must be done before rings are +restarted. Protocol features -----------------@@ -259,11 +301,11 @@ Message types * VHOST_USER_RESET_OWNER
docs/specs/vhost-user.txt Message types · line 308
Quoted 10 lines
docs/specs/vhost-user.txt +3
-3Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER Master payload: N/A - Issued when a new connection is about to be closed. The Master will no - longer own this connection (and will usually close it). + This is no longer used. Used to be sent to request stopping + all rings, but some clients interpreted it to also discard + connection state (this interpretation would lead to bugs).
The new code uses VHOST_USER_RESET_DEVICE as the name for request Id 4. Also it is not clear whether it is recommended just to ignore the reset device request now?
Quoted 14 lines
docs/specs/vhost-user.txt +2
-0* VHOST_USER_SET_MEM_TABLE@@ -388,6 +430,8 @@ Message types Master payload: vring state description Signal slave to enable or disable corresponding vring. + This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES + has been negotiated. * VHOST_USER_SEND_RARPSignature · 1 line
MST
--Victor
Quoted 115 lines · Victor Kaplansky
Quoted 114 lines · Michael S. Tsirkin
Clarify logging setup to make sure all clients comply in a way that is future-proof. Document how rings are started/stopped.
Signed-off-by Michael S. Tsirkin <[email protected]>--- docs/specs/vhost-user.txt | 62 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 9 deletions(-)docs/specs/vhost-user.txt +48
-7@@ -87,6 +87,14 @@ Depending on the request type, payload can be: User address: a 64-bit user address mmap offset: 64-bit offset where region starts in the mapped memory +* Log description + --------------------------- + | log size | log offset | + --------------------------- + log size: size of area used for logging + log offset: offset from start of supplied file descriptor + where logging starts (i.e. where guest address 0 would be logged) + In QEMU the vhost-user message is implemented with the following struct: typedef struct VhostUserMsg {@@ -138,6 +146,23 @@ As older slaves don't support negotiating protocol features, a feature bit was dedicated for this purpose: #define VHOST_USER_F_PROTOCOL_FEATURES 30 +Starting and stopping rings +---------------------- +Each ring is initialized in a stopped state, client must not process it until +ring is enabled. + +If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, client must start and +stop ring processing upon receiving VHOST_USER_SET_VRING_ENABLE with parameters +1 and 0 respoectively. + +If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, client must start +ring processing upon receiving a kick (that is, detecting that file descriptor +is readable) on the descriptor specified by VHOST_USER_SET_VRING_KICK, and stop +ring processing upon receiving VHOST_USER_GET_VRING_BASE. + +While rings are running, client must support changing some configuration +aspects on the fly. + Multiple queue support ----------------------@@ -162,9 +187,13 @@ the slave makes to the memory mapped regions. The client should mark the dirty pages in a log. Once it complies to this logging, it may declare the VHOST_F_LOG_ALL vhost feature. +To start/stop logging of data/used ring writes, server may send messages +VHOST_USER_SET_FEATURES with VHOST_F_LOG_ALL and VHOST_USER_SET_VRING_ADDR with +VHOST_VRING_F_LOG in ring's flags set to 1/0, respectively. + All the modifications to memory pointed by vring "descriptor" should be marked. Modifications to "used" vring should be marked if -VHOST_VRING_F_LOG is part of ring's features. +VHOST_VRING_F_LOG is part of ring's flags. Dirty pages are of size: #define VHOST_LOG_PAGE 0x1000@@ -173,22 +202,35 @@ The log memory fd is provided in the ancillary data of VHOST_USER_SET_LOG_BASE message when the slave has VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature. -The size of the log may be computed by using all the known guest -addresses. The log covers from address 0 to the maximum of guest +The size of the log is supplied as part of VhostUserMsg +which should be large enough to cover all known guest +addresses. Log starts at the supplied offset in the +supplied file descriptor. +The log covers from address 0 to the maximum of guest regions. In pseudo-code, to mark page at "addr" as dirty: page = addr / VHOST_LOG_PAGE log[page / 8] |= 1 << page % 8 +Where addr is the guest physical address. + Use atomic operations, as the log may be concurrently manipulated. +Note that when logging modifications to the used ring (when VHOST_VRING_F_LOG +is set for this ring), log_guest_addr should be used to calculate the log +offset: the write to first byte of the used ring is logged at this offset from +log start. Also note that this value might be outside the legal guest physical +address range (i.e. does not have to be covered by the VhostUserMemory table), +but the bit offset of the last byte of the ring must fall within +the size supplied by VhostUserLog. + VHOST_USER_SET_LOG_FD is an optional message with an eventfd in ancillary data, it may be used to inform the master that the log has been modified. -Once the source has finished migration, VHOST_USER_RESET_OWNER message -will be sent by the source. No further update must be done before the -destination takes over with new regions & rings. +Once the source has finished migration, rings will be stopped by +the source. No further update must be done before rings are +restarted. Protocol features -----------------@@ -259,11 +301,11 @@ Message types * VHOST_USER_RESET_OWNER Id: 4 - Equivalent ioctl: VHOST_RESET_OWNER Master payload: N/A
Quoted 8 lines
docs/specs/vhost-user.txt Message types · line 308
Quoted 5 lines
docs/specs/vhost-user.txt +3
-2- Issued when a new connection is about to be closed. The Master will no - longer own this connection (and will usually close it). + This is no longer used. Used to be sent to request stopping + all rings, but some clients interpreted it to also discard + connection state (this interpretation would lead to bugs).The new code uses
VHOST_USER_RESET_DEVICEas the name for request Id 4.
This was reverted before rc0.
Quoted 2 lines
Also it is not clear whether it is recommended just to
ignore the reset device request now?
Since it's never sent, does it matter?
Quoted 15 lines
Quoted 14 lines
docs/specs/vhost-user.txt +2
-0* VHOST_USER_SET_MEM_TABLE@@ -388,6 +430,8 @@ Message types Master payload: vring state description Signal slave to enable or disable corresponding vring. + This request should be sent only when VHOST_USER_F_PROTOCOL_FEATURES + has been negotiated. * VHOST_USER_SEND_RARPSignature · 1 line
MST--Victor