Postcopy migration and vhost-user errors
cover Postcopy migration and vhost-user errors
From: Prasad Pandit <[email protected]>
Hello,
* virsh(1) offers multiple options to initiate Postcopy migration:
1) virsh migrate --postcopy --postcopy-after-precopy
2) virsh migrate --postcopy + virsh migrate-postcopy
3) virsh migrate --postcopy --timeout <N> --timeout-postcopy
When Postcopy migration is invoked via method (2) or (3) above, the guest on the destination host seems to hang or get stuck sometimes.
* During Postcopy migration, multiple threads are spawned on the destination host to start the guest and setup devices. One such thread starts vhost device via vhost_dev_start() function and another called fault_thread handles page faults in user space using kernel's userfaultfd(2) system.
When fault_thread exits upon completion of Postcopy migration, it sends a 'postcopy_end' message to the vhost-user device. But sometimes 'postcopy_end' message is sent while vhost device is being setup via vhost_dev_start().
Thread-1 Thread-2
vhost_dev_start postcopy_ram_incoming_cleanup
vhost_device_iotlb_miss postcopy_notify
vhost_backend_update_device_iotlb vhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msg vhost_user_postcopy_end
process_message_reply process_message_reply
vhost_user_read vhost_user_read
vhost_user_read_header vhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply to postcopy_end"
This creates confusion when vhost device receives 'postcopy_end' message while it is still trying to update IOTLB entries.
This seems to leave the guest in a stranded/hung state because fault_thread
has exited saying Postcopy migration has ended, but vhost-device is probably
still expecting updates. QEMU logs following errors on the destination host
===
...
qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm: vhost_user_postcopy_end: 700871,700900: Failed to receive reply to postcopy_end
qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x8 instead of 0x5.
qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x16 instead of 0x5.
qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm: vhost_user_read_header: 700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
qemu-kvm: vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
===
* Couple of patches here help to fix/handle these errors.
Thank you.
---
Prasad Pandit (2):
vhost-user: add a write-read lock
vhost: fail device start if iotlb update fails
hw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- hw/virtio/vhost.c | 6 +- include/hw/virtio/vhost-user.h | 3 + 3 files changed, 259 insertions(+), 173 deletions(-)
--
2.45.2
PATCH 1/2 vhost-user: add a write-read lock
From: Prasad Pandit <[email protected]>
QEMU threads use vhost_user_write/read calls to send and receive messages from a vhost-user device. When multiple threads communicate with the same vhost-user device, they can receive each other's messages, resulting in an erroneous state.
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
vhost_device_iotlb_miss:
700871,700871: Fail to update device iotlb
vhost_user_postcopy_end:
700871,700900: Failed to receive reply to postcopy_end
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags 0x0 instead of 0x5.
Here fault thread seems to end the postcopy migration while another thread is starting the vhost-user device.
Add a rw lock to hold for one vhost_user_write/read cycle and avoid such race conditions.
--- hw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- include/hw/virtio/vhost-user.h | 3 + 2 files changed, 254 insertions(+), 172 deletions(-)
hw/virtio/vhost-user.c +251 -172
@@ -24,6 +24,7 @@#include "qemu/main-loop.h" #include "qemu/uuid.h" #include "qemu/sockets.h" +#include "qemu/lockable.h" #include "sysemu/runstate.h" #include "sysemu/cryptodev.h" #include "migration/postcopy-ram.h"
@@ -433,6 +434,8 @@ int vhost_user_gpu_set_socket(struct vhost_dev *dev, int fd)static int vhost_user_set_log_base (struct vhost_dev *dev,uint64_t base,struct vhost_log *log) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_USER_MAX_RAM_SLOTS ];size_t fd_num =0 ; bool shmfd = virtio_has_feature(dev->protocol_features,
@@ -455,23 +458,25 @@ static int vhost_user_set_log_base(struct vhost_dev *dev, uint64_t base, fds[fd_num++] = log->fd; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } - -if (shmfd) { - msg.hdr.size =0 ; - ret =vhost_user_read (dev, &msg); + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num);if (ret <0 ) {return ret; } -if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { -error_report ("Received unexpected msg type. " -"Expected %d received %d" , -VHOST_USER_SET_LOG_BASE , msg.hdr.request); -return -EPROTO ; +if (shmfd) { + msg.hdr.size =0 ; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } + +if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { +error_report ("Received unexpected msg type. " +"Expected %d received %d" , +VHOST_USER_SET_LOG_BASE , msg.hdr.request); +return -EPROTO ; + } } }
@@ -664,6 +669,7 @@ static int send_remove_regions(struct vhost_dev *dev, bool reply_supported) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;struct vhost_memory_region *shadow_reg;int i, fd, shadow_reg_idx, ret;ram_addr_t offset;
@@ -685,16 +691,18 @@ static int send_remove_regions(struct vhost_dev *dev,vhost_user_fill_msg_region (®ion_buffer, shadow_reg,0 ); msg->payload.mem_reg.region = region_buffer; - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } - -if (reply_supported) { - ret =process_message_reply (dev, msg); -if (ret) { + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) {return ret; } + +if (reply_supported) { + ret =process_message_reply (dev, msg); +if (ret) { +return ret; + } + } } }
@@ -725,6 +733,9 @@ static int send_add_regions(struct vhost_dev *dev,VhostUserMsg msg_reply;VhostUserMemoryRegion region_buffer; +struct VhostUserState *us = u->user; +QEMU_LOCK_GUARD (&us->vhost_user_rw_lock); +for (i =0 ; i < nr_add_reg; i++) { reg = add_reg[i].region; reg_idx = add_reg[i].reg_idx;
@@ -893,6 +904,7 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev, bool config_mem_slots) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;VhostUserMsg msg_reply;
@@ -926,14 +938,16 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg_reply); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg_reply); +if (ret <0 ) { +return ret; + } }if (msg_reply.hdr.request !=VHOST_USER_SET_MEM_TABLE ) {
@@ -1005,6 +1019,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev, struct vhost_memory *mem) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;bool do_postcopy = u->postcopy_listen && u->postcopy_fd.handler;
@@ -1044,13 +1059,15 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } } }
@@ -1080,6 +1097,8 @@ static int vhost_user_set_vring_endian(struct vhost_dev *dev,static int vhost_user_get_u64 (struct vhost_dev *dev,int request,uint64_t *u64) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserMsg msg = { .hdr.request = request, .hdr.flags =VHOST_USER_VERSION ,
@@ -1089,14 +1108,16 @@ static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t *u64)return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request != request) {
@@ -1129,6 +1150,8 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, bool wait_for_reply) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (wait_for_reply) { bool reply_supported = virtio_has_feature(dev->protocol_features,
@@ -1138,25 +1161,27 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, } } - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (wait_for_reply) { -uint64_t dummy; +if (wait_for_reply) { +uint64_t dummy; -if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { -return process_message_reply (dev, msg); - } +if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { +return process_message_reply (dev, msg); + } -/* -* We need to wait for a reply but the backend does not -* support replies for the command we just sent. -* Send VHOST_USER_GET_FEATURES which makes all backends -* send a reply. -*/ -return vhost_user_get_features (dev, &dummy); +/* +* We need to wait for a reply but the backend does not +* support replies for the command we just sent. +* Send VHOST_USER_GET_FEATURES which makes all backends +* send a reply. +*/ +return vhost_user_get_features (dev, &dummy); + } }return 0 ;
@@ -1277,20 +1302,23 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, .hdr.size = sizeof(msg.payload.state), };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserHostNotifier *n =fetch_notifier (u->user, ring->index);if (n) {vhost_user_host_notifier_remove (n, dev->vdev); } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_VRING_BASE ) {
@@ -1661,6 +1689,7 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, int *dmabuf_fd) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ret;VhostUserMsg msg = {
@@ -1669,14 +1698,16 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, };memcpy (msg.payload.object.uuid, uuid,sizeof (msg.payload.object.uuid)); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_SHARED_OBJECT ) {
@@ -1858,6 +1889,7 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) .hdr.flags = VHOST_USER_VERSION, };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int sv[2 ], ret =0 ;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );
@@ -1889,15 +1921,16 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); -if (ret) { - goto out; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); +if (ret) { + goto out; + } -if (reply_supported) { - ret =process_message_reply (dev, &msg); +if (reply_supported) { + ret =process_message_reply (dev, &msg); + } } - out:close (sv[1 ]);if (ret) {
@@ -1985,6 +2018,7 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) {#ifdef CONFIG_LINUX struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ufd;int ret;
@@ -1993,16 +2027,18 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) .hdr.flags = VHOST_USER_VERSION, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_advise to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_advise to vhost" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_POSTCOPY_ADVISE ) {
@@ -2051,16 +2087,19 @@ static int vhost_user_postcopy_listen(struct vhost_dev *dev, Error **errp)trace_vhost_user_postcopy_listen (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_listen to vhost" ); -return ret; - } +struct VhostUserState *us = u->user; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_listen to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_listen" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_listen" ); +return ret; + } }return 0 ;
@@ -2077,19 +2116,22 @@ static int vhost_user_postcopy_end(struct vhost_dev *dev, Error **errp) };int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;trace_vhost_user_postcopy_end_entry (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_end to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_end to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_end" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_end" ); +return ret; + } }postcopy_unregister_shared_ufd (&u->postcopy_fd);close (u->postcopy_fd.fd);
@@ -2359,6 +2401,8 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_NET_MTU ))) {return 0 ;
@@ -2372,14 +2416,16 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -/* If reply_ack supported, backend has to ack specified MTU is valid */ -if (reply_supported) { -return process_message_reply (dev, &msg); +/* If reply_ack supported, backend has to ack specified MTU is valid */ +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;
@@ -2396,12 +2442,19 @@ static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev, .payload.iotlb = *imsg, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; + + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } + + ret =process_message_reply (dev, &msg); } -return process_message_reply (dev, &msg); +return ret; }
@@ -2414,6 +2467,8 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, uint32_t config_len, Error **errp) { int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; VhostUserMsg msg = { .hdr.request = VHOST_USER_GET_CONFIG, .hdr.flags = VHOST_USER_VERSION,
@@ -2430,16 +2485,19 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, msg.payload.config.offset =0 ; msg.payload.config.size = config_len; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_CONFIG ) {
@@ -2464,6 +2522,8 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, {int ret;uint8_t *p; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );
@@ -2492,13 +2552,15 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, p = msg.payload.config.region;memcpy (p, data, size); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;
@@ -2509,6 +2571,8 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, uint64_t *session_id) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool crypto_session =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_CRYPTO_SESSION );CryptoDevBackendSessionInfo *backend_info = session_info;
@@ -2572,18 +2636,21 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, msg.payload.session.op_code = backend_info->op_code; msg.payload.session.session_id = backend_info->session_id; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_report ("vhost_user_write() return %d, create session failed" , - ret); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_report ("vhost_user_read() return %d, create session failed" , - ret); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_report ("vhost_user_write() return %d, create session failed" , + ret); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_report ("vhost_user_read() return %d, create session failed" , + ret); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CREATE_CRYPTO_SESSION ) {
@@ -2648,6 +2715,7 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,int fd;int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;VhostUserMsg msg = { .hdr.request =VHOST_USER_GET_INFLIGHT_FD ,
@@ -2662,14 +2730,16 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_INFLIGHT_FD ) {
@@ -2757,6 +2827,7 @@ bool vhost_user_init(VhostUserState *user, CharBackend *chr, Error **errp) user->memory_slots =0 ; user->notifiers =g_ptr_array_new_full (VIRTIO_QUEUE_MAX /4 , &vhost_user_state_destroy); +qemu_mutex_init (&user->vhost_user_rw_lock);return true; }
@@ -2769,6 +2840,7 @@ void vhost_user_cleanup(VhostUserState *user) user->notifiers = (GPtrArray *)g_ptr_array_free (user->notifiers, true);memory_region_transaction_commit (); user->chr =NULL ; +qemu_mutex_destroy (&user->vhost_user_rw_lock); }
@@ -2882,6 +2954,7 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev, {int ret;struct vhost_user *vu = dev->opaque; +struct VhostUserState *us = vu->user;VhostUserMsg msg = { .hdr = { .request =VHOST_USER_SET_DEVICE_STATE_FD ,
@@ -2902,19 +2975,21 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg, &fd,1 ); -close (fd); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send SET_DEVICE_STATE_FD message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &fd,1 ); +close (fd); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send SET_DEVICE_STATE_FD message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive SET_DEVICE_STATE_FD reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive SET_DEVICE_STATE_FD reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_SET_DEVICE_STATE_FD ) {
@@ -2951,6 +3026,8 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,static int vhost_user_check_device_state (struct vhost_dev *dev,Error **errp) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int ret;VhostUserMsg msg = { .hdr = {
@@ -2965,18 +3042,20 @@ static int vhost_user_check_device_state(struct vhost_dev *dev, Error **errp)return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send CHECK_DEVICE_STATE message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send CHECK_DEVICE_STATE message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive CHECK_DEVICE_STATE reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive CHECK_DEVICE_STATE reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CHECK_DEVICE_STATE ) {
include/hw/virtio/vhost-user.h +3 -0
@@ -67,6 +67,9 @@ typedef struct VhostUserState {GPtrArray *notifiers;int memory_slots;bool supports_config; + +/* Hold lock for vhost_user_write/read cycle */ +QemuMutex vhost_user_rw_lock; } VhostUserState;/**
Signature · 1 line
2.45.2
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 16 lines · Prasad Pandit
From: Prasad Pandit <[email protected]>
QEMU threads use vhost_user_write/read calls to send and receive messages from a vhost-user device. When multiple threads communicate with the same vhost-user device, they can receive each other's messages, resulting in an erroneous state.
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.
vhost_device_iotlb_miss:
700871,700871: Fail to update device iotlb
vhost_user_postcopy_end:
700871,700900: Failed to receive reply topostcopy_end
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.
Quoted 8 lines
Here fault thread seems to end the postcopy migration while another thread is starting the vhost-user device.
Add a rw lock to hold for one vhost_user_write/read cycle and avoid such race conditions.
Suggested-by Peter Xu <[email protected]>Signed-off-by Prasad Pandit <[email protected]>
Could you supply a Fixes tag here? What commit introduced the race?
Quoted 765 lines
--- hw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- include/hw/virtio/vhost-user.h | 3 + 2 files changed, 254 insertions(+), 172 deletions(-)hw/virtio/vhost-user.c +251
-172@@ -24,6 +24,7 @@#include "qemu/main-loop.h" #include "qemu/uuid.h" #include "qemu/sockets.h" +#include "qemu/lockable.h" #include "sysemu/runstate.h" #include "sysemu/cryptodev.h" #include "migration/postcopy-ram.h" @@ -433,6 +434,8 @@ int vhost_user_gpu_set_socket(struct vhost_dev *dev, int fd)static int vhost_user_set_log_base (struct vhost_dev *dev,uint64_t base,struct vhost_log *log) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_USER_MAX_RAM_SLOTS ];size_t fd_num =0 ; bool shmfd = virtio_has_feature(dev->protocol_features,@@ -455,23 +458,25 @@ static int vhost_user_set_log_base(struct vhost_dev *dev, uint64_t base, fds[fd_num++] = log->fd; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } - -if (shmfd) { - msg.hdr.size =0 ; - ret =vhost_user_read (dev, &msg); + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num);if (ret <0 ) {return ret; } -if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { -error_report ("Received unexpected msg type. " -"Expected %d received %d" , -VHOST_USER_SET_LOG_BASE , msg.hdr.request); -return -EPROTO ; +if (shmfd) { + msg.hdr.size =0 ; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } + +if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { +error_report ("Received unexpected msg type. " +"Expected %d received %d" , +VHOST_USER_SET_LOG_BASE , msg.hdr.request); +return -EPROTO ; + } } }@@ -664,6 +669,7 @@ static int send_remove_regions(struct vhost_dev *dev, bool reply_supported) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;struct vhost_memory_region *shadow_reg;int i, fd, shadow_reg_idx, ret;ram_addr_t offset;@@ -685,16 +691,18 @@ static int send_remove_regions(struct vhost_dev *dev,vhost_user_fill_msg_region (®ion_buffer, shadow_reg,0 ); msg->payload.mem_reg.region = region_buffer; - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } - -if (reply_supported) { - ret =process_message_reply (dev, msg); -if (ret) { + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) {return ret; } + +if (reply_supported) { + ret =process_message_reply (dev, msg); +if (ret) { +return ret; + } + } } }@@ -725,6 +733,9 @@ static int send_add_regions(struct vhost_dev *dev,VhostUserMsg msg_reply;VhostUserMemoryRegion region_buffer; +struct VhostUserState *us = u->user; +QEMU_LOCK_GUARD (&us->vhost_user_rw_lock); +for (i =0 ; i < nr_add_reg; i++) { reg = add_reg[i].region; reg_idx = add_reg[i].reg_idx;@@ -893,6 +904,7 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev, bool config_mem_slots) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;VhostUserMsg msg_reply;@@ -926,14 +938,16 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg_reply); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg_reply); +if (ret <0 ) { +return ret; + } }if (msg_reply.hdr.request !=VHOST_USER_SET_MEM_TABLE ) {@@ -1005,6 +1019,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev, struct vhost_memory *mem) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;bool do_postcopy = u->postcopy_listen && u->postcopy_fd.handler;@@ -1044,13 +1059,15 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } } }@@ -1080,6 +1097,8 @@ static int vhost_user_set_vring_endian(struct vhost_dev *dev,static int vhost_user_get_u64 (struct vhost_dev *dev,int request,uint64_t *u64) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserMsg msg = { .hdr.request = request, .hdr.flags =VHOST_USER_VERSION ,@@ -1089,14 +1108,16 @@ static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t *u64)return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request != request) {@@ -1129,6 +1150,8 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, bool wait_for_reply) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (wait_for_reply) { bool reply_supported = virtio_has_feature(dev->protocol_features,@@ -1138,25 +1161,27 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, } } - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (wait_for_reply) { -uint64_t dummy; +if (wait_for_reply) { +uint64_t dummy; -if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { -return process_message_reply (dev, msg); - } +if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { +return process_message_reply (dev, msg); + } -/* -* We need to wait for a reply but the backend does not -* support replies for the command we just sent. -* Send VHOST_USER_GET_FEATURES which makes all backends -* send a reply. -*/ -return vhost_user_get_features (dev, &dummy); +/* +* We need to wait for a reply but the backend does not +* support replies for the command we just sent. +* Send VHOST_USER_GET_FEATURES which makes all backends +* send a reply. +*/ +return vhost_user_get_features (dev, &dummy); + } }return 0 ;@@ -1277,20 +1302,23 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, .hdr.size = sizeof(msg.payload.state), };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserHostNotifier *n =fetch_notifier (u->user, ring->index);if (n) {vhost_user_host_notifier_remove (n, dev->vdev); } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_VRING_BASE ) {@@ -1661,6 +1689,7 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, int *dmabuf_fd) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ret;VhostUserMsg msg = {@@ -1669,14 +1698,16 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, };memcpy (msg.payload.object.uuid, uuid,sizeof (msg.payload.object.uuid)); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_SHARED_OBJECT ) {@@ -1858,6 +1889,7 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) .hdr.flags = VHOST_USER_VERSION, };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int sv[2 ], ret =0 ;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );@@ -1889,15 +1921,16 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); -if (ret) { - goto out; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); +if (ret) { + goto out; + } -if (reply_supported) { - ret =process_message_reply (dev, &msg); +if (reply_supported) { + ret =process_message_reply (dev, &msg); + } } - out:close (sv[1 ]);if (ret) {@@ -1985,6 +2018,7 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) {#ifdef CONFIG_LINUX struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ufd;int ret;@@ -1993,16 +2027,18 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) .hdr.flags = VHOST_USER_VERSION, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_advise to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_advise to vhost" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_POSTCOPY_ADVISE ) {@@ -2051,16 +2087,19 @@ static int vhost_user_postcopy_listen(struct vhost_dev *dev, Error **errp)trace_vhost_user_postcopy_listen (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_listen to vhost" ); -return ret; - } +struct VhostUserState *us = u->user; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_listen to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_listen" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_listen" ); +return ret; + } }return 0 ;@@ -2077,19 +2116,22 @@ static int vhost_user_postcopy_end(struct vhost_dev *dev, Error **errp) };int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;trace_vhost_user_postcopy_end_entry (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_end to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_end to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_end" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_end" ); +return ret; + } }postcopy_unregister_shared_ufd (&u->postcopy_fd);close (u->postcopy_fd.fd);@@ -2359,6 +2401,8 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_NET_MTU ))) {return 0 ;@@ -2372,14 +2416,16 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -/* If reply_ack supported, backend has to ack specified MTU is valid */ -if (reply_supported) { -return process_message_reply (dev, &msg); +/* If reply_ack supported, backend has to ack specified MTU is valid */ +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;@@ -2396,12 +2442,19 @@ static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev, .payload.iotlb = *imsg, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; + + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } + + ret =process_message_reply (dev, &msg); } -return process_message_reply (dev, &msg); +return ret; }@@ -2414,6 +2467,8 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, uint32_t config_len, Error **errp) { int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; VhostUserMsg msg = { .hdr.request = VHOST_USER_GET_CONFIG, .hdr.flags = VHOST_USER_VERSION,@@ -2430,16 +2485,19 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, msg.payload.config.offset =0 ; msg.payload.config.size = config_len; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_CONFIG ) {@@ -2464,6 +2522,8 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, {int ret;uint8_t *p; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );@@ -2492,13 +2552,15 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, p = msg.payload.config.region;memcpy (p, data, size); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;@@ -2509,6 +2571,8 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, uint64_t *session_id) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool crypto_session =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_CRYPTO_SESSION );CryptoDevBackendSessionInfo *backend_info = session_info;@@ -2572,18 +2636,21 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, msg.payload.session.op_code = backend_info->op_code; msg.payload.session.session_id = backend_info->session_id; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_report ("vhost_user_write() return %d, create session failed" , - ret); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_report ("vhost_user_read() return %d, create session failed" , - ret); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_report ("vhost_user_write() return %d, create session failed" , + ret); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_report ("vhost_user_read() return %d, create session failed" , + ret); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CREATE_CRYPTO_SESSION ) {@@ -2648,6 +2715,7 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,int fd;int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;VhostUserMsg msg = { .hdr.request =VHOST_USER_GET_INFLIGHT_FD ,@@ -2662,14 +2730,16 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_INFLIGHT_FD ) {@@ -2757,6 +2827,7 @@ bool vhost_user_init(VhostUserState *user, CharBackend *chr, Error **errp) user->memory_slots =0 ; user->notifiers =g_ptr_array_new_full (VIRTIO_QUEUE_MAX /4 , &vhost_user_state_destroy); +qemu_mutex_init (&user->vhost_user_rw_lock);return true; }@@ -2769,6 +2840,7 @@ void vhost_user_cleanup(VhostUserState *user) user->notifiers = (GPtrArray *)g_ptr_array_free (user->notifiers, true);memory_region_transaction_commit (); user->chr =NULL ; +qemu_mutex_destroy (&user->vhost_user_rw_lock); }@@ -2882,6 +2954,7 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev, {int ret;struct vhost_user *vu = dev->opaque; +struct VhostUserState *us = vu->user;VhostUserMsg msg = { .hdr = { .request =VHOST_USER_SET_DEVICE_STATE_FD ,@@ -2902,19 +2975,21 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg, &fd,1 ); -close (fd); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send SET_DEVICE_STATE_FD message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &fd,1 ); +close (fd); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send SET_DEVICE_STATE_FD message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive SET_DEVICE_STATE_FD reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive SET_DEVICE_STATE_FD reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_SET_DEVICE_STATE_FD ) {@@ -2951,6 +3026,8 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,static int vhost_user_check_device_state (struct vhost_dev *dev,Error **errp) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int ret;VhostUserMsg msg = { .hdr = {@@ -2965,18 +3042,20 @@ static int vhost_user_check_device_state(struct vhost_dev *dev, Error **errp)return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send CHECK_DEVICE_STATE message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send CHECK_DEVICE_STATE message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive CHECK_DEVICE_STATE reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive CHECK_DEVICE_STATE reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CHECK_DEVICE_STATE ) {include/hw/virtio/vhost-user.h +3
-0@@ -67,6 +67,9 @@ typedef struct VhostUserState {GPtrArray *notifiers;int memory_slots;bool supports_config; + +/* Hold lock for vhost_user_write/read cycle */ +QemuMutex vhost_user_rw_lock; } VhostUserState;/** Signature · 1 line
2.45.2
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 1 line · Michael S. Tsirkin
Could you supply a Fixes tag here? What commit introduced the race?
'postcopy_end' message was added by:
-> https://github.com/qemu/qemu/commit/46343570c06e63b4499f619011df80f91349cd49
Not sure if its race condition also began with it. It is not clear if the front-end should allow multiple threads to talk to the same vhost-user device.
Thank you.
---
- Prasad
Re: [PATCH 1/2] vhost-user: add a write-read lock
In reply to Prasad PanditQuoted 19 lines · Prasad Pandit
From: Prasad Pandit <[email protected]>
QEMU threads use vhost_user_write/read calls to send and receive messages from a vhost-user device. When multiple threads communicate with the same vhost-user device, they can receive each other's messages, resulting in an erroneous state.
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.
vhost_device_iotlb_miss:
700871,700871: Fail to update device iotlb
vhost_user_postcopy_end:
700871,700900: Failed to receive reply topostcopy_end
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.Here fault thread seems to end the postcopy migration while another thread is starting the vhost-user device.
Quoted 8 lines
Add a rw lock to hold for one vhost_user_write/read cycle and avoid such race conditions.
Suggested-by Peter Xu <[email protected]>Signed-off-by Prasad Pandit <[email protected]>--- hw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- include/hw/virtio/vhost-user.h | 3 +
I apologize if I suggested WITH_QEMU_LOCK_GUARD when we talked.. I don't remember which one I suggested, but in this case IIUC it'll be much easier to review if you use the other sister function QEMU_LOCK_GUARD() instead.. That should make the diff much, much less.
Signature · 1 line
Peter Xu
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 4 lines · Peter Xu
I apologize if I suggested
WITH_QEMU_LOCK_GUARDwhen we talked.. I don't remember which one I suggested, but in this case IIUC it'll be much easier to review if you use the other sister functionQEMU_LOCK_GUARD()instead.. That should make the diff much, much less.
* Yes, QEMU_LOCK_GUARD simplifies the diff, but it may extend the time for which lock is held, delaying other threads, is that okay?
Thank you.
---
- Prasad
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 8 lines · Prasad Pandit
Quoted 4 lines · Peter Xu
I apologize if I suggested
WITH_QEMU_LOCK_GUARDwhen we talked.. I don't remember which one I suggested, but in this case IIUC it'll be much easier to review if you use the other sister functionQEMU_LOCK_GUARD()instead.. That should make the diff much, much less.* Yes,
QEMU_LOCK_GUARDsimplifies the diff, but it may extend the time for which lock is held, delaying other threads, is that okay?
I think it shouldn't be a major deal in most cases, if the extended cycles only cover a bunch of instructions. In special case we can still use WITH_QEMU_LOCK_GUARD, but I'd start with the simple first and only switch if necessary.
Thanks,
Signature · 1 line
Peter Xu
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 4 lines · Peter Xu
I think it shouldn't be a major deal in most cases, if the extended cycles only cover a bunch of instructions. In special case we can still use
WITH_QEMU_LOCK_GUARD, but I'd start with the simple first and only switch if necessary.
* Okay, will send patch v2.
Thank you.
---
- Prasad
Re: [PATCH 1/2] vhost-user: add a write-read lock
In reply to Prasad PanditQuoted 778 lines · Prasad Pandit
From: Prasad Pandit <[email protected]>
QEMU threads use vhost_user_write/read calls to send and receive messages from a vhost-user device. When multiple threads communicate with the same vhost-user device, they can receive each other's messages, resulting in an erroneous state.
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.
vhost_device_iotlb_miss:
700871,700871: Fail to update device iotlb
vhost_user_postcopy_end:
700871,700900: Failed to receive reply topostcopy_end
vhost_user_read_header:
700871,700871: Failed to read msg header. Flags0x0instead of0x5.Here fault thread seems to end the postcopy migration while another thread is starting the vhost-user device.
Add a rw lock to hold for one vhost_user_write/read cycle and avoid such race conditions.
Suggested-by Peter Xu <[email protected]>Signed-off-by Prasad Pandit <[email protected]>--- hw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- include/hw/virtio/vhost-user.h | 3 + 2 files changed, 254 insertions(+), 172 deletions(-)hw/virtio/vhost-user.c +251
-172@@ -24,6 +24,7 @@#include "qemu/main-loop.h" #include "qemu/uuid.h" #include "qemu/sockets.h" +#include "qemu/lockable.h" #include "sysemu/runstate.h" #include "sysemu/cryptodev.h" #include "migration/postcopy-ram.h" @@ -433,6 +434,8 @@ int vhost_user_gpu_set_socket(struct vhost_dev *dev, int fd)static int vhost_user_set_log_base (struct vhost_dev *dev,uint64_t base,struct vhost_log *log) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_USER_MAX_RAM_SLOTS ];size_t fd_num =0 ; bool shmfd = virtio_has_feature(dev->protocol_features,@@ -455,23 +458,25 @@ static int vhost_user_set_log_base(struct vhost_dev *dev, uint64_t base, fds[fd_num++] = log->fd; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } - -if (shmfd) { - msg.hdr.size =0 ; - ret =vhost_user_read (dev, &msg); + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num);if (ret <0 ) {return ret; } -if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { -error_report ("Received unexpected msg type. " -"Expected %d received %d" , -VHOST_USER_SET_LOG_BASE , msg.hdr.request); -return -EPROTO ; +if (shmfd) { + msg.hdr.size =0 ; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } + +if (msg.hdr.request !=VHOST_USER_SET_LOG_BASE ) { +error_report ("Received unexpected msg type. " +"Expected %d received %d" , +VHOST_USER_SET_LOG_BASE , msg.hdr.request); +return -EPROTO ; + } } }@@ -664,6 +669,7 @@ static int send_remove_regions(struct vhost_dev *dev, bool reply_supported) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;struct vhost_memory_region *shadow_reg;int i, fd, shadow_reg_idx, ret;ram_addr_t offset;@@ -685,16 +691,18 @@ static int send_remove_regions(struct vhost_dev *dev,vhost_user_fill_msg_region (®ion_buffer, shadow_reg,0 ); msg->payload.mem_reg.region = region_buffer; - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } - -if (reply_supported) { - ret =process_message_reply (dev, msg); -if (ret) { + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) {return ret; } + +if (reply_supported) { + ret =process_message_reply (dev, msg); +if (ret) { +return ret; + } + } } }@@ -725,6 +733,9 @@ static int send_add_regions(struct vhost_dev *dev,VhostUserMsg msg_reply;VhostUserMemoryRegion region_buffer; +struct VhostUserState *us = u->user; +QEMU_LOCK_GUARD (&us->vhost_user_rw_lock); +for (i =0 ; i < nr_add_reg; i++) { reg = add_reg[i].region; reg_idx = add_reg[i].reg_idx;@@ -893,6 +904,7 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev, bool config_mem_slots) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;VhostUserMsg msg_reply;@@ -926,14 +938,16 @@ static int vhost_user_set_mem_table_postcopy(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg_reply); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg_reply); +if (ret <0 ) { +return ret; + } }if (msg_reply.hdr.request !=VHOST_USER_SET_MEM_TABLE ) {@@ -1005,6 +1019,7 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev, struct vhost_memory *mem) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int fds[VHOST_MEMORY_BASELINE_NREGIONS ];size_t fd_num =0 ;bool do_postcopy = u->postcopy_listen && u->postcopy_fd.handler;@@ -1044,13 +1059,15 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev,return ret; } - ret =vhost_user_write (dev, &msg, fds, fd_num); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, fds, fd_num); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } } }@@ -1080,6 +1097,8 @@ static int vhost_user_set_vring_endian(struct vhost_dev *dev,static int vhost_user_get_u64 (struct vhost_dev *dev,int request,uint64_t *u64) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserMsg msg = { .hdr.request = request, .hdr.flags =VHOST_USER_VERSION ,@@ -1089,14 +1108,16 @@ static int vhost_user_get_u64(struct vhost_dev *dev, int request, uint64_t *u64)return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request != request) {@@ -1129,6 +1150,8 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, bool wait_for_reply) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (wait_for_reply) { bool reply_supported = virtio_has_feature(dev->protocol_features,@@ -1138,25 +1161,27 @@ static int vhost_user_write_sync(struct vhost_dev *dev, VhostUserMsg *msg, } } - ret =vhost_user_write (dev, msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (wait_for_reply) { -uint64_t dummy; +if (wait_for_reply) { +uint64_t dummy; -if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { -return process_message_reply (dev, msg); - } +if (msg->hdr.flags &VHOST_USER_NEED_REPLY_MASK ) { +return process_message_reply (dev, msg); + } -/* -* We need to wait for a reply but the backend does not -* support replies for the command we just sent. -* Send VHOST_USER_GET_FEATURES which makes all backends -* send a reply. -*/ -return vhost_user_get_features (dev, &dummy); +/* +* We need to wait for a reply but the backend does not +* support replies for the command we just sent. +* Send VHOST_USER_GET_FEATURES which makes all backends +* send a reply. +*/ +return vhost_user_get_features (dev, &dummy); + } }return 0 ;@@ -1277,20 +1302,23 @@ static int vhost_user_get_vring_base(struct vhost_dev *dev, .hdr.size = sizeof(msg.payload.state), };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;VhostUserHostNotifier *n =fetch_notifier (u->user, ring->index);if (n) {vhost_user_host_notifier_remove (n, dev->vdev); } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_VRING_BASE ) {@@ -1661,6 +1689,7 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, int *dmabuf_fd) {struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ret;VhostUserMsg msg = {@@ -1669,14 +1698,16 @@ int vhost_user_get_shared_object(struct vhost_dev *dev, unsigned char *uuid, };memcpy (msg.payload.object.uuid, uuid,sizeof (msg.payload.object.uuid)); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_SHARED_OBJECT ) {@@ -1858,6 +1889,7 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) .hdr.flags = VHOST_USER_VERSION, };struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int sv[2 ], ret =0 ;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );@@ -1889,15 +1921,16 @@ static int vhost_setup_backend_channel(struct vhost_dev *dev) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); -if (ret) { - goto out; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &sv[1 ],1 ); +if (ret) { + goto out; + } -if (reply_supported) { - ret =process_message_reply (dev, &msg); +if (reply_supported) { + ret =process_message_reply (dev, &msg); + } } - out:close (sv[1 ]);if (ret) {@@ -1985,6 +2018,7 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) {#ifdef CONFIG_LINUX struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;int ufd;int ret;@@ -1993,16 +2027,18 @@ static int vhost_user_postcopy_advise(struct vhost_dev *dev, Error **errp) .hdr.flags = VHOST_USER_VERSION, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_advise to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_advise to vhost" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg (errp,"Failed to get postcopy_advise reply from vhost" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_POSTCOPY_ADVISE ) {@@ -2051,16 +2087,19 @@ static int vhost_user_postcopy_listen(struct vhost_dev *dev, Error **errp)trace_vhost_user_postcopy_listen (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_listen to vhost" ); -return ret; - } +struct VhostUserState *us = u->user; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_listen to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_listen" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_listen" ); +return ret; + } }return 0 ;@@ -2077,19 +2116,22 @@ static int vhost_user_postcopy_end(struct vhost_dev *dev, Error **errp) };int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;trace_vhost_user_postcopy_end_entry (); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg (errp,"Failed to send postcopy_end to vhost" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg (errp,"Failed to send postcopy_end to vhost" ); +return ret; + } - ret =process_message_reply (dev, &msg); -if (ret) { -error_setg (errp,"Failed to receive reply to postcopy_end" ); -return ret; + ret =process_message_reply (dev, &msg); +if (ret) { +error_setg (errp,"Failed to receive reply to postcopy_end" ); +return ret; + } }postcopy_unregister_shared_ufd (&u->postcopy_fd);close (u->postcopy_fd.fd);@@ -2359,6 +2401,8 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;if (!(dev->protocol_features & (1ULL <<VHOST_USER_PROTOCOL_F_NET_MTU ))) {return 0 ;@@ -2372,14 +2416,16 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu) msg.hdr.flags |=VHOST_USER_NEED_REPLY_MASK ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -/* If reply_ack supported, backend has to ack specified MTU is valid */ -if (reply_supported) { -return process_message_reply (dev, &msg); +/* If reply_ack supported, backend has to ack specified MTU is valid */ +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;@@ -2396,12 +2442,19 @@ static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev, .payload.iotlb = *imsg, }; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; + + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } + + ret =process_message_reply (dev, &msg); } -return process_message_reply (dev, &msg); +return ret; }@@ -2414,6 +2467,8 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, uint32_t config_len, Error **errp) { int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user; VhostUserMsg msg = { .hdr.request = VHOST_USER_GET_CONFIG, .hdr.flags = VHOST_USER_VERSION,@@ -2430,16 +2485,19 @@ static int vhost_user_get_config(struct vhost_dev *dev, uint8_t *config, msg.payload.config.offset =0 ; msg.payload.config.size = config_len; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret,"vhost_get_config failed" ); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret,"vhost_get_config failed" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_CONFIG ) {@@ -2464,6 +2522,8 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, {int ret;uint8_t *p; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool reply_supported =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_REPLY_ACK );@@ -2492,13 +2552,15 @@ static int vhost_user_set_config(struct vhost_dev *dev, const uint8_t *data, p = msg.payload.config.region;memcpy (p, data, size); - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } -if (reply_supported) { -return process_message_reply (dev, &msg); +if (reply_supported) { +return process_message_reply (dev, &msg); + } }return 0 ;@@ -2509,6 +2571,8 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, uint64_t *session_id) {int ret; +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;bool crypto_session =virtio_has_feature (dev->protocol_features,VHOST_USER_PROTOCOL_F_CRYPTO_SESSION );CryptoDevBackendSessionInfo *backend_info = session_info;@@ -2572,18 +2636,21 @@ static int vhost_user_crypto_create_session(struct vhost_dev *dev, msg.payload.session.op_code = backend_info->op_code; msg.payload.session.session_id = backend_info->session_id; - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_report ("vhost_user_write() return %d, create session failed" , - ret); -return ret; - } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_report ("vhost_user_read() return %d, create session failed" , - ret); -return ret; + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_report ("vhost_user_write() return %d, create session failed" , + ret); +return ret; + } + + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_report ("vhost_user_read() return %d, create session failed" , + ret); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CREATE_CRYPTO_SESSION ) {@@ -2648,6 +2715,7 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,int fd;int ret;struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;CharBackend *chr = u->user->chr;VhostUserMsg msg = { .hdr.request =VHOST_USER_GET_INFLIGHT_FD ,@@ -2662,14 +2730,16 @@ static int vhost_user_get_inflight_fd(struct vhost_dev *dev,return 0 ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +return ret; + } }if (msg.hdr.request !=VHOST_USER_GET_INFLIGHT_FD ) {@@ -2757,6 +2827,7 @@ bool vhost_user_init(VhostUserState *user, CharBackend *chr, Error **errp) user->memory_slots =0 ; user->notifiers =g_ptr_array_new_full (VIRTIO_QUEUE_MAX /4 , &vhost_user_state_destroy); +qemu_mutex_init (&user->vhost_user_rw_lock);return true; }@@ -2769,6 +2840,7 @@ void vhost_user_cleanup(VhostUserState *user) user->notifiers = (GPtrArray *)g_ptr_array_free (user->notifiers, true);memory_region_transaction_commit (); user->chr =NULL ; +qemu_mutex_destroy (&user->vhost_user_rw_lock); }@@ -2882,6 +2954,7 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev, {int ret;struct vhost_user *vu = dev->opaque; +struct VhostUserState *us = vu->user;VhostUserMsg msg = { .hdr = { .request =VHOST_USER_SET_DEVICE_STATE_FD ,@@ -2902,19 +2975,21 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg, &fd,1 ); -close (fd); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send SET_DEVICE_STATE_FD message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg, &fd,1 ); +close (fd); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send SET_DEVICE_STATE_FD message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive SET_DEVICE_STATE_FD reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive SET_DEVICE_STATE_FD reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_SET_DEVICE_STATE_FD ) {@@ -2951,6 +3026,8 @@ static int vhost_user_set_device_state_fd(struct vhost_dev *dev,static int vhost_user_check_device_state (struct vhost_dev *dev,Error **errp) { +struct vhost_user *u = dev->opaque; +struct VhostUserState *us = u->user;int ret;VhostUserMsg msg = { .hdr = {@@ -2965,18 +3042,20 @@ static int vhost_user_check_device_state(struct vhost_dev *dev, Error **errp)return -ENOTSUP ; } - ret =vhost_user_write (dev, &msg,NULL ,0 ); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to send CHECK_DEVICE_STATE message" ); -return ret; - } + WITH_QEMU_LOCK_GUARD(&us->vhost_user_rw_lock) { + ret =vhost_user_write (dev, &msg,NULL ,0 ); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to send CHECK_DEVICE_STATE message" ); +return ret; + } - ret =vhost_user_read (dev, &msg); -if (ret <0 ) { -error_setg_errno (errp, -ret, -"Failed to receive CHECK_DEVICE_STATE reply" ); -return ret; + ret =vhost_user_read (dev, &msg); +if (ret <0 ) { +error_setg_errno (errp, -ret, +"Failed to receive CHECK_DEVICE_STATE reply" ); +return ret; + } }if (msg.hdr.request !=VHOST_USER_CHECK_DEVICE_STATE ) {include/hw/virtio/vhost-user.h +0
-0
include/hw/virtio/vhost-user.h typedef struct VhostUserState { · line 73
Quoted 7 lines
include/hw/virtio/vhost-user.h +3
-0@@ -67,6 +67,9 @@ typedef struct VhostUserState {GPtrArray *notifiers;int memory_slots;bool supports_config; + +/* Hold lock for vhost_user_write/read cycle */ +QemuMutex vhost_user_rw_lock; } VhostUserState;
So it's not a rw lock. It's just a mutex. Lock should be named after what they protect, not after where they are held.
In this case, this ensures only 1 request is outstanding at a time.
So vhost_user_request_reply_lock
Quoted 4 lines
include/hw/virtio/vhost-user.h +0
-0/** Signature · 1 line
2.45.2
Re: [PATCH 1/2] vhost-user: add a write-read lock
Quoted 6 lines · Michael S. Tsirkin
So it's not a rw lock. It's just a mutex. Lock should be named after what they protect, not after where they are held.
In this case, this ensures only 1 request is outstanding at a time.
Sovhost_user_request_reply_lock
Okay, will do this change. Thank you.
---
- Prasad
PATCH 2/2 vhost: fail device start if iotlb update fails
In reply to Prasad PanditFrom: Prasad Pandit <[email protected]>
While starting a vhost device, updating iotlb entries via 'vhost_device_iotlb_miss' may return an error.
qemu-kvm: vhost_device_iotlb_miss:
700871,700871: Fail to update device iotlb
Fail device start when such an error occurs.
--- hw/virtio/vhost.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
hw/virtio/vhost.c +5 -1
@@ -2151,7 +2151,11 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice *vdev, bool vrings)* vhost-kernel code requires for this.*/ for (i =0 ; i < hdev->nvqs; ++i) {struct vhost_virtqueue *vq = hdev->vqs + i; -vhost_device_iotlb_miss (hdev, vq->used_phys, true); + r =vhost_device_iotlb_miss (hdev, vq->used_phys, true); +if (r) { +VHOST_OPS_DEBUG (r,"vhost_device_iotlb_miss failed" ); + goto fail_start; + } } }vhost_start_config_intr (hdev);
Signature · 1 line
2.45.2
Quoted 10 lines · Prasad Pandit
From: Prasad Pandit <[email protected]>
Hello,
* virsh(1) offers multiple options to initiate Postcopy migration:
1) virsh migrate --postcopy --postcopy-after-precopy
2) virsh migrate --postcopy + virsh migrate-postcopy
3) virsh migrate --postcopy --timeout <N> --timeout-postcopy
Quoted 6 lines
When Postcopy migration is invoked via method (2) or (3) above, the guest on the destination host seems to hang or get stuck sometimes.
* During Postcopy migration, multiple threads are spawned on the destination host to start the guest and setup devices. One such thread starts vhost device via
vhost_dev_start()function and another calledfault_threadhandles
Hmm, I thought it was one of the vcpu threads that invoked vhost_dev_start(), rather than any migration thread?
Quoted 38 lines
page faults in user space using kernel's userfaultfd(2) system.
When
fault_threadexits upon completion of Postcopy migration, it sends a 'postcopy_end' message to the vhost-user device. But sometimes 'postcopy_end' message is sent while vhost device is being setup viavhost_dev_start().Thread-1 Thread-2
vhost_dev_startpostcopy_ram_incoming_cleanup
vhost_device_iotlb_misspostcopy_notify
vhost_backend_update_device_iotlbvhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msgvhost_user_postcopy_end
process_message_replyprocess_message_reply
vhost_user_readvhost_user_read
vhost_user_read_headervhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply topostcopy_end"This creates confusion when vhost device receives '
postcopy_end' message while it is still trying to update IOTLB entries.This seems to leave the guest in a stranded/hung state because
fault_thread
has exited saying Postcopy migration has ended, but vhost-device is probably
still expecting updates. QEMU logs following errors on the destination host
===
...
qemu-kvm:vhost_user_read_header: 700871,700871: Failed to read msg header. Flags0x0instead of0x5.
qemu-kvm:vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm:vhost_user_postcopy_end: 700871,700900: Failed to receive reply topostcopy_end
qemu-kvm:vhost_user_read_header: 700871,700871: Failed to read msg header. Flags0x0instead of0x5.
qemu-kvm:vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm:vhost_user_read_header: 700871,700871: Failed to read msg header. Flags0x8instead of0x5.
qemu-kvm:vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm:vhost_user_read_header: 700871,700871: Failed to read msg header. Flags0x16instead of0x5.
qemu-kvm:vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
qemu-kvm:vhost_user_read_header: 700871,700871: Failed to read msg header. Flags0x0instead of0x5.
qemu-kvm:vhost_device_iotlb_miss: 700871,700871: Fail to update device iotlb
===
Quoted 1 line
* Couple of patches here help to fix/handle these errors.
I remember after you added the rwlock, there's still a hang issue.
Did you investigated that? Or do you mean this series will fix all the problems?
Thanks,
Quoted 15 lines
Thank you.
---
Prasad Pandit (2):
vhost-user: add a write-read lock
vhost: fail device start if iotlb update failshw/virtio/vhost-user.c | 423 +++++++++++++++++++-------------- hw/virtio/vhost.c | 6 +- include/hw/virtio/vhost-user.h | 3 + 3 files changed, 259 insertions(+), 173 deletions(-)--
2.45.2
Signature · 1 line
Peter Xu
Quoted 2 lines · Peter Xu
Hmm, I thought it was one of the vcpu threads that invoked
vhost_dev_start(), rather than any migration thread?
[QEMU=vhost-user-front-end] <===========> [QEMU=vhost-user-front-end]
^
|
|
|
|
|
|
V[external-process=vhost-user-back-end]
[external-process=vhost-user-back-end]
===
vhost-user-protocol:
-> https://www.qemu.org/docs/master/interop/vhost-user.html#vhost-user-proto
* It is not clear which thread calls vhost_dev_start() routine, it could be a vCPU thread. Sending 'postcopy_end' message to the 'vhost-user-back-end', hints that the device was being migrated and migration finished before the device set-up was done. The protocol above says
"...The nature of the channel is implementation-defined, but it
must generally behave like a pipe: The writing end will write all the data it has into it, signalling the end of data by closing its end. The reading end must read all of this data (until encountering the end of file) and process it."
* It does not mention sending the 'postcopy_end' message. But it talks about the front-end sending 'VHOST_USER_CHECK_DEVICE_STATE' to the back-end to check if the migration of the device state was successful or not.
Quoted 2 lines
I remember after you added the rwlock, there's still a hang issue. Did you investigated that? Or do you mean this series will fix all the problems?
* No, this series does not fix the guest hang issue. Root cause of that is still a mystery. If migration is ending abruptly before all of the guest state is migrated, the guest hang scenario seems possible. Adding vhost-user-rw-lock does not address the issue of end of migration.
* From the protocol page above, it is not clear if the front-end should allow/have multiple threads talking to the same vhost-user device.
Thank you.
---
- Prasad
Quoted 8 lines · Prasad Pandit
Quoted 2 lines
I remember after you added the rwlock, there's still a hang issue. Did you investigated that? Or do you mean this series will fix all the problems?
* No, this series does not fix the guest hang issue. Root cause of that is still a mystery. If migration is ending abruptly before all of the guest state is migrated, the guest hang scenario seems possible. Adding vhost-user-rw-lock does not address the issue of end of migration.
IMHO it's better we debug and fix all the issues before merging this one, otherwise we may overlook something. You could pass over the patch to whoever going to debug this, so it will be included in the whole set to be posted when the bug is completely fixed.
Quoted 3 lines
* From the protocol page above, it is not clear if the front-end should allow/have multiple threads talking to the same vhost-user device.
The protocol should have no restriction on the thread model of a front-end. It only describes the wire protocol.
IIUC the protocol was designed to be serialized by nature (where there's no request ID, so we can't match reply to any of the previous response), then the front-end can manage the threads well to serialize all the requests, like using this rwlock.
Thanks,
Signature · 1 line
Peter Xu
Hello Peter,
Quoted 2 lines · Peter Xu
IMHO it's better we debug and fix all the issues before merging this one, otherwise we may overlook something.
* Well we don't know where the issue is, not sure where the fix may go in, ex. if the issue turns out to be how virsh(1) invokes migrate-postcopy, fix may go in virsh(1). Patches in this series anyway don't help to fix the migration convergence issue, so they could be reviewed independently I guess.
Quoted 2 lines
You could pass over the patch to whoever going to debug this, so it will be included in the whole set to be posted when the bug is completely fixed.
* Yes, this patch series is linked there.
Quoted 7 lines
The protocol should have no restriction on the thread model of a front-end. It only describes the wire protocol.
IIUC the protocol was designed to be serialized by nature (where there's no request ID, so we can't match reply to any of the previous response), then the front-end can manage the threads well to serialize all the requests, like using this rwlock.
* I see, okay. The simple protocol definition seems to indicate that it is meant for one front-end/back-end pair. If we are dividing the front-end across multiple threads, maybe we need a document to describe those threads and how they work, at least for the QEMU (front-end) side. Because the back-end could be a non-QEMU process, we can not do much there. (just thinking)
Thank you.
---
- Prasad
Quoted 11 lines · Prasad Pandit
Hello Peter,
Quoted 2 lines · Peter Xu
IMHO it's better we debug and fix all the issues before merging this one, otherwise we may overlook something.
* Well we don't know where the issue is, not sure where the fix may go in, ex. if the issue turns out to be how virsh(1) invokes migrate-postcopy, fix may go in virsh(1). Patches in this series anyway don't help to fix the migration convergence issue, so they could be reviewed independently I guess.
I still think we should find a complete solution before merging anything, because I'm not 100% confident the issue to be further investigated is irrelevant to this patch.
No strong opinions, I'll leave that to Michael to decide.
Quoted 14 lines
Quoted 3 lines
You could pass over the patch to whoever going to debug this, so it will be included in the whole set to be posted when the bug is completely fixed.
* Yes, this patch series is linked there.
Quoted 7 lines
The protocol should have no restriction on the thread model of a front-end. It only describes the wire protocol.
IIUC the protocol was designed to be serialized by nature (where there's no request ID, so we can't match reply to any of the previous response), then the front-end can manage the threads well to serialize all the requests, like using this rwlock.
Quoted 6 lines
* I see, okay. The simple protocol definition seems to indicate that it is meant for one front-end/back-end pair. If we are dividing the front-end across multiple threads, maybe we need a document to describe those threads and how they work, at least for the QEMU (front-end) side. Because the back-end could be a non-QEMU process, we can not do much there. (just thinking)
IMHO that's not part of the protocol but impl details, so the current doc looks all fine to me.
Thanks,
Signature · 1 line
Peter Xu
Quoted 40 lines · Peter Xu
Quoted 11 lines · Prasad Pandit
Hello Peter,
Quoted 2 lines · Peter Xu
IMHO it's better we debug and fix all the issues before merging this one, otherwise we may overlook something.
* Well we don't know where the issue is, not sure where the fix may go in, ex. if the issue turns out to be how virsh(1) invokes migrate-postcopy, fix may go in virsh(1). Patches in this series anyway don't help to fix the migration convergence issue, so they could be reviewed independently I guess.
I still think we should find a complete solution before merging anything, because I'm not 100% confident the issue to be further investigated is irrelevant to this patch.
No strong opinions, I'll leave that to Michael to decide.
Quoted 14 lines
Quoted 3 lines
You could pass over the patch to whoever going to debug this, so it will be included in the whole set to be posted when the bug is completely fixed.
* Yes, this patch series is linked there.
Quoted 7 lines
The protocol should have no restriction on the thread model of a front-end. It only describes the wire protocol.
IIUC the protocol was designed to be serialized by nature (where there's no request ID, so we can't match reply to any of the previous response), then the front-end can manage the threads well to serialize all the requests, like using this rwlock.
Quoted 6 lines
* I see, okay. The simple protocol definition seems to indicate that it is meant for one front-end/back-end pair. If we are dividing the front-end across multiple threads, maybe we need a document to describe those threads and how they work, at least for the QEMU (front-end) side. Because the back-end could be a non-QEMU process, we can not do much there. (just thinking)
Quoted 7 lines
IMHO that's not part of the protocol but impl details, so the current doc looks all fine to me.
Thanks,
Signature · 1 line
Peter Xu
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Signature · 1 line
MST
Hi, Michael,
Quoted 5 lines · Michael S. Tsirkin
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Prasad mentioned how the race happened in the cover letter:
https://lore.kernel.org/r/[email protected]
Thread-1 Thread-2
vhost_dev_start postcopy_ram_incoming_cleanup
vhost_device_iotlb_miss postcopy_notify
vhost_backend_update_device_iotlb vhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msg vhost_user_postcopy_end
process_message_reply process_message_reply
vhost_user_read vhost_user_read
vhost_user_read_header vhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply to postcopy_end"
The normal case should be that thread-2 is postcopy_ram_listen_thread(), and this happens when postcopy migration is close to the end.
Thanks,
Signature · 1 line
Peter Xu
Quoted 24 lines · Peter Xu
Hi, Michael,
Quoted 5 lines · Michael S. Tsirkin
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Prasad mentioned how the race happened in the cover letter:
https://lore.kernel.org/r/[email protected]
Thread-1 Thread-2
vhost_dev_startpostcopy_ram_incoming_cleanup
vhost_device_iotlb_misspostcopy_notify
vhost_backend_update_device_iotlbvhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msgvhost_user_postcopy_end
process_message_replyprocess_message_reply
vhost_user_readvhost_user_read
vhost_user_read_headervhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply topostcopy_end"
Quoted 7 lines
The normal case should be that thread-2 is
postcopy_ram_listen_thread(), and this happens when postcopy migration is close to the end.Thanks,
Signature · 1 line
Peter Xu
OK, so postcopy_ram_ things run without the BQL?
Signature · 1 line
MST
Quoted 28 lines · Michael S. Tsirkin
Quoted 27 lines · Peter Xu
Hi, Michael,
Quoted 5 lines · Michael S. Tsirkin
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Prasad mentioned how the race happened in the cover letter:
https://lore.kernel.org/r/[email protected]
Thread-1 Thread-2
vhost_dev_startpostcopy_ram_incoming_cleanup
vhost_device_iotlb_misspostcopy_notify
vhost_backend_update_device_iotlbvhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msgvhost_user_postcopy_end
process_message_replyprocess_message_reply
vhost_user_readvhost_user_read
vhost_user_read_headervhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply topostcopy_end"The normal case should be that thread-2 is
postcopy_ram_listen_thread(), and this happens when postcopy migration is close to the end.
Quoted 7 lines
Quoted 4 lines
Thanks,
Signature · 1 line
Peter XuOK, so
postcopy_ram_things run without the BQL?
There are a lot of postcopy_ram_* functions, I didn't check all of them but I think it's true in this case. Thanks.
Signature · 1 line
Peter Xu
Quoted 31 lines · Peter Xu
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Peter Xu
Hi, Michael,
Quoted 5 lines · Michael S. Tsirkin
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Prasad mentioned how the race happened in the cover letter:
https://lore.kernel.org/r/[email protected]
Thread-1 Thread-2
vhost_dev_startpostcopy_ram_incoming_cleanup
vhost_device_iotlb_misspostcopy_notify
vhost_backend_update_device_iotlbvhost_user_postcopy_notifier
vhost_user_send_device_iotlb_msgvhost_user_postcopy_end
process_message_replyprocess_message_reply
vhost_user_readvhost_user_read
vhost_user_read_headervhost_user_read_header
"Fail to update device iotlb" "Failed to receive reply topostcopy_end"The normal case should be that thread-2 is
postcopy_ram_listen_thread(), and this happens when postcopy migration is close to the end.Thanks,
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
Signature · 1 line
Peter XuOK, so
postcopy_ram_things run without the BQL?There are a lot of postcopy_ram_* functions, I didn't check all of them but I think it's true in this case. Thanks.
Signature · 1 line
Peter Xu
So pls document this in the commit log.
Quoted 7 lines · Michael S. Tsirkin
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
OK, so
postcopy_ram_things run without the BQL?There are a lot of postcopy_ram_* functions, I didn't check all of them but I think it's true in this case. Thanks.
So pls document this in the commit log.
* ie. IIUC, if we take BQL in postcop_ram_* functions, we don't need this 'vhost_user_request_reply_lock patch'? I'll check the postcopy_ram_* functions to see what's happening there.
Thank you.
---
- Prasad
Quoted 10 lines · Prasad Pandit
Quoted 9 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 8 lines
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
OK, so
postcopy_ram_things run without the BQL?
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
There are a lot of postcopy_ram_* functions, I didn't check all of them but I think it's true in this case. Thanks.
So pls document this in the commit log.
* ie. IIUC, if we take BQL in postcop_ram_* functions, we don't need this '
vhost_user_request_reply_lockpatch'? I'll check the postcopy_ram_* functions to see what's happening there.
Go ahead, Prasad. But just to mention postcopy stuff doesn't take BQL may be because there's some good reason. So if you're not confident on some changes, you can share a patch before testing in an involved environment.
Personally, I'd avoid using BQL as long as it can be replaced by a finer grained lock. Otherwise the lock semantics can be more unclear, and we'll never get rid of BQL. IOW, I'd prefer a smaller lock in general if possible to avoid BQL, but we can see how the patch look like when it's ready.
Thanks,
Signature · 1 line
Peter Xu
Hi,
Quoted 11 lines · Prasad Pandit
Quoted 8 lines · Michael S. Tsirkin
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines · Michael S. Tsirkin
Quoted 5 lines
I just want to understand how we managed to have two threads talking in parallel. BQL is normally enough, which path manages to invoke vhost-user with BQL not taken? Just check BQL taken on each vhost user invocation and you will figure it out.
OK, so
postcopy_ram_things run without the BQL?I'll check the postcopy_ram_* functions to see what's happening there.
===
2024-07-23T17:11:25.934756Z qemu-kvm: vhost_user_postcopy_end: 161184:161213: BQL not taken 2024-07-23T17:11:25.934994Z qemu-kvm: vhost_user_postcopy_end: 161184:161213: BQL not taken 2024-07-23T17:11:25.935095Z qemu-kvm: vhost_user_postcopy_end:
161184:161213: BQL not taken
===
* postcopy_ram_listen_thread() does not take the BQL. Trying to see where to take it.
Thank you.
---
- Prasad