Skip to contentSkip to conversation

vhost-user: fix the issue of vhost deadlock in nested virtualization

PATCH In review Hao Chen 3 participants 6 messages Jan 26, 2024 – Feb 21, 2024

lore
Share & tools
lore

Hao Chen

author
#0 lore raw

PATCH 1/1 vhost-user: fix the issue of vhost deadlock in nested virtualization

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")
---
 hw/virtio/vhost-user.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)
 static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                             struct vhost_iotlb_msg *imsg)
 {
-    int ret;
     VhostUserMsg msg = {
         .hdr.request = VHOST_USER_IOTLB_MSG,
         .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
         .payload.iotlb = *imsg,
     };
 
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
 }
 
 
Signature · 1 line
2.27.0
#1 lore raw
Quoted 15 lines · Hao Chen

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

Quoted 8 lines

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")

I would be very worried that IOTLB becomes stale and guest memory is corrupted if we just proceed without waiting.

Maxime what do you think? How would you address the issue?

Quoted 33 lines
---
 hw/virtio/vhost-user.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)
 static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                             struct vhost_iotlb_msg *imsg)
 {
-    int ret;
     VhostUserMsg msg = {
         .hdr.request = VHOST_USER_IOTLB_MSG,
         .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
         .payload.iotlb = *imsg,
     };
 
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
 }
 
 
Signature · 1 line
2.27.0
#2 lore raw
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Hao Chen

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Quoted 8 lines
Quoted 3 lines
Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")

I would be very worried that IOTLB becomes stale and guest memory is corrupted if we just proceed without waiting.

Maxime what do you think? How would you address the issue?

I agree with you, this is not possible.
For example, in case of IOTLB invalidate, the frontend relies on the backend reply to ensure it is no more accessing the memory before proceeding.

The reply-ack for VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG request is less important, if it fails the host notifications won't work but would not risk corruption. Maybe on Qemu side we could fail init if processing the request fails, as I think that if negotiated, we can expect it to succeed.

What do you think about this proposal?

Regards,
Maxime

Quoted 36 lines
Quoted 35 lines

---
  hw/virtio/vhost-user.c | 10 ++--------
  1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)
  static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                              struct vhost_iotlb_msg *imsg)
  {
-    int ret;
      VhostUserMsg msg = {
          .hdr.request = VHOST_USER_IOTLB_MSG,
          .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
          .payload.iotlb = *imsg,
      };
 
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
  }
 
 
Signature · 1 line
2.27.0
#3 lore raw
Quoted 44 lines · Maxime Coquelin
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Hao Chen

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Quoted 8 lines
Quoted 3 lines
Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")

I would be very worried that IOTLB becomes stale and guest memory is corrupted if we just proceed without waiting.

Maxime what do you think? How would you address the issue?

I agree with you, this is not possible.
For example, in case of IOTLB invalidate, the frontend relies on the backend reply to ensure it is no more accessing the memory before proceeding.

The reply-ack for VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG request is less important, if it fails the host notifications won't work but would not risk corruption. Maybe on Qemu side we could fail init if processing the request fails, as I think that if negotiated, we can expect it to succeed.

Quoted 4 lines

What do you think about this proposal?

Regards,
Maxime

Fundamentally, I think that if qemu blocks guest waiting for a rely that is ok but it really has to process incoming messages meanwhile. Same should apply to backend I think ...

Quoted 33 lines
Quoted 33 lines
Quoted 32 lines

---
  hw/virtio/vhost-user.c | 10 ++--------
  1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)
  static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                              struct vhost_iotlb_msg *imsg)
  {
-    int ret;
      VhostUserMsg msg = {
          .hdr.request = VHOST_USER_IOTLB_MSG,
          .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
          .payload.iotlb = *imsg,
      };
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
  }
Signature · 1 line
2.27.0
#4 lore raw
Quoted 45 lines · Michael S. Tsirkin
Quoted 44 lines · Maxime Coquelin
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Hao Chen

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Quoted 8 lines
Quoted 3 lines
Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")

I would be very worried that IOTLB becomes stale and guest memory is corrupted if we just proceed without waiting.

Maxime what do you think? How would you address the issue?

I agree with you, this is not possible.
For example, in case of IOTLB invalidate, the frontend relies on the backend reply to ensure it is no more accessing the memory before proceeding.

The reply-ack for VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG request is less important, if it fails the host notifications won't work but would not risk corruption. Maybe on Qemu side we could fail init if processing the request fails, as I think that if negotiated, we can expect it to succeed.

Quoted 8 lines
Quoted 4 lines

What do you think about this proposal?

Regards,
Maxime

Fundamentally, I think that if qemu blocks guest waiting for a rely that is ok but it really has to process incoming messages meanwhile. Same should apply to backend I think ...

I understand your point.
For DPDK Vhost library, it will likely imply ABI breakage as it would require to asynchronous handling of Vhost-user requests. We would only be able to do it at next LTS release.

Hao, as your driver is not available upstream it will be difficult to assist you more. But if you look at other DPDK vDPA driver like SFC for instance, the way they implemented host notification control should be safe against this kind of deadlocks.

Quoted 35 lines
Quoted 34 lines
Quoted 34 lines
Quoted 33 lines


---
   hw/virtio/vhost-user.c | 10 ++--------
   1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct vhost_dev *dev, uint16_t mtu)
   static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                               struct vhost_iotlb_msg *imsg)
   {
-    int ret;
       VhostUserMsg msg = {
           .hdr.request = VHOST_USER_IOTLB_MSG,
           .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
           .payload.iotlb = *imsg,
       };
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
   }
Signature · 1 line
2.27.0
author
#5 lore raw
Quoted 62 lines · Maxime Coquelin
Quoted 45 lines · Michael S. Tsirkin
Quoted 44 lines · Maxime Coquelin
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Hao Chen

I run "dpdk-vdpa" and "qemur-L2" in "qemu-L1".

In a nested virtualization environment, "qemu-L2" vhost-user socket sends a "VHOST_USER_IOTLB_MSG" message to "dpdk-vdpa" and blocks waiting for "dpdk-vdpa" to process the message.
If "dpdk-vdpa" doesn't complete the processing of the "VHOST_USER_IOTLB_MSG" message and sends a message that needs to be replied in another thread, such as "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", "dpdk-vdpa" will also block and wait for "qemu-L2" to process this message. However, "qemu-L2" vhost-user's socket is blocking while waiting for a reply from "dpdk-vdpa" after processing the message "VHOSTr_USER_IOTLB_MSG", and "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" will not be processed. In this case, both "dpdk-vdpa" and "qemu-L2" are blocked on the vhost read, resulting in a deadlock.

You can modify "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG" or "VHOST_USER_IOTLB_MSG" to "no need reply" to fix this issue. There are too many messages in dpdk that are similar to "VHOST_USER_SLAVE_VRING_HOST_NOTIFIER_MSG", and I would prefer the latter.

Quoted 8 lines
Quoted 3 lines
Fixes 24e34754eb78 ("vhost-user: factor out msg head and payload")

I would be very worried that IOTLB becomes stale and guest memory is corrupted if we just proceed without waiting.

Maxime what do you think? How would you address the issue?

I agree with you, this is not possible.
For example, in case of IOTLB invalidate, the frontend relies on the backend reply to ensure it is no more accessing the memory before proceeding.

The reply-ack for VHOST_USER_BACKEND_VRING_HOST_NOTIFIER_MSG request is less important, if it fails the host notifications won't work but would not risk corruption. Maybe on Qemu side we could fail init if processing the request fails, as I think that if negotiated, we can expect it to succeed.

Quoted 8 lines
Quoted 4 lines

What do you think about this proposal?

Regards,
Maxime

Fundamentally, I think that if qemu blocks guest waiting for a rely that is ok but it really has to process incoming messages meanwhile. Same should apply to backend I think ...

I understand your point.
For DPDK Vhost library, it will likely imply ABI breakage as it would require to asynchronous handling of Vhost-user requests. We would only be able to do it at next LTS release.

Quoted 4 lines

Hao, as your driver is not available upstream it will be difficult to assist you more. But if you look at other DPDK vDPA driver like SFC for instance, the way they implemented host notification control should be safe against this kind of deadlocks.

Okay, I can also avoid this issue by sending the "VHOST_USER_SLAVE_VRING_HOSTNOTIFIER_MSG" message as late as possible to avoid conflicts with the "VHOST-USER-IOTLB-MSG" message. In summary, thank you.

Quoted 39 lines
Quoted 38 lines
Quoted 37 lines
Quoted 37 lines
Quoted 36 lines

---
   hw/virtio/vhost-user.c | 10 ++--------

   1 file changed, 2 insertions(+), 8 deletions(-)
hw/virtio/vhost-user.c +2 -8
index f214df804b..02caa94b6c 100644
@@ -2371,20 +2371,14 @@ static int vhost_user_net_set_mtu(struct 
 vhost_dev *dev, uint16_t mtu)
    static int vhost_user_send_device_iotlb_msg(struct vhost_dev *dev,
                                                struct vhost_iotlb_msg *imsg)
    {
-    int ret;
        VhostUserMsg msg = {
            .hdr.request = VHOST_USER_IOTLB_MSG,
            .hdr.size = sizeof(msg.payload.iotlb),
-        .hdr.flags = VHOST_USER_VERSION | VHOST_USER_NEED_REPLY_MASK,
+        .hdr.flags = VHOST_USER_VERSION,
            .payload.iotlb = *imsg,
        };
-    ret = vhost_user_write(dev, &msg, NULL, 0);
-    if (ret < 0) {
-        return ret;
-    }
-
-    return process_message_reply(dev, &msg);
+    return vhost_user_write(dev, &msg, NULL, 0);
    }
Signature · 1 line
2.27.0