Skip to contentSkip to conversation

vdpa/ifc: add live migration for block device

PATCH Applied Andy Pei 2 participants 3 messages Dec 14, 2022 – Feb 3, 2023 virtio core

lore
Share & tools
lore

Andy Pei

author
#0 lore raw

PATCH 1/1 vdpa/ifc: add live migration for block device

When we use "sw-live-migration=1" in arguments, we run in SW assisted live migration mode.

In SW assisted live migration mode, driver will stop the device and setup a mediated virtio ring to relay the communication between the virtio driver and the VDPA device.

For block devices, we have to make sure that each IO is completed. When we terminate the above mediated virtio ring, we stop notifying queue about new IOs, wait for all the in-flight IOs to be completed.

---
 drivers/vdpa/ifc/ifcvf_vdpa.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
drivers/vdpa/ifc/ifcvf_vdpa.c +22 -0
index 49d68ad..17838ac 100644
@@ -865,8 +865,30 @@ struct rte_vdpa_dev_info {
 	struct ifcvf_hw *hw = &internal->hw;
 	uint64_t m_vring_iova = IFCVF_MEDIATED_VRING;
 	uint64_t size, len;
+	u32 ring_state = 0;
 
 	vid = internal->vid;
+
+	/* to make sure no packet is lost for blk device
+	 * do not stop until last_avail_idx == last_used_idx
+	 */
+	if (internal->hw.device_type == IFCVF_BLK) {
+		for (i = 0; i < hw->nr_vring; i++) {
+			do {
+				if (hw->lm_cfg != NULL)
+					ring_state = *(u32 *)(hw->lm_cfg +
+						IFCVF_LM_RING_STATE_OFFSET +
+						i * IFCVF_LM_CFG_SIZE);
+				hw->vring[i].last_avail_idx =
+					(u16)(ring_state & IFCVF_16_BIT_MASK);
+				hw->vring[i].last_used_idx =
+					(u16)(ring_state >> 16);
+				usleep(10);
+			} while (hw->vring[i].last_avail_idx !=
+				hw->vring[i].last_used_idx);
+		}
+	}
+
 	ifcvf_stop_hw(hw);
 
 	for (i = 0; i < hw->nr_vring; i++) {
Signature · 1 line
1.8.3.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 12 lines · Andy Pei

When we use "sw-live-migration=1" in arguments, we run in SW assisted live migration mode.

In SW assisted live migration mode, driver will stop the device and setup a mediated virtio ring to relay the communication between the virtio driver and the VDPA device.

For block devices, we have to make sure that each IO is completed. When we terminate the above mediated virtio ring, we stop notifying queue about new IOs, wait for all the in-flight IOs to be completed.

Quoted 5 lines
---
  drivers/vdpa/ifc/ifcvf_vdpa.c | 22 ++++++++++++++++++++++
  1 file changed, 22 insertions(+)

Thanks,
Maxime

#2 lore raw
In reply to Andy Pei
Quoted 12 lines · Andy Pei

When we use "sw-live-migration=1" in arguments, we run in SW assisted live migration mode.

In SW assisted live migration mode, driver will stop the device and setup a mediated virtio ring to relay the communication between the virtio driver and the VDPA device.

For block devices, we have to make sure that each IO is completed. When we terminate the above mediated virtio ring, we stop notifying queue about new IOs, wait for all the in-flight IOs to be completed.

Quoted 5 lines
---
  drivers/vdpa/ifc/ifcvf_vdpa.c | 22 ++++++++++++++++++++++
  1 file changed, 22 insertions(+)

Applied to dpdk-next-virtio/main.

Thanks,
Maxime