vq_assert_lock__ fail in vhost_user_set_vring_addr during live migration with HW vDPA

https://bugs.dpdk.org/show_bug.cgi?id=1394

Bug ID: 1394
Summary: vq_assert_lock__ fail in vhost_user_set_vring_addr
during live migration with HW vDPA
Product: DPDK
Version: unspecified
Hardware: x86
OS: Linux
Status: UNCONFIRMED
Severity: critical
Priority: Normal
Component: vhost/virtio
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---

DPDK version: v24.03-rc1

1. boot vDPA and qemu(8.1)
build/examples/dpdk-vdpa -a 5e:00.2,class=vdpa --file-prefix vf0 --log-level=.,info -- --client --iface /tmp/vfe-net

<qemu:arg value='-chardev'/>
<qemu:arg value='socket,id=charnet1,path=/tmp/vfe-net0,server'/>
<qemu:arg value='-netdev'/>
<qemu:arg value='vhost-user,chardev=charnet1,queues=2,id=hostnet1'/>
<qemu:arg value='-device'/>
<qemu:arg value='virtio-net-pci,mq=on,vectors=6,netdev=hostnet1,id=net1,mac=00:00:00:00:33:00,bus=pci.0,addr=0x8,page-per-vq=on'/>

2. live migration VM to another server
sudo virsh migrate --verbose --live --persistent gen-l-vrt-295-005-CentOS-7.4 qemu+ssh://gen-l-vrt-294/system --unsafe

3. dpdk crash

After device configured, vhost_user_lock_all_queue_pairs won't be called. The vq_assert_lock__ failed in vhost_user_set_vring_addr for vDPA case.

related to commit:
commit 741dc052eaf9459cc576b0d87e96a40069485c32 (HEAD)

Author: David Marchand [email protected]
Date:   Tue Dec 5 10:45:34 2023 +0100

vhost: annotate virtqueue access checks

Modifying vq->access_ok should be done with a write lock taken. Annotate vring_translate() and vring_invalidate().

new port /tmp/vfe-net0, device : 5e:00.2
mlx5_vdpa: MTU cannot be set on device 5e:00.2.
mlx5_vdpa: Region 0: HVA 0x7fff00000000, GPA 0x0, size 0xc0000000.
mlx5_vdpa: Region 1: HVA 0x7ffdc0000000, GPA 0x100000000, size 0x140000000.
mlx5_vdpa: Indirect mkey mode is KLM Fixed Buffer Size.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 0.
mlx5_vdpa: Virtq 0 notifier state is enabled.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 1.
mlx5_vdpa: Virtq 1 notifier state is enabled.
[New Thread 0x7fffe7b9b400 (LWP 962699)]
mlx5_vdpa: vDPA device 0 was configured.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 0
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 1
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 2
mlx5_vdpa: Update virtq 2 status disable -> enable.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 2.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: (/tmp/vfe-net0) set queue enable: 1 to qp idx: 3
mlx5_vdpa: Update virtq 3 status disable -> enable.
mlx5_vdpa: Virtq 2 notifier state is enabled.
mlx5_vdpa: vid 0: Init last_avail_idx=0, last_used_idx=0 for virtq 3.
mlx5_vdpa: Virtq 3 notifier state is enabled.
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_LOG_BASE
VHOST_CONFIG: (/tmp/vfe-net0) log mmap size: 294912, offset: 0
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: (/tmp/vfe-net0) negotiated Virtio features: 0x144601803
mlx5_vdpa: mlx5 vdpa: enabling dirty logging...
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_GET_STATUS
VHOST_CONFIG: (/tmp/vfe-net0) read message VHOST_USER_SET_VRING_ADDR
EAL: PANIC in vq_assert_lock__():
VHOST_CONFIG: (/tmp/vfe-net0) vhost_user_set_vring_addr() called without access lock taken.
0: /images/vdpa/dpdk/build/examples/dpdk-vdpa (rte_dump_stack+0x1f) [aadfca] 1: /images/vdpa/dpdk/build/examples/dpdk-vdpa (__rte_panic+0xe2) [a8032b] 2: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x3bac83) [7bac83] 3: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x3bd0ef) [7bd0ef] 4: /images/vdpa/dpdk/build/examples/dpdk-vdpa (vhost_user_msg_handler+0x508) [7c264f]
5: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x36b797) [76b797] 6: /images/vdpa/dpdk/build/examples/dpdk-vdpa (fdset_event_dispatch+0x1cd) [769793]
7: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x6970ca) [a970ca] 8: /images/vdpa/dpdk/build/examples/dpdk-vdpa (400000+0x6af41c) [aaf41c] 9: /lib64/libpthread.so.0 (7ffff6426000+0x814a) [7ffff642e14a]
10: /lib64/libc.so.6 (clone+0x43) [7ffff615ddc3]

Thread 29 "dpdk-vhost-evt" received signal SIGABRT, Aborted. [Switching to Thread 0x7fffe839c400 (LWP 962487)] __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 return ret;
Missing separate debuginfos, use: yum debuginfo-install elfutils-libelf-0.182-3.el8.x86_64 libgcc-8.4.1-1.el8.x86_64 libibverbs-2307mlnx47-1.2310007.x86_64 libnl3-3.5.0-1.el8.x86_64 libpcap-1.9.1-5.el8.x86_64 numactl-libs-2.0.12-11.el8.x86_64 openssl-libs-1.1.1k-5.el8_5.x86_64 zlib-1.2.11-17.el8.x86_64 (gdb) bt

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff6082db5 in __GI_abort () at abort.c:79
#2  0x0000000000a80330 in __rte_panic (funcname=0x3461330 <__func__.33032>
"vq_assert_lock__", 
    format=0x345eb90 "VHOST_CONFIG: (%s) %s() called without access lock
taken.\n%.0s") at ../lib/eal/common/eal_common_debug.c:26
#3  0x00000000007bac83 in vq_assert_lock__ (dev=0x17ffc2500, vq=0x17ffa2280,
func=0x3461370 <__func__.33676> "vhost_user_set_vring_addr")
    at ../lib/vhost/vhost.h:547

#4 0x00000000007bd0ef in vhost_user_set_vring_addr (pdev=0x7fffe8399270, ctx=0x7fffe8398fc0, main_fd=115) at ../lib/vhost/vhost_user.c:930 #5 0x00000000007c264f in vhost_user_msg_handler (vid=0, fd=115) at ../lib/vhost/vhost_user.c:3197
#6 0x000000000076b797 in vhost_user_read_cb (connfd=115, dat=0x7fffe0000dd0, remove=0x7fffe8399354) at ../lib/vhost/socket.c:318
#7 0x0000000000769793 in fdset_event_dispatch (arg=0x3cb3940 <vhost_user+8192>) at ../lib/vhost/fd_man.c:282
#8 0x0000000000a970ca in control_thread_start (arg=0x9a5ed30) at ../lib/eal/common/eal_common_thread.c:282
#9 0x0000000000aaf41c in thread_start_wrapper (arg=0x7fffffffe090) at ../lib/eal/unix/rte_thread.c:114
#10 0x00007ffff642e14a in start_thread (arg=<optimized out>) at pthread_create.c:479
#11 0x00007ffff615ddc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

You are receiving this mail because:
You are the assignee for the bug.