vhost: fix wrong IOTLB initialization

PATCH In review Chenbo Xia 5 participants 11 messages May 13, 2021 – May 17, 2021

Share & tools

Chenbo Xia

author

2021-05-13 12:38 UTC #0 lore raw

PATCH 1/1 vhost: fix wrong IOTLB initialization

This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.

iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.

Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")

Signed-off-by Chenbo Xia <[email protected]>

---
 lib/vhost/vhost_user.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

lib/vhost/vhost_user.c +9 -4

index 611ff209e3..ae4df8eb69 100644
@@ -311,6 +311,7 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
 	uint64_t features = msg->payload.u64;
 	uint64_t vhost_features = 0;
 	struct rte_vdpa_device *vdpa_dev;
+	uint32_t i;
 
 	if (validate_msg_fds(msg, 0) != 0)
 		return RTE_VHOST_MSG_RESULT_ERR;

@@ -389,6 +390,14 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
 		vdpa_dev->ops->set_features(dev->vid);
 
 	dev->flags &= ~VIRTIO_DEV_FEATURES_FAILED;
+
+	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+		for (i = 0; i < dev->nr_vring; i++) {
+			if (vhost_user_iotlb_init(dev, i))
+				return RTE_VHOST_MSG_RESULT_ERR;
+		}
+	}
+
 	return RTE_VHOST_MSG_RESULT_OK;
 }

@@ -469,10 +478,6 @@ vhost_user_set_vring_num(struct virtio_net **pdev,
 		return RTE_VHOST_MSG_RESULT_ERR;
 	}
 
-	if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
-		if (vhost_user_iotlb_init(dev, msg->payload.state.index))
-			return RTE_VHOST_MSG_RESULT_ERR;
-	}
 	return RTE_VHOST_MSG_RESULT_OK;
 }

Signature · 1 line

2.17.1

Ferruh Yigit

2021-05-13 13:10 UTC #1 lore raw

Quoted 11 lines · Chenbo Xia

This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.
Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")
Signed-off-by Chenbo Xia <[email protected]>

Hi Maxime, David,

Can Red Hat QA verify this patch please?

Btw, the issue is for defect:
https://bugs.dpdk.org/show_bug.cgi?id=703

Kevin Traynor

2021-05-13 13:26 UTC #2 lore raw

Quoted 9 lines · Ferruh Yigit

Quoted 8 lines · Chenbo Xia
This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.

Quoted 9 lines

Quoted 3 lines
Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")
Signed-off-by Chenbo Xia <[email protected]>
Hi Maxime, David,
Can Red Hat QA verify this patch please?

I've requested for QA to test RC3+this patch.

Quoted 3 lines

Btw, the issue is for defect:
https://bugs.dpdk.org/show_bug.cgi?id=703

David Marchand

2021-05-13 14:12 UTC #3 lore raw

In reply to Chenbo Xia

Quoted 8 lines · Chenbo Xia

This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.

This makes the problem I reproduced disappear at init, but I noticed the segfault after restarting testpmd once.
And a little bit after this, my vm crashed.

This is not systematic, so I guess there is some condition with how the virtio device is initialised in the vm.

One question below.

Bugzilla ID: 703

Quoted 2 lines

Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")

Reported-by Pei Zhang <[email protected]>

Quoted 18 lines

Signed-off-by Chenbo Xia <[email protected]>

---
 lib/vhost/vhost_user.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

lib/vhost/vhost_user.c +1 -0

index 611ff209e3..ae4df8eb69 100644
@@ -311,6 +311,7 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
        uint64_t features = msg->payload.u64;
        uint64_t vhost_features = 0;
        struct rte_vdpa_device *vdpa_dev;
+       uint32_t i;
 
        if (validate_msg_fds(msg, 0) != 0)
                return RTE_VHOST_MSG_RESULT_ERR;

lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg, · line 395

Quoted 6 lines

lib/vhost/vhost_user.c +3 -0

@@ -389,6 +390,14 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
                vdpa_dev->ops->set_features(dev->vid);
 
        dev->flags &= ~VIRTIO_DEV_FEATURES_FAILED;
+
+       if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+               for (i = 0; i < dev->nr_vring; i++) {

I don't know the vhost-user protocol.
At this point of the device init/life, are we sure nr_vring is set to the max number of vring?
The logs I have tend to say it is the case, but is there a guarantee in the protocol?

Another way to fix would be to allocate on the first VHOST_USER_IOTLB_MSG message received for a vring.

Quoted 8 lines

lib/vhost/vhost_user.c +5 -0

+                       if (vhost_user_iotlb_init(dev, i))
+                               return RTE_VHOST_MSG_RESULT_ERR;
+               }
+       }
+
        return RTE_VHOST_MSG_RESULT_OK;
 }

Signature · 1 line

David Marchand

Kevin Traynor

2021-05-13 14:39 UTC #4 lore raw

Quoted 10 lines · David Marchand

Quoted 8 lines · Chenbo Xia
This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.

Quoted 7 lines

This makes the problem I reproduced disappear at init, but I noticed the segfault after restarting testpmd once.
And a little bit after this, my vm crashed.
This is not systematic, so I guess there is some condition with how the virtio device is initialised in the vm.

Ok, no point in Red Hat QA testing RC3 yet, if it is still faulty.

fyi - if you want to fix with a new patch it will likely delay Red Hat QA testing RC3 (maybe others?) and probably they will only have cycles for one RC3 test run.

If you choose to revert, we can ask Red Hat QA to test RC3 without further delay. Please let us know when you consider the options.

Quoted 57 lines

One question below.
Bugzilla ID: 703
Quoted 2 lines
Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")
Reported-by Pei Zhang <[email protected]>
Quoted 18 lines
Signed-off-by Chenbo Xia <[email protected]>
---
 lib/vhost/vhost_user.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
lib/vhost/vhost_user.c +1 -0
index 611ff209e3..ae4df8eb69 100644
@@ -311,6 +311,7 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
        uint64_t features = msg->payload.u64;
        uint64_t vhost_features = 0;
        struct rte_vdpa_device *vdpa_dev;
+       uint32_t i;
 
        if (validate_msg_fds(msg, 0) != 0)
                return RTE_VHOST_MSG_RESULT_ERR;
lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg, · line 395
Quoted 6 lines
lib/vhost/vhost_user.c +3 -0
@@ -389,6 +390,14 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
                vdpa_dev->ops->set_features(dev->vid);
 
        dev->flags &= ~VIRTIO_DEV_FEATURES_FAILED;
+
+       if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+               for (i = 0; i < dev->nr_vring; i++) {
I don't know the vhost-user protocol.
At this point of the device init/life, are we sure nr_vring is set to the max number of vring?
The logs I have tend to say it is the case, but is there a guarantee in the protocol?
Another way to fix would be to allocate on the first VHOST_USER_IOTLB_MSG message received for a vring.
Quoted 8 lines
lib/vhost/vhost_user.c +5 -0
+                       if (vhost_user_iotlb_init(dev, i))
+                               return RTE_VHOST_MSG_RESULT_ERR;
+               }
+       }
+
        return RTE_VHOST_MSG_RESULT_OK;
 }
 

Ferruh Yigit

2021-05-13 15:03 UTC #5 lore raw

Quoted 19 lines · Kevin Traynor

Quoted 10 lines · David Marchand
Quoted 8 lines · Chenbo Xia
This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.
Quoted 7 lines
This makes the problem I reproduced disappear at init, but I noticed the segfault after restarting testpmd once.
And a little bit after this, my vm crashed.
This is not systematic, so I guess there is some condition with how the virtio device is initialised in the vm.

Quoted 9 lines

Ok, no point in Red Hat QA testing RC3 yet, if it is still faulty.
fyi - if you want to fix with a new patch it will likely delay Red Hat QA testing RC3 (maybe others?) and probably they will only have cycles for one RC3 test run.
If you choose to revert, we can ask Red Hat QA to test RC3 without further delay. Please let us know when you consider the options.

If the patch is not good to go as it is I suggest reverting it, as far as I know Chenbo will be off for Friday & Monday, so it doesn't leave much time to update/test a new version.

Quoted 58 lines

Quoted 57 lines
One question below.
Bugzilla ID: 703
Quoted 2 lines
Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")
Reported-by Pei Zhang <[email protected]>
Quoted 18 lines
Signed-off-by Chenbo Xia <[email protected]>
---
 lib/vhost/vhost_user.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
lib/vhost/vhost_user.c +1 -0
index 611ff209e3..ae4df8eb69 100644
@@ -311,6 +311,7 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
        uint64_t features = msg->payload.u64;
        uint64_t vhost_features = 0;
        struct rte_vdpa_device *vdpa_dev;
+       uint32_t i;
 
        if (validate_msg_fds(msg, 0) != 0)
                return RTE_VHOST_MSG_RESULT_ERR;
lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg, · line 395
Quoted 6 lines
lib/vhost/vhost_user.c +3 -0
@@ -389,6 +390,14 @@ vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg,
                vdpa_dev->ops->set_features(dev->vid);
 
        dev->flags &= ~VIRTIO_DEV_FEATURES_FAILED;
+
+       if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+               for (i = 0; i < dev->nr_vring; i++) {
I don't know the vhost-user protocol.
At this point of the device init/life, are we sure nr_vring is set to the max number of vring?
The logs I have tend to say it is the case, but is there a guarantee in the protocol?
Another way to fix would be to allocate on the first VHOST_USER_IOTLB_MSG message received for a vring.
Quoted 8 lines
lib/vhost/vhost_user.c +5 -0
+                       if (vhost_user_iotlb_init(dev, i))
+                               return RTE_VHOST_MSG_RESULT_ERR;
+               }
+       }
+
        return RTE_VHOST_MSG_RESULT_OK;
 }
 

David Marchand

2021-05-14 08:18 UTC #6 lore raw

Quoted 7 lines · Ferruh Yigit

Quoted 3 lines
If you choose to revert, we can ask Red Hat QA to test RC3 without further delay. Please let us know when you consider the options.
If the patch is not good to go as it is I suggest reverting it, as far as I know Chenbo will be off for Friday & Monday, so it doesn't leave much time to update/test a new version.

Yes, reverting is the safer, and the author proposed the same.

Signature · 1 line

David Marchand

Ferruh Yigit

2021-05-14 09:09 UTC #7 lore raw

Quoted 11 lines · David Marchand

Quoted 7 lines · Ferruh Yigit
Quoted 3 lines
If you choose to revert, we can ask Red Hat QA to test RC3 without further delay. Please let us know when you consider the options.
If the patch is not good to go as it is I suggest reverting it, as far as I know Chenbo will be off for Friday & Monday, so it doesn't leave much time to update/test a new version.
Yes, reverting is the safer, and the author proposed the same.

ack. Chenbo mentioned this is an optimization, so it should be OK to revert.

Thomas,

Can you revert the patch [1] in the main repo, or do you prefer a patch for it?

[1]
968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")

Thomas Monjalon

2021-05-14 09:25 UTC #8 lore raw

14/05/2021 11:09, Ferruh Yigit:

Quoted 16 lines · David Marchand

Quoted 11 lines · Ferruh Yigit
Quoted 7 lines
Quoted 3 lines
If you choose to revert, we can ask Red Hat QA to test RC3 without further delay. Please let us know when you consider the options.
If the patch is not good to go as it is I suggest reverting it, as far as I know Chenbo will be off for Friday & Monday, so it doesn't leave much time to update/test a new version.
Yes, reverting is the safer, and the author proposed the same.
ack. Chenbo mentioned this is an optimization, so it should be OK to revert.

Quoted 6 lines

Thomas,
Can you revert the patch [1] in the main repo, or do you prefer a patch for it?
[1]
968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU disabled")

It would be better to have a patch with the correct explanation of the issue and few acks if possible.

David Marchand

2021-05-14 08:17 UTC #9 lore raw

In reply to David Marchand

Quoted 10 lines · David Marchand

Quoted 8 lines · Chenbo Xia
This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.

Quoted 6 lines

This makes the problem I reproduced disappear at init, but I noticed the segfault after restarting testpmd once.
And a little bit after this, my vm crashed.
This is not systematic, so I guess there is some condition with how the virtio device is initialised in the vm.

The crash is systematic (not sure what I missed yesterday, but I always get it with simple steps below).

Full logs:

# dpdk-testpmd --vdev
net_vhost0,iface=/var/lib/vhost_sockets/vhost0,client=1,iommu-support=1,queues=2
-w 0:0:0.0 --log-level=lib.vhost.config:debug -- -ia --rxq=2
VHOST_CONFIG: vhost-user client: socket created, fd: 31
VHOST_CONFIG: failed to connect to /var/lib/vhost_sockets/vhost0: No
such file or directory
VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: reconnecting...

# start vm (with virtio device in the guest OS bound to kernel kmod,
i.e. no special configuration)

# testpmd logs:
testpmd> VHOST_CONFIG: /var/lib/vhost_sockets/vhost0: connected
VHOST_CONFIG: new device, handle is 0
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:36
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded
and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:35
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:39
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 0 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region size: 0x80000000
     guest physical addr: 0x0
     guest virtual  addr: 0x7f2400000000
     host  virtual  addr: 0x7fff40000000
     mmap addr : 0x7fff40000000
     mmap size : 0x80000000
     mmap align: 0x40000000
     mmap off  : 0x0
VHOST_CONFIG: guest memory region size: 0x180000000
     guest physical addr: 0x100000000
     guest virtual  addr: 0x7f2480000000
     host  virtual  addr: 0x7ffdc0000000
     mmap addr : 0x7ffd40000000
     mmap size : 0x200000000
     mmap align: 0x40000000
     mmap off  : 0x80000000
VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:42
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:43
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:36
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:44
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_100873_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:2 file:37
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:45
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:3 file:35
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:46
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map avail ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map avail ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d886000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d887000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d887240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d89e000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d89f000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d89f240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1d8da000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1d8db000
VHOST_CONFIG: (0) mapped address used: 0x7fff1d8db240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.

Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) mapped address desc: 0x7fff1e198000
VHOST_CONFIG: (0) mapped address avail: 0x7fff1e199000
VHOST_CONFIG: (0) mapped address used: 0x7fff1e199240
VHOST_CONFIG: (0) log_guest_addr: 0
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.

Port 0: queue state event
VHOST_CONFIG: virtio is now ready for processing.

Port 0: link state change event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 2
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.

Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 3
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.

Port 0: queue state event
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.

So far, everything looks good.
Now I quit testpmd.

VHOST_CONFIG: free connfd = 31 for device '/var/lib/vhost_sockets/vhost0'

And I restart testpmd with the same command as above:

VHOST_CONFIG: vhost-user client: socket created, fd: 31
VHOST_CONFIG: new device, handle is 0
Port 0: 56:48:4F:53:54:00
Checking link statuses...
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
VHOST_CONFIG: Processing VHOST_USER_GET_QUEUE_NUM succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_OWNER
VHOST_CONFIG: Processing VHOST_USER_SET_OWNER succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:36
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_PROTOCOL_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
VHOST_CONFIG: Processing VHOST_USER_SET_PROTOCOL_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
VHOST_CONFIG: Processing VHOST_USER_SET_SLAVE_REQ_FD succeeded.
VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
VHOST_CONFIG: Processing VHOST_USER_GET_FEATURES succeeded and needs reply.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:35
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:39
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x370607f83
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 on
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_0
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_1
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_2
VHOST_CONFIG: IOTLB cache name: iotlb_101328_0_3
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
VHOST_CONFIG: guest memory region size: 0x80000000

     guest physical addr: 0x0
     guest virtual  addr: 0x7f2400000000
     host  virtual  addr: 0x7fff40000000
     mmap addr : 0x7fff40000000
     mmap size : 0x80000000
     mmap align: 0x40000000
     mmap off  : 0x0
VHOST_CONFIG: guest memory region size: 0x180000000
     guest physical addr: 0x100000000
     guest virtual  addr: 0x7f2480000000
     host  virtual  addr: 0x7ffdc0000000
     mmap addr : 0x7ffd40000000
     mmap size : 0x200000000
     mmap align: 0x40000000
     mmap off  : 0x80000000

VHOST_CONFIG: Processing VHOST_USER_SET_MEM_TABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:0 file:42
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:0 file:43
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:1 file:36
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:1 file:44
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_IOTLB_MSG succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 0
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
VHOST_CONFIG: set queue enable: 1 to qp idx: 1
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ENABLE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
VHOST_CONFIG: negotiated Virtio features: 0x40000000
VHOST_CONFIG: (0) mergeable RX buffers off, virtio 1 off
VHOST_CONFIG: Processing VHOST_USER_SET_FEATURES succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:2 file:35
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:2 file:37
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_NUM succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_BASE succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_ADDR succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
VHOST_CONFIG: vring kick idx:3 file:39
VHOST_CONFIG: (0) failed to map desc ring.
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_KICK succeeded.
VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
VHOST_CONFIG: vring call idx:3 file:45
VHOST_CONFIG: Processing VHOST_USER_SET_VRING_CALL succeeded.
VHOST_CONFIG: read message VHOST_USER_IOTLB_MSG

Thread 9 "vhost-events" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffee7fc400 (LWP 101339)] 0x0000000000e30803 in rte_mempool_default_cache (lcore_id=4294967295, mp=0x0) at ../lib/mempool/rte_mempool.h:1244
1244 if (mp->cache_size == 0)
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-127.el8_3.2.x86_64 libibverbs-29.0-3.el8.x86_64 libnl3-3.5.0-1.el8.x86_64 numactl-libs-2.0.12-11.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64
(gdb) bt full
#0 0x0000000000e30803 in rte_mempool_default_cache (lcore_id=4294967295, mp=0x0) at ../lib/mempool/rte_mempool.h:1244 No locals.
#1 rte_mempool_get_bulk (n=1, obj_table=0x7fffee7f5e98, mp=0x0) at ../lib/mempool/rte_mempool.h:1533
cache = 0x0
cache = <optimized out>
#2 rte_mempool_get (obj_p=0x7fffee7f5e98, mp=0x0) at ../lib/mempool/rte_mempool.h:1561
No locals.
#3 vhost_user_iotlb_cache_insert (vq=0x1673bff00, iova=10159157248, uaddr=140733688868864, size=4096, perm=3 '\003') at ../lib/vhost/iotlb.c:164

        node = 0x1673c08c0
        new_node = 0x1673671f0
        ret = 0
#4  0x000000000100d51d in vhost_user_iotlb_msg (pdev=0x7fffee7f92b8,
msg=0x7fffee7f9010, main_fd=31) at ../lib/vhost/vhost_user.c:2409
        vq = 0x1673bff00
        dev = 0x1673c08c0
        imsg = 0x7fffee7f901c
        i = 2
        vva = 140733688868864
        len = 4096
#5  0x000000000100e31e in vhost_user_msg_handler (vid=0, fd=31) at
../lib/vhost/vhost_user.c:2882
        dev = 0x1673c08c0
        msg = {request = {master = 22, slave = 22}, flags = 9, size =
32, payload = {u64 = 10159157248, state = {index = 1569222656, num =
2}, addr = {index = 1569222656, flags = 2, desc_user_addr = 4096,
              used_user_addr = 139800607223808, avail_user_addr =
7907112783476097539, log_guest_addr = 10159256128}, memory = {nregions
= 1569222656, padding = 2, regions = {{guest_phys_addr = 4096,
                  memory_size = 139800607223808, userspace_addr =
7907112783476097539, mmap_offset = 10159256128}, {guest_phys_addr =
4294967296, memory_size = 6442450944, userspace_addr =
139794743033856,
                  mmap_offset = 2147483648}, {guest_phys_addr = 0,

memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 4294967296}, {
guest_phys_addr = 0, memory_size = 0, userspace_addr
= 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0,
userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0, memory_size = 4294967296, userspace_addr = 0, mmap_offset = 3419188017980506112}}}, log = {mmap_size = 10159157248,

              mmap_offset = 4096}, iotlb = {iova = 10159157248, size =
4096, uaddr = 139800607223808, perm = 3 '\003', type = 2 '\002'},
crypto_session = {session_id = 10159157248, op_code = 4096,
              cipher_algo = 0, cipher_key_len = 3716706304, hash_algo
= 32549, digest_len = 3523936771, auth_key_len = 1841018158, aad_len =
1569321536, op_type = 2 '\002', dir = 0 '\000',
              hash_mode = 0 '\000', chaining_dir = 0 '\000', ciphe_key

= 0x100000000 "\356o.\001\003", auth_key = 0x180000000 "",
cipher_key_buf =
"\000\000\000\200$\177\000\000\000\000\000\200", '\000' <repeats 51 times>,

              auth_key_buf = '\000' <repeats 12 times>, "\001", '\000'
<repeats 111 times>, "\001", '\000' <repeats 15 times>,
"ces/system/node", '\000' <repeats 209 times>...}, area = {u64 =
10159157248,
              size = 4096, offset = 139800607223808}, inflight =

{mmap_size = 10159157248, mmap_offset = 4096, num_queues = 28672, queue_size = 56712}}, fds = {-1, -1, -1, -1, -1, -1, -1, -1}, fd_num = 0}
vdpa_dev = 0x0
ret = 0
unlock_required = 0
handled = false
request = 22
i = 4
#6 0x0000000000e41f75 in vhost_user_read_cb (connfd=31, dat=0x6e28510, remove=0x7fffee7f9394) at ../lib/vhost/socket.c:309

        conn = 0x6e28510
        vsocket = 0x6e1f140
        ret = 1
#7  0x0000000000e0f9c4 in fdset_event_dispatch (arg=0x66f9a60
<vhost_user+8192>) at ../lib/vhost/fd_man.c:286
        i = 1
        pfd = 0x66f9a68 <vhost_user+8200>
        pfdentry = 0x66fba88 <vhost_user+16424>
        rcb = 0xe41f3c <vhost_user_read_cb>
        wcb = 0x0
        dat = 0x6e28510
        fd = 31
        numfds = 2

...

And qemu crashes right after this:

(gdb) bt full
#0 0x000056269a497160 in vhost_device_iotlb_miss (dev=dev@entry=0x56269d6ed800, iova=10159497216, write=<optimized out>)

    at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost.c:944
        iotlb = <optimized out>
        uaddr = <optimized out>
        len = <optimized out>
        ret = -14
        _rcu_read_auto = 0x1

#1 0x000056269a499361 in vhost_backend_handle_iotlb_msg
(imsg=0x7ffd5eddeea0, dev=0x56269d6ed800) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:351
ret = <optimized out>
ret = <optimized out>
#2 vhost_backend_handle_iotlb_msg (dev=dev@entry=0x56269d6ed800, imsg=imsg@entry=0x7ffd5eddeea0) at /usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-backend.c:344

        ret = 0
#3  0x000056269a499f0b in slave_read (opaque=0x56269d6ed800) at
/usr/src/debug/qemu-kvm-4.2.0-34.module+el8.3.0+10437+1ca0c2ba.5.x86_64/hw/virtio/vhost-user.c:1048
        dev = 0x56269d6ed800
        u = 0x56269cd3f7b0
        hdr = {request = VHOST_USER_GET_FEATURES, flags = 1, size = 32}
        payload = {u64 = 10159497216, state = {index = 1569562624, num
= 2}, addr = {index = 1569562624, flags = 2, desc_user_addr = 0,
used_user_addr = 0, avail_user_addr = 259, log_guest_addr = 0}, memory
= {
            nregions = 1569562624, padding = 2, regions =

{{guest_phys_addr = 0, memory_size = 0, userspace_addr = 259, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0,
mmap_offset = 0}, {guest_phys_addr = 0, memory_size =
0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0,
memory_size = 0, userspace_addr = 0, mmap_offset = 0},
{guest_phys_addr = 0, memory_size = 0, userspace_addr = 0, mmap_offset = 0}, {guest_phys_addr = 0, memory_size = 0, userspace_addr = 0,
mmap_offset = 0}, {guest_phys_addr = 0, memory_size =
0, userspace_addr = 0, mmap_offset = 0}}}, log = {mmap_size = 10159497216, mmap_offset = 0}, iotlb = {iova = 10159497216, size = 0,

            uaddr = 0, perm = 3 '\003', type = 1 '\001'}, config =
{offset = 1569562624, size = 2, flags = 0, region = '\000' <repeats 12
times>, "\003\001", '\000' <repeats 241 times>}, session = {
            session_id = 10159497216, session_setup_data = {op_code =
0, cipher_alg = 0, key_len = 0, hash_alg = 0, hash_result_len = 259,
auth_key_len = 0, add_len = 0, op_type = 0 '\000',
              direction = 0 '\000', hash_mode = 0 '\000',
alg_chain_order = 0 '\000', cipher_key = 0x0, auth_key = 0x0}, key =
'\000' <repeats 63 times>, auth_key = '\000' <repeats 511 times>},
area = {
            u64 = 10159497216, size = 0, offset = 0}, inflight =
{mmap_size = 10159497216, mmap_offset = 0, num_queues = 0, queue_size
= 0}}
        size = <optimized out>
        ret = 0
        iov = {iov_base = 0x7ffd5eddee04, iov_len = 12}
        msgh = {msg_name = 0x0, msg_namelen = 0, msg_iov =
0x7ffd5eddee10, msg_iovlen = 1, msg_control = 0x7ffd5eddf120,
msg_controllen = 0, msg_flags = 0}
        fd = {-1, -1, -1, -1, -1, -1, -1, -1}
        control =

"^\a\022\000\000\000\000\000\226\070\270\t\000\000\000\000\334\345\061\234&V\000\000\000x\v\322.\265\273mP\347\061\234&V\000\000\000\000\000\000\000\000\000"
cmsg = <optimized out>
i = <optimized out>
fdsize = 0
...

Signature · 1 line

David Marchand

Xia, Chenbo

author

2021-05-17 12:46 UTC #10 lore raw

In reply to David Marchand

Hi David,

Quoted 15 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Thursday, May 13, 2021 10:12 PM
To: Xia, Chenbo <[email protected]>; Maxime Coquelin <[email protected]>
Cc: dev <[email protected]>; Kevin Traynor <[email protected]>; Pei Zhang <[email protected]>; Yigit, Ferruh <[email protected]>; Thomas Monjalon <[email protected]>
Subject: Re: [PATCH] vhost: fix wrong IOTLB initialization
Quoted 4 lines · Chenbo Xia
This patch fixes an issue of application crash because of vhost iotlb not initialized when virtio has multiqueue enabled.

Quoted 8 lines

Quoted 4 lines
iotlb messages will be sent when some queues are not enabled. If we initialize iotlb in vhost_user_set_vring_num, it could happen that iotlb update comes when iotlb pool of disabled queues are not initialized.
This makes the problem I reproduced disappear at init, but I noticed the segfault after restarting testpmd once.
And a little bit after this, my vm crashed.

Oops.. Maybe there's some env difference. My env works well with the 'restart' test.

After checking the logs you provided, is the segfault still because of iotlb cache not init? IMHO, based on the message sequence, the cache should be inited.

Quoted 43 lines

This is not systematic, so I guess there is some condition with how the virtio device is initialised in the vm.
One question below.
Bugzilla ID: 703
Quoted 1 line
Fixes 968bbc7e2e50 ("vhost: avoid IOTLB mempool allocation while IOMMU
disabled")
Quoted 1 line
Reported-by Pei Zhang <[email protected]>
lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev,
Quoted 10 lines
Signed-off-by Chenbo Xia <[email protected]>
---
 lib/vhost/vhost_user.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
lib/vhost/vhost_user.c +0 -0
index 611ff209e3..ae4df8eb69 100644
@@ -311,6 +311,7 @@ vhost_user_set_features(struct virtio_net **pdev,
struct VhostUserMsg *msg,
lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev,
Quoted 8 lines
lib/vhost/vhost_user.c +1 -0
        uint64_t features = msg->payload.u64;
        uint64_t vhost_features = 0;
        struct rte_vdpa_device *vdpa_dev;
+       uint32_t i;
 
        if (validate_msg_fds(msg, 0) != 0)
                return RTE_VHOST_MSG_RESULT_ERR;
@@ -389,6 +390,14 @@ vhost_user_set_features(struct virtio_net **pdev,
struct VhostUserMsg *msg,
lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg, · line 395
Quoted 6 lines
lib/vhost/vhost_user.c +3 -0
                vdpa_dev->ops->set_features(dev->vid);
 
        dev->flags &= ~VIRTIO_DEV_FEATURES_FAILED;
+
+       if (dev->features & (1ULL << VIRTIO_F_IOMMU_PLATFORM)) {
+               for (i = 0; i < dev->nr_vring; i++) {

Quoted 5 lines

I don't know the vhost-user protocol.
At this point of the device init/life, are we sure nr_vring is set to the max number of vring?
The logs I have tend to say it is the case, but is there a guarantee in the protocol?

I think you are correct.. Based on current QEMU implementation, nr_vring should be the correct value (Correct me if there're corner cases). But I don't think there is a guarantee as vhost-user protocol doesn't mention about 'SET_FEATURES' comes after per-vring messages. @Maxime Coquelin Do I miss anything?

Quoted 4 lines

Another way to fix would be to allocate on the first VHOST_USER_IOTLB_MSG message received for a vring.

Emmm.. Could there be a case that some hypervisor init certain queue after the first IOTLB msg? If there is, we may also need to check nr_vring is not changed/there's new queue inited.

And David, thanks for testing and writing the revert patch for me during my leave. That's much appreciated!

Thanks,
Chenbo

Quoted 14 lines

lib/vhost/vhost_user.c vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg *msg, · line 403
Quoted 10 lines
lib/vhost/vhost_user.c +5 -0
+                       if (vhost_user_iotlb_init(dev, i))
+                               return RTE_VHOST_MSG_RESULT_ERR;
+               }
+       }
+
        return RTE_VHOST_MSG_RESULT_OK;
 }
 
--
David Marchand