vdpa/sfc: resolve race between libvhost and dev_conf
PATCH 1/1 vdpa/sfc: resolve race between libvhost and dev_conf
From: Abhimanyu Saini <[email protected]>
libvhost calls dev_conf() before prosessing the VHOST_USER_SET_VRING_CALL message for the last VQ. So this message is processed after dev_conf() returns.
However, the dev_conf() function spawns a thread to set rte_vhost_host_notifier_ctrl() before returning control to libvhost. This parallel thread in turn invokes get_notify_area(). To get the notify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.
But at the same time libvhost is processing the last VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.
Hence there is a race b/w the libvhost and the vdpa driver.
To resolve this race condition, query the HW and cache notify_area inside dev_conf() instead of doing it the parallel thread.
--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
drivers/vdpa/sfc/sfc_vdpa_ops.c +18 -18
@@ -222,6 +222,7 @@ sfc_vdpa_virtq_start(struct sfc_vdpa_ops_data *ops_data, int vq_num) { int rc; +uint32_t doorbell;efx_virtio_vq_t *vq;struct sfc_vdpa_vring_info vring;efx_virtio_vq_cfg_t vq_cfg;
@@ -270,22 +271,35 @@/* Start virtqueue */ rc =efx_virtio_qstart (vq, &vq_cfg, &vq_dyncfg);if (rc !=0 ) { -/* destroy virtqueue */ sfc_vdpa_err (ops_data->dev_handle,"virtqueue start failed: %s" ,rte_strerror (rc)); -efx_virtio_qdestroy (vq); goto fail_virtio_qstart; }sfc_vdpa_info (ops_data->dev_handle,"virtqueue started successfully for vq_num %d" , vq_num); + rc =efx_virtio_get_doorbell_offset (vq, &doorbell); +if (rc !=0 ) { +sfc_vdpa_err (ops_data->dev_handle, +"failed to get doorbell offset: %s" , +rte_strerror (rc)); + goto fail_doorbell; + } + +/* +* Cache the bar_offset here for each VQ here, it will come +* in handy when sfc_vdpa_get_notify_area() is invoked. +*/ + ops_data->vq_cxt[vq_num].doorbell = (void *)(uintptr_t )doorbell; ops_data->vq_cxt[vq_num].enable =B_TRUE ;return rc; +fail_doorbell: fail_virtio_qstart: +efx_virtio_qdestroy (vq); fail_vring_info:return rc; }
@@ -792,8 +806,6 @@int ret;efx_nic_t *nic;int vfio_dev_fd; -efx_rc_t rc; -unsigned int bar_offset;volatile void *doorbell;struct rte_pci_device *pci_dev;struct rte_vdpa_device *vdpa_dev;
@@ -824,19 +836,6 @@return -1 ; } -if (ops_data->vq_cxt[qid].enable !=B_TRUE ) { -sfc_vdpa_err (dev,"vq is not enabled" ); -return -1 ; - } - - rc =efx_virtio_get_doorbell_offset (ops_data->vq_cxt[qid].vq, - &bar_offset); -if (rc !=0 ) { -sfc_vdpa_err (dev,"failed to get doorbell offset: %s" , -rte_strerror (rc)); -return rc; - } - reg.index =sfc_vdpa_adapter_by_dev_handle (dev)->mem_bar.esb_rid; ret =ioctl (vfio_dev_fd,VFIO_DEVICE_GET_REGION_INFO , ®);if (ret !=0 ) {
@@ -845,7 +844,8 @@return ret; } - *offset = reg.offset + bar_offset; +/* Use bar_offset that was cached during sfc_vdpa_virtq_start() */ + *offset = reg.offset + (uint64_t )ops_data->vq_cxt[qid].doorbell; len = (1U << encp->enc_vi_window_shift) /2 ;if (len >=sysconf (_SC_PAGESIZE)) {
drivers/vdpa/sfc/sfc_vdpa_ops.h +1 -0
@@ -35,6 +35,7 @@ struct sfc_vdpa_vring_info { }; typedef struct sfc_vdpa_vq_context_s { + volatile void *doorbell; uint8_t enable; uint32_t pidx; uint32_t cidx;
Signature · 1 line
1.8.3.1
Hi Abhimanyu,
Quoted 17 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
Quoted 8 lines
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
We are really late in the v22.07 release cycle.
Does this issue reproduces easily, i.e. is it a blocker if not applied in v22.07?
How confident are you about this fix? If we take it in -rc4 and it introduces a regression, we might not be able to fix it on time for final v22.07.
Also, it misses the Fixes tag, and stable is not CC'ed (The driver was introduced in v22.11).
Regards,
Maxime
Quoted 101 lines
drivers/vdpa/sfc/sfc_vdpa_ops.c +18
-18@@ -222,6 +222,7 @@ sfc_vdpa_virtq_start(struct sfc_vdpa_ops_data *ops_data, int vq_num) { int rc; +uint32_t doorbell;efx_virtio_vq_t *vq;struct sfc_vdpa_vring_info vring;efx_virtio_vq_cfg_t vq_cfg;@@ -270,22 +271,35 @@/* Start virtqueue */ rc =efx_virtio_qstart (vq, &vq_cfg, &vq_dyncfg);if (rc !=0 ) { -/* destroy virtqueue */ sfc_vdpa_err (ops_data->dev_handle,"virtqueue start failed: %s" ,rte_strerror (rc)); -efx_virtio_qdestroy (vq); goto fail_virtio_qstart; }sfc_vdpa_info (ops_data->dev_handle,"virtqueue started successfully for vq_num %d" , vq_num); + rc =efx_virtio_get_doorbell_offset (vq, &doorbell); +if (rc !=0 ) { +sfc_vdpa_err (ops_data->dev_handle, +"failed to get doorbell offset: %s" , +rte_strerror (rc)); + goto fail_doorbell; + } + +/* +* Cache the bar_offset here for each VQ here, it will come +* in handy when sfc_vdpa_get_notify_area() is invoked. +*/ + ops_data->vq_cxt[vq_num].doorbell = (void *)(uintptr_t )doorbell; ops_data->vq_cxt[vq_num].enable =B_TRUE ;return rc; +fail_doorbell: fail_virtio_qstart: +efx_virtio_qdestroy (vq); fail_vring_info:return rc; }@@ -792,8 +806,6 @@int ret;efx_nic_t *nic;int vfio_dev_fd; -efx_rc_t rc; -unsigned int bar_offset;volatile void *doorbell;struct rte_pci_device *pci_dev;struct rte_vdpa_device *vdpa_dev;@@ -824,19 +836,6 @@return -1 ; } -if (ops_data->vq_cxt[qid].enable !=B_TRUE ) { -sfc_vdpa_err (dev,"vq is not enabled" ); -return -1 ; - } - - rc =efx_virtio_get_doorbell_offset (ops_data->vq_cxt[qid].vq, - &bar_offset); -if (rc !=0 ) { -sfc_vdpa_err (dev,"failed to get doorbell offset: %s" , -rte_strerror (rc)); -return rc; - } - reg.index =sfc_vdpa_adapter_by_dev_handle (dev)->mem_bar.esb_rid; ret =ioctl (vfio_dev_fd,VFIO_DEVICE_GET_REGION_INFO , ®);if (ret !=0 ) {@@ -845,7 +844,8 @@return ret; } - *offset = reg.offset + bar_offset; +/* Use bar_offset that was cached during sfc_vdpa_virtq_start() */ + *offset = reg.offset + (uint64_t )ops_data->vq_cxt[qid].doorbell; len = (1U << encp->enc_vi_window_shift) /2 ;if (len >=sysconf (_SC_PAGESIZE)) {drivers/vdpa/sfc/sfc_vdpa_ops.h +1
-0@@ -35,6 +35,7 @@ struct sfc_vdpa_vring_info { }; typedef struct sfc_vdpa_vq_context_s { + volatile void *doorbell; uint8_t enable; uint32_t pidx; uint32_t cidx;
Quoted 17 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
Quoted 9 lines
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
During today's Release status meeting, Andrew mentioned that this patch has been for a log time already in your internal tree.
So it gives a bit of confidence in taking it in -rc4.
Thanks,
Maxime
Hello Abhimanyu, Vijay,
Quoted 28 lines · Maxime Coquelin
Quoted 17 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
Quoted 9 lines
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
Quoted 4 lines
During today's Release status meeting, Andrew mentioned that this patch has been for a log time already in your internal tree.
So it gives a bit of confidence in taking it in -rc4.
- But it is neither reviewed, nor acked by the driver maintainer.
Vijay, as this driver maintainer, your opinion matters. We are in rc4 stage and we merge only critical fixes now. There won't be much time to test this fix once merged (and I am not talking about fixing a regression).
Are you confident with this fix? is it required for the 22.07 release?
If we don't get an answer, the safer is to let those fixes slip to a next release.
- Besides, I see there is a new fix for some sfc driver.
https://patches.dpdk.org/project/dpdk/patch/[email protected]/
The same questions will be asked.
Signature · 1 line
David Marchand
Srivastava, Vijay
Acked-by Vijay SrivastavaQuoted 53 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Friday, July 8, 2022 1:43 PM
To: [email protected]; [email protected]
Cc: Maxime Coquelin <[email protected]>; dev <[email protected]>; Xia, Chenbo <[email protected]>; Andrew Rybchenko <[email protected]>; Saini, Abhimanyu <[email protected]>
Subject: Re: [PATCH] vdpa/sfc: resolve race between libvhost anddev_confCAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
Hello Abhimanyu, Vijay,
Quoted 28 lines · Maxime Coquelin
Quoted 17 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
Quoted 9 lines
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)Quoted 4 lines
During today's Release status meeting, Andrew mentioned that this patch has been for a log time already in your internal tree.
So it gives a bit of confidence in taking it in -rc4.
Quoted 8 lines
- But it is neither reviewed, nor acked by the driver maintainer.
Vijay, as this driver maintainer, your opinion matters.
We are in rc4 stage and we merge only critical fixes now.
There won't be much time to test this fix once merged (and I am not talking about fixing a regression).Are you confident with this fix?
Yes.
Quoted 1 line
is it required for the 22.07 release?
It is not a blocker issue, but it would be good to have in this release.
Quoted 11 lines
If we don't get an answer, the safer is to let those fixes slip to a next release.
- Besides, I see there is a new fix for some sfc driver.
https://patches.dpdk.org/project/dpdk/patch/20220708073702.29391-1-
[email protected]/
The same questions will be asked.--
David Marchand
Maxime Coquelin
AppliedQuoted 17 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
Quoted 9 lines
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
Applied to dpdk-next-virtio/main.
Thanks,
Maxime
Quoted 31 lines · Maxime Coquelin
Quoted 20 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Quoted 7 lines
Signed-off-by Abhimanyu Saini <[email protected]>---
drivers/vdpa/sfc/sfc_vdpa_ops.c| 36
++++++++++++++++++------------------
drivers/vdpa/sfc/sfc_vdpa_ops.h| 1 +2 files changed, 19 insertions(+), 18 deletions(-)
Quoted 1 line
Applied to dpdk-next-virtio/main.
Sorry, I notice it is missing the Fixes tag, and cc'ing stable. Can you confirm this is needed and provide the faulty commit?
Thanks,
Maxime
Quoted 2 lines
Thanks,
Maxime
[AMD Official Use Only - General]
Sorry Maxime, I forgot to copy stable.
Yes it's required, the faulty commit is : 630be406dcbfc26260e9d9688c40a381d0f012db
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, July 8, 2022 2:55 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; Saini, Abhimanyu <[email protected]>
Subject: Re: [PATCH] vdpa/sfc: resolve race between libvhost and dev_conf
[CAUTION: External Email]
Quoted 31 lines · Maxime Coquelin
Quoted 20 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Quoted 7 lines
Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)
Quoted 1 line
Applied to dpdk-next-virtio/main.
Sorry, I notice it is missing the Fixes tag, and cc'ing stable. Can you confirm this is needed and provide the faulty commit?
Thanks,
Maxime
Quoted 2 lines
Thanks,
Maxime
Quoted 4 lines · Abhimanyu
[AMD Official Use Only - General]
Sorry Maxime, I forgot to copy stable.
Yes it's required, the faulty commit is :630be406dcbfc26260e9d9688c40a381d0f012db
Thanks, ammended the commit message and applied to dpdk-next-virtio/main.
Quoted 52 lines
-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, July 8, 2022 2:55 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; Saini, Abhimanyu <[email protected]>
Subject: Re: [PATCH] vdpa/sfc: resolve race between libvhost anddev_conf[CAUTION: External Email]
Quoted 31 lines · Maxime Coquelin
Quoted 20 lines · On 7/6/22 11:24, [email protected] wrote:
From: Abhimanyu Saini <[email protected]>
libvhost calls
dev_conf()before prosessing theVHOST_USER_SET_VRING_CALLmessage for the last VQ. So this message is processed afterdev_conf()returns.However, the
dev_conf()function spawns a thread to setrte_vhost_host_notifier_ctrl()before returning control to libvhost. This parallel thread in turn invokesget_notify_area(). To get thenotify_area, the vdpa driver needs to query the HW and for this query it needs an enabled VQ.But at the same time libvhost is processing the last
VHOST_USER_SET_VRING_CALL, and to do that it disables the last VQ.Hence there is a race b/w the libvhost and the vdpa driver.
To resolve this race condition, query the HW and cache
notify_areainsidedev_conf()instead of doing it the parallel thread.Quoted 7 lines
Signed-off-by Abhimanyu Saini <[email protected]>--- drivers/vdpa/sfc/sfc_vdpa_ops.c | 36 ++++++++++++++++++------------------ drivers/vdpa/sfc/sfc_vdpa_ops.h | 1 + 2 files changed, 19 insertions(+), 18 deletions(-)Quoted 1 line
Applied to dpdk-next-virtio/main.
Sorry, I notice it is missing the Fixes tag, and cc'ing stable. Can you confirm this is needed and provide the faulty commit?
Thanks,
MaximeQuoted 2 lines
Thanks,
Maxime