Skip to contentSkip to conversation

MLX Adress Space ID + Control VirtQueue fixes

RFC In review Eugenio Pérez 2 participants 12 messages Jan 12, 2023 – Jan 17, 2023

Subsystems (2)
lore
Share & tools
lore

Eugenio Pérez

author maintainer
#0 lore raw

cover MLX Adress Space ID + Control VirtQueue fixes

Proposal for fixes detected when using CVQ in a different ASID.

Only tested with qemu and vhost_vdpa.

Eugenio Pérez (3):
vdpa/mlx5: reset iotlb at dup_iotlb
vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr vdpa/mlx5: take iommu_lock at dup_iotlb

 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
 drivers/vdpa/mlx5/core/mr.c        | 20 ++++++++++++++------
 drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
 3 files changed, 21 insertions(+), 13 deletions(-)
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

RFC 1/3 vdpa/mlx5: reset iotlb at dup_iotlb

Regular memory region changes already reset cvq iotlb at set_map. However this is not true if CVQ and data VQs are in different ASID.

Clean the CVQ iotlb every time we hit dup_iotlb.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for control and data")
---
 drivers/vdpa/mlx5/core/mr.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/mlx5/core/mr.c +2 -0
index 0a1e0b0dc37e..ae34dcac9a3f 100644
@@ -456,6 +456,8 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src)
 	u64 start = 0, last = ULLONG_MAX;
 	int err;
 
+	vhost_iotlb_reset(mvdev->cvq.iotlb);
+
 	if (!src) {
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOST_ACCESS_RW);
 		return err;
Signature · 1 line
2.31.1
#2 lore raw

RE: [RFC 1/3] vdpa/mlx5: reset iotlb at dup_iotlb

Quoted 25 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 1/3] vdpa/mlx5: reset iotlb at dup_iotlb

Regular memory region changes already reset cvq iotlb at set_map. However this is not true if CVQ and data VQs are in different ASID.

Clean the CVQ iotlb every time we hit dup_iotlb.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
 drivers/vdpa/mlx5/core/mr.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/mlx5/core/mr.c +0 -0
index 0a1e0b0dc37e..ae34dcac9a3f 100644

drivers/vdpa/mlx5/core/mr.c static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, · line 461

Quoted 6 lines
drivers/vdpa/mlx5/core/mr.c +2 -0
@@ -456,6 +456,8 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
 	u64 start = 0, last = ULLONG_MAX;
 	int err;
 
+	vhost_iotlb_reset(mvdev->cvq.iotlb);
+

As far as I can see, mlx5_vdpa_destroy_mr() is called independently of the asid and it will calls prune_iotlb() which resets the cvq iotlb. Am I missing something?

Quoted 6 lines
drivers/vdpa/mlx5/core/mr.c +0 -0
 	if (!src) {
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last,

start, VHOST_ACCESS_RW);
return err;
--
2.31.1

author maintainer
#3 lore raw

Re: [RFC 1/3] vdpa/mlx5: reset iotlb at dup_iotlb

Quoted 30 lines · Eli Cohen
Quoted 30 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 1/3] vdpa/mlx5: reset iotlb at dup_iotlb

Regular memory region changes already reset cvq iotlb at set_map. However this is not true if CVQ and data VQs are in different ASID.

Clean the CVQ iotlb every time we hit dup_iotlb.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
 drivers/vdpa/mlx5/core/mr.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/vdpa/mlx5/core/mr.c +0 -0
index 0a1e0b0dc37e..ae34dcac9a3f 100644
@@ -456,6 +456,8 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
      u64 start = 0, last = ULLONG_MAX;
      int err;
 
Quoted 6 lines

drivers/vdpa/mlx5/core/mr.c static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src) · line 460

Quoted 2 lines
drivers/vdpa/mlx5/core/mr.c +2 -0
+     vhost_iotlb_reset(mvdev->cvq.iotlb);
+

As far as I can see, mlx5_vdpa_destroy_mr() is called independently of the asid and it will calls prune_iotlb() which resets the cvq iotlb. Am I missing something?

Sorry, my fault. It is not obvious from the patch but this prepares the code for the next patch in the series.

With the next patch applied, prune_iotlb is not called in all contexts. This patch may be dropped in future series.

Thanks!

Quoted 7 lines
Quoted 6 lines
drivers/vdpa/mlx5/core/mr.c +0 -0
      if (!src) {
              err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last,
start, VHOST_ACCESS_RW);
              return err;

--
2.31.1

author maintainer
#4 lore raw

RFC 2/3 vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

In reply to Eugenio Pérez

mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again.

Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for control and data")
---
 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
 drivers/vdpa/mlx5/core/mr.c        |  5 +++--
 drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
 3 files changed, 10 insertions(+), 9 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +1 -1
index 058fbe28107e..000b144019ec 100644
@@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io
 			     bool *change_map, unsigned int asid);
 int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb,
 			unsigned int asid);
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool delete_cvq_iotlb);
 
 #define mlx5_vdpa_warn(__dev, format, ...)                                                         \
 	dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__,     \
drivers/vdpa/mlx5/core/mr.c +3 -2
index ae34dcac9a3f..878ee94efa78 100644
@@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr
 	}
 }
 
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool delete_cvq_iotlb)
 {
 	struct mlx5_vdpa_mr *mr = &mvdev->mr;
 
@@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
 	if (!mr->initialized)
 		goto out;
 
-	prune_iotlb(mvdev);
+	if (delete_cvq_iotlb)
+		prune_iotlb(mvdev);
 	if (mr->user_mr)
 		destroy_user_mr(mvdev, mr);
 	else
drivers/vdpa/mlx5/net/mlx5_vnet.c +6 -6
index 6632651b1e54..1f1f341f602b 100644
@@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,
 		goto err_mr;
 
 	teardown_driver(ndev);
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
 	err = mlx5_vdpa_create_mr(mvdev, iotlb, asid);
 	if (err)
 		goto err_mr;
@@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,
 	return 0;
 
 err_setup:
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
 err_mr:
 	return err;
 }
@@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status)
 	return;
 
 err_setup:
-	mlx5_vdpa_destroy_mr(&ndev->mvdev);
+	mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
 	ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED;
 err_clear:
 	up_write(&ndev->reslock);
@@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev)
 	down_write(&ndev->reslock);
 	teardown_driver(ndev);
 	clear_vqs_ready(ndev);
-	mlx5_vdpa_destroy_mr(&ndev->mvdev);
+	mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
 	ndev->mvdev.status = 0;
 	ndev->cur_num_vqs = 0;
 	ndev->mvdev.cvq.received_desc = 0;
@@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev)
 	ndev = to_mlx5_vdpa_ndev(mvdev);
 
 	free_resources(ndev);
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, true);
 	if (!is_zero_ether_addr(ndev->config.mac)) {
 		pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
 		mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
@@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
 err_res2:
 	free_resources(ndev);
 err_mr:
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, true);
 err_res:
 	mlx5_vdpa_free_resources(&ndev->mvdev);
 err_mpfs:
Signature · 1 line
2.31.1
#5 lore raw

RE: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

Quoted 75 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again.

Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
 drivers/vdpa/mlx5/core/mr.c        |  5 +++--
 drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
 3 files changed, 10 insertions(+), 9 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +0 -0
index 058fbe28107e..000b144019ec 100644
@@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct

mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io
bool *change_map, unsigned int asid);
int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb
*iotlb,

drivers/vdpa/mlx5/core/mlx5_vdpa.h +1 -1
 			unsigned int asid);
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool

delete_cvq_iotlb);

#define mlx5_vdpa_warn(__dev, format, ...)
\
dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: "
format, __func__, __LINE__, \

drivers/vdpa/mlx5/core/mr.c +1 -1
index ae34dcac9a3f..878ee94efa78 100644
@@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev
 *mvdev, struct mlx5_vdpa_mr *mr
 	}
 }
 
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool
 delete_cvq_iotlb)
 {
drivers/vdpa/mlx5/core/mr.c +2 -1
 	struct mlx5_vdpa_mr *mr = &mvdev->mr;
@@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev
 *mvdev)
 	if (!mr->initialized)
 		goto out;
 
-	prune_iotlb(mvdev);
+	if (delete_cvq_iotlb)
+		prune_iotlb(mvdev);
 	if (mr->user_mr)
 		destroy_user_mr(mvdev, mr);
drivers/vdpa/mlx5/core/mr.c +0 -0
 	else
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
index 6632651b1e54..1f1f341f602b 100644

drivers/vdpa/mlx5/net/mlx5_vnet.c static int mlx5_vdpa_change_map(struct · line 2438

Quoted 7 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
 		goto err_mr;
 
 	teardown_driver(ndev);
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, mvdev-
 >group2asid[MLX5_VDPA_CVQ_GROUP] == asid);

Looks to me we need to handle this in a more generic manner. The asid should be used conditionally for either CVQ or data VQ updates. You are protecting CVQ but same thing should hold also for data VQs iotlb. Meaning, if qemu wants to update only CVQ than data VQ translation must not be affected.

Quoted 56 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
 	err = mlx5_vdpa_create_mr(mvdev, iotlb, asid);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
 	if (err)
 		goto err_mr;
@@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
 	return 0;
 
 err_setup:
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, mvdev-
 >group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
 err_mr:
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
 	return err;
 }
@@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct
 vdpa_device *vdev, u8 status)
 	return;
 
 err_setup:
-	mlx5_vdpa_destroy_mr(&ndev->mvdev);
+	mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
 	ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED;
 err_clear:
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
 	up_write(&ndev->reslock);
@@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device
 *vdev)
 	down_write(&ndev->reslock);
 	teardown_driver(ndev);
 	clear_vqs_ready(ndev);
-	mlx5_vdpa_destroy_mr(&ndev->mvdev);
+	mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
 	ndev->mvdev.status = 0;
 	ndev->cur_num_vqs = 0;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
 	ndev->mvdev.cvq.received_desc = 0;
@@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device
 *vdev)
 	ndev = to_mlx5_vdpa_ndev(mvdev);
 
 	free_resources(ndev);
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, true);
 	if (!is_zero_ether_addr(ndev->config.mac)) {
 		pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
 		mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
@@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct
 vdpa_mgmt_dev *v_mdev, const char *name,
 err_res2:
 	free_resources(ndev);
 err_mr:
-	mlx5_vdpa_destroy_mr(mvdev);
+	mlx5_vdpa_destroy_mr(mvdev, true);
 err_res:
 	mlx5_vdpa_free_resources(&ndev->mvdev);

err_mpfs:
--
2.31.1

author maintainer
#6 lore raw

Re: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

Quoted 79 lines · Eli Cohen
Quoted 79 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again.

Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
 drivers/vdpa/mlx5/core/mr.c        |  5 +++--
 drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
 3 files changed, 10 insertions(+), 9 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +0 -0
index 058fbe28107e..000b144019ec 100644
@@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct

mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io

                           bool *change_map, unsigned int asid);
 int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb
*iotlb,
                      unsigned int asid);
drivers/vdpa/mlx5/core/mlx5_vdpa.h +1 -1
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool

delete_cvq_iotlb);

#define mlx5_vdpa_warn(__dev, format, ...)

\
      dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: "

format, __func__, __LINE__, \

drivers/vdpa/mlx5/core/mr.c +1 -1
index ae34dcac9a3f..878ee94efa78 100644
@@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev
 *mvdev, struct mlx5_vdpa_mr *mr
      }
 }
 
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool
 delete_cvq_iotlb)
 {
      struct mlx5_vdpa_mr *mr = &mvdev->mr;
drivers/vdpa/mlx5/core/mr.c +2 -1
@@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev
 *mvdev)
      if (!mr->initialized)
              goto out;
 
-     prune_iotlb(mvdev);
+     if (delete_cvq_iotlb)
+             prune_iotlb(mvdev);
      if (mr->user_mr)
              destroy_user_mr(mvdev, mr);

else

drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
index 6632651b1e54..1f1f341f602b 100644
@@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
              goto err_mr;
 
Quoted 6 lines

drivers/vdpa/mlx5/net/mlx5_vnet.c static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,

Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
      teardown_driver(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 >group2asid[MLX5_VDPA_CVQ_GROUP] == asid);

Looks to me we need to handle this in a more generic manner. The asid should be used conditionally for either CVQ or data VQ updates. You are protecting CVQ but same thing should hold also for data VQs iotlb.

I agree. Maybe the best option is to replace the boolean indicating the ASID we want to destroy mr? Then, at cleanup, we can iterate by all vq groups / ASID.

Quoted 1 line

Meaning, if qemu wants to update only CVQ than data VQ translation must not be affected.

_mlx5_vdpa_create_mr. is the one that checks it If I recall correctly. It is not obvious in this change though.

Thanks!

Quoted 58 lines
Quoted 57 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
      err = mlx5_vdpa_create_mr(mvdev, iotlb, asid);
      if (err)
              goto err_mr;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
      return 0;
 
 err_setup:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 >group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
 err_mr:
      return err;

}

drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct
 vdpa_device *vdev, u8 status)
      return;
 
 err_setup:
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
      ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED;
 err_clear:
      up_write(&ndev->reslock);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device
 *vdev)
      down_write(&ndev->reslock);
      teardown_driver(ndev);
      clear_vqs_ready(ndev);
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
      ndev->mvdev.status = 0;
      ndev->cur_num_vqs = 0;
      ndev->mvdev.cvq.received_desc = 0;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device
 *vdev)
      ndev = to_mlx5_vdpa_ndev(mvdev);
 
      free_resources(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
      if (!is_zero_ether_addr(ndev->config.mac)) {
              pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
              mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct
 vdpa_mgmt_dev *v_mdev, const char *name,
 err_res2:
      free_resources(ndev);
 err_mr:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
 err_res:
      mlx5_vdpa_free_resources(&ndev->mvdev);

err_mpfs:
--
2.31.1

#7 lore raw

Re: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

Quoted 79 lines · Eugenio Perez Martin
Quoted 78 lines · Eli Cohen
Quoted 78 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again.

Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
  drivers/vdpa/mlx5/core/mr.c        |  5 +++--
  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
  3 files changed, 10 insertions(+), 9 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +0 -0
index 058fbe28107e..000b144019ec 100644
@@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct

mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io

                            bool *change_map, unsigned int asid);
  int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb
*iotlb,
                       unsigned int asid);
drivers/vdpa/mlx5/core/mlx5_vdpa.h +1 -1
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool

delete_cvq_iotlb);

#define mlx5_vdpa_warn(__dev, format, ...)

\
       dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: "

format, __func__, __LINE__, \

drivers/vdpa/mlx5/core/mr.c +1 -1
index ae34dcac9a3f..878ee94efa78 100644
@@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev
 *mvdev, struct mlx5_vdpa_mr *mr
       }
  }
 
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool
 delete_cvq_iotlb)
  {
       struct mlx5_vdpa_mr *mr = &mvdev->mr;
drivers/vdpa/mlx5/core/mr.c +2 -1
@@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev
 *mvdev)
       if (!mr->initialized)
               goto out;
 
-     prune_iotlb(mvdev);
+     if (delete_cvq_iotlb)
+             prune_iotlb(mvdev);
       if (mr->user_mr)
               destroy_user_mr(mvdev, mr);

else

drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
index 6632651b1e54..1f1f341f602b 100644
@@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
               goto err_mr;
 
Quoted 8 lines
Quoted 5 lines

drivers/vdpa/mlx5/net/mlx5_vnet.c static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,

Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
       teardown_driver(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 > group2asid[MLX5_VDPA_CVQ_GROUP] == asid);

Looks to me we need to handle this in a more generic manner. The asid should be used conditionally for either CVQ or data VQ updates. You are protecting CVQ but same thing should hold also for data VQs iotlb.

I agree. Maybe the best option is to replace the boolean indicating the ASID we want to destroy mr? Then, at cleanup, we can iterate by all vq groups / ASID.

I think mlx5_vdpa_destroy_mr() should get the asid as an argument and have a logic such as:

if (asid == data asid)

    destroy_data_mr

if (asid == ctrl_vq_asid)

    prune_iotlb()

return

Since we have only two groups, one for data and for control, I don't think we need to iterate.

Quoted 63 lines
Quoted 2 lines

Meaning, if qemu wants to update only CVQ than data VQ translation must not be affected.

_mlx5_vdpa_create_mr. is the one that checks it If I recall correctly. It is not obvious in this change though.

Thanks!

Quoted 56 lines
Quoted 56 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
       err = mlx5_vdpa_create_mr(mvdev, iotlb, asid);
       if (err)
               goto err_mr;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
       return 0;
 
  err_setup:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 > group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
  err_mr:
       return err;

}

drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct
 vdpa_device *vdev, u8 status)
       return;
 
  err_setup:
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
       ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED;
  err_clear:
       up_write(&ndev->reslock);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device
 *vdev)
       down_write(&ndev->reslock);
       teardown_driver(ndev);
       clear_vqs_ready(ndev);
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
       ndev->mvdev.status = 0;
       ndev->cur_num_vqs = 0;
       ndev->mvdev.cvq.received_desc = 0;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device
 *vdev)
       ndev = to_mlx5_vdpa_ndev(mvdev);
 
       free_resources(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
       if (!is_zero_ether_addr(ndev->config.mac)) {
               pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
               mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct
 vdpa_mgmt_dev *v_mdev, const char *name,
  err_res2:
       free_resources(ndev);
  err_mr:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
  err_res:
       mlx5_vdpa_free_resources(&ndev->mvdev);

err_mpfs:
--
2.31.1

author maintainer
#8 lore raw

Re: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

Quoted 99 lines · Eli Cohen
Quoted 79 lines · Eugenio Perez Martin
Quoted 78 lines · Eli Cohen
Quoted 78 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr

mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again.

Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory.

Fixes 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for

control and data")

---
  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  2 +-
  drivers/vdpa/mlx5/core/mr.c        |  5 +++--
  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 12 ++++++------
  3 files changed, 10 insertions(+), 9 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +0 -0
index 058fbe28107e..000b144019ec 100644
@@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct

mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io

                            bool *change_map, unsigned int asid);
  int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb
*iotlb,
                       unsigned int asid);
drivers/vdpa/mlx5/core/mlx5_vdpa.h +1 -1
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev);
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool

delete_cvq_iotlb);

#define mlx5_vdpa_warn(__dev, format, ...)

\
       dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: "

format, __func__, __LINE__, \

drivers/vdpa/mlx5/core/mr.c +1 -1
index ae34dcac9a3f..878ee94efa78 100644
@@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev
 *mvdev, struct mlx5_vdpa_mr *mr
       }
  }
 
-void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev)
+void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool
 delete_cvq_iotlb)
  {
       struct mlx5_vdpa_mr *mr = &mvdev->mr;
drivers/vdpa/mlx5/core/mr.c +2 -1
@@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev
 *mvdev)
       if (!mr->initialized)
               goto out;
 
-     prune_iotlb(mvdev);
+     if (delete_cvq_iotlb)
+             prune_iotlb(mvdev);
       if (mr->user_mr)
               destroy_user_mr(mvdev, mr);

else

drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
index 6632651b1e54..1f1f341f602b 100644
@@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
               goto err_mr;
 
Quoted 8 lines
Quoted 5 lines

drivers/vdpa/mlx5/net/mlx5_vnet.c static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,

Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
       teardown_driver(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 > group2asid[MLX5_VDPA_CVQ_GROUP] == asid);

Looks to me we need to handle this in a more generic manner. The asid should be used conditionally for either CVQ or data VQ updates. You are protecting CVQ but same thing should hold also for data VQs iotlb.

I agree. Maybe the best option is to replace the boolean indicating the ASID we want to destroy mr? Then, at cleanup, we can iterate by all vq groups / ASID.

I think mlx5_vdpa_destroy_mr() should get the asid as an argument and have a logic such as:

if (asid == data asid)

destroy_data_mr

Quoted 7 lines

if (asid == ctrl_vq_asid)

prune_iotlb()

return

I agree.

Quoted 3 lines

Since we have only two groups, one for data and for control, I don't think we need to iterate.

I mean to iterate on mlx5_vdpa_reset and mlx5_vdpa_free. Or, at least, to call mlx5_vdpa_destroy_mr(asid=data) and mlx5_vdpa_destroy_mr(asid=control) one after another.

Thanks!

Quoted 64 lines
Quoted 63 lines
Quoted 2 lines

Meaning, if qemu wants to update only CVQ than data VQ translation must not be affected.

_mlx5_vdpa_create_mr. is the one that checks it If I recall correctly. It is not obvious in this change though.

Thanks!

Quoted 56 lines
Quoted 56 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -0
       err = mlx5_vdpa_create_mr(mvdev, iotlb, asid);
       if (err)
               goto err_mr;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct
 mlx5_vdpa_dev *mvdev,
       return 0;
 
  err_setup:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, mvdev-
 > group2asid[MLX5_VDPA_CVQ_GROUP] == asid);
  err_mr:
       return err;

}

drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct
 vdpa_device *vdev, u8 status)
       return;
 
  err_setup:
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
       ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED;
  err_clear:
       up_write(&ndev->reslock);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device
 *vdev)
       down_write(&ndev->reslock);
       teardown_driver(ndev);
       clear_vqs_ready(ndev);
-     mlx5_vdpa_destroy_mr(&ndev->mvdev);
+     mlx5_vdpa_destroy_mr(&ndev->mvdev, true);
       ndev->mvdev.status = 0;
       ndev->cur_num_vqs = 0;
       ndev->mvdev.cvq.received_desc = 0;
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device
 *vdev)
       ndev = to_mlx5_vdpa_ndev(mvdev);
 
       free_resources(ndev);
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
       if (!is_zero_ether_addr(ndev->config.mac)) {
               pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev));
               mlx5_mpfs_del_mac(pfmdev, ndev->config.mac);
drivers/vdpa/mlx5/net/mlx5_vnet.c +1 -1
@@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct
 vdpa_mgmt_dev *v_mdev, const char *name,
  err_res2:
       free_resources(ndev);
  err_mr:
-     mlx5_vdpa_destroy_mr(mvdev);
+     mlx5_vdpa_destroy_mr(mvdev, true);
  err_res:
       mlx5_vdpa_free_resources(&ndev->mvdev);

err_mpfs:
--
2.31.1

author maintainer
#9 lore raw

RFC 3/3 vdpa/mlx5: take iommu_lock at dup_iotlb

In reply to Eugenio Pérez

Both iommu changes and lookup are protected by mlx5_vdpa_net->reslock at this moment, but:
* These iotlb changes / queries are not in the fast data path.
* reslock belongs to netdev, while dup_iotlb seems generic.
* It's located in a different file than the lock it needs to hold

Justifies the lock acquisition.

Fixes 5262912ef3cf ("vdpa/mlx5: Add support for control VQ and MAC setting")
---
 drivers/vdpa/mlx5/core/mr.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/core/mr.c +9 -4
index 878ee94efa78..e9c8a7f8ee1d 100644
@@ -454,13 +454,15 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src)
 {
 	struct vhost_iotlb_map *map;
 	u64 start = 0, last = ULLONG_MAX;
-	int err;
+	int err = 0;
+
+	spin_lock(&mvdev->cvq.iommu_lock);
 
 	vhost_iotlb_reset(mvdev->cvq.iotlb);
 
 	if (!src) {
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOST_ACCESS_RW);
-		return err;
+		goto out;
 	}
 
 	for (map = vhost_iotlb_itree_first(src, start, last); map;
@@ -468,9 +470,12 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src)
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start, map->last,
 					    map->addr, map->perm);
 		if (err)
-			return err;
+			goto out;
 	}
-	return 0;
+
+out:
+	spin_unlock(&mvdev->cvq.iommu_lock);
+	return err;
 }
 
 static void prune_iotlb(struct mlx5_vdpa_dev *mvdev)
Signature · 1 line
2.31.1
#10 lore raw

RE: [RFC 3/3] vdpa/mlx5: take iommu_lock at dup_iotlb

Quoted 8 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 3/3] vdpa/mlx5: take iommu_lock at dup_iotlb

Quoted 8 lines

Both iommu changes and lookup are protected by mlx5_vdpa_net->reslock at this moment, but:
* These iotlb changes / queries are not in the fast data path.
* reslock belongs to netdev, while dup_iotlb seems generic.
* It's located in a different file than the lock it needs to hold

Justifies the lock acquisition.

Following this reasoning we should take the spinlock wherever we reference an iotlb. Question if it make sense that the iotlb could change while it is being referenced. Can you identify a specific case for this?

Quoted 50 lines
Fixes 5262912ef3cf ("vdpa/mlx5: Add support for control VQ and MAC

setting")

---
 drivers/vdpa/mlx5/core/mr.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/core/mr.c +4 -2
index 878ee94efa78..e9c8a7f8ee1d 100644
@@ -454,13 +454,15 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
 {
 	struct vhost_iotlb_map *map;
 	u64 start = 0, last = ULLONG_MAX;
-	int err;
+	int err = 0;
+
+	spin_lock(&mvdev->cvq.iommu_lock);
 
 	vhost_iotlb_reset(mvdev->cvq.iotlb);
 
 	if (!src) {
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last,
 start, VHOST_ACCESS_RW);
-		return err;
+		goto out;
 	}
drivers/vdpa/mlx5/core/mr.c +5 -2
 	for (map = vhost_iotlb_itree_first(src, start, last); map;
@@ -468,9 +470,12 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
 		err = vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start,
 map->last,
 					    map->addr, map->perm);
 		if (err)
-			return err;
+			goto out;
 	}
-	return 0;
+
+out:
+	spin_unlock(&mvdev->cvq.iommu_lock);
+	return err;
 }

static void prune_iotlb(struct mlx5_vdpa_dev *mvdev)
--
2.31.1

author maintainer
#11 lore raw

Re: [RFC 3/3] vdpa/mlx5: take iommu_lock at dup_iotlb

Quoted 15 lines · Eli Cohen
Quoted 15 lines

From: Eugenio Pérez <[email protected]>
Sent: Thursday, 12 January 2023 16:22
To: [email protected]; Eli Cohen <[email protected]>
Cc: [email protected]; Parav Pandit <[email protected]>; [email protected]; [email protected]; [email protected] foundation.org; [email protected]; [email protected]
Subject: [RFC 3/3] vdpa/mlx5: take iommu_lock at dup_iotlb

Both iommu changes and lookup are protected by mlx5_vdpa_net->reslock at this moment, but:
* These iotlb changes / queries are not in the fast data path.
* reslock belongs to netdev, while dup_iotlb seems generic.
* It's located in a different file than the lock it needs to hold

Quoted 4 lines
Quoted 2 lines

Justifies the lock acquisition.

Following this reasoning we should take the spinlock wherever we reference an iotlb.

vring.c:iotlb_translate takes it.

Quoted 3 lines

Question if it make sense that the iotlb could change while it is being referenced. Can you identify a specific case for this?

Not at this moment, because both are protected by mlx5_vdpa_net->reslock before access or change iotlb. So this would require changes to be exploitable, that's true.

However, to take the lock is the expected usage for vringh, so future changes to either mlx or vringh could miss it.

Thanks!

Quoted 51 lines
Quoted 50 lines
Fixes 5262912ef3cf ("vdpa/mlx5: Add support for control VQ and MAC

setting")

---
 drivers/vdpa/mlx5/core/mr.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/vdpa/mlx5/core/mr.c +4 -2
index 878ee94efa78..e9c8a7f8ee1d 100644
@@ -454,13 +454,15 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
 {
      struct vhost_iotlb_map *map;
      u64 start = 0, last = ULLONG_MAX;
-     int err;
+     int err = 0;
+
+     spin_lock(&mvdev->cvq.iommu_lock);
 
      vhost_iotlb_reset(mvdev->cvq.iotlb);
 
      if (!src) {
              err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last,
 start, VHOST_ACCESS_RW);
-             return err;
+             goto out;
      }
      for (map = vhost_iotlb_itree_first(src, start, last); map;
drivers/vdpa/mlx5/core/mr.c +5 -2
@@ -468,9 +470,12 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev,
 struct vhost_iotlb *src)
              err = vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start,
 map->last,
                                          map->addr, map->perm);
              if (err)
-                     return err;
+                     goto out;
      }
-     return 0;
+
+out:
+     spin_unlock(&mvdev->cvq.iommu_lock);
+     return err;
 }

static void prune_iotlb(struct mlx5_vdpa_dev *mvdev)
--
2.31.1