Skip to contentSkip to conversation

net/virtio: fix legacy device IO port map in secondary process

PATCH Superseded Miao Li 5 participants 11 messages Jun 28, 2023 – Jul 7, 2023 Network drivers

lore
Share & tools
lore

Miao Li

author
#0 lore raw

PATCH 1/2 net/virtio: fix legacy device IO port map in secondary process

When doing IO port map for legacy device in sencondary process, vfio_cfg setup for leagacy device like vfio_group_fd and vfio_dev_fd is missing. So, in sencondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill region info like in primary process.

Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI ethdev init")
---
 drivers/net/virtio/virtio_pci_ethdev.c | 34 +++++++++++++-------------
 1 file changed, 17 insertions(+), 17 deletions(-)
drivers/net/virtio/virtio_pci_ethdev.c +17 -17
index 9b4b846f8a..dc11a6e82f 100644
@@ -44,23 +44,23 @@ virtio_remap_pci(struct rte_pci_device *pci_dev, struct virtio_pci_dev *dev)
 {
 	struct virtio_hw *hw = &dev->hw;
 
-	if (dev->modern) {
-		/*
-		 * We don't have to re-parse the PCI config space, since
-		 * rte_pci_map_device() makes sure the mapped address
-		 * in secondary process would equal to the one mapped in
-		 * the primary process: error will be returned if that
-		 * requirement is not met.
-		 *
-		 * That said, we could simply reuse all cap pointers
-		 * (such as dev_cfg, common_cfg, etc.) parsed from the
-		 * primary process, which is stored in shared memory.
-		 */
-		if (rte_pci_map_device(pci_dev)) {
-			PMD_INIT_LOG(DEBUG, "failed to map pci device!");
-			return -1;
-		}
-	} else {
+	/*
+	 * We don't have to re-parse the PCI config space, since
+	 * rte_pci_map_device() makes sure the mapped address
+	 * in secondary process would equal to the one mapped in
+	 * the primary process: error will be returned if that
+	 * requirement is not met.
+	 *
+	 * That said, we could simply reuse all cap pointers
+	 * (such as dev_cfg, common_cfg, etc.) parsed from the
+	 * primary process, which is stored in shared memory.
+	 */
+	if (rte_pci_map_device(pci_dev)) {
+		PMD_INIT_LOG(DEBUG, "failed to map pci device!");
+		return -1;
+	}
+
+	if (!dev->modern) {
 		if (vtpci_legacy_ioport_map(hw) < 0)
 			return -1;
 	}
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 2/2 bus/pci: add IO port region check before region map

This patch adds IO port region check to skip region map when doing IO port map for legacy device in sencondary process.

Fixes 33604c31354a ("vfio: refactor PCI BAR mapping")
---
 drivers/bus/pci/linux/pci_vfio.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/bus/pci/linux/pci_vfio.c +9 -0
index e634de8322..dd4ca46120 100644
@@ -1099,6 +1099,15 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 	maps = vfio_res->maps;
 
 	for (i = 0; i < vfio_res->nb_maps; i++) {
+		/* chk for io port region */
+		ret = pci_vfio_is_ioport_bar(dev, vfio_dev_fd, i);
+		if (ret < 0) {
+			goto err_vfio_dev_fd;
+		} else if (ret) {
+			RTE_LOG(INFO, EAL, "Ignore mapping IO port bar(%d)\n", i);
+			continue;
+		}
+
 		if (maps[i].nr_areas > 0) {
 			ret = pci_vfio_sparse_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
 			if (ret < 0) {
Signature · 1 line
2.25.1
author
#2 lore raw

PATCH v2 1/2 net/virtio: fix legacy device IO port map in secondary process

In reply to Miao Li

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI ethdev init")
---
 drivers/net/virtio/virtio_pci_ethdev.c | 34 +++++++++++++-------------
 1 file changed, 17 insertions(+), 17 deletions(-)
drivers/net/virtio/virtio_pci_ethdev.c +17 -17
index 9b4b846f8a..dc11a6e82f 100644
@@ -44,23 +44,23 @@ virtio_remap_pci(struct rte_pci_device *pci_dev, struct virtio_pci_dev *dev)
 {
 	struct virtio_hw *hw = &dev->hw;
 
-	if (dev->modern) {
-		/*
-		 * We don't have to re-parse the PCI config space, since
-		 * rte_pci_map_device() makes sure the mapped address
-		 * in secondary process would equal to the one mapped in
-		 * the primary process: error will be returned if that
-		 * requirement is not met.
-		 *
-		 * That said, we could simply reuse all cap pointers
-		 * (such as dev_cfg, common_cfg, etc.) parsed from the
-		 * primary process, which is stored in shared memory.
-		 */
-		if (rte_pci_map_device(pci_dev)) {
-			PMD_INIT_LOG(DEBUG, "failed to map pci device!");
-			return -1;
-		}
-	} else {
+	/*
+	 * We don't have to re-parse the PCI config space, since
+	 * rte_pci_map_device() makes sure the mapped address
+	 * in secondary process would equal to the one mapped in
+	 * the primary process: error will be returned if that
+	 * requirement is not met.
+	 *
+	 * That said, we could simply reuse all cap pointers
+	 * (such as dev_cfg, common_cfg, etc.) parsed from the
+	 * primary process, which is stored in shared memory.
+	 */
+	if (rte_pci_map_device(pci_dev)) {
+		PMD_INIT_LOG(DEBUG, "failed to map pci device!");
+		return -1;
+	}
+
+	if (!dev->modern) {
 		if (vtpci_legacy_ioport_map(hw) < 0)
 			return -1;
 	}
Signature · 1 line
2.25.1
author
#3 lore raw

PATCH v2 2/2 bus/pci: add IO port region check before region map

This patch adds IO port region check to skip region map when doing IO port map for legacy device in secondary process.

Fixes 33604c31354a ("vfio: refactor PCI BAR mapping")
---
 drivers/bus/pci/linux/pci_vfio.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/bus/pci/linux/pci_vfio.c +9 -0
index e634de8322..dd4ca46120 100644
@@ -1099,6 +1099,15 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)
 	maps = vfio_res->maps;
 
 	for (i = 0; i < vfio_res->nb_maps; i++) {
+		/* chk for io port region */
+		ret = pci_vfio_is_ioport_bar(dev, vfio_dev_fd, i);
+		if (ret < 0) {
+			goto err_vfio_dev_fd;
+		} else if (ret) {
+			RTE_LOG(INFO, EAL, "Ignore mapping IO port bar(%d)\n", i);
+			continue;
+		}
+
 		if (maps[i].nr_areas > 0) {
 			ret = pci_vfio_sparse_mmap_bar(vfio_dev_fd, vfio_res, i, MAP_FIXED);
 			if (ret < 0) {
Signature · 1 line
2.25.1
Tested-by on v2 1/2
#4 lore raw
In reply to Miao Li
Quoted 15 lines

-----Original Message-----
From: Miao Li <[email protected]>
Sent: Thursday, June 29, 2023 10:27 AM
To: [email protected]
Cc: [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>; David Marchand <[email protected]>
Subject: [PATCH v2 1/2] net/virtio: fix legacy device IO port map in secondary process

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

Quoted 6 lines
Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI ethdev

init")
Cc: [email protected]

---

#5 lore raw
In reply to Miao Li
Quoted 6 lines · Miao Li

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map() function.

Quoted 1 line
Quoted 1 line
Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI ethdev init")

This commit only moved code, and at this point, there was no need for a call to rte_pci_map_device in the secondary process case. It seems unlikely this is a faulty change.

The recent addition on the vfio side seems a better culprit, but I am fine with being proven wrong :-).

Quoted 3 lines
Signature · 1 line
David Marchand
author
#6 lore raw

Hi,

Quoted 17 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 3, 2023 3:48 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v2 1/2] net/virtio: fix legacy device IO port map in secondary process

Quoted 6 lines · Miao Li

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

Quoted 6 lines

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map() function.

If adding vfio setup in the pci_vfio_ioport_map() function, vfio will be setup twice in primary process because rte_pci_map_device will be called for legacy device in primary process. I add IO port region check to skip region map in the next patch.

Quoted 11 lines
Quoted 3 lines
Quoted 3 lines
Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI

ethdev init")

This commit only moved code, and at this point, there was no need for a call to rte_pci_map_device in the secondary process case.
It seems unlikely this is a faulty change.

The recent addition on the vfio side seems a better culprit, but I am fine with being proven wrong :-).

Yes, the fix commit is wrong, but not the recent addition commit on the vfio side. Because the root cause is missing a vfio setup. After adding recent addition commit, the uninitialized vfio_cfg info(like vfio_dev_fd, region info) will be used so this bug will be found. I think the correct fix commit will be 6d890f8ab512("net/virtio: fix multiple process support").

Quoted 8 lines
Quoted 4 lines

--
David Marchand

Thanks,
Miao Li

#7 lore raw
Quoted 6 lines · Miao
Quoted 6 lines
Quoted 5 lines

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

Quoted 8 lines
Quoted 6 lines

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map() function.

If adding vfio setup in the pci_vfio_ioport_map() function, vfio will be setup twice in primary process because rte_pci_map_device will be called for legacy device in primary process. I add IO port region check to skip region map in the next patch.

Well, something must be done so that it is not mapped twice, I did not look into the details.
This current patch looks wrong to me and I understand this is not a virtio only issue.

Signature · 1 line
David Marchand
#8 lore raw

+Nipun

Quoted 25 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 3, 2023 4:58 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v2 1/2] net/virtio: fix legacy device IO port map in secondary process

Quoted 2 lines · Miao
Quoted 2 lines
Quoted 2 lines

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd

is

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map()

function.

Quoted 2 lines
Quoted 1 line

If adding vfio setup in the pci_vfio_ioport_map() function, vfio will be

Quoted 8 lines

setup twice in primary process because rte_pci_map_device will be called for legacy device in primary process.

Quoted 1 line

I add IO port region check to skip region map in the next patch.

Well, something must be done so that it is not mapped twice, I did not look into the details.
This current patch looks wrong to me and I understand this is not a virtio only issue.

I think we could have some way to improve this:

1. Make rte_pci_map_device map either PIO or MMIO (Based on my knowledge, it's doable for vfio. For UIO, I am no expert and not sure). For ioport, it's only about setting up the ioport offset and size.
2. rte_pci_ioport_map may not be needed anymore.
3. struct rte_pci_ioport may not be needed anymore as the info could be saved in struct rte_pci_device_internal.
4. ioport device uses bar #, len, offset to RW specific BAR.

Then for virtio device, either primary or secondary process only calls rte_pci_map_device once.

Any comments?

Thanks,
Chenbo

Quoted 4 lines

--
David Marchand

#9 lore raw
Quoted 46 lines · Chenbo

+Nipun

Quoted 25 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 3, 2023 4:58 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v2 1/2] net/virtio: fix legacy device IO port map in secondary process

Quoted 2 lines · Miao
Quoted 2 lines
Quoted 2 lines

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd

is

Quoted 9 lines
Quoted 9 lines
Quoted 3 lines

missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map()

function.

Quoted 2 lines
Quoted 1 line

If adding vfio setup in the pci_vfio_ioport_map() function, vfio will be

Quoted 8 lines

setup twice in primary process because rte_pci_map_device will be called for legacy device in primary process.

Quoted 1 line

I add IO port region check to skip region map in the next patch.

Well, something must be done so that it is not mapped twice, I did not look into the details.
This current patch looks wrong to me and I understand this is not a virtio only issue.

I think we could have some way to improve this:

1. Make rte_pci_map_device map either PIO or MMIO (Based on my knowledge, it's doable for vfio. For UIO, I am no expert and not sure). For ioport, it's only about setting up the ioport offset and size.
2. rte_pci_ioport_map may not be needed anymore.
3. struct rte_pci_ioport may not be needed anymore as the info could be saved in struct rte_pci_device_internal.
4. ioport device uses bar #, len, offset to RW specific BAR.

Quoted 4 lines

Then for virtio device, either primary or secondary process only calls rte_pci_map_device once.

Any comments?

Wouldn't a call to API rte_vfio_setup_device() to setup vfio_cfg, vfio_group_fd, vfio_dev_fd under a secondary process check suffice to handle IO port map for legacy device in secondary process?

I do not have much info on legacy Virtio device, and I am not clear on why and how for these devices rte_pci_map_device() would be called in case of primary process, but not in case of secondary process as mentioned by Miao Li?

Steps you have mentioned looks fine but note that this would cause an ABI breakage and as you mentioned may need changes in UIO (even I am not an expert in UIO).

Thanks,
Nipun

Quoted 9 lines

Thanks,
Chenbo

Quoted 4 lines

--
David Marchand

#10 lore raw
In reply to David Marchand
Quoted 17 lines

-----Original Message-----
From: David Marchand <[email protected]>
Sent: Monday, July 3, 2023 3:48 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Subject: Re: [PATCH v2 1/2] net/virtio: fix legacy device IO port map in secondary process

Quoted 6 lines · Miao Li

When doing IO port map for legacy device in secondary process, vfio_cfg setup for legacy device like vfio_group_fd and vfio_dev_fd is missing. So, in secondary process, rte_pci_map_device is added for legacy device to setup vfio_cfg and fill in region info like in primary process.

Quoted 5 lines

I think, in legacy mode, there is no PCI mappable memory. So there should be no need for this call to rte_pci_map_device.

What is missing is a vfio setup, is this correct?
I'd rather see this issue be fixed in the pci_vfio_ioport_map() function.

Thinking about this again: pci_vfio_ioport_map is defined to map specific ioport so it does not make sense to do any device setup in such function. Any reason why we can't call rte_pci_map_device in secondary/legacy? This function rte_pci_map_device is defined to set-up device and set up BAR mapping if needed. Secondary process for any driver needs set-up device and BAR mapping again (right?). For legacy device it can skip the BAR mapping part, which rte_pci_map_device is already doing.

Any comments?

Thanks,
Chenbo

Quoted 20 lines
Quoted 3 lines
Quoted 3 lines
Fixes 512e27eeb743 ("net/virtio: move PCI specific dev init to PCI

ethdev init")

This commit only moved code, and at this point, there was no need for a call to rte_pci_map_device in the secondary process case. It seems unlikely this is a faulty change.

The recent addition on the vfio side seems a better culprit, but I am fine with being proven wrong :-).

Quoted 3 lines

--
David Marchand