Skip to contentSkip to conversation

net/virtio-user: fix multi-process issue

PATCH In review Ami Sabo 4 participants 20 messages Feb 23, 2017 – Apr 14, 2017 Network drivers

lore
Share & tools
lore

Ami Sabo

author
#0 lore raw

PATCH 1/1 net/virtio-user: fix multi-process issue

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

Accessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.

So, most of the rte_eth_dev_data fields are not initialized. For pci devices these fields are initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().
To fix it: Allow non-pci drivers call to dev_attach_secondary() and call it (for secondary process) from virtio_user_pmd_probe.

---
 drivers/net/virtio/virtio_user_ethdev.c | 29 +++++++++++++++++------------
 lib/librte_ether/rte_ethdev.c           |  6 +++---
 lib/librte_ether/rte_ethdev.h           | 11 +++++++++++
 lib/librte_ether/rte_ether_version.map  |  1 +
 4 files changed, 32 insertions(+), 15 deletions(-)
drivers/net/virtio/virtio_user_ethdev.c +17 -12
index e544acc..d388b92 100644
@@ -418,18 +418,23 @@ virtio_user_pmd_probe(const char *name, const char *params)
 		goto end;
 	}
 
-	eth_dev = virtio_user_eth_dev_alloc(name);
-	if (!eth_dev) {
-		PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
-		goto end;
-	}
-
-	hw = eth_dev->data->dev_private;
-	if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
-				 queue_size, mac_addr) < 0) {
-		PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
-		virtio_user_eth_dev_free(eth_dev);
-		goto end;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		eth_dev = virtio_user_eth_dev_alloc(name);
+		if (!eth_dev) {
+			PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
+			goto end;
+		}
+		hw = eth_dev->data->dev_private;
+		if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
+			queue_size, mac_addr) < 0) {
+			PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
+			virtio_user_eth_dev_free(eth_dev);
+			goto end;
+		}
+	} else {
+		eth_dev = rte_eth_dev_attach_secondary(name);
+		if (!eth_dev)
+			goto end;
 	}
 
 	/* previously called by rte_eal_pci_probe() for physical dev */
lib/librte_ether/rte_ethdev.c +3 -3
index 61f44e2..ea4f76c 100644
@@ -239,8 +239,8 @@ rte_eth_dev_allocate(const char *name)
  * makes sure that the same device would have the same port id both
  * in the primary and secondary process.
  */
-static struct rte_eth_dev *
-eth_dev_attach_secondary(const char *name)
+struct rte_eth_dev *
+rte_eth_dev_attach_secondary(const char *name)
 {
 	uint8_t i;
 	struct rte_eth_dev *eth_dev;
@@ -302,7 +302,7 @@ rte_eth_dev_pci_probe(struct rte_pci_driver *pci_drv,
 		if (eth_dev->data->dev_private == NULL)
 			rte_panic("Cannot allocate memzone for private port data\n");
 	} else {
-		eth_dev = eth_dev_attach_secondary(ethdev_name);
+		eth_dev = rte_eth_dev_attach_secondary(ethdev_name);
 		if (eth_dev == NULL) {
 			/*
 			 * if we failed to attach a device, it means the
lib/librte_ether/rte_ethdev.h +11 -0
index c17bbda..3281205 100644
@@ -1762,6 +1762,17 @@ struct rte_eth_dev *rte_eth_dev_allocate(const char *name);
 
 /**
  * @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param	name	Ethernet device's name.
+  @return
+ *   - Slot in the rte_dev_devices array for attached device;
+ */
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +1 -0
index c6c9d0d..f8bf2ee 100644
@@ -152,5 +152,6 @@ DPDK_17.02 {
 	rte_flow_flush;
 	rte_flow_query;
 	rte_flow_validate;
+	rte_eth_dev_attach_secondary;
 
 } DPDK_16.11;
Signature · 1 line
2.7.4
#1 lore raw
Quoted 18 lines · Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

Accessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.

So, most of the rte_eth_dev_data fields are not initialized. For pci devices these fields are initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().
To fix it: Allow non-pci drivers call to dev_attach_secondary() and call it (for secondary process) from virtio_user_pmd_probe.

Quoted 1 line

Firstly, two minor comments:

- A fix path needs a fixline (check dpdk.org/dev for HOWTO)

- It fixes a bug in former releases, thus it need be picked into a stable release. Then you need add following just before you Signed-off-by:

Quoted 8 lines

/**
* @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param name Ethernet device's name.
+ @return
+ * - Slot in the rte_dev_devices array for attached device;

Yes, that's what it returns on success. You also need to add the case when it fails.

Quoted 13 lines

+ */

lib/librte_ether/rte_ethdev.h +4 -0
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +0 -0
index c6c9d0d..f8bf2ee 100644

lib/librte_ether/rte_ether_version.map DPDK_17.02 { · line 157

Quoted 6 lines
lib/librte_ether/rte_ether_version.map +1 -0
@@ -152,5 +152,6 @@ DPDK_17.02 {
 	rte_flow_flush;
 	rte_flow_query;
 	rte_flow_validate;
+	rte_eth_dev_attach_secondary;
 
 } DPDK_16.11;

17.02 is released, you should add a new table for 17.05 and add it there.

Besides, I would suggest you to split this patch into two:

- one for exporting rte_eth_dev_attach_secondary
- another for fixing the bug

	--yliu
author
#2 lore raw

cover Fix virtio-user multi-process crash.

In reply to Ami Sabo

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

Ami Sabo (2):
lib/librte_ether: export secondary attach function net/virtio-user: fix multi-process issue

 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 lib/librte_ether/rte_ethdev.c           |  6 +++---
 lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map  |  7 +++++++
 4 files changed, 39 insertions(+), 13 deletions(-)
Signature · 1 line
2.7.4
author
#3 lore raw

PATCH 1/2 lib/librte_ether: export secondary attach function

Today eth_dev_attach_secondary is defined as static and can only be called by pci drivers. However, the functionality is also required for non-pci drivers - so the patch export the function.

---
 lib/librte_ether/rte_ethdev.c          |  6 +++---
 lib/librte_ether/rte_ethdev.h          | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map |  7 +++++++
 3 files changed, 23 insertions(+), 3 deletions(-)
lib/librte_ether/rte_ethdev.c +3 -3
index eb0a94a..86ee5bb 100644
@@ -239,8 +239,8 @@ rte_eth_dev_allocate(const char *name)
  * makes sure that the same device would have the same port id both
  * in the primary and secondary process.
  */
-static struct rte_eth_dev *
-eth_dev_attach_secondary(const char *name)
+struct rte_eth_dev *
+rte_eth_dev_attach_secondary(const char *name)
 {
 	uint8_t i;
 	struct rte_eth_dev *eth_dev;
@@ -302,7 +302,7 @@ rte_eth_dev_pci_probe(struct rte_pci_driver *pci_drv,
 		if (eth_dev->data->dev_private == NULL)
 			rte_panic("Cannot allocate memzone for private port data\n");
 	} else {
-		eth_dev = eth_dev_attach_secondary(ethdev_name);
+		eth_dev = rte_eth_dev_attach_secondary(ethdev_name);
 		if (eth_dev == NULL) {
 			/*
 			 * if we failed to attach a device, it means the
lib/librte_ether/rte_ethdev.h +13 -0
index 97f3e2d..9d5848b 100644
@@ -1762,6 +1762,19 @@ struct rte_eth_dev *rte_eth_dev_allocate(const char *name);
 
 /**
  * @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param       name    Ethernet device's name.
+  @return
+ *   - Success: Slot in the rte_dev_devices array for attached
+ *        device.
+ *   - Error: Null pointer.
+ */
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +7 -0
index c6c9d0d..d34c57a 100644
@@ -154,3 +154,10 @@ DPDK_17.02 {
 	rte_flow_validate;
 
 } DPDK_16.11;
+
+DPDK_17.05 {
+        global:
+
+        rte_eth_dev_attach_secondary;
+
+} DPDK_17.02;
Signature · 1 line
2.7.4
Reviewed-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] lib/librte_ether: export secondary attach function

Cc Thomas, the librte_ether maintainer.

Quoted 7 lines · Ami Sabo

/**
* @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param name Ethernet device's name.
+ @return

mailformed comment: missing *.

Quoted 16 lines

+ * - Success: Slot in the rte_dev_devices array for attached
+ * device.
+ * - Error: Null pointer.
+ */

lib/librte_ether/rte_ethdev.h +4 -0
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +0 -0
index c6c9d0d..d34c57a 100644

lib/librte_ether/rte_ether_version.map DPDK_17.02 { · line 161

Quoted 8 lines
lib/librte_ether/rte_ether_version.map +5 -0
@@ -154,3 +154,10 @@ DPDK_17.02 {
 	rte_flow_validate;
 
 } DPDK_16.11;
+
+DPDK_17.05 {
+        global:
+
+        rte_eth_dev_attach_secondary;

I saw whitespace chars. Like code, it should be TABs. Other than those minor nits, it looks good to me.

	--yliu
author
#5 lore raw

v2 cover Fix virtio-user multi-process crash.

In reply to Ami Sabo

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because the secondary process calls, at virtio_user_pmd_probe() to virtio_user_eth_dev_alloc()-> rte_eth_dev_allocate() instead of eth_dev_attach_secondary(), as it's done from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized + the device data maybe overridden by the secondary process.

The patchset contains 3 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.
3. Code style fixes following Yuanhan Lio's comments.

Ami Sabo (3):
lib/librte_ether: export secondary attach function net/virtio-user: fix multi-process issue lib/librte_ether: fix code style issues

 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 lib/librte_ether/rte_ethdev.c           |  6 +++---
 lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map  |  7 +++++++
 4 files changed, 39 insertions(+), 13 deletions(-)

--
v2:
* Fix code style issues following Yuanhan Liu's review.

Signature · 1 line
2.7.4
author
#6 lore raw

PATCH v2 1/3 lib/librte_ether: export secondary attach function

Today eth_dev_attach_secondary is defined as static and can only be called by pci drivers. However, the functionality is also required for non-pci drivers - so the patch export the function.

---
 lib/librte_ether/rte_ethdev.c          |  6 +++---
 lib/librte_ether/rte_ethdev.h          | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map |  7 +++++++
 3 files changed, 23 insertions(+), 3 deletions(-)
lib/librte_ether/rte_ethdev.c +3 -3
index eb0a94a..86ee5bb 100644
@@ -239,8 +239,8 @@ rte_eth_dev_allocate(const char *name)
  * makes sure that the same device would have the same port id both
  * in the primary and secondary process.
  */
-static struct rte_eth_dev *
-eth_dev_attach_secondary(const char *name)
+struct rte_eth_dev *
+rte_eth_dev_attach_secondary(const char *name)
 {
 	uint8_t i;
 	struct rte_eth_dev *eth_dev;
@@ -302,7 +302,7 @@ rte_eth_dev_pci_probe(struct rte_pci_driver *pci_drv,
 		if (eth_dev->data->dev_private == NULL)
 			rte_panic("Cannot allocate memzone for private port data\n");
 	} else {
-		eth_dev = eth_dev_attach_secondary(ethdev_name);
+		eth_dev = rte_eth_dev_attach_secondary(ethdev_name);
 		if (eth_dev == NULL) {
 			/*
 			 * if we failed to attach a device, it means the
lib/librte_ether/rte_ethdev.h +13 -0
index 97f3e2d..9d5848b 100644
@@ -1762,6 +1762,19 @@ struct rte_eth_dev *rte_eth_dev_allocate(const char *name);
 
 /**
  * @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param       name    Ethernet device's name.
+  @return
+ *   - Success: Slot in the rte_dev_devices array for attached
+ *        device.
+ *   - Error: Null pointer.
+ */
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +7 -0
index c6c9d0d..d34c57a 100644
@@ -154,3 +154,10 @@ DPDK_17.02 {
 	rte_flow_validate;
 
 } DPDK_16.11;
+
+DPDK_17.05 {
+        global:
+
+        rte_eth_dev_attach_secondary;
+
+} DPDK_17.02;
Signature · 1 line
2.7.4
author
#7 lore raw

PATCH v2 2/3 net/virtio-user: fix multi-process issue

In reply to Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

ccessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.
For pci devices the struct is initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().

The fix is to call rte_eth_dev_attach_secondary(), for secondary process, from virtio_user_pmd_probe.

Fixes 7f95f78a8aea ("ethdev: clear data when allocating device")
---
 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)
drivers/net/virtio/virtio_user_ethdev.c +16 -10
index 0b226ac..6033908 100644
@@ -418,18 +418,24 @@ virtio_user_pmd_probe(const char *name, const char *params)
 		goto end;
 	}
 
-	eth_dev = virtio_user_eth_dev_alloc(name);
-	if (!eth_dev) {
-		PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
-		goto end;
-	}
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		eth_dev = virtio_user_eth_dev_alloc(name);
+		if (!eth_dev) {
+			PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
+			goto end;
+		}
 
-	hw = eth_dev->data->dev_private;
-	if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
+		hw = eth_dev->data->dev_private;
+		if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
 				 queue_size, mac_addr) < 0) {
-		PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
-		virtio_user_eth_dev_free(eth_dev);
-		goto end;
+			PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
+			virtio_user_eth_dev_free(eth_dev);
+			goto end;
+		}
+	} else {
+		eth_dev = rte_eth_dev_attach_secondary(name);
+		if (!eth_dev)
+			goto end;
 	}
 
 	/* previously called by rte_eal_pci_probe() for physical dev */
Signature · 1 line
2.7.4
author
#8 lore raw

PATCH v2 3/3 lib/librte_ether: fix code style issues

In reply to Ami Sabo

Fixed code style issues following Yuanhan Liu's review.

---
 lib/librte_ether/rte_ethdev.h          | 2 +-
 lib/librte_ether/rte_ether_version.map | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
lib/librte_ether/rte_ethdev.h +1 -1
index 9d5848b..6c9af46 100644
@@ -1766,7 +1766,7 @@ struct rte_eth_dev *rte_eth_dev_allocate(const char *name);
  * process.
  *
  * @param       name    Ethernet device's name.
-  @return
+ * @return
  *   - Success: Slot in the rte_dev_devices array for attached
  *        device.
  *   - Error: Null pointer.
lib/librte_ether/rte_ether_version.map +2 -2
index d34c57a..28fb7d9 100644
@@ -156,8 +156,8 @@ DPDK_17.02 {
 } DPDK_16.11;
 
 DPDK_17.05 {
-        global:
+	global:
 
-        rte_eth_dev_attach_secondary;
+	rte_eth_dev_attach_secondary;
 
 } DPDK_17.02;
Signature · 1 line
2.7.4
#9 lore raw

Re: [PATCH v2 3/3] lib/librte_ether: fix code style issues

Quoted 1 line · Ami Sabo

Fixed code style issues following Yuanhan Liu's review.

Patch 1 is not merged yet. You should squash this fix into patch 1, instead of sending another patch.

Meanwhile, when you do so, you should add version log, and put it at .....>

Quoted 3 lines

---

... here (right below the ---). It could be something like:

- v2: fix coding style issues (Yuanhan Liu)

If you are still unsure, just find some examples from the mailing list.

	--yliu
author
#10 lore raw

cover Fix virtio-user multi-process crash.

In reply to Ami Sabo

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

Ami Sabo (2):
lib/librte_ether: export secondary attach function net/virtio-user: fix multi-process issue

 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 lib/librte_ether/rte_ethdev.c           |  6 +++---
 lib/librte_ether/rte_ethdev.h           | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map  |  6 ++++++
 4 files changed, 38 insertions(+), 13 deletions(-)
Signature · 1 line
2.7.4
author
#11 lore raw

PATCH 1/2 lib/librte_ether: export secondary attach function

Today eth_dev_attach_secondary is defined as static and can only be called by pci drivers. However, the functionality is also required for non-pci drivers - so the patch export the function.

---
v2: Fix coding style issues (Yuanhan Liu)

---
 lib/librte_ether/rte_ethdev.c          |  6 +++---
 lib/librte_ether/rte_ethdev.h          | 13 +++++++++++++
 lib/librte_ether/rte_ether_version.map |  6 ++++++
 3 files changed, 22 insertions(+), 3 deletions(-)
lib/librte_ether/rte_ethdev.c +3 -3
index eb0a94a..86ee5bb 100644
@@ -239,8 +239,8 @@ rte_eth_dev_allocate(const char *name)
  * makes sure that the same device would have the same port id both
  * in the primary and secondary process.
  */
-static struct rte_eth_dev *
-eth_dev_attach_secondary(const char *name)
+struct rte_eth_dev *
+rte_eth_dev_attach_secondary(const char *name)
 {
 	uint8_t i;
 	struct rte_eth_dev *eth_dev;
@@ -302,7 +302,7 @@ rte_eth_dev_pci_probe(struct rte_pci_driver *pci_drv,
 		if (eth_dev->data->dev_private == NULL)
 			rte_panic("Cannot allocate memzone for private port data\n");
 	} else {
-		eth_dev = eth_dev_attach_secondary(ethdev_name);
+		eth_dev = rte_eth_dev_attach_secondary(ethdev_name);
 		if (eth_dev == NULL) {
 			/*
 			 * if we failed to attach a device, it means the
lib/librte_ether/rte_ethdev.h +13 -0
index 97f3e2d..6c9af46 100644
@@ -1762,6 +1762,19 @@ struct rte_eth_dev *rte_eth_dev_allocate(const char *name);
 
 /**
  * @internal
+ * Attach to the ethdev already initialized by the primary
+ * process.
+ *
+ * @param       name    Ethernet device's name.
+ * @return
+ *   - Success: Slot in the rte_dev_devices array for attached
+ *        device.
+ *   - Error: Null pointer.
+ */
+struct rte_eth_dev *rte_eth_dev_attach_secondary(const char *name);
+
+/**
+ * @internal
  * Release the specified ethdev port.
  *
  * @param eth_dev
lib/librte_ether/rte_ether_version.map +6 -0
index c6c9d0d..85b0402 100644
@@ -154,3 +154,9 @@ DPDK_17.02 {
 	rte_flow_validate;
 
 } DPDK_16.11;
+
+DPDK_17.05 {
+	global:
+
+	rte_eth_dev_attach_secondary;
+} DPDK_17.02;
Signature · 1 line
2.7.4
author
#12 lore raw

PATCH 2/2 net/virtio-user: fix multi-process issue

In reply to Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

ccessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.
For pci devices the struct is initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().

The fix is to call rte_eth_dev_attach_secondary(), for secondary process, from virtio_user_pmd_probe.

Fixes 7f95f78a8aea ("ethdev: clear data when allocating device")
---
 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)
drivers/net/virtio/virtio_user_ethdev.c +16 -10
index 0b226ac..6033908 100644
@@ -418,18 +418,24 @@ virtio_user_pmd_probe(const char *name, const char *params)
 		goto end;
 	}
 
-	eth_dev = virtio_user_eth_dev_alloc(name);
-	if (!eth_dev) {
-		PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
-		goto end;
-	}
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		eth_dev = virtio_user_eth_dev_alloc(name);
+		if (!eth_dev) {
+			PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
+			goto end;
+		}
 
-	hw = eth_dev->data->dev_private;
-	if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
+		hw = eth_dev->data->dev_private;
+		if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
 				 queue_size, mac_addr) < 0) {
-		PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
-		virtio_user_eth_dev_free(eth_dev);
-		goto end;
+			PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
+			virtio_user_eth_dev_free(eth_dev);
+			goto end;
+		}
+	} else {
+		eth_dev = rte_eth_dev_attach_secondary(name);
+		if (!eth_dev)
+			goto end;
 	}
 
 	/* previously called by rte_eal_pci_probe() for physical dev */
Signature · 1 line
2.7.4
#13 lore raw

Re: [PATCH 0/2] Fix virtio-user multi-process crash.

In reply to Ami Sabo

2017-03-02 11:00, Ami Sabo:

Quoted 8 lines

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

Quoted 4 lines

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

I do not understand why nobody complains for other virtual devices. We should have the same issue with pcap, tap, ring, af_packet, etc. Probably that other drivers are broken in secondary processes. Or should we make a fix to handle every secondary vdev in rte_eth_dev_allocate() ?

#14 lore raw

Re: [PATCH 0/2] Fix virtio-user multi-process crash.

Ping

2017-03-08 12:40, Thomas Monjalon:

Quoted 14 lines

2017-03-02 11:00, Ami Sabo:

Quoted 8 lines

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

Quoted 4 lines

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

Quoted 5 lines

I do not understand why nobody complains for other virtual devices. We should have the same issue with pcap, tap, ring, af_packet, etc. Probably that other drivers are broken in secondary processes. Or should we make a fix to handle every secondary vdev in rte_eth_dev_allocate() ?

Any opinion?

#15 lore raw

Re: [PATCH 0/2] Fix virtio-user multi-process crash.

Hi Thomas,

Quoted 18 lines · Thomas Monjalon

Ping

2017-03-08 12:40, Thomas Monjalon:

Quoted 15 lines

2017-03-02 11:00, Ami Sabo:

Quoted 8 lines

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

Quoted 4 lines

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

I do not understand why nobody complains for other virtual devices. We should have the same issue with pcap, tap, ring, af_packet, etc.

Yes, none of vdev except ring supports primary/secondary mode.

Quoted 3 lines
Quoted 3 lines

Probably that other drivers are broken in secondary processes. Or should we make a fix to handle every secondary vdev in rte_eth_dev_allocate() ?

Agreed. We can change the rte_eth_dev_allocate() like this:
if (primary)

    allocate();
else
    lookup(name);

Besides, we need each vdev to handle its private. For example, pcap should share the selectable unix fd with secondary process; virtio-user should share callfds and kickfds; tap should share the fd pointing to /dev/tun.

Thanks,
Jianfeng

#16 lore raw

Re: [PATCH 0/2] Fix virtio-user multi-process crash.

In reply to Ami Sabo

2017-03-02 11:00, Ami Sabo:

Quoted 8 lines

The patchset fixes secondary process crash issue when it tries to access virtio-user pmd (e.g. via rte_eth_rx_burst).

The crash happens because in virtio_user probing, eth_dev_attach_secondary is not being called, as it does from rte_eth_dev_pci_probe. Therefore, the device is not properly initialized.

Quoted 8 lines

The patchset contains 2 patches:
1. Export rte_eth_dev_attach_secondary, so non-pci drivers will be allowed to call it.
2. Fix the actual bug by calling the function during virtio_user probe.

Ami Sabo (2):
lib/librte_ether: export secondary attach function net/virtio-user: fix multi-process issue

Applied, thanks

author
#17 lore raw

PATCH 2/2 net/virtio-user: fix multi-process issue

In reply to Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

ccessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.
For pci devices the struct is initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().

The fix is to call rte_eth_dev_attach_secondary(), for secondary process, from virtio_user_pmd_probe.

Fixes 7f95f78a8aea ("ethdev: clear data when allocating device")
---
 drivers/net/virtio/virtio_user_ethdev.c | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)
drivers/net/virtio/virtio_user_ethdev.c +16 -10
index 0b226ac..6033908 100644
@@ -418,18 +418,24 @@ virtio_user_pmd_probe(const char *name, const char *params)
 		goto end;
 	}
 
-	eth_dev = virtio_user_eth_dev_alloc(name);
-	if (!eth_dev) {
-		PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
-		goto end;
-	}
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		eth_dev = virtio_user_eth_dev_alloc(name);
+		if (!eth_dev) {
+			PMD_INIT_LOG(ERR, "virtio_user fails to alloc device");
+			goto end;
+		}
 
-	hw = eth_dev->data->dev_private;
-	if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
+		hw = eth_dev->data->dev_private;
+		if (virtio_user_dev_init(hw->virtio_user_dev, path, queues, cq,
 				 queue_size, mac_addr) < 0) {
-		PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
-		virtio_user_eth_dev_free(eth_dev);
-		goto end;
+			PMD_INIT_LOG(ERR, "virtio_user_dev_init fails");
+			virtio_user_eth_dev_free(eth_dev);
+			goto end;
+		}
+	} else {
+		eth_dev = rte_eth_dev_attach_secondary(name);
+		if (!eth_dev)
+			goto end;
 	}
 
 	/* previously called by rte_eal_pci_probe() for physical dev */
Signature · 1 line
2.7.4
#18 lore raw
Quoted 14 lines · Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

ccessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.
For pci devices the struct is initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().

Quoted 4 lines

The fix is to call rte_eth_dev_attach_secondary(), for secondary process, from virtio_user_pmd_probe.

Fixes 7f95f78a8aea ("ethdev: clear data when allocating device")

Are you sure that's the real culprit? As I'm aware of, virtio-user is not built with multiple process support in the beginning. That said, it's likely that the first commit introduces virtio-user is the "culprit" commit.

Besides that, the code looks good to me. If Thomas is fine with your first patch, I could merge them to my tree.

	--yliu
author
#19 lore raw

Hi,
You are right, the commit I mentioned didn't cause the issue - it just flooded it.
The real issue is that rte_eth_dev_allocate should be called only from the primary process.
Tomas's commit flood the issue by resseting rte_eth_dev_data, so now, when the virtio-user secondary process comes up and calls rte_eth_dev_allocate It clears the ethdev->data struct (so fields like rx_queues, mac_addrs, etc will be 0, plus this may cause race condition between the primary and secondary processes...)

	--ami

-----Original Message-----
From: Yuanhan Liu [mailto:[email protected]]
Sent: Tuesday, February 28, 2017 8:40 AM
To: Ami Sabo
Cc: [email protected]; [email protected]
Subject: Re: [PATCH 2/2] net/virtio-user: fix multi-process issue

Quoted 14 lines · Ami Sabo

Secondary process doesn't properly attach to the rte_eth_device initialized by the primary process.

ccessing device from secondary process (e.g. via rte_eth_rx_burst), causes process to crash. because rte_eth_dev_data is not properly set.

The issue was flood by
'commit 7f95f78a8aea ("ethdev: clear data when allocating device")' which now clears rte_eth_dev_data entry.
For pci devices the struct is initialized by rte_eth_dev_pci_probe ->eth_dev_attach_secondary().
However, for virtio-user virtio_user_pmd_probe() is called instead of rte_eth_dev_pci_probe().

Quoted 4 lines

The fix is to call rte_eth_dev_attach_secondary(), for secondary process, from virtio_user_pmd_probe.

Fixes 7f95f78a8aea ("ethdev: clear data when allocating device")

Are you sure that's the real culprit? As I'm aware of, virtio-user is not built with multiple process support in the beginning. That said, it's likely that the first commit introduces virtio-user is the "culprit" commit.

Besides that, the code looks good to me. If Thomas is fine with your first patch, I could merge them to my tree.

	--yliu