Skip to contentSkip to conversation

bus/pci: probe PCI devices in whitelisted order

PATCH In review [email protected] 14 participants 64 messages Sep 23, 2019 – Jun 26, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bus/pci: probe PCI devices in whitelisted order

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

It ensures the eth devices that application uses are probed in device whitelisted sequence, in turn it facilitates the packet forwarding applications to work without any packet loss or performance drop when the underneath network ports have different bandwidths. By altering the whitelist order applications like testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.

---
 drivers/bus/pci/pci_common.c | 67 ++++++++++++++++++++++++++++++--------------
 1 file changed, 46 insertions(+), 21 deletions(-)
drivers/bus/pci/pci_common.c +46 -21
index 6b46b4f..c27a0e9 100644
@@ -293,32 +293,57 @@ rte_pci_probe(void)
 	struct rte_pci_device *dev = NULL;
 	size_t probed = 0, failed = 0;
 	struct rte_devargs *devargs;
-	int probe_all = 0;
+	struct rte_pci_addr addr;
 	int ret = 0;
 
-	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST)
-		probe_all = 1;
-
-	FOREACH_DEVICE_ON_PCIBUS(dev) {
-		probed++;
+	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST) {
+		/* Probe all devices */
+		FOREACH_DEVICE_ON_PCIBUS(dev) {
+			probed++;
 
-		devargs = dev->device.devargs;
-		/* probe all or only whitelisted devices */
-		if (probe_all)
 			ret = pci_probe_all_drivers(dev);
-		else if (devargs != NULL &&
-			devargs->policy == RTE_DEV_WHITELISTED)
-			ret = pci_probe_all_drivers(dev);
-		if (ret < 0) {
-			if (ret != -EEXIST) {
-				RTE_LOG(ERR, EAL, "Requested device "
-					PCI_PRI_FMT " cannot be used\n",
-					dev->addr.domain, dev->addr.bus,
-					dev->addr.devid, dev->addr.function);
-				rte_errno = errno;
-				failed++;
+			if (ret < 0) {
+				if (ret != -EEXIST) {
+					RTE_LOG(ERR, EAL, "Requested device "
+						PCI_PRI_FMT " cannot be used\n",
+						dev->addr.domain, dev->addr.bus,
+						dev->addr.devid,
+						dev->addr.function);
+					rte_errno = errno;
+					failed++;
+				}
+				ret = 0;
+			}
+		}
+	} else {
+		/* Probe only whitelisted devices */
+		RTE_EAL_DEVARGS_FOREACH("pci", devargs) {
+			if (devargs->policy != RTE_DEV_WHITELISTED)
+				continue;
+
+			devargs->bus->parse(devargs->name, &addr);
+
+			FOREACH_DEVICE_ON_PCIBUS(dev) {
+				if (rte_pci_addr_cmp(&dev->addr, &addr))
+					continue;
+				probed++;
+				ret = pci_probe_all_drivers(dev);
+				if (ret < 0) {
+					if (ret != -EEXIST) {
+						RTE_LOG(ERR, EAL,
+							"Requested device "
+							 PCI_PRI_FMT
+							"cannot be used\n",
+							dev->addr.domain,
+							dev->addr.bus,
+							dev->addr.devid,
+							dev->addr.function);
+						rte_errno = errno;
+						failed++;
+					}
+					ret = 0;
+				}
 			}
-			ret = 0;
 		}
 	}
 
Signature · 1 line
2.8.4
Acked-by on v1 1/1
#1 lore raw
Quoted 10 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 23, 2019 14:57
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; [email protected]; Vamsi Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

Quoted 7 lines

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order, it might be useful for bonded devices and representors either.

Quoted 103 lines

It ensures the eth devices that application uses are probed in device whitelisted sequence, in turn it facilitates the packet forwarding applications to work without any packet loss or performance drop when the underneath network ports have different bandwidths. By altering the whitelist order applications like testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.

---
 drivers/bus/pci/pci_common.c | 67 ++++++++++++++++++++++++++++++----
----------
 1 file changed, 46 insertions(+), 21 deletions(-)
drivers/bus/pci/pci_common.c +8 -20
index 6b46b4f..c27a0e9 100644
@@ -293,32 +293,57 @@ rte_pci_probe(void)
 	struct rte_pci_device *dev = NULL;
 	size_t probed = 0, failed = 0;
 	struct rte_devargs *devargs;
-	int probe_all = 0;
+	struct rte_pci_addr addr;
 	int ret = 0;
 
-	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST)
-		probe_all = 1;
-
-	FOREACH_DEVICE_ON_PCIBUS(dev) {
-		probed++;
+	if (rte_pci_bus.bus.conf.scan_mode != RTE_BUS_SCAN_WHITELIST) {
+		/* Probe all devices */
+		FOREACH_DEVICE_ON_PCIBUS(dev) {
+			probed++;
 
-		devargs = dev->device.devargs;
-		/* probe all or only whitelisted devices */
-		if (probe_all)
 			ret = pci_probe_all_drivers(dev);
-		else if (devargs != NULL &&
-			devargs->policy == RTE_DEV_WHITELISTED)
-			ret = pci_probe_all_drivers(dev);
-		if (ret < 0) {
-			if (ret != -EEXIST) {
-				RTE_LOG(ERR, EAL, "Requested device "
-					PCI_PRI_FMT " cannot be used\n",
-					dev->addr.domain, dev->addr.bus,
-					dev->addr.devid, dev-
 >addr.function);
-				rte_errno = errno;
-				failed++;
+			if (ret < 0) {
+				if (ret != -EEXIST) {
+					RTE_LOG(ERR, EAL, "Requested

device "

drivers/bus/pci/pci_common.c +1 -0
+						PCI_PRI_FMT " cannot be

used\n",

drivers/bus/pci/pci_common.c +1 -0
+						dev->addr.domain, dev-
Quoted 1 line

addr.bus,

drivers/bus/pci/pci_common.c +36 -2
+						dev->addr.devid,
+						dev->addr.function);
+					rte_errno = errno;
+					failed++;
+				}
+				ret = 0;
+			}
+		}
+	} else {
+		/* Probe only whitelisted devices */
+		RTE_EAL_DEVARGS_FOREACH("pci", devargs) {
+			if (devargs->policy != RTE_DEV_WHITELISTED)
+				continue;
+
+			devargs->bus->parse(devargs->name, &addr);
+
+			FOREACH_DEVICE_ON_PCIBUS(dev) {
+				if (rte_pci_addr_cmp(&dev->addr, &addr))
+					continue;
+				probed++;
+				ret = pci_probe_all_drivers(dev);
+				if (ret < 0) {
+					if (ret != -EEXIST) {
+						RTE_LOG(ERR, EAL,
+							"Requested device "
+							 PCI_PRI_FMT
+							"cannot be used\n",
+							dev->addr.domain,
+							dev->addr.bus,
+							dev->addr.devid,
+							dev->addr.function);
+						rte_errno = errno;
+						failed++;
+					}
+					ret = 0;
+				}
 			}
-			ret = 0;
 		}
 	}
 
--

2.8.4

#2 lore raw
Quoted 15 lines · Slava Ovsiienko
Quoted 15 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 23, 2019 14:57
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; [email protected]; Vamsi Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Quoted 17 lines
Quoted 2 lines

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order, it might be useful for bonded devices and representors either.

Quoted 9 lines

It ensures the eth devices that application uses are probed in device whitelisted sequence, in turn it facilitates the packet forwarding applications to work without any packet loss or performance drop when the underneath network ports have different bandwidths. By altering the whitelist order applications like testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better handled. The port-mapping part however should be managed at the app level.

Vamsi, you gave the example of l2fwd and testpmd, being able to properly configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports. This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream rework from Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.

As you say Viacheslav, there are use-cases that will rely on fine-grained probe order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.

A solution, would be an EAL parameter (I propose --no-dev), that disable probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.

This will keep the existing behavior stable for current apps, while allowing flexibility for the more advanced ones.

Signature · 2 lines
Gaëtan Rivet
6WIND
#3 lore raw

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaëtan Rivet
Sent: Wednesday, September 25, 2019 2:37 PM
To: Slava Ovsiienko <[email protected]>
Cc: Vamsi Krishna Attunuru <[email protected]>; [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

Quoted 15 lines · Slava Ovsiienko
Quoted 15 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 23, 2019 14:57
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; [email protected]; Vamsi Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Quoted 17 lines
Quoted 2 lines

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order, it might be useful for bonded devices and representors either.

Quoted 9 lines

It ensures the eth devices that application uses are probed in
device whitelisted sequence, in turn it facilitates the packet
forwarding applications to work without any packet loss or
performance drop when the underneath network ports have different
bandwidths. By altering the whitelist order applications like
testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better handled. The port-mapping part however should be managed at the app level.

Vamsi, you gave the example of l2fwd and testpmd, being able to properly configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports. This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream rework from Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.

As you say Viacheslav, there are use-cases that will rely on fine-grained probe order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.

A solution, would be an EAL parameter (I propose --no-dev), that disable probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.

This will keep the existing behavior stable for current apps, while allowing flexibility for the more advanced ones.

Hi Gaetan,

Thanks, vdev part was not taken care in this patch. Rather than imposing hotplug for every application which requires port mapping, If vdev probing order is also handled same as pdevs(in whitelist order), existing whitelisting feature will serve the port mapping requirement, right. Also the existing applications get benefited instead of overloading them with more configuration options. If these probing order is not needed by default, it can be triggered using an EAL parameter(not added yet).

Regards,
A Vamsi

--
Gaëtan Rivet
6WIND

#4 lore raw
Quoted 59 lines · Vamsi Krishna Attunuru

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaëtan Rivet
Sent: Wednesday, September 25, 2019 2:37 PM
To: Slava Ovsiienko <[email protected]>
Cc: Vamsi Krishna Attunuru <[email protected]>; [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

Quoted 15 lines · Slava Ovsiienko
Quoted 15 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 23, 2019 14:57
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; [email protected]; Vamsi Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Quoted 17 lines
Quoted 2 lines

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order, it might be useful for bonded devices and representors either.

Quoted 9 lines

It ensures the eth devices that application uses are probed in
device whitelisted sequence, in turn it facilitates the packet
forwarding applications to work without any packet loss or
performance drop when the underneath network ports have different
bandwidths. By altering the whitelist order applications like
testpmd, l2fwd can forward the ingress traffic to egress port that has of equivalent bandwidth.

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better handled. The port-mapping part however should be managed at the app level.

Vamsi, you gave the example of l2fwd and testpmd, being able to properly configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports. This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream rework from Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.

As you say Viacheslav, there are use-cases that will rely on fine-grained probe order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.

A solution, would be an EAL parameter (I propose --no-dev), that disable probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.

This will keep the existing behavior stable for current apps, while allowing flexibility for the more advanced ones.

Quoted 6 lines

Hi Gaetan,

Thanks, vdev part was not taken care in this patch. Rather than imposing hotplug for every application which requires port mapping, If vdev probing order is also handled same as pdevs(in whitelist order), existing whitelisting feature will serve the port mapping requirement, right. Also the existing applications get benefited instead of overloading them with more configuration options. If these probing order is not needed by default, it can be triggered using an EAL parameter(not added yet).

Regards,
A Vamsi

Hi,

The way buses are written right now, they will each do a whole scan, then they each probe all their devices.

You cannot intersperse probes across several buses, i.e. probe a PCI device, then a vdev, then another PCI device.

Changing this structure could be difficult. A possible way to do what you want without breaking everything would be to do what the app would have done in my solution above, but from within the EAL: block all probes, then go over a mixed list of (-w) and (--vdev) parameters and hotplug them in order. This would require the --no-dev (or --no-probe, or --no-auto-probe) flag anyway (or as a conf item, or something at least telling the EAL to behave this way).

Would this way of doing it work for you?

In any case, controlling the probe order should be fixed properly for all buses and the general use-case if possible, instead of limiting the patch to the PCI bus.

Kind regards,

Signature · 2 lines
Gaëtan Rivet
6WIND
#5 lore raw
Quoted 118 lines

-----Original Message-----
From: Gaëtan Rivet <[email protected]>
Sent: Thursday, September 26, 2019 1:34 PM
To: Vamsi Krishna Attunuru <[email protected]>
Cc: Slava Ovsiienko <[email protected]>; [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [EXT] Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

External Email

----------------------------------------------------------------------

Quoted 12 lines · Vamsi Krishna Attunuru

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaëtan Rivet
Sent: Wednesday, September 25, 2019 2:37 PM
To: Slava Ovsiienko <[email protected]>
Cc: Vamsi Krishna Attunuru <[email protected]>; [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

Quoted 32 lines
Quoted 31 lines · Slava Ovsiienko
Quoted 11 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of [email protected]
Sent: Monday, September 23, 2019 14:57
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; Thomas Monjalon <[email protected]>; [email protected]; Vamsi Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v1 1/1] bus/pci: probe PCI devices in whitelisted order

Quoted 7 lines

From: Vamsi Attunuru <[email protected]>

Current pci bus driver scans pci devices in the order that it read from sysfs. Accordingly all or whitelisted devices are getting probed.

Patch modifies the probing order of whitelisted pci devices in a sequence the devices are whitelisted(using EAL flags).

Thanks, it would be nice to have opportunity to control probing order, it might be useful for bonded devices and representors either.

Quoted 7 lines

It ensures the eth devices that application uses are probed in device whitelisted sequence, in turn it facilitates the packet forwarding applications to work without any packet loss or performance drop when the underneath network ports have different bandwidths. By altering the whitelist order applications like testpmd, l2fwd can forward the ingress traffic to egress port that has of

equivalent bandwidth.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

Hello Vamsi, Viacheslav,

This is a nice patch. I agree that port dependency could be better handled. The

port-mapping part however should be managed at the app level.

Quoted 2 lines

Vamsi, you gave the example of l2fwd and testpmd, being able to properly

configure forwarding directions implicitly. I think the better approach here is to add these configurations items within the apps directly. Configuring the mapping at the port level is not precise enough. The proper control is about cores, port and queues, not only ports.

Quoted 3 lines

This patch only solves a limited part of this issue with testpmd.

I wrote a command to do this, that collided with some stream rework from

Intel at the time (3, 4 years back?), so I did not take the time to force it through. If there is a need we could discuss about adding this back. I had needed it to write a PMD, that could be useful to others.

Quoted 2 lines

As you say Viacheslav, there are use-cases that will rely on fine-grained probe

order. However, this patch solves this issue only regarding PCI devices, depending on other PCI devices. We have in EAL an improper hack about it, forcing the vdev probe last, because usually ports depending on others are virtual ones. As this patch shows, the hack is not sufficient, and as the hack shows, this patch does not cover everything.

Quoted 2 lines

A solution, would be an EAL parameter (I propose --no-dev), that disable

probing for all buses. Applications and devices requiring a fine-grained probe order, are then free to start in this mode (and maybe force it through EAL conf), then hotplug ports as they see fit.

Quoted 2 lines

This will keep the existing behavior stable for current apps, while allowing

flexibility for the more advanced ones.

Quoted 5 lines

Hi Gaetan,

Thanks, vdev part was not taken care in this patch. Rather than imposing

hotplug for every application which requires port mapping, If vdev probing order is also handled same as pdevs(in whitelist order), existing whitelisting feature will serve the port mapping requirement, right. Also the existing applications get benefited instead of overloading them with more configuration options. If these probing order is not needed by default, it can be triggered using an EAL parameter(not added yet).

Quoted 3 lines

Regards,
A Vamsi

Hi,

The way buses are written right now, they will each do a whole scan, then they each probe all their devices.

You cannot intersperse probes across several buses, i.e. probe a PCI device, then a vdev, then another PCI device.

Quoted 8 lines

Changing this structure could be difficult. A possible way to do what you want without breaking everything would be to do what the app would have done in my solution above, but from within the EAL: block all probes, then go over a mixed list of (-w) and (--vdev) parameters and hotplug them in order. This would require the --no-dev (or --no-probe, or --no-auto-probe) flag anyway (or as a conf item, or something at least telling the EAL to behave this way).

Would this way of doing it work for you?

Yes, above approach sounds fine to me and it works without breaking everything.

Quoted 3 lines

In any case, controlling the probe order should be fixed properly for all buses and the general use-case if possible, instead of limiting the patch to the PCI bus.

Ack

Quoted 5 lines

Kind regards,
--
Gaëtan Rivet
6WIND

#6 lore raw

PATCH 1/1 eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..3996e429d 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL wil now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0

47 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +17 -0
index b7cf91214..bd9087bcc 100644
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +5 -0
index d53f0fe69..2daa63b23 100644
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
lib/librte_eal/linux/eal/eal.c +5 -0
index 946222ccd..476fc2e4a 100644
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
lib/librte_eal/rte_eal_version.map +7 -0
index 7cbf82d37..fe5395c24 100644
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.05 {
+	global:
+
+	rte_eal_manual_probe
+
+} DPDK_19.11;
+
 EXPERIMENTAL {
 	global:
 
Signature · 1 line
2.11.0
#7 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 276 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..3996e429d 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL wil now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0
index 9e4f09d83..3b7769a3c 100644
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0 || !rte_dev_is_probed(dev)) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +17 -0
index b7cf91214..bd9087bcc 100644
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +5 -0
index d53f0fe69..2daa63b23 100644
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
lib/librte_eal/linux/eal/eal.c +5 -0
index 946222ccd..476fc2e4a 100644
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..fe5395c24 100644

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 318

Quoted 7 lines
lib/librte_eal/rte_eal_version.map +4 -0
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.05 {
+	global:
+
+	rte_eal_manual_probe

Syntax error - needs a ';' at the end of this symbol.

Quoted 5 lines
lib/librte_eal/rte_eal_version.map +3 -0
+
+} DPDK_19.11;
+
 EXPERIMENTAL {
 	global:
#8 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 279 lines · Aaron Conole
Quoted 277 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..3996e429d 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL wil now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0
index 9e4f09d83..3b7769a3c 100644
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0 || !rte_dev_is_probed(dev)) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +17 -0
index b7cf91214..bd9087bcc 100644
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +5 -0
index d53f0fe69..2daa63b23 100644
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
lib/librte_eal/linux/eal/eal.c +5 -0
index 946222ccd..476fc2e4a 100644
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..fe5395c24 100644
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
Quoted 9 lines

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 318

Quoted 6 lines
lib/librte_eal/rte_eal_version.map +4 -0
 } DPDK_19.05;
 
+DPDK_19.05 {
+	global:
+
+	rte_eal_manual_probe

Syntax error - needs a ';' at the end of this symbol.

Ah yes, sorry.
I'm thinking there is a script or a validation tool using this, I should try it.

Quoted 5 lines
Quoted 5 lines
lib/librte_eal/rte_eal_version.map +3 -0
+
+} DPDK_19.11;
+
 EXPERIMENTAL {
 	global:
Signature · 2 lines
Gaëtan Rivet
6WIND
#9 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 290 lines · Gaëtan Rivet
Quoted 279 lines · Aaron Conole
Quoted 277 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..3996e429d 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL wil now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0
index 9e4f09d83..3b7769a3c 100644
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char *devname,
 }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not supported by bus %s, "
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0 || !rte_dev_is_probed(dev)) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device %s\n",
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +17 -0
index b7cf91214..bd9087bcc 100644
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +5 -0
index d53f0fe69..2daa63b23 100644
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
lib/librte_eal/linux/eal/eal.c +5 -0
index 946222ccd..476fc2e4a 100644
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..fe5395c24 100644
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
Quoted 9 lines

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 318

Quoted 6 lines
lib/librte_eal/rte_eal_version.map +4 -0
 } DPDK_19.05;
 
+DPDK_19.05 {
+	global:
+
+	rte_eal_manual_probe

Syntax error - needs a ';' at the end of this symbol.

Quoted 2 lines

Ah yes, sorry.
I'm thinking there is a script or a validation tool using this, I should try it.

One method I use is to push to my personal github account, linked to Travis-CI - then I get emails when it fails ;)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
lib/librte_eal/rte_eal_version.map +3 -0
+
+} DPDK_19.11;
+
 EXPERIMENTAL {
 	global:
#10 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

In reply to Gaetan Rivet
Quoted 6 lines · On Mon, 30 Sep 2019 14:51:03 +0200 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Quoted 7 lines

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

I have no problems with the patch, but it would help if there was better way to handle device naming policy in DPDK. Applications that depend on particular port number are prone to get broken by changes in surrounding OS or hardware environment. Just like Linux applications that are built to depend on "eth0"; which is unfortunately all too common.

#11 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 15 lines · Stephen Hemminger
Quoted 12 lines · On Mon, 30 Sep 2019 14:51:03 +0200 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

Quoted 7 lines
Quoted 1 line

I have no problems with the patch, but it would help if there was better way to handle device naming policy in DPDK. Applications that depend on particular port number are prone to get broken by changes in surrounding OS or hardware environment. Just like Linux applications that are built to depend on "eth0"; which is unfortunately all too common.

Hello Stephen,

This patch is a way to avoid having the PCI bus defining the probe order with the current hardware environment. It seems to be a step in the right direction for the issue you identify.

There is a tight coupling between device names and driver matches for the vdev bus, but that seems difficult to avoid.

Do you see other EAL APIs fostering an over reliance of downstream systems on device names?

I pushed a few months back a way to iterate / match devices by their properties. If you identify other pain points, this could certainly be improved as well.

Signature · 2 lines
Gaëtan Rivet
6WIND
#12 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 34 lines · Gaëtan Rivet
Quoted 15 lines · Stephen Hemminger
Quoted 12 lines · On Mon, 30 Sep 2019 14:51:03 +0200 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

Quoted 7 lines
Quoted 1 line

I have no problems with the patch, but it would help if there was better way to handle device naming policy in DPDK. Applications that depend on particular port number are prone to get broken by changes in surrounding OS or hardware environment. Just like Linux applications that are built to depend on "eth0"; which is unfortunately all too common.

Hello Stephen,

This patch is a way to avoid having the PCI bus defining the probe order with the current hardware environment. It seems to be a step in the right direction for the issue you identify.

There is a tight coupling between device names and driver matches for the vdev bus, but that seems difficult to avoid.

Quoted 6 lines

Do you see other EAL APIs fostering an over reliance of downstream systems on device names?

I pushed a few months back a way to iterate / match devices by their properties. If you identify other pain points, this could certainly be improved as well.

And this mode will be kicked in only when "--manual-probe" selected on eal arguments.
So it won't change the behavior of the existing applications.

Quoted 4 lines

--
Gaėtan Rivet
6WIND

#13 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 38 lines · Jerin Jacob
Quoted 37 lines · Gaëtan Rivet
Quoted 15 lines · Stephen Hemminger
Quoted 12 lines · On Mon, 30 Sep 2019 14:51:03 +0200 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

Quoted 7 lines
Quoted 1 line

I have no problems with the patch, but it would help if there was better way to handle device naming policy in DPDK. Applications that depend on particular port number are prone to get broken by changes in surrounding OS or hardware environment. Just like Linux applications that are built to depend on "eth0"; which is unfortunately all too common.

Hello Stephen,

This patch is a way to avoid having the PCI bus defining the probe order with the current hardware environment. It seems to be a step in the right direction for the issue you identify.

There is a tight coupling between device names and driver matches for the vdev bus, but that seems difficult to avoid.

Do you see other EAL APIs fostering an over reliance of downstream systems on device names?

Quoted 8 lines
Quoted 3 lines

I pushed a few months back a way to iterate / match devices by their properties. If you identify other pain points, this could certainly be improved as well.

And this mode will be kicked in only when "--manual-probe" selected on eal arguments.
So it won't change the behavior of the existing applications.

If I read you correctly, if hardware independence is the proper way to function, we should switch entirely to it.

I agree, but that means rewriting entirely the probe step of rte_bus. This patch is a incremental step, I preferred to keep risks low.

It is not clear from your remark whether you are considering this limitation a good or a bad thing however :)
Can you be a bit more explicit?

Signature · 2 lines
Gaëtan Rivet
6WIND
#14 lore raw

Re: [dpdk-dev] [PATCH v1] eal: add manual probing option

Quoted 49 lines · Gaëtan Rivet
Quoted 38 lines · Jerin Jacob
Quoted 37 lines · Gaėtan Rivet
Quoted 15 lines · Stephen Hemminger
Quoted 12 lines · On Mon, 30 Sep 2019 14:51:03 +0200 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

Quoted 7 lines
Quoted 1 line

I have no problems with the patch, but it would help if there was better way to handle device naming policy in DPDK. Applications that depend on particular port number are prone to get broken by changes in surrounding OS or hardware environment. Just like Linux applications that are built to depend on "eth0"; which is unfortunately all too common.

Hello Stephen,

This patch is a way to avoid having the PCI bus defining the probe order with the current hardware environment. It seems to be a step in the right direction for the issue you identify.

There is a tight coupling between device names and driver matches for the vdev bus, but that seems difficult to avoid.

Do you see other EAL APIs fostering an over reliance of downstream systems on device names?

Quoted 8 lines
Quoted 3 lines

I pushed a few months back a way to iterate / match devices by their properties. If you identify other pain points, this could certainly be improved as well.

And this mode will be kicked in only when "--manual-probe" selected on eal arguments.
So it won't change the behavior of the existing applications.

Quoted 5 lines

If I read you correctly, if hardware independence is the proper way to function, we should switch entirely to it.

I agree, but that means rewriting entirely the probe step of rte_bus. This patch is a incremental step, I preferred to keep risks low.

+1

Quoted 3 lines

It is not clear from your remark whether you are considering this limitation a good or a bad thing however :)

I am considering it as a good step.

Quoted 5 lines

Can you be a bit more explicit?

--
Gaėtan Rivet
6WIND

#15 lore raw

PATCH v2 1/1 eal: add manual probing option

In reply to Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 17 +++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           |  5 ++++
 lib/librte_eal/linux/eal/eal.c             |  5 ++++
 lib/librte_eal/rte_eal_version.map         |  7 +++++
 11 files changed, 110 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..88d06e82e 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now propose to enforce manual probing of devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0

47 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +17 -0
index b7cf91214..bd9087bcc 100644
@@ -465,6 +465,23 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option.
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ */
+int rte_eal_manual_probe(void);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +5 -0
index d53f0fe69..2daa63b23 100644
@@ -961,6 +961,11 @@ int rte_eal_has_pci(void)
 	return !internal_config.no_pci;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 int rte_eal_create_uio_dev(void)
 {
 	return internal_config.create_uio_dev;
lib/librte_eal/linux/eal/eal.c +5 -0
index 946222ccd..476fc2e4a 100644
@@ -1376,6 +1376,11 @@ int rte_eal_create_uio_dev(void)
 	return internal_config.create_uio_dev;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
 enum rte_intr_mode
 rte_eal_vfio_intr_mode(void)
 {
lib/librte_eal/rte_eal_version.map +7 -0
index 7cbf82d37..77f34542b 100644
@@ -312,6 +312,13 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+	global:
+
+	rte_eal_manual_probe;
+
+} DPDK_19.05;
+
 EXPERIMENTAL {
 	global:
 
Signature · 1 line
2.11.0
#16 lore raw

PATCH v3 1/1 eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (herited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
 lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
 lib/librte_eal/rte_eal_version.map         |  8 ++++++
 11 files changed, 138 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..700f9a726 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow manual probing devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +41 -0

47 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +34 -0
index b7cf91214..da86fdb6b 100644
@@ -465,6 +465,40 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +10 -0
index d53f0fe69..cc6d846bd 100644
@@ -972,6 +972,16 @@ rte_eal_vfio_intr_mode(void)
 	return RTE_INTR_MODE_NONE;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
 int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
 		      __rte_unused const char *dev_addr,
 		      __rte_unused int *vfio_dev_fd,
lib/librte_eal/linux/eal/eal.c +10 -0
index 946222ccd..da00eb14d 100644
@@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
 	return internal_config.vfio_intr_mode;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
 int
 rte_eal_check_module(const char *module_name)
 {
lib/librte_eal/rte_eal_version.map +8 -0
index 7cbf82d37..ccc4ffb21 100644
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+	global:
+
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
+
+} DPDK_19.08;
+
 EXPERIMENTAL {
 	global:
 
Signature · 1 line
2.11.0
author Acked-by Tested-by on v3 1/1
#17 lore raw

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

Quoted 44 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaetan Rivet
Sent: Friday, October 4, 2019 6:25 PM
To: [email protected]; Vamsi Krishna Attunuru <[email protected]>
Cc: Gaetan Rivet <[email protected]>; Ferruh Yigit <[email protected]>; Anatoly Burakov <[email protected]>; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; David Marchand <[email protected]>
Subject: [dpdk-dev] [PATCH v3] eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

Quoted 7 lines

v3: properly fixed the map file (herited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in
it!

@Gaetan: Thanks a lot for providing this patch. I have tested it in our use cases and it works as expected, also verified multiple combinations by passing whitelisted pdevs, vdevs and mix of both. Probing sequence happens as required.

Quoted 319 lines

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41
++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 34
+++++++++++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
 lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
 lib/librte_eal/rte_eal_version.map         |  8 ++++++
 11 files changed, 138 insertions(+)
doc/guides/rel_notes/release_19_11.rst +4 -0
index 27cfbd9e3..700f9a726 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow manual probing devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being

present.

doc/guides/rel_notes/release_19_11.rst +5 -0
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+ also be used to disable automatic probing from the ``PCI`` bus without
+ having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +19 -0
index 9e4f09d83..f4ce1c56c 100644
@@ -109,6 +109,47 @@ build_devargs(const char *busname, const char
 *devname,  }
 
 int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name, da-
 >name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s on bus
 %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug) not

supported by bus %s, "

lib/librte_eal/common/eal_common_dev.c +12 -0
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus. Probing
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device

%s\n",

lib/librte_eal/common/eal_common_dev.c +10 -0
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+
+int
 rte_eal_hotplug_add(const char *busname, const char *devname,
 		    const char *drvargs)
 {
lib/librte_eal/common/eal_common_options.c +1 -0
index 05cae5f75..66c232b14 100644
@@ -81,6 +81,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
 OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
 OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }

};

lib/librte_eal/common/eal_common_options.c +3 -0
@@ -1408,6 +1409,9 @@ eal_parse_common_option(int opt, const char
 *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		conf->manual_probe = 1;
+		break;
 
 	/* don't know what to do, leave this to caller */
lib/librte_eal/common/eal_common_options.c +3 -0
 	default:
@@ -1634,6 +1638,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug

API.\n"

lib/librte_eal/common/eal_common_options.c +1 -0
+	       "                      PCI and vdev declarations will be treated in order as

hotplug commands.\n"

lib/librte_eal/common/eal_common_options.c +0 -0
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device

probing. */
volatile unsigned vmware_tsc_map; /**< true to use VMware TSC
mapping

* instead of native TSC */
volatile unsigned no_shconf; /**< true if there is no shared config
*/

lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);  void  rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +34 -0
index b7cf91214..da86fdb6b 100644
@@ -465,6 +465,40 @@ int rte_eal_has_hugepages(void);  int
 rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/freebsd/eal/eal.c +8 -0
index d53f0fe69..cc6d846bd 100644
@@ -972,6 +972,16 @@ rte_eal_vfio_intr_mode(void)
 	return RTE_INTR_MODE_NONE;
 }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled) {
+	internal_config.manual_probe = !!enabled; }
+
 int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
 		      __rte_unused const char *dev_addr,
 		      __rte_unused int *vfio_dev_fd,
lib/librte_eal/linux/eal/eal.c index renamed from lib/librte_eal/linux/eal/eal.c +0 -0

946222ccd..da00eb14d 100644

lib/librte_eal/linux/eal/eal.c +8 -0
@@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
 	return internal_config.vfio_intr_mode;  }
 
+int rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled) {
+	internal_config.manual_probe = !!enabled; }
+
 int
 rte_eal_check_module(const char *module_name)  { diff --git

a/lib/librte_eal/rte_eal_version.map b/lib/librte_eal/rte_eal_version.map index 7cbf82d37..ccc4ffb21 100644

lib/librte_eal/rte_eal_version.map +8 -1
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+	global:
+
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
+
+} DPDK_19.08;
+
 EXPERIMENTAL {
 	global:
 
--

2.11.0

#18 lore raw

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

Hi,

Quoted 52 lines · Vamsi Krishna Attunuru
Quoted 44 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaetan Rivet
Sent: Friday, October 4, 2019 6:25 PM
To: [email protected]; Vamsi Krishna Attunuru <[email protected]>
Cc: Gaetan Rivet <[email protected]>; Ferruh Yigit <[email protected]>; Anatoly Burakov <[email protected]>; Thomas Monjalon <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; David Marchand <[email protected]>
Subject: [dpdk-dev] [PATCH v3] eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL.
This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

Quoted 7 lines

v3: properly fixed the map file (herited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in
it!

Quoted 17 lines

@Gaetan: Thanks a lot for providing this patch. I have tested it in our use cases and it works as expected, also verified multiple combinations by passing whitelisted pdevs, vdevs and mix of both. Probing sequence happens as required.

Quoted 10 lines

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

Any further comment for this patch? Can someone review it?

Signature · 2 lines
Gaëtan Rivet
6WIND
#19 lore raw

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

In reply to Gaetan Rivet
Quoted 93 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (herited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
 lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
 lib/librte_eal/rte_eal_version.map         |  8 ++++++
 11 files changed, 138 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..700f9a726 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow manual probing devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +1 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>

lib/librte_eal/common/eal_common_bus.c rte_bus_probe(void) · line 67

Quoted 4 lines
lib/librte_eal/common/eal_common_bus.c +1 -0
@@ -63,6 +64,11 @@ rte_bus_probe(void)
        int ret;
        struct rte_bus *bus, *vbus = NULL;
 
+       if (rte_eal_manual_probe()) {

See below,

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +0 -0
 int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
                      __rte_unused const char *dev_addr,
                      __rte_unused int *vfio_dev_fd,
lib/librte_eal/linux/eal/eal.c +0 -0
index 946222ccd..da00eb14d 100644
@@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
        return internal_config.vfio_intr_mode;
 }

lib/librte_eal/linux/eal/eal.c rte_eal_vfio_intr_mode(void) · line 1393

Quoted 10 lines
lib/librte_eal/linux/eal/eal.c +9 -0
 
+int rte_eal_manual_probe(void)
+{
+       return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+       internal_config.manual_probe = !!enabled;
+}

I don't think FreeBSD and Linux specific implementation is NOT required as internal_config accessible in eal/common. example diff.

[master][dpdk.org] $ git diff

(quoted diff) +0 -0

b/lib/librte_eal/common/eal_common_bus.c
index 145a96812..db4257cf1 100644

lib/librte_eal/common/eal_common_bus.c +2 -1
@@ -13,6 +13,7 @@
 #include <rte_errno.h>
 
 #include "eal_private.h"
+#include "eal_internal_cfg.h"
 
 static struct rte_bus_list rte_bus_list =
        TAILQ_HEAD_INITIALIZER(rte_bus_list);
@@ -64,7 +65,7 @@ rte_bus_probe(void)
        int ret;
        struct rte_bus *bus, *vbus = NULL;
 
-       if (rte_eal_manual_probe()) {
+       if (internal_config.manual_probe) {
                RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
                return rte_dev_probe_devargs_list();
        }
Quoted 8 lines
lib/librte_eal/linux/eal/eal.c +0 -0
 int
 rte_eal_check_module(const char *module_name)
 {
lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..ccc4ffb21 100644

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 319

Quoted 8 lines
lib/librte_eal/rte_eal_version.map +5 -0
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
+DPDK_19.11 {
+       global:
+
+       rte_eal_manual_probe;
+       rte_eal_manual_probe_set;

Do we need public API for this? it is only used by eal lib.

checkpatch complains the following too.
ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11 section, but is expected to be added in the EXPERIMENTAL section of the version map

#20 lore raw

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

Quoted 162 lines · Jerin Jacob
Quoted 93 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (herited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++++
 lib/librte_eal/common/eal_common_dev.c     | 41 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++++
 lib/librte_eal/common/include/rte_eal.h    | 34 +++++++++++++++++++++++++
 lib/librte_eal/freebsd/eal/eal.c           | 10 ++++++++
 lib/librte_eal/linux/eal/eal.c             | 10 ++++++++
 lib/librte_eal/rte_eal_version.map         |  8 ++++++
 11 files changed, 138 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index 27cfbd9e3..700f9a726 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow manual probing devices.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 
 Removed Items
 -------------
lib/librte_eal/common/eal_common_bus.c +1 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>

lib/librte_eal/common/eal_common_bus.c rte_bus_probe(void) · line 67

Quoted 4 lines
lib/librte_eal/common/eal_common_bus.c +1 -0
@@ -63,6 +64,11 @@ rte_bus_probe(void)
        int ret;
        struct rte_bus *bus, *vbus = NULL;
 
+       if (rte_eal_manual_probe()) {

See below,

Quoted 10 lines
lib/librte_eal/common/eal_common_bus.c +0 -0
 int rte_vfio_setup_device(__rte_unused const char *sysfs_base,
                      __rte_unused const char *dev_addr,
                      __rte_unused int *vfio_dev_fd,
lib/librte_eal/linux/eal/eal.c +0 -0
index 946222ccd..da00eb14d 100644
@@ -1382,6 +1382,16 @@ rte_eal_vfio_intr_mode(void)
        return internal_config.vfio_intr_mode;
 }

lib/librte_eal/linux/eal/eal.c rte_eal_vfio_intr_mode(void) · line 1393

Quoted 10 lines
lib/librte_eal/linux/eal/eal.c +9 -0
 
+int rte_eal_manual_probe(void)
+{
+       return internal_config.manual_probe;
+}
+
+void rte_eal_manual_probe_set(int enabled)
+{
+       internal_config.manual_probe = !!enabled;
+}

I don't think FreeBSD and Linux specific implementation is NOT required as internal_config accessible in eal/common. example diff.

[master][dpdk.org] $ git diff

lib/librte_eal/common/eal_common_bus.c +2 -1
index 145a96812..db4257cf1 100644
@@ -13,6 +13,7 @@
 #include <rte_errno.h>
 
 #include "eal_private.h"
+#include "eal_internal_cfg.h"
 
 static struct rte_bus_list rte_bus_list =
        TAILQ_HEAD_INITIALIZER(rte_bus_list);
@@ -64,7 +65,7 @@ rte_bus_probe(void)
        int ret;
        struct rte_bus *bus, *vbus = NULL;
 
-       if (rte_eal_manual_probe()) {
+       if (internal_config.manual_probe) {
                RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
                return rte_dev_probe_devargs_list();
        }
Quoted 12 lines
lib/librte_eal/linux/eal/eal.c +0 -0
 int
 rte_eal_check_module(const char *module_name)
 {
lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..ccc4ffb21 100644
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
Quoted 8 lines

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 319

Quoted 5 lines
lib/librte_eal/rte_eal_version.map +5 -0
+DPDK_19.11 {
+       global:
+
+       rte_eal_manual_probe;
+       rte_eal_manual_probe_set;

Do we need public API for this? it is only used by eal lib.

I think some PMDs and some applications would need to issue warnings to the user if manual probing is not enabled (i.e. a port representor parameter was used in a PMD, but probing is automatic --> on some platform it would work and others it would fail with the same config).

Some applications would also prefer forcing it enabled without having to use the option on the command line.

This motivates the public API. Given that there is a public API, I think it's better to use it even from within the EAL.

However I agree that it does not need two separate implementations. I could move it to some eal_common_* part probably, maybe bus or dev?

Quoted 4 lines

checkpatch complains the following too.
ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11 section, but is expected to be added in the EXPERIMENTAL section of the version map

The API is simple enough IMO to make it immediately stable: it will allow applications forbidding the use of experimental API to make use of it.

This is only for the sake of simplicity for some users. I'm not opposed to making it experimental first if it's expected for all new APIs.

Signature · 2 lines
Gaëtan Rivet
6WIND
#21 lore raw

Re: [dpdk-dev] [PATCH v3] eal: add manual probing option

Quoted 24 lines
Quoted 10 lines
Quoted 10 lines

rte_eal_check_module(const char *module_name)
{

lib/librte_eal/rte_eal_version.map +0 -0
index 7cbf82d37..ccc4ffb21 100644
@@ -312,6 +312,14 @@ DPDK_19.08 {
 
 } DPDK_19.05;
 
Quoted 8 lines

lib/librte_eal/rte_eal_version.map DPDK_19.08 { · line 319

Quoted 5 lines
lib/librte_eal/rte_eal_version.map +5 -0
+DPDK_19.11 {
+       global:
+
+       rte_eal_manual_probe;
+       rte_eal_manual_probe_set;

Do we need public API for this? it is only used by eal lib.

I think some PMDs and some applications would need to issue warnings to the user if manual probing is not enabled (i.e. a port representor parameter was used in a PMD, but probing is automatic --> on some platform it would work and others it would fail with the same config).

Quoted 4 lines

Some applications would also prefer forcing it enabled without having to use the option on the command line.

This motivates the public API. Given that there is a public API, I think

it's better to use it even from within the EAL.

No issue from my side to expose through public API as well. Please rte_eal_manual_probe_set() when setting in flags in eal as well. Currently rte_eal_manual_probe_set() not used anywhere.

Quoted 3 lines

However I agree that it does not need two separate implementations. I could move it to some eal_common_* part probably, maybe bus or dev?

eal should be fine. No strong opinion on the place.

Quoted 6 lines
Quoted 5 lines

checkpatch complains the following too.
ERROR: symbol rte_eal_manual_probe_set is added in the DPDK_19.11 section, but is expected to be added in the EXPERIMENTAL section of the version map

Quoted 7 lines

The API is simple enough IMO to make it immediately stable: it will allow applications forbidding the use of experimental API to make use of it.

This is only for the sake of simplicity for some users. I'm not opposed to making it experimental first if it's expected for all new APIs.

All new APIs are treated as experimental. I don't think, we need to make an exception here.

Other these nits, This patch looks good.

Quoted 4 lines

--
Gaėtan Rivet
6WIND

#22 lore raw

PATCH v4 1/1 eal: add manual probing option

In reply to Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

 doc/guides/rel_notes/release_19_11.rst     |  9 +++++
 lib/librte_eal/common/eal_common_bus.c     |  6 ++++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 +++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++
 lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 +++
 9 files changed, 129 insertions(+)
doc/guides/rel_notes/release_19_11.rst +9 -0
index e77d226b5..7251a17b2 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **FreeBSD now supports `--base-virtaddr` EAL option.**
 
   FreeBSD version now also supports setting base virtual address for mapping
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +54 -0

66 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index a7f9c5f9b..de73970fe 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +36 -0
index b7cf91214..71d9b4ca6 100644
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/rte_eal_version.map +4 -0
index 7cbf82d37..7d7ff6531 100644
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
Signature · 1 line
2.11.0
Reviewed-by on v4 1/1
#23 lore raw

Re: [dpdk-dev] [PATCH v4] eal: add manual probing option

Quoted 16 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

Quoted 9 lines
 doc/guides/rel_notes/release_19_11.rst     |  9 +++++
 lib/librte_eal/common/eal_common_bus.c     |  6 ++++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 +++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++
 lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 +++

Missed to update new eal argument in
doc/guides/linux_gsg/eal_args.include.rst file.

With the above change:

#24 lore raw

PATCH v5 1/1 eal: add manual probing option

In reply to Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

 doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
 doc/guides/rel_notes/release_19_11.rst     |  9 +++++
 lib/librte_eal/common/eal_common_bus.c     |  6 ++++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 +++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 ++
 lib/librte_eal/common/eal_private.h        |  9 +++++
 lib/librte_eal/common/include/rte_eal.h    | 36 ++++++++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 +++
 10 files changed, 142 insertions(+)
doc/guides/linux_gsg/eal_args.include.rst +13 -0
index ed8b0e35b..d0717d4a0 100644
@@ -69,6 +69,19 @@ Device-related options
 
        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
     Load external drivers. An argument can be a single shared object file, or a
doc/guides/rel_notes/release_19_11.rst +9 -0
index e77d226b5..7251a17b2 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **FreeBSD now supports `--base-virtaddr` EAL option.**
 
   FreeBSD version now also supports setting base virtual address for mapping
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +54 -0

66 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index a7f9c5f9b..de73970fe 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);
 void
 rte_option_usage(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +36 -0
index b7cf91214..71d9b4ca6 100644
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/rte_eal_version.map +4 -0
index 7cbf82d37..7d7ff6531 100644
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
Signature · 1 line
2.11.0
Reviewed-by on v5 1/1
#25 lore raw

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

Quoted 7 lines · Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Quoted 7 lines

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

#26 lore raw

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

In reply to Gaetan Rivet

Ping
@[email protected]

Since, you weren't reachable on regular email I rebased the patch on ToT and sent a v6
http://patches.dpdk.org/patch/64235/

Quoted 395 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaetan Rivet
Sent: Friday, October 25, 2019 9:17 PM
To: [email protected]
Cc: Gaetan Rivet <[email protected]>; Vamsi Krishna Attunuru <[email protected]>; David Marchand <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [dpdk-dev] [PATCH v5] eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and -- vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one
interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
doc/guides/rel_notes/release_19_11.rst     |  9 +++++
lib/librte_eal/common/eal_common_bus.c     |  6 ++++
lib/librte_eal/common/eal_common_dev.c     | 54
++++++++++++++++++++++++++++++
lib/librte_eal/common/eal_common_options.c |  8 +++++
lib/librte_eal/common/eal_internal_cfg.h   |  1 +
lib/librte_eal/common/eal_options.h        |  2 ++
lib/librte_eal/common/eal_private.h        |  9 +++++
lib/librte_eal/common/include/rte_eal.h    | 36
++++++++++++++++++++
lib/librte_eal/rte_eal_version.map         |  4 +++
10 files changed, 142 insertions(+)
doc/guides/linux_gsg/eal_args.include.rst +13 -0
index ed8b0e35b..d0717d4a0 100644
@@ -69,6 +69,19 @@ Device-related options
 
       --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
    Load external drivers. An argument can be a single shared object file,

or a

doc/guides/rel_notes/release_19_11.rst +0 -0
index e77d226b5..7251a17b2 100644
@@ -56,6 +56,15 @@ New Features
     Also, make sure to start the actual text at the margin.
 

======================================================
===

doc/guides/rel_notes/release_19_11.rst +3 -0
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared

devices.
+ This could cause issues for drivers depending on another device being present.

doc/guides/rel_notes/release_19_11.rst +5 -0
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+

* **FreeBSD now supports --base-virtaddr EAL option.**

FreeBSD version now also supports setting base virtual address for
mapping

lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +33 -0
index 9e4f09d83..368afa273 100644
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name,
 da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s
 on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug)

not supported by bus %s, "

lib/librte_eal/common/eal_common_dev.c +8 -0
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus.

Probing

lib/librte_eal/common/eal_common_dev.c +4 -0
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device

%s\n",

lib/librte_eal/common/eal_common_dev.c +9 -0
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+

/* helper function to build devargs, caller should free the memory */ static int
build_devargs(const char *busname, const char *devname,

lib/librte_eal/common/eal_common_options.c +1 -0
index a7f9c5f9b..de73970fe 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
 },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
 OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
 OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL,

OPT_MANUAL_PROBE_NUM },

	{0,                     0, NULL, 0                        }
};
lib/librte_eal/common/eal_common_options.c +3 -0
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const
 char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */

default:

lib/librte_eal/common/eal_common_options.c +0 -0
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is

<driver><id>[,key=val,...]\n"

	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
lib/librte_eal/common/eal_common_options.c +3 -0
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug

API.\n"

lib/librte_eal/common/eal_common_options.c +1 -0
+	       "                      PCI and vdev declarations will be treated in

order as hotplug commands.\n"
" --"OPT_IOVA_MODE" Set IOVA mode. 'pa' for
IOVA_PA\n"
" 'va' for IOVA_VA\n"
" -d LIB.so|DIR Add a driver or driver directory\n"

lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files
 */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual

device probing. */
volatile unsigned vmware_tsc_map; /**< true to use VMware
TSC mapping

* instead of native TSC */
volatile unsigned no_shconf; /**< true if there is no shared
config */

lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +0 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);

void
rte_option_usage(void);

lib/librte_eal/common/eal_private.h +9 -0
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+

#endif /* _EAL_PRIVATE_H_ */

lib/librte_eal/common/include/rte_eal.h +36 -0
index b7cf91214..71d9b4ca6 100644
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
 * Whether the EAL was asked to create UIO device.
 *
 * @return
lib/librte_eal/rte_eal_version.map +4 -1
index 7cbf82d37..7d7ff6531 100644
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
--

2.11.0

#27 lore raw

Re: [dpdk-dev] [PATCH v5] eal: add manual probing option

Quoted 6 lines · Pavan Nikhilesh Bhagavatula

Ping
@[email protected]

Since, you weren't reachable on regular email I rebased the patch on ToT and sent a v6
http://patches.dpdk.org/patch/64235/

Hello Pavan, indeed sorry about this.
I have not heard remarks about the patchset anyway in the meantime, not sure whether there is anything to change.

Thanks for the rebase, I will follow-up if there are remarks blocking the integration.

Quoted 396 lines
Quoted 395 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaetan Rivet
Sent: Friday, October 25, 2019 9:17 PM
To: [email protected]
Cc: Gaetan Rivet <[email protected]>; Vamsi Krishna Attunuru <[email protected]>; David Marchand <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [dpdk-dev] [PATCH v5] eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and -- vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

I haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one
interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

doc/guides/linux_gsg/eal_args.include.rst  | 13 +++++++
doc/guides/rel_notes/release_19_11.rst     |  9 +++++
lib/librte_eal/common/eal_common_bus.c     |  6 ++++
lib/librte_eal/common/eal_common_dev.c     | 54
++++++++++++++++++++++++++++++
lib/librte_eal/common/eal_common_options.c |  8 +++++
lib/librte_eal/common/eal_internal_cfg.h   |  1 +
lib/librte_eal/common/eal_options.h        |  2 ++
lib/librte_eal/common/eal_private.h        |  9 +++++
lib/librte_eal/common/include/rte_eal.h    | 36
++++++++++++++++++++
lib/librte_eal/rte_eal_version.map         |  4 +++
10 files changed, 142 insertions(+)
doc/guides/linux_gsg/eal_args.include.rst +13 -0
index ed8b0e35b..d0717d4a0 100644
@@ -69,6 +69,19 @@ Device-related options
 
        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
     Load external drivers. An argument can be a single shared object file,

or a

doc/guides/rel_notes/release_19_11.rst +0 -0
index e77d226b5..7251a17b2 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
 

======================================================
===

doc/guides/rel_notes/release_19_11.rst +3 -0
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared

devices.
+ This could cause issues for drivers depending on another device being present.

doc/guides/rel_notes/release_19_11.rst +5 -0
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+

* **FreeBSD now supports --base-virtaddr EAL option.**

FreeBSD version now also supports setting base virtual address for
mapping

lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +33 -0
index 9e4f09d83..368afa273 100644
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name,
 da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s
 on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug)

not supported by bus %s, "

lib/librte_eal/common/eal_common_dev.c +8 -0
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus.

Probing

lib/librte_eal/common/eal_common_dev.c +4 -0
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device

%s\n",

lib/librte_eal/common/eal_common_dev.c +9 -0
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+

/* helper function to build devargs, caller should free the memory */ static int
build_devargs(const char *busname, const char *devname,

lib/librte_eal/common/eal_common_options.c +1 -0
index a7f9c5f9b..de73970fe 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
 },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
 OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
 OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL,

OPT_MANUAL_PROBE_NUM },

	{0,                     0, NULL, 0                        }
};
lib/librte_eal/common/eal_common_options.c +3 -0
@@ -1446,6 +1447,9 @@ eal_parse_common_option(int opt, const
 char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */

default:

lib/librte_eal/common/eal_common_options.c +0 -0
@@ -1672,6 +1676,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is

<driver><id>[,key=val,...]\n"

	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
lib/librte_eal/common/eal_common_options.c +3 -0
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug

API.\n"

lib/librte_eal/common/eal_common_options.c +1 -0
+	       "                      PCI and vdev declarations will be treated in

order as hotplug commands.\n"
" --"OPT_IOVA_MODE" Set IOVA mode. 'pa' for
IOVA_PA\n"
" 'va' for IOVA_VA\n"
" -d LIB.so|DIR Add a driver or driver directory\n"

lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files
 */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual

device probing. */
volatile unsigned vmware_tsc_map; /**< true to use VMware
TSC mapping

* instead of native TSC */
volatile unsigned no_shconf; /**< true if there is no shared
config */

lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +0 -0
index 798ede553..fd7ac8e37 100644
@@ -381,4 +381,13 @@ rte_option_init(void);

void
rte_option_usage(void);

lib/librte_eal/common/eal_private.h +9 -0
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+

#endif /* _EAL_PRIVATE_H_ */

lib/librte_eal/common/include/rte_eal.h +36 -0
index b7cf91214..71d9b4ca6 100644
@@ -465,6 +465,42 @@ int rte_eal_has_hugepages(void);
 int rte_eal_has_pci(void);
 
 /**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
+/**
  * Whether the EAL was asked to create UIO device.
  *
  * @return
lib/librte_eal/rte_eal_version.map +4 -1
index 7cbf82d37..7d7ff6531 100644
@@ -419,4 +419,8 @@ EXPERIMENTAL {
 	rte_mcfg_timer_lock;
 	rte_mcfg_timer_unlock;
 	rte_rand_max;
+
+	# added in 19.11
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
--

2.11.0

#28 lore raw

PATCH v7 1/1 eal: add manual probing option

In reply to Gaetan Rivet

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

v6: Rebased on master

see http://mails.dpdk.org/archives/dev/2020-January/154178.html
for reference to this version, linking v7 to v5 thread.

v7: Updated author and SoB.

 doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
 doc/guides/rel_notes/release_20_02.rst     |  9 ++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 +
 lib/librte_eal/common/eal_private.h        |  9 ++++
 lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 ++
 10 files changed, 142 insertions(+)
doc/guides/linux_gsg/eal_args.include.rst +13 -0
index ed8b0e35b..d0717d4a0 100644
@@ -69,6 +69,19 @@ Device-related options
 
        --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
     Load external drivers. An argument can be a single shared object file, or a
doc/guides/rel_notes/release_20_02.rst +9 -0
index 50e2c1484..f6b3b3def 100644
@@ -56,6 +56,15 @@ New Features
      Also, make sure to start the actual text at the margin.
      =========================================================
 
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared devices.
+  This could cause issues for drivers depending on another device being present.
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+
 * **Added Wait Until Equal API.**
 
   A new API has been added to wait for a memory location to be updated with a
lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +54 -0

66 lines not on this page · raw message

lib/librte_eal/common/eal_common_options.c +8 -0
index 5920233bc..f899eea4d 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM       },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL, OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL, OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL, OPT_MANUAL_PROBE_NUM     },
 	{0,                     0, NULL, 0                        }
 };
 
@@ -1443,6 +1444,9 @@ eal_parse_common_option(int opt, const char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */
 	default:
@@ -1669,6 +1673,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is <driver><id>[,key=val,...]\n"
 	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug API.\n"
+	       "                      PCI and vdev declarations will be treated in order as hotplug commands.\n"
 	       "  --"OPT_IOVA_MODE"   Set IOVA mode. 'pa' for IOVA_PA\n"
 	       "                      'va' for IOVA_VA\n"
 	       "  -d LIB.so|DIR       Add a driver or driver directory\n"
lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual device probing. */
 	volatile unsigned vmware_tsc_map; /**< true to use VMware TSC mapping
 										* instead of native TSC */
 	volatile unsigned no_shconf;      /**< true if there is no shared config */
lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +9 -0
index ddcfbe2e4..680c7db88 100644
@@ -443,4 +443,13 @@ rte_option_usage(void);
 uint64_t
 eal_get_baseaddr(void);
 
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+
 #endif /* _EAL_PRIVATE_H_ */
lib/librte_eal/common/include/rte_eal.h +36 -0
index 2f9ed298d..7195f6859 100644
@@ -421,6 +421,42 @@ int rte_eal_has_hugepages(void);
  */
 int rte_eal_has_pci(void);
 
+/**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
 /**
  * Whether the EAL was asked to create UIO device.
  *
lib/librte_eal/rte_eal_version.map +4 -0
index e38d02530..13d04a8bc 100644
@@ -332,4 +332,8 @@ EXPERIMENTAL {
 	# added in 19.11
 	rte_log_get_stream;
 	rte_mcfg_get_single_file_segments;
+
+	# added in 20.02
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
Signature · 1 line
2.25.0
#29 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Quoted 407 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Gaetan Rivet
Sent: Thursday, January 23, 2020 3:28 PM
To: [email protected]
Cc: Vamsi Krishna Attunuru <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>
Subject: [dpdk-dev] [PATCH v7] eal: add manual probing option

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and -- vdev),
will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual
probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one
interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

v6: Rebased on master

see https://urldefense.proofpoint.com/v2/url?u=http-
3A__mails.dpdk.org_archives_dev_2020- 2DJanuary_154178.html&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r= 1cjuAHrGh745jHNmj2fD85sUMIJ2IPIDsIJzo6FN6Z0&m=BsQe7kO_de- Kb6YFvVZPpZgIhGnWWpW8Pou2KkH_Cjk&s=oFGrj6beplYlYJWodLDGY GCUExSbvb1iJEERsA18QnA&e=
for reference to this version, linking v7 to v5 thread.

v7: Updated author and SoB.

doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
doc/guides/rel_notes/release_20_02.rst     |  9 ++++
lib/librte_eal/common/eal_common_bus.c     |  6 +++
lib/librte_eal/common/eal_common_dev.c     | 54
++++++++++++++++++++++
lib/librte_eal/common/eal_common_options.c |  8 ++++
lib/librte_eal/common/eal_internal_cfg.h   |  1 +
lib/librte_eal/common/eal_options.h        |  2 +
lib/librte_eal/common/eal_private.h        |  9 ++++
lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
lib/librte_eal/rte_eal_version.map         |  4 ++
10 files changed, 142 insertions(+)
doc/guides/linux_gsg/eal_args.include.rst +13 -0
index ed8b0e35b..d0717d4a0 100644
@@ -69,6 +69,19 @@ Device-related options
 
       --vdev 'net_pcap0,rx_pcap=input.pcap,tx_pcap=output.pcap'
 
+*   ``--manual-probe``
+
+    Switch the ``EAL`` probe mode to manual. The main bus probe step
+    is disabled and applications are expected to manually probe
+    devices using ``rte_dev_probe()``.
+
+    Devices declared on the command-line using ``-w`` and ``-vdev``
+    are interpreted as hotplug commands. They are thus probed in the
+    order they are declared.
+
+    This makes this option useful to enforce a specific device probe
+    order, instead of relying on each bus scan implementation details.
+
 *   ``-d <path to shared object or directory>``
 
    Load external drivers. An argument can be a single shared object file,

or a

doc/guides/rel_notes/release_20_02.rst +0 -0
index 50e2c1484..f6b3b3def 100644
@@ -56,6 +56,15 @@ New Features
     Also, make sure to start the actual text at the margin.
 

======================================================
===

doc/guides/rel_notes/release_20_02.rst +3 -0
+* **EAL will now allow probing devices manually.**
+
+  Previously, a user could not force an order when probing declared

devices.
+ This could cause issues for drivers depending on another device being present.

doc/guides/rel_notes/release_20_02.rst +5 -0
+  A new option ``--manual-probe`` is now available to do just that.
+  This new option relies on the device bus supporting hotplug. It can
+  also be used to disable automatic probing from the ``PCI`` bus without
+  having to disable the whole bus.
+

* **Added Wait Until Equal API.**

A new API has been added to wait for a memory location to be
updated with a

lib/librte_eal/common/eal_common_bus.c +6 -0
index baa5b532a..145a96812 100644
@@ -6,6 +6,7 @@
 #include <string.h>
 #include <sys/queue.h>
 
+#include <rte_eal.h>
 #include <rte_bus.h>
 #include <rte_debug.h>
 #include <rte_string_fns.h>
@@ -63,6 +64,11 @@ rte_bus_probe(void)
 	int ret;
 	struct rte_bus *bus, *vbus = NULL;
 
+	if (rte_eal_manual_probe()) {
+		RTE_LOG(DEBUG, EAL, "Manual probing enabled.\n");
+		return rte_dev_probe_devargs_list();
+	}
+
 	TAILQ_FOREACH(bus, &rte_bus_list, next) {
 		if (!strcmp(bus->name, "vdev")) {
 			vbus = bus;
lib/librte_eal/common/eal_common_dev.c +33 -0
index 9e4f09d83..368afa273 100644
@@ -21,6 +21,7 @@
 #include <rte_malloc.h>
 #include <rte_string_fns.h>
 
+#include "eal_internal_cfg.h"
 #include "eal_private.h"
 #include "hotplug_mp.h"
 
@@ -83,6 +84,59 @@ rte_dev_is_probed(const struct rte_device *dev)
 	return dev->driver != NULL;
 }
 
+int
+rte_eal_manual_probe(void)
+{
+	return internal_config.manual_probe;
+}
+
+void
+rte_eal_manual_probe_set(int enabled)
+{
+	internal_config.manual_probe = !!enabled;
+}
+
+int
+rte_dev_probe_devargs_list(void)
+{
+	struct rte_device *dev;
+	struct rte_devargs *da;
+	int ret;
+
+	RTE_EAL_DEVARGS_FOREACH(NULL, da) {
+		dev = da->bus->find_device(NULL, cmp_dev_name,
 da->name);
+		if (dev == NULL) {
+			RTE_LOG(ERR, EAL, "Unable to find device %s
 on bus %s\n",
+				da->name, da->bus->name);
+			continue;
+		}
+
+		if (rte_dev_is_probed(dev))
+			continue;
+
+		if (dev->bus->plug == NULL) {
+			RTE_LOG(ERR, EAL, "Manual probing (hotplug)

not supported by bus %s, "

lib/librte_eal/common/eal_common_dev.c +8 -0
+					  "required by device %s\n",
+				dev->bus->name, dev->name);
+			continue;
+		}
+
+		ret = dev->bus->plug(dev);
+		/* Ignore positive return values, they are possibly
+		 * triggered by blacklisted devices on the PCI bus.

Probing

lib/librte_eal/common/eal_common_dev.c +4 -0
+		 * should then continue.
+		 */
+		if (ret < 0) {
+			RTE_LOG(ERR, EAL, "Driver cannot attach device

%s\n",

lib/librte_eal/common/eal_common_dev.c +9 -0
+				dev->name);
+			/* Fail on first real probe error. */
+			return ret;
+		}
+	}
+
+	return 0;
+}
+

/* helper function to build devargs, caller should free the memory */ static int
build_devargs(const char *busname, const char *devname,

lib/librte_eal/common/eal_common_options.c +1 -0
index 5920233bc..f899eea4d 100644
@@ -82,6 +82,7 @@ eal_long_options[] = {
 	{OPT_LEGACY_MEM,        0, NULL, OPT_LEGACY_MEM_NUM
 },
 	{OPT_SINGLE_FILE_SEGMENTS, 0, NULL,
 OPT_SINGLE_FILE_SEGMENTS_NUM},
 	{OPT_MATCH_ALLOCATIONS, 0, NULL,
 OPT_MATCH_ALLOCATIONS_NUM},
+	{OPT_MANUAL_PROBE,      0, NULL,

OPT_MANUAL_PROBE_NUM },

	{0,                     0, NULL, 0                        }
};
lib/librte_eal/common/eal_common_options.c +3 -0
@@ -1443,6 +1444,9 @@ eal_parse_common_option(int opt, const
 char *optarg,
 			return -1;
 		}
 		break;
+	case OPT_MANUAL_PROBE_NUM:
+		rte_eal_manual_probe_set(1);
+		break;
 
 	/* don't know what to do, leave this to caller */

default:

lib/librte_eal/common/eal_common_options.c +0 -0
@@ -1669,6 +1673,10 @@ eal_common_usage(void)
 	       "  --"OPT_VDEV"              Add a virtual device.\n"
 	       "                      The argument format is

<driver><id>[,key=val,...]\n"

	       "                      (ex: --vdev=net_pcap0,iface=eth2).\n"
lib/librte_eal/common/eal_common_options.c +3 -0
+	       "  --"OPT_MANUAL_PROBE"      Enable manual probing.\n"
+	       "                      Disable probe step for all buses.\n"
+	       "                      Devices will need to be probed using the hotplug

API.\n"

lib/librte_eal/common/eal_common_options.c +1 -0
+	       "                      PCI and vdev declarations will be treated in

order as hotplug commands.\n"
" --"OPT_IOVA_MODE" Set IOVA mode. 'pa' for
IOVA_PA\n"
" 'va' for IOVA_VA\n"
" -d LIB.so|DIR Add a driver or driver directory\n"

lib/librte_eal/common/eal_internal_cfg.h +1 -0
index a42f34923..0006f903f 100644
@@ -44,6 +44,7 @@ struct internal_config {
 	unsigned hugepage_unlink;         /**< true to unlink backing files
 */
 	volatile unsigned no_pci;         /**< true to disable PCI */
 	volatile unsigned no_hpet;        /**< true to disable HPET */
+	volatile unsigned manual_probe;   /**< true to enable manual

device probing. */
volatile unsigned vmware_tsc_map; /**< true to use VMware
TSC mapping

* instead of native TSC */
volatile unsigned no_shconf; /**< true if there is no shared
config */

lib/librte_eal/common/eal_options.h +2 -0
index 9855429e5..588fa32a6 100644
@@ -69,6 +69,8 @@ enum {
 	OPT_IOVA_MODE_NUM,
 #define OPT_MATCH_ALLOCATIONS  "match-allocations"
 	OPT_MATCH_ALLOCATIONS_NUM,
+#define OPT_MANUAL_PROBE "manual-probe"
+	OPT_MANUAL_PROBE_NUM,
 	OPT_LONG_MAX_NUM
 };
 
lib/librte_eal/common/eal_private.h +0 -0
index ddcfbe2e4..680c7db88 100644
@@ -443,4 +443,13 @@ rte_option_usage(void);

uint64_t
eal_get_baseaddr(void);

lib/librte_eal/common/eal_private.h +9 -0
+/**
+ * Go through the devargs list and probe everything in order.
+ *
+ * @return
+ *   0 on success, negative on error.
+ */
+int
+rte_dev_probe_devargs_list(void);
+

#endif /* _EAL_PRIVATE_H_ */

lib/librte_eal/common/include/rte_eal.h +36 -0
index 2f9ed298d..7195f6859 100644
@@ -421,6 +421,42 @@ int rte_eal_has_hugepages(void);
 */
 int rte_eal_has_pci(void);
 
+/**
+ * Whether EAL probe is manual.
+ * Enabled by the --manual-probe option or by
+ * using rte_eal_manual_probe_set().
+ *
+ * When manual probing is enabled, batched bus probe of
+ * their devices is disabled. All devices need to be probed
+ * using the proper rte_dev API.
+ *
+ * In this mode, devices declared on the command line will
+ * be probed using the bus hotplug API. It is used to enforce
+ * a specific probe order.
+ *
+ * @return
+ *   Nonzero if manual device probing is enabled.
+ *
+ * @see rte_eal_manual_probe_set
+ */
+__rte_experimental
+int rte_eal_manual_probe(void);
+
+/**
+ * Configure EAL probe mode -- manual or automatic.
+ *
+ * Enable or disable manual probe mode in EAL.
+ * This function can be called at any time, but must be used
+ * before calling rte_eal_init() to have any effect.
+ *
+ * @param enabled
+ *   zero to disable manual probe, non-zero to enable it.
+ *
+ * @see rte_eal_manual_probe
+ */
+__rte_experimental
+void rte_eal_manual_probe_set(int enabled);
+
 /**
 * Whether the EAL was asked to create UIO device.
 *
lib/librte_eal/rte_eal_version.map +4 -1
index e38d02530..13d04a8bc 100644
@@ -332,4 +332,8 @@ EXPERIMENTAL {
 	# added in 19.11
 	rte_log_get_stream;
 	rte_mcfg_get_single_file_segments;
+
+	# added in 20.02
+	rte_eal_manual_probe;
+	rte_eal_manual_probe_set;
 };
--

2.25.0

#30 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

#31 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 9 lines · Thomas Monjalon

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

Quoted 11 lines

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

#32 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 9 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

Quoted 9 lines

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

#33 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 21 lines · Thomas Monjalon

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 9 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

       http://mails.dpdk.org/archives/dev/2019-September/144166.html
     My arguments:
       http://mails.dpdk.org/archives/dev/2019-September/144564.html
#34 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

04/02/2020 13:43, Gaetan Rivet:

Quoted 44 lines · Thomas Monjalon
Quoted 21 lines

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 7 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

Quoted 8 lines

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

        http://mails.dpdk.org/archives/dev/2019-September/144166.html
      My arguments:
        http://mails.dpdk.org/archives/dev/2019-September/144564.html

OK so there are two needs:

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here. Unfortunately neither you or me had time to work on it, and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications. This must be done in applications.
Tweaking the probe order is a hack.

#35 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 57 lines · Thomas Monjalon

04/02/2020 13:43, Gaetan Rivet:

Quoted 44 lines · Thomas Monjalon
Quoted 21 lines

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 7 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

Quoted 8 lines

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

         http://mails.dpdk.org/archives/dev/2019-September/144166.html
       My arguments:
         http://mails.dpdk.org/archives/dev/2019-September/144564.html

OK so there are two needs:

Quoted 10 lines

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here. Unfortunately neither you or me had time to work on it, and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications. This must be done in applications.
Tweaking the probe order is a hack.

An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.

Instead, with this patch this app can do

  rte_eal_manual_probe_set(1);
  rte_eal_init();

to have the same behavior and be able to hotplug ports as it sees fit.

You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.

This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

#36 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 74 lines · Gaetan Rivet
Quoted 57 lines · Thomas Monjalon

04/02/2020 13:43, Gaetan Rivet:

Quoted 44 lines · Thomas Monjalon
Quoted 21 lines

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 7 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

Quoted 8 lines

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

         http://mails.dpdk.org/archives/dev/2019-September/144166.html
       My arguments:
         http://mails.dpdk.org/archives/dev/2019-September/144564.html

OK so there are two needs:

Quoted 10 lines

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here. Unfortunately neither you or me had time to work on it, and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications. This must be done in applications.
Tweaking the probe order is a hack.

An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.

Instead, with this patch this app can do

Quoted 8 lines
   rte_eal_manual_probe_set(1);
   rte_eal_init();

to have the same behavior and be able to hotplug ports as it sees fit.

You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.

This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

Thomas,

What would be the resolution of this? What is your recommendation to fix the issue as you have the concern of this patch?

Issue:
1) When l2fwd does the forwarding for simplicity and performance reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and other 10G) then forwarding will drop the packets.

So, either
a) We need to control the probing order

b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table OR
2) Have the command-line option to specify destination port like l3fwd.

We can fix it in the application, but do we need to complicate l2fwd? I am fine with that, if that is consensus.

Thoughts? If you think, there is a rework needed in eal then could you enumerate the items for the rework.

#37 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

10/02/2020 15:51, Jerin Jacob:

Quoted 98 lines · Gaetan Rivet
Quoted 74 lines · Thomas Monjalon
Quoted 57 lines

04/02/2020 13:43, Gaetan Rivet:

Quoted 44 lines · Thomas Monjalon
Quoted 21 lines

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 7 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

Quoted 8 lines

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

         http://mails.dpdk.org/archives/dev/2019-September/144166.html
       My arguments:
         http://mails.dpdk.org/archives/dev/2019-September/144564.html

OK so there are two needs:

Quoted 10 lines

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here. Unfortunately neither you or me had time to work on it, and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications. This must be done in applications.
Tweaking the probe order is a hack.

An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.

Instead, with this patch this app can do

Quoted 8 lines
   rte_eal_manual_probe_set(1);
   rte_eal_init();

to have the same behavior and be able to hotplug ports as it sees fit.

You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.

This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

Thomas,

What would be the resolution of this? What is your recommendation to fix the issue as you have the concern of this patch?

Issue:
1) When l2fwd does the forwarding for simplicity and performance reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and other 10G) then forwarding will drop the packets.

So, either
a) We need to control the probing order

Quoted 6 lines

b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table OR
2) Have the command-line option to specify destination port like l3fwd.

We can fix it in the application, but do we need to complicate l2fwd? I am fine with that, if that is consensus.

You are describing an application issue,
that's why I believe it should be fixed in the application.

Should we have a command line option to configure the forwarding rules in the application (2)? I think yes.
Should we implement an application logic to automatically create the best forwarding rules (1)? It would be nice, but anyway, I think we need manual config (2) as a fallback.

Quoted 2 lines

Thoughts? If you think, there is a rework needed in eal then could you enumerate the items for the rework.

Sorry I don't have time to describe dive into EAL probing and
enumerate the items to rework.
The most important issues I remind are:
- white/blacklist policy is a mess and should be done in a higher layer
- devargs syntax should allow generic matching (thanks to class awareness and generic syntax)

Starting from these 2 items, we could imagine a generic path to disable automatic probing, but I think the l2fwd logic should not rely on probing order anyway.

#38 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 104 lines · Thomas Monjalon

10/02/2020 15:51, Jerin Jacob:

Quoted 102 lines · Gaetan Rivet
Quoted 74 lines · Thomas Monjalon
Quoted 57 lines

04/02/2020 13:43, Gaetan Rivet:

Quoted 44 lines · Thomas Monjalon
Quoted 21 lines

04/02/2020 11:03, Gaetan Rivet:

Quoted 20 lines · Thomas Monjalon
Quoted 18 lines

03/02/2020 06:16, Pavan Nikhilesh Bhagavatula:

Quoted 5 lines

@David Marchand @[email protected]

Ping?

Are there any more changes required for this patch? It's been in queue since last October.

Sorry we have not decided whether it is a good idea or not.

All changes related to probing are very sensitive, and we know a big refactoring would be better than stacking more and more options and corner cases.

As we are busy with ABI stability stuff, we did not allocate enough time to properly think about this feature. Please accept our apologies, and let's consider it as a high priority for 20.05 cycle.

Quoted 8 lines
Quoted 5 lines

Hello Thomas,

This is unfortunate. I pushed Pavan to accept an alternative implementation of this functionality that was less obtrusive, to make the integration smoother. I took care to alleviate those risks from the common path.

The big refactoring is needed yes, but considering the current path I'm not seeing it happen in 20.05. If that means taking this patch as-is in 20.05 for Marvell users, I'm not sure much is gained from waiting 3 months, except minimal risk avoidance.

Yes, life is full of bad decisions and consequences.

Ah, yes, but I stand by my initial opinion, the first implementation [1] was riskier and less useful.

Quoted 7 lines

I still think there is a risk in adding new user expectations, and maintaining some code to workaround unknown issues.

The real question here is to know why this patch? Is it to workaround a broken driver?
Or to workaround a broken design in EAL and bus drivers?

Two birds - one stone here: OVS needed a way to disable automatic probing cleanly (current workaround seen in multiple deployment is to add a dummy whitelisted device, which will be ignored by the PCI bus --> it sets the bus in whitelist mode but avoid probing anything), and as a bonus this option allows using devices that depends on other devices being probed already (LAG, representors, failsafe, etc).

Quoted 8 lines

I'm not sure having a dependent-probe by default is good, and that would be a big change.

If we are doing the genesis of this patch, the initial motivation should be asked for more details from Marvell people and David for the OVS side.

[1]: First proposal:

         http://mails.dpdk.org/archives/dev/2019-September/144166.html
       My arguments:
         http://mails.dpdk.org/archives/dev/2019-September/144564.html

OK so there are two needs:

Quoted 10 lines

1/ Better control whitelist/blacklist mode.
We already know that a rework is needed here. Unfortunately neither you or me had time to work on it, and others who were interested disappeared.

2/ Associate ports with equivalent properties in applications. This must be done in applications.
Tweaking the probe order is a hack.

An application that want to tightly control the port init order, currently (by doing exactly like me here, hotpluging one by one the ports), would still need the bigger hack that consist in inserting a whitelist PCI devargs with a dummy address, depending on a undocumented PCI bus feature consisting in ignoring matching errors but keeping probing policy from failed devargs processing.

Instead, with this patch this app can do

Quoted 8 lines
   rte_eal_manual_probe_set(1);
   rte_eal_init();

to have the same behavior and be able to hotplug ports as it sees fit.

You are worried about creating user expectations about this behavior (being forced to replicate in some way the functionality during the rewrite, as I understand it?), but then you are currently forcing users to expect this workaround to exist in the PCI bus, blocking devs from touching it as it will thus break current app configurations. I've seen systemd unit file using this -w dummy flag, as well as the programmatic equivalent. Which is better, to have to rework it cutting short these configs, or to propose beforehand a deprecation path?.

This rework won't happen in 20.05, nor in the medium term unless you decide to drive this change. This workaround serves three needs (PCI normalization, port congruence and port dependency) in a low-risk implementation.

Thomas,

What would be the resolution of this? What is your recommendation to fix the issue as you have the concern of this patch?

Issue:
1) When l2fwd does the forwarding for simplicity and performance reason it just xor the port to find the destination port to forward.
2) If the adjacent ports are not symmetrical(example: one is 40G and other 10G) then forwarding will drop the packets.

So, either
a) We need to control the probing order

b) Or Application need
1) To track the symmetrical ports and maintain the forwarding table OR
2) Have the command-line option to specify destination port like l3fwd.

Quoted 5 lines
Quoted 2 lines

We can fix it in the application, but do we need to complicate l2fwd? I am fine with that, if that is consensus.

You are describing an application issue,
that's why I believe it should be fixed in the application.

Thanks for the quick reply and I agree.

Quoted 17 lines

Should we have a command line option to configure the forwarding rules in the application (2)? I think yes.
Should we implement an application logic to automatically create the best forwarding rules (1)? It would be nice, but anyway, I think we need manual config (2) as a fallback.

Quoted 2 lines

Thoughts? If you think, there is a rework needed in eal then could you enumerate the items for the rework.

Sorry I don't have time to describe dive into EAL probing and
enumerate the items to rework.
The most important issues I remind are:
- white/blacklist policy is a mess and should be done in a higher layer
- devargs syntax should allow generic matching (thanks to class awareness and generic syntax)

Quoted 3 lines

Starting from these 2 items, we could imagine a generic path to disable automatic probing, but I think the l2fwd logic should not rely on probing order anyway.

+ Bruce as l2fwd maintainer.

Since in any case, l2fwd needs to be updated, We will focus on l2fwd change for v20.05 and leaving the fate of this patch to EAL maintainers.
Let us know, Are we are OK with below change in l2fwd as
- Introduce an array-based port lookup table instead of hardcoding to xor based lookup.
- if no argument specified fill dest port as xor of source
- If argument is specified override the lookup table with a user-specified destination port.

Quoted 2 lines
#39 lore raw

PATCH 1/1 examples/l2fwd: add cmdline option for forwarding port info

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5).

---
 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
 3 files changed, 177 insertions(+), 21 deletions(-)
doc/guides/rel_notes/release_20_05.rst +6 -0
index 000bbf5..645400b 100644
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config(port, port)[,(port, port)]`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +16 -2
index 39d6b00..e708f88 100644
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config(port, port)[,(port, port)]
 
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --config(port,port)[,(port,port)]: Determines which ports are mapped to
+    which ports for packet forwarding.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
examples/l2fwd/main.c +155 -19

248 lines not on this page · raw message

Signature · 1 line
2.8.4
Reviewed-by on v1 1/1
#40 lore raw

Re: [dpdk-dev] [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

Quoted 34 lines · On 4/3/20 5:30 AM, [email protected] wrote:

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5).

---
 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
 3 files changed, 177 insertions(+), 21 deletions(-)
doc/guides/rel_notes/release_20_05.rst +0 -0
index 000bbf5..645400b 100644

doc/guides/rel_notes/release_20_05.rst New Features · line 67

Quoted 6 lines
doc/guides/rel_notes/release_20_05.rst +3 -0
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config(port, port)[,(port, port)]`` to

Maybe use the actual syntax with '=' or space after 'config'?

Quoted 11 lines
doc/guides/rel_notes/release_20_05.rst +3 -0
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b00..e708f88 100644

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line options: · line 97

Quoted 8 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -1
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config(port, port)[,(port, port)]

Ditto

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 108

Quoted 12 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -1
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --config(port,port)[,(port,port)]: Determines which ports are mapped to
+    which ports for packet forwarding.

Ditto

Quoted 32 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +8 -0
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
examples/l2fwd/main.c +1 -0
index 09257aa..a8b98b9 100644
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 78

Quoted 10 lines
examples/l2fwd/main.c +8 -0
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params port_pair_params_array[RTE_MAX_ETHPORTS];
+static struct port_pair_params *port_pair_params;
+static uint16_t nb_port_pair_params = 1;

Why this initialization to 1? Below (during parsing) it is initialized to 0, so I would leave this default initialized.

Quoted 27 lines
examples/l2fwd/main.c +20 -0
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 
 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg)
+{
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask) · line 360

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		snprintf(s, sizeof(s), "%.*s", size, p);

Same comment as for recent l2fwd-event - this can be simple memcpy.

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask) · line 368

Quoted 8 lines
examples/l2fwd/main.c +8 -0
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] > RTE_MAX_ETHPORTS)

I think this check should be >=, RTE_MAX_ETHPORTS is an invalid port id.

Quoted 16 lines
examples/l2fwd/main.c +16 -0
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params: %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}

[...]

Quoted 15 lines
examples/l2fwd/main.c +4 -7
@@ -565,26 +686,40 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx;
+

examples/l2fwd/main.c main(int argc, char **argv) · line 702

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			if (idx % 2 == 0) {
+				portid = port_pair_params[idx >> 1].port[0];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[1];
+			} else {
+				portid = port_pair_params[idx >> 1].port[1];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[0];
+			}

I think this conditional can be rephrased like:

	uint16_t idx, p;
...
	p = idx & 1;
	port_id = port_pair_params[idx >> 1].port[p];
	l2fwd_dst_ports[port_id] =
		port_pair_params[idx >> 1].port[p ^ 1];

but I'm not sure which one is more readable - I leave you the decision,
feel free to skip this comment altogether :).

Quoted 35 lines
examples/l2fwd/main.c +17 -10
+		}
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
+
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
 
+			nb_ports_in_mask++;
+		}
 		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
 		}
-		else
-			last_port = portid;
-
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
 	}
 
 	rx_lcore_id = 0;

examples/l2fwd/main.c main(int argc, char **argv) · line 755

Quoted 9 lines
examples/l2fwd/main.c +2 -1
@@ -613,7 +748,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd + MAX_PKT_BURST +

Apart from these comments:

With regards
Andrzej Ostruszka

#41 lore raw

Re: [dpdk-dev] [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

Quoted 306 lines

-----Original Message-----
From: Andrzej Ostruszka [C] <[email protected]>
Sent: Friday, April 3, 2020 6:22 PM
To: Vamsi Krishna Attunuru <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Jerin Jacob Kollanukkaran <[email protected]>; Pavan Nikhilesh Bhagavatula <[email protected]>; [email protected]
Subject: Re: [PATCH] [v1 1/1] examples/l2fwd: add cmdline option for forwarding port info

Quoted 35 lines · On 4/3/20 5:30 AM, [email protected] wrote:

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5).

---
 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 174 ++++++++++++++++++---
 3 files changed, 177 insertions(+), 21 deletions(-)
doc/guides/rel_notes/release_20_05.rst +0 -0
index 000bbf5..645400b 100644

doc/guides/rel_notes/release_20_05.rst New Features · line 68

Quoted 7 lines
doc/guides/rel_notes/release_20_05.rst +4 -0
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config(port, port)[,(port,
+ port)]`` to

Maybe use the actual syntax with '=' or space after 'config'?

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line

Quoted 12 lines
doc/guides/rel_notes/release_20_05.rst +3 -0
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
+ more  details of this parameter usage.
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b00..e708f88 100644
@@ -91,7 +91,10 @@ The application requires a number of command line

options:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line options:

Quoted 9 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +5 -1
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config(port, port)[,(port,
+ port)]

Ditto

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 102

Quoted 8 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -1
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled by default).

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 105

Quoted 1 line
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +1 -0
+*   --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled by default)

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 107

Quoted 2 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
+
+*   --config(port,port)[,(port,port)]: Determines which ports are mapped

to

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 108

Quoted 1 line
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +1 -0
+    which ports for packet forwarding.

Ditto

Quoted 3 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
 To run the application in linux environment with 4 lcores, 16 ports

and 8 RX queues per lcore and MAC address updating enabled, issue the

command:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 119

Quoted 7 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +3 -0
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8
+RX queues per lcore and MAC address updating enabled, to forward RX
+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue the

command:

Quoted 23 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +5 -0
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on

running applications and the Environment Abstraction Layer (EAL) options.

examples/l2fwd/main.c index renamed from examples/l2fwd/main.c +0 -0

09257aa..a8b98b9 100644

examples/l2fwd/main.c +1 -0
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 79

Quoted 10 lines
examples/l2fwd/main.c +9 -0
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params
+port_pair_params_array[RTE_MAX_ETHPORTS];
+static struct port_pair_params *port_pair_params; static uint16_t
+nb_port_pair_params = 1;

Why this initialization to 1? Below (during parsing) it is initialized to 0, so I would leave this default initialized.

Quoted 26 lines
examples/l2fwd/main.c +19 -0
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 
 #define MAX_RX_QUEUE_PER_LCORE 16
@@ -319,6 +329,60 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg) {
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask) · line 359

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		snprintf(s, sizeof(s), "%.*s", size, p);

Same comment as for recent l2fwd-event - this can be simple memcpy.

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask) · line 368

Quoted 8 lines
examples/l2fwd/main.c +8 -0
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] > RTE_MAX_ETHPORTS)

I think this check should be >=, RTE_MAX_ETHPORTS is an invalid port id.

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask)

Quoted 4 lines
examples/l2fwd/main.c +4 -0
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params:

%hu\n",

Quoted 12 lines
examples/l2fwd/main.c +12 -0
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}

[...]

Quoted 15 lines
examples/l2fwd/main.c +4 -7
@@ -565,26 +686,40 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx;
+

examples/l2fwd/main.c main(int argc, char **argv) · line 702

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			if (idx % 2 == 0) {
+				portid = port_pair_params[idx >> 1].port[0];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[1];
+			} else {
+				portid = port_pair_params[idx >> 1].port[1];
+				l2fwd_dst_ports[portid] =
+					port_pair_params[idx >> 1].port[0];
+			}

I think this conditional can be rephrased like:

	uint16_t idx, p;
...
	p = idx & 1;
	port_id = port_pair_params[idx >> 1].port[p];
	l2fwd_dst_ports[port_id] =
		port_pair_params[idx >> 1].port[p ^ 1];

but I'm not sure which one is more readable - I leave you the decision, feel free to skip this comment altogether :).

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+		}
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
+
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;

examples/l2fwd/main.c main(int argc, char **argv)

Quoted 10 lines
examples/l2fwd/main.c +6 -2
+			} else {
+				last_port = portid;
+			}
 
+			nb_ports_in_mask++;
+		}
 		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+			printf("Notice: odd number of ports in

portmask.\n");

Quoted 14 lines
examples/l2fwd/main.c +1 -8
+			l2fwd_dst_ports[last_port] = last_port;
 		}
-		else
-			last_port = portid;
-
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
 	}
 
 	rx_lcore_id = 0;

examples/l2fwd/main.c main(int argc, char **argv) · line 755

Quoted 9 lines
examples/l2fwd/main.c +2 -1
@@ -613,7 +748,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +

MAX_PKT_BURST +

Quoted 1 line
Quoted 6 lines

Apart from these comments:

With regards
Andrzej Ostruszka

Thanks Andrzej,
Acked all the comments in V2.

#42 lore raw

PATCH v2 1/1 examples/l2fwd: add cmdline option for forwarding port info

In reply to [email protected]

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5).

---
V2 Changes:

* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 169 ++++++++++++++++++---
 3 files changed, 172 insertions(+), 21 deletions(-)
doc/guides/rel_notes/release_20_05.rst +6 -0
index 000bbf5..69e1798 100644
@@ -62,6 +62,12 @@ New Features
 
   * Added support for matching on IPv4 Time To Live and IPv6 Hop Limit.
 
+* **Added --config command line parameter to l2fwd example.**
+
+  Added new command line option ``--config="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +16 -2
index 39d6b00..5c0ee05 100644
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --config="(port, port)[,(port, port)]"
 
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --config="(port,port)[,(port,port)]": Determines which ports are mapped to
+    which ports for packet forwarding.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --config="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
examples/l2fwd/main.c +150 -19

243 lines not on this page · raw message

Signature · 1 line
2.8.4
Acked-by on v2 1/1
#43 lore raw

Re: [dpdk-dev] [PATCH] [v2 1/1] examples/l2fwd: add cmdline option for forwarding port info

Quoted 26 lines · On 4/5/20 5:52 AM, [email protected] wrote:

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

With above config option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5).

---
V2 Changes:

Quoted 8 lines

* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst             |   6 +
 .../sample_app_ug/l2_forward_real_virtual.rst      |  18 ++-
 examples/l2fwd/main.c                              | 169 ++++++++++++++++++---
 3 files changed, 172 insertions(+), 21 deletions(-)
#44 lore raw

Re: [dpdk-dev] [PATCH] [v2 1/1] examples/l2fwd: add cmdline option for forwarding port info

In reply to [email protected]

05/04/2020 05:52, [email protected]:

Quoted 10 lines

From: Vamsi Attunuru <[email protected]>

Current l2fwd application configures adjacent ports as destination ports for forwarding the traffic which is a kind of static mapping that can not be altered by the command line options.

Patch adds a config option to pass the forwarding port pair mapping as a command line parameter which allows the user to pass required forwarding port mapping.

Quoted 7 lines

If no config argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

When port pair mapping is passed in config option, destination port map is configured and traffic gets forwarded accordingly.

Ex: ./l2fwd -c 0xff -- -p 0x3f --config="(0,3)(1,4)(2,5)"

The option name "--config" is vague.
Could we call it "--portmap", or something similar?

#45 lore raw

PATCH v3 1/1 examples/l2fwd: add cmdline option for forwarding port info

In reply to [email protected]

From: Vamsi Attunuru <[email protected]>

Current l2fwd application statically configures adjacent ports as destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port map, port/queue configuration of each lcore gets modified when portmap option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

With above portmap option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5)

---
v3 Changes:
* s/config/portmap/
* rebase on master

v2 Changes:
* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst        |   7 +
 .../sample_app_ug/l2_forward_real_virtual.rst |  18 +-
 examples/l2fwd/main.c                         | 182 +++++++++++++++---
 3 files changed, 181 insertions(+), 26 deletions(-)
doc/guides/rel_notes/release_20_05.rst +7 -0
index b124c3f28..019bec5d7 100644
@@ -212,6 +212,13 @@ New Features
   * Added IPsec inbound load-distribution support for ipsec-secgw application
     using NIC load distribution feature(Flow Director).
 
+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
+
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +16 -2
index 39d6b0067..b54321b5b 100644
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --portmap="(port, port)[,(port, port)]"
 
 where,
 
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --portmap="(port,port)[,(port,port)]": Determines which ports are mapped to
+    which ports for packet forwarding.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore and MAC address updating enabled, to forward RX traffic of ports 0 & 1
+on ports 2 & 3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
examples/l2fwd/main.c +158 -24

262 lines not on this page · raw message

--
2.17.1

#46 lore raw

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

Quoted 84 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Monday, April 27, 2020 1:30 PM
To: Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; John McNamara <[email protected]>; Marko Kovacevic <[email protected]>; Ori Kam <[email protected]>; Bruce Richardson <[email protected]>; Radu Nicolau <[email protected]>; Akhil Goyal <[email protected]>; Tomasz Kantecki <[email protected]>; Sunil Kumar Kori <[email protected]>; Pavan Nikhilesh Bhagavatula <[email protected]>
Cc: Andrzej Ostruszka [C] <[email protected]>; [email protected]; Vamsi Krishna Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

From: Vamsi Attunuru <[email protected]>

Current l2fwd application statically configures adjacent ports as destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port map, port/queue configuration of each lcore gets modified when portmap option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

With above portmap option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5)

---
v3 Changes:
* s/config/portmap/
* rebase on master

v2 Changes:
* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

doc/guides/rel_notes/release_20_05.rst        |   7 +
.../sample_app_ug/l2_forward_real_virtual.rst |  18 +-
examples/l2fwd/main.c                         | 182 +++++++++++++++---
3 files changed, 181 insertions(+), 26 deletions(-)
doc/guides/rel_notes/release_20_05.rst +7 -0
index b124c3f28..019bec5d7 100644
@@ -212,6 +212,13 @@ New Features
  * Added IPsec inbound load-distribution support for ipsec-secgw application
    using NIC load distribution feature(Flow Director).
 
+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port,
+ port)]"`` to  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
+ more  details of this parameter usage.
+
 

Removed Items
-------------

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b0067..b54321b5b 100644
@@ -91,7 +91,10 @@ The application requires a number of command line

options:

.. code-block:: console

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line options: · line 97

Quoted 5 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -1
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --portmap="(port, port)[,(port, port)]"

Will it be better to represent as [--portmap="(port, port)[,(port, port)]] as it is a optional parameter ?

Quoted 13 lines

where,

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -1
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled

by default).
+* --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
by default)

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
+
+*   --portmap="(port,port)[,(port,port)]": Determines which ports are
Quoted 2 lines

mapped to
+ which ports for packet forwarding.

May be rephrased a bit shorter as "Determines forwarding ports mapping".

Quoted 11 lines

To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address updating enabled, issue the command:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +3 -0
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8
+RX queues per lcore and MAC address updating enabled, to forward RX
+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue the

command:

IMO, No need to mentioned information about MAC address update enabled because it is enabled by default and no param is passed in below command. So information is not relevant here.

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 123

Quoted 4 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -0
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"

It looks like comma is missed in between two port map information.

Quoted 17 lines

+
Refer to the *DPDK Getting Started Guide* for general information on running applications and the Environment Abstraction Layer (EAL) options.

examples/l2fwd/main.c index renamed from examples/l2fwd/main.c +0 -0

88ddfe589..81861a22a 100644

examples/l2fwd/main.c +1 -0
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 74

Quoted 7 lines
examples/l2fwd/main.c +4 -0
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports instead of following

struct port_pair_params {
     uint16_t port1;
     uint16_t port2;
}; 

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 77

Quoted 3 lines
examples/l2fwd/main.c +3 -0
+
+static struct port_pair_params
+port_pair_params_array[RTE_MAX_ETHPORTS];

Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed ? As I understood it is used to store CLI port mapping, I think it is better to Use any other MACRO which define maximum number of port map parameter. If you consider it, then it can also be updated in documentation.

Quoted 28 lines
examples/l2fwd/main.c +3 -0
+static struct port_pair_params *port_pair_params; static uint16_t
+nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 

#define MAX_RX_QUEUE_PER_LCORE 16

examples/l2fwd/main.c +0 -1
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
-		   "  -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +0 -1
-		   "  --[no-]mac-updating: Enable or disable MAC addresses

updating (enabled by default)\n"

examples/l2fwd/main.c +0 -2
-		   "      When enabled:\n"
-		   "       - The source MAC address is replaced by the TX port

MAC address\n"

examples/l2fwd/main.c +0 -1
-		   "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n",

examples/l2fwd/main.c +1 -0
+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to

disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +1 -0
+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled by default)\n"

examples/l2fwd/main.c +2 -0
+	       "      When enabled:\n"
+	       "       - The source MAC address is replaced by the TX port MAC

address\n"

examples/l2fwd/main.c +1 -0
+	       "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n"

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 313

Quoted 2 lines
examples/l2fwd/main.c +2 -0
+	       "  --portmap: Configure forwarding port pair mapping\n"
+	       "	      Default: alternate port pairs\n\n",

IMO, Indentation changes for other parameters should in separate patch.

Quoted 241 lines
	       prgname);
}
examples/l2fwd/main.c +54 -0
@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg) {
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		memcpy(s, p, size);
+		s[size] = '\0';
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] >= RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params:
 %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+

static unsigned int
l2fwd_parse_nqueue(const char *q_arg)
{

examples/l2fwd/main.c +1 -0
@@ -361,6 +428,7 @@ static const char short_options[] =
 
 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"
 
 enum {
 	/* long options mapped to a short option */ @@ -368,11 +436,13 @@
enum {
	/* first long only option value must be >= 256, so that we won't
	 * conflict with short options */
	CMD_LINE_OPT_MIN_NUM = 256,
examples/l2fwd/main.c +1 -0
+	CMD_LINE_OPT_PORTMAP_NUM,

};

static const struct option lgopts[] = {
{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating,
1},
{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument,
&mac_updating, 0},

examples/l2fwd/main.c +1 -0
+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0,

CMD_LINE_OPT_PORTMAP_NUM},

	{NULL, 0, 0, 0}
};
examples/l2fwd/main.c +2 -0
@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];
 
 	argvopt = argv;
+	port_pair_params = NULL;
 
 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) { @@ -423,7
+494,13 @@ l2fwd_parse_args(int argc, char **argv)
			break;

		/* long options */
examples/l2fwd/main.c +7 -1
-		case 0:
+		case CMD_LINE_OPT_PORTMAP_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
			break;

		default:
examples/l2fwd/main.c +42 -0
@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }
 
+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port
 mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the
 board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */  static

void check_all_ports_link_status(uint32_t port_mask) @@ -555,6 +674,11 @@ main(int argc, char **argv)

	if (nb_ports == 0)
		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
examples/l2fwd/main.c +5 -0
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
	/* check port mask to possible port mask */
	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
examples/l2fwd/main.c +28 -18
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;
 
-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}
 
 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +

MAX_PKT_BURST +
--
2.17.1

#47 lore raw

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

Quoted 2 lines · Sunil Kumar Kori
Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]>

[...]

Quoted 10 lines

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 74

Quoted 8 lines
examples/l2fwd/main.c +4 -0
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports instead of following

Quoted 4 lines
struct port_pair_params {
      uint16_t port1;
      uint16_t port2;
};

Initially it was so, but I made a comment that this leads to code duplication in check_port_pair_config() (same checks for port1 and port2
- now handled via loop). I still stand by this comment :)

With regards
Andrzej Ostruszka

#48 lore raw

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

Quoted 34 lines

-----Original Message-----
From: Andrzej Ostruszka [C] <[email protected]>
Sent: Monday, April 27, 2020 3:06 PM
To: Sunil Kumar Kori <[email protected]>; Pavan Nikhilesh Bhagavatula <[email protected]>; Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; John McNamara <[email protected]>; Marko Kovacevic <[email protected]>; Ori Kam <[email protected]>; Bruce Richardson <[email protected]>; Radu Nicolau <[email protected]>; Akhil Goyal <[email protected]>; Tomasz Kantecki <[email protected]>
Cc: [email protected]; Vamsi Krishna Attunuru <[email protected]>
Subject: Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

Quoted 2 lines · Sunil Kumar Kori
Quoted 2 lines

-----Original Message-----
From: [email protected] <[email protected]>

[...]

Quoted 10 lines

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 74

Quoted 8 lines
examples/l2fwd/main.c +4 -0
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports

Quoted 4 lines
instead of following struct port_pair_params {
      uint16_t port1;
      uint16_t port2;
};
Quoted 4 lines

Initially it was so, but I made a comment that this leads to code duplication in check_port_pair_config() (same checks for port1 and port2
- now handled via loop). I still stand by this comment :)

Okay, I think having for two variables only is not a big deal. if it is some higher number then it look fine. IMO, it is more readable when using suggested way. You can take on this.

Quoted 2 lines

With regards
Andrzej Ostruszka

#49 lore raw

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

In reply to Sunil Kumar Kori
Quoted 12 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command

Quoted 6 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b0067..b54321b5b 100644
@@ -91,7 +91,10 @@ The application requires a number of command

line

Quoted 5 lines

options:

.. code-block:: console

- ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-

Quoted 9 lines

updating

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line options: · line 97

Quoted 4 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -0
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --portmap="(port, port)[,(port, port)]"

Will it be better to represent as [--portmap="(port, port)[,(port, port)]] as it is a optional parameter ?

Will change in v4.

Quoted 12 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 102

Quoted 8 lines

where,

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -1
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled

Quoted 2 lines

by default).
+* --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled

Quoted 8 lines
Quoted 5 lines

by default)

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
+
+*   --portmap="(port,port)[,(port,port)]": Determines which ports are

mapped to
+ which ports for packet forwarding.

May be rephrased a bit shorter as "Determines forwarding ports mapping".

Will change in v4.

Quoted 13 lines
Quoted 2 lines

To run the application in linux environment with 4 lcores, 16 ports and

8 RX

Quoted 1 line

queues per lcore and MAC address updating enabled, issue the

command:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 118

Quoted 6 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8
+RX queues per lcore and MAC address updating enabled, to forward

RX

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 119

Quoted 1 line
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +1 -0
+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue
Quoted 9 lines

the

Quoted 1 line

command:

IMO, No need to mentioned information about MAC address update enabled because it
is enabled by default and no param is passed in below command. So information is not
relevant here.

Will change in v4.

Quoted 33 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 123

Quoted 4 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -0
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"

It looks like comma is missed in between two port map information.

Quoted 2 lines

+
Refer to the *DPDK Getting Started Guide* for general information

on running

Quoted 15 lines

applications and the Environment Abstraction Layer (EAL) options.

examples/l2fwd/main.c index renamed from examples/l2fwd/main.c +0 -0

88ddfe589..81861a22a 100644

examples/l2fwd/main.c +1 -0
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 74

Quoted 7 lines
examples/l2fwd/main.c +4 -0
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports instead of following

Quoted 5 lines
struct port_pair_params {
    uint16_t port1;
    uint16_t port2;
};

It was done to reduce code duplication. Will leave as is.

Quoted 5 lines

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 77

Quoted 3 lines
examples/l2fwd/main.c +3 -0
+
+static struct port_pair_params
+port_pair_params_array[RTE_MAX_ETHPORTS];

Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed ? As I understood it is used to store CLI port mapping,

RTE_MAX_ETHPORTS/2 will be sufficient.

Quoted 37 lines

I think it is better to
Use any other MACRO which define maximum number of port map parameter.
If you consider it, then it can also be updated in documentation.

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 305

Quoted 9 lines
examples/l2fwd/main.c +3 -0
+static struct port_pair_params *port_pair_params; static uint16_t
+nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 

#define MAX_RX_QUEUE_PER_LCORE 16

examples/l2fwd/main.c +0 -0
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
 	       "  -p PORTMASK: hexadecimal bitmask of ports to

configure\n"

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 306

Quoted 1 line
examples/l2fwd/main.c +0 -0
 	       "  -q NQ: number of queue (=ports) per lcore (default is

1)\n"

examples/l2fwd/main.c

Quoted 3 lines
examples/l2fwd/main.c +0 -1
-		   "  -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +0 -1
-		   "  --[no-]mac-updating: Enable or disable MAC

addresses

examples/l2fwd/main.c

Quoted 3 lines

updating (enabled by default)\n"

examples/l2fwd/main.c +0 -2
-		   "      When enabled:\n"
-		   "       - The source MAC address is replaced by the TX

port

examples/l2fwd/main.c

Quoted 4 lines

MAC address\n"

examples/l2fwd/main.c +0 -1
-		   "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n",

examples/l2fwd/main.c +1 -0
+	       "  -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to

examples/l2fwd/main.c

Quoted 2 lines

disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +1 -0
+	       "  --[no-]mac-updating: Enable or disable MAC addresses

updating

examples/l2fwd/main.c

Quoted 3 lines

(enabled by default)\n"

examples/l2fwd/main.c +2 -0
+	       "      When enabled:\n"
+	       "       - The source MAC address is replaced by the TX port

MAC

Quoted 8 lines

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 313

Quoted 5 lines

address\n"

examples/l2fwd/main.c +1 -0
+	       "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n"

examples/l2fwd/main.c +2 -0
+	       "  --portmap: Configure forwarding port pair mapping\n"
+	       "	      Default: alternate port pairs\n\n",

IMO, Indentation changes for other parameters should in separate patch.

It's not a functionality change it just minor code formatting change and doesn't deserve a separate patch.

Quoted 8 lines

examples/l2fwd/main.c l2fwd_parse_portmask(const char

Quoted 4 lines
	       prgname);
}
examples/l2fwd/main.c +0 -0
@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char

*portmask)

Quoted 3 lines
examples/l2fwd/main.c +0 -0
 	return pm;
 }
 
#50 lore raw

Re: [dpdk-dev] [PATCH v3] examples/l2fwd: add cmdline option for forwarding port info

Sent from Workspace ONE Boxer

Quoted 112 lines · Pavan Nikhilesh Bhagavatula
Quoted 13 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command

Quoted 7 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b0067..b54321b5b 100644
@@ -91,7 +91,10 @@ The application requires a number of command

line

Quoted 5 lines

options:

.. code-block:: console

- ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-

Quoted 9 lines

updating

doc/guides/sample_app_ug/l2_forward_real_virtual.rst The application requires a number of command line options: · line 97

Quoted 4 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -0
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   --portmap="(port, port)[,(port, port)]"

Will it be better to represent as [--portmap="(port, port)[,(port, port)]] as it is a optional parameter ?

Will change in v4.

Quoted 12 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst where, · line 102

Quoted 8 lines

where,

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -1
@@ -99,7 +102,10 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled

Quoted 2 lines

by default).
+* --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled

Quoted 8 lines
Quoted 5 lines

by default)

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
+
+*   --portmap="(port,port)[,(port,port)]": Determines which ports are

mapped to
+ which ports for packet forwarding.

May be rephrased a bit shorter as "Determines forwarding ports mapping".

Will change in v4.

Quoted 13 lines
Quoted 2 lines

To run the application in linux environment with 4 lcores, 16 ports and

8 RX

Quoted 1 line

queues per lcore and MAC address updating enabled, issue the

command:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 118

Quoted 6 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
@@ -108,6 +114,14 @@ updating enabled, issue the command:
 
    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8
+RX queues per lcore and MAC address updating enabled, to forward

RX

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 119

Quoted 1 line
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +1 -0
+traffic of ports 0 & 1 on ports 2 & 3 respectively and vice versa, issue
Quoted 9 lines

the

Quoted 1 line

command:

IMO, No need to mentioned information about MAC address update enabled because it
is enabled by default and no param is passed in below command. So information is not
relevant here.

Will change in v4.

Quoted 33 lines

doc/guides/sample_app_ug/l2_forward_real_virtual.rst updating enabled, issue the command: · line 123

Quoted 4 lines
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +4 -0
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"

It looks like comma is missed in between two port map information.

Quoted 2 lines

+
Refer to the *DPDK Getting Started Guide* for general information

on running

Quoted 15 lines

applications and the Environment Abstraction Layer (EAL) options.

examples/l2fwd/main.c index renamed from examples/l2fwd/main.c +0 -0

88ddfe589..81861a22a 100644

examples/l2fwd/main.c +1 -0
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 74

Quoted 7 lines
examples/l2fwd/main.c +4 -0
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS    2
+     uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;

Is there any specific reason to use this syntax to declare two ports instead of following

Quoted 5 lines
struct port_pair_params {
    uint16_t port1;
    uint16_t port2;
};
Quoted 1 line

It was done to reduce code duplication. Will leave as is.

Ack.

Quoted 8 lines
Quoted 6 lines

examples/l2fwd/main.c static uint32_t l2fwd_enabled_port_mask = 0; · line 77

Quoted 4 lines
examples/l2fwd/main.c +3 -0
+
+static struct port_pair_params
+port_pair_params_array[RTE_MAX_ETHPORTS];

Should not be RTE_MAX_ETHPORTS/2 if only 1:1 mapping is allowed ? As I understood it is used to store CLI port mapping,

RTE_MAX_ETHPORTS/2 will be sufficient.

Ack.

Quoted 44 lines
Quoted 44 lines

I think it is better to
Use any other MACRO which define maximum number of port map parameter.
If you consider it, then it can also be updated in documentation.

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 305

Quoted 9 lines
examples/l2fwd/main.c +3 -0
+static struct port_pair_params *port_pair_params; static uint16_t
+nb_port_pair_params;
+
 static unsigned int l2fwd_rx_queue_per_lcore = 1;
 

#define MAX_RX_QUEUE_PER_LCORE 16

examples/l2fwd/main.c +0 -0
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
      printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
             "  -p PORTMASK: hexadecimal bitmask of ports to

configure\n"

examples/l2fwd/main.c l2fwd_usage(const char *prgname) · line 306

Quoted 1 line
examples/l2fwd/main.c +0 -0
             "  -q NQ: number of queue (=ports) per lcore (default is

1)\n"

Quoted 3 lines
examples/l2fwd/main.c +0 -1
-                "  -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to disable, 10 default, 86400 maximum)\n"
- " --[no-]mac-updating: Enable or disable MAC

addresses

examples/l2fwd/main.c

Quoted 3 lines

updating (enabled by default)\n"

examples/l2fwd/main.c +0 -2
-                "      When enabled:\n"
-                "       - The source MAC address is replaced by the TX

port

Quoted 4 lines

MAC address\n"
- " - The destination MAC address is replaced by
02:00:00:00:00:TX_PORT_ID\n",
+ " -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to

Quoted 2 lines

disable, 10 default, 86400 maximum)\n"
+ " --[no-]mac-updating: Enable or disable MAC addresses

updating

examples/l2fwd/main.c

Quoted 3 lines

(enabled by default)\n"

examples/l2fwd/main.c +2 -0
+            "      When enabled:\n"
+            "       - The source MAC address is replaced by the TX port

MAC

Quoted 5 lines

address\n"
+ " - The destination MAC address is replaced by
02:00:00:00:00:TX_PORT_ID\n"

examples/l2fwd/main.c +2 -0
+            "  --portmap: Configure forwarding port pair mapping\n"
+            "       Default: alternate port pairs\n\n",
Quoted 5 lines
Quoted 2 lines

IMO, Indentation changes for other parameters should in separate patch.

It's not a functionality change it just minor code formatting change and doesn’t deserve a separate patch.

Ack.

Quoted 9 lines
Quoted 9 lines

examples/l2fwd/main.c l2fwd_parse_portmask(const char

Quoted 5 lines
             prgname);
}
examples/l2fwd/main.c +0 -0
@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char

*portmask)

Quoted 3 lines
examples/l2fwd/main.c +0 -0
      return pm;
 }
 
#51 lore raw

PATCH v4 1/1 examples/l2fwd: add cmdline option for forwarding port info

In reply to [email protected]

From: Vamsi Attunuru <[email protected]>

Current l2fwd application statically configures adjacent ports as destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port map, port/queue configuration of each lcore gets modified when portmap option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

With above portmap option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5)

---
v4 Changes:
* Documentation changes. (Sunil)
* reduce port_pair_params_array size. (Sunil)

v3 Changes:
* s/config/portmap/
* rebase on master

v2 Changes:
* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

 doc/guides/rel_notes/release_20_05.rst        |   7 +
 .../sample_app_ug/l2_forward_real_virtual.rst |  17 +-
 examples/l2fwd/main.c                         | 182 +++++++++++++++---
 3 files changed, 180 insertions(+), 26 deletions(-)
doc/guides/rel_notes/release_20_05.rst +7 -0
index b124c3f28..019bec5d7 100644
@@ -212,6 +212,13 @@ New Features
   * Added IPsec inbound load-distribution support for ipsec-secgw application
     using NIC load distribution feature(Flow Director).
 
+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port, port)]"`` to
+  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for more
+  details of this parameter usage.
+
 
 Removed Items
 -------------
doc/guides/sample_app_ug/l2_forward_real_virtual.rst +15 -2
index 39d6b0067..90ca609d6 100644
@@ -91,7 +91,10 @@ The application requires a number of command line options:
 
 .. code-block:: console
 
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   [--portmap="(port, port)[,(port, port)]"]
 
 where,
 
@@ -99,7 +102,9 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default).
+*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled by default)
+
+*   --portmap="(port,port)[,(port,port)]": Determines forwarding ports mapping.
 
 To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address
 updating enabled, issue the command:
@@ -108,6 +113,14 @@ updating enabled, issue the command:
 
     $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8 RX queues
+per lcore, to forward RX traffic of ports 0 & 1 on ports 2 & 3 respectively and
+vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
+
 Refer to the *DPDK Getting Started Guide* for general information on running applications
 and the Environment Abstraction Layer (EAL) options.
 
examples/l2fwd/main.c +158 -24

262 lines not on this page · raw message

--
2.17.1

Acked-by on v4 1/1
#52 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Looks okay.

Regards
Sunil Kumar Kori

Quoted 33 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Tuesday, April 28, 2020 12:01 AM
To: Jerin Jacob Kollanukkaran <[email protected]>; [email protected]; John McNamara <[email protected]>; Marko Kovacevic <[email protected]>; Ori Kam <[email protected]>; Bruce Richardson <[email protected]>; Radu Nicolau <[email protected]>; Akhil Goyal <[email protected]>; Tomasz Kantecki <[email protected]>; Sunil Kumar Kori <[email protected]>; Pavan Nikhilesh Bhagavatula <[email protected]>
Cc: Andrzej Ostruszka [C] <[email protected]>; [email protected]; Vamsi Krishna Attunuru <[email protected]>
Subject: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

From: Vamsi Attunuru <[email protected]>

Current l2fwd application statically configures adjacent ports as destination ports for forwarding the traffic.

Add a portmap option to pass the forwarding port pair mapping which allows the user to configure forwarding port mapping.

If no portmap argument is specified, destination port map is not changed and traffic gets forwarded with existing mapping.

To align port/queue configuration of each lcore with destination port map, port/queue configuration of each lcore gets modified when portmap option is specified.

Ex: ./l2fwd -c 0xff -- -p 0x3f -q 2 --portmap="(0,3)(1,4)(2,5)"

Quoted 7 lines

With above portmap option, traffic received from portid = 0 gets forwarded to port = 3 and vice versa, similarly traffic gets forwarded on other port pairs (1,4) and (2,5)

Quoted 380 lines

---
v4 Changes:
* Documentation changes. (Sunil)
* reduce port_pair_params_array size. (Sunil)

v3 Changes:
* s/config/portmap/
* rebase on master

v2 Changes:
* Fix command option format in docs.
* Use memcpy instead of snprintf.
* Rephrase if-else condition.

doc/guides/rel_notes/release_20_05.rst        |   7 +
.../sample_app_ug/l2_forward_real_virtual.rst |  17 +-
examples/l2fwd/main.c                         | 182 +++++++++++++++---
3 files changed, 180 insertions(+), 26 deletions(-)
doc/guides/rel_notes/release_20_05.rst +7 -0
index b124c3f28..019bec5d7 100644
@@ -212,6 +212,13 @@ New Features
  * Added IPsec inbound load-distribution support for ipsec-secgw application
    using NIC load distribution feature(Flow Director).
 
+* **Added --portmap command line parameter to l2fwd example.**
+
+  Added new command line option ``--portmap="(port, port)[,(port,
+ port)]"`` to  pass forwarding port details.
+  See the :doc:`doc/guides/sample_app_ug/l2_forward_real_virtual` for
+ more  details of this parameter usage.
+
 

Removed Items
-------------

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -0
index 39d6b0067..90ca609d6 100644
@@ -91,7 +91,10 @@ The application requires a number of command line

options:

.. code-block:: console

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +5 -1
-    ./build/l2fwd [EAL options] -- -p PORTMASK [-q NQ] --[no-]mac-updating
+    ./build/l2fwd [EAL options] -- -p PORTMASK
+                                   [-q NQ]
+                                   --[no-]mac-updating
+                                   [--portmap="(port, port)[,(port,
+ port)]"]

where,

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +0 -1
@@ -99,7 +102,9 @@ where,
 
 *   q NQ: A number of queues (=ports) per lcore (default is 1)
 
-*   --[no-]mac-updating: Enable or disable MAC addresses updating (enabled

by default).
+* --[no-]mac-updating: Enable or disable MAC addresses updating (enabled
by default)

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +2 -0
+
+*   --portmap="(port,port)[,(port,port)]": Determines forwarding ports

mapping.

To run the application in linux environment with 4 lcores, 16 ports and 8 RX queues per lcore and MAC address updating enabled, issue the command:

doc/guides/sample_app_ug/l2_forward_real_virtual.rst +8 -0
@@ -108,6 +113,14 @@ updating enabled, issue the command:
 
    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p ffff
 
+To run the application in linux environment with 4 lcores, 4 ports, 8
+RX queues per lcore, to forward RX traffic of ports 0 & 1 on ports 2 &
+3 respectively and vice versa, issue the command:
+
+.. code-block:: console
+
+    $ ./build/l2fwd -l 0-3 -n 4 -- -q 8 -p f --portmap="(0,2)(1,3)"
+

Refer to the *DPDK Getting Started Guide* for general information on running applications and the Environment Abstraction Layer (EAL) options.

examples/l2fwd/main.c index renamed from examples/l2fwd/main.c +0 -0

88ddfe589..1d84cd789 100644

examples/l2fwd/main.c +7 -0
@@ -38,6 +38,7 @@
 #include <rte_ethdev.h>
 #include <rte_mempool.h>
 #include <rte_mbuf.h>
+#include <rte_string_fns.h>
 
 static volatile bool force_quit;
 
@@ -67,6 +68,15 @@ static uint32_t l2fwd_enabled_port_mask = 0;
 /* list of enabled ports */
 static uint32_t l2fwd_dst_ports[RTE_MAX_ETHPORTS];
 
+struct port_pair_params {
+#define NUM_PORTS	2
+	uint16_t port[NUM_PORTS];
+} __rte_cache_aligned;
+
+static struct port_pair_params

port_pair_params_array[RTE_MAX_ETHPORTS

examples/l2fwd/main.c +3 -0
+/ 2]; static struct port_pair_params *port_pair_params; static uint16_t
+nb_port_pair_params;
+

static unsigned int l2fwd_rx_queue_per_lcore = 1;

#define MAX_RX_QUEUE_PER_LCORE 16

examples/l2fwd/main.c +0 -1
@@ -294,11 +304,13 @@ l2fwd_usage(const char *prgname)
 	printf("%s [EAL options] -- -p PORTMASK [-q NQ]\n"
 	       "  -p PORTMASK: hexadecimal bitmask of ports to configure\n"
 	       "  -q NQ: number of queue (=ports) per lcore (default is 1)\n"
-		   "  -T PERIOD: statistics will be refreshed each PERIOD

seconds (0 to disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +0 -1
-		   "  --[no-]mac-updating: Enable or disable MAC addresses

updating (enabled by default)\n"

examples/l2fwd/main.c +0 -2
-		   "      When enabled:\n"
-		   "       - The source MAC address is replaced by the TX port

MAC address\n"

examples/l2fwd/main.c +0 -1
-		   "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n",

examples/l2fwd/main.c +1 -0
+	       "  -T PERIOD: statistics will be refreshed each PERIOD seconds (0 to

disable, 10 default, 86400 maximum)\n"

examples/l2fwd/main.c +1 -0
+	       "  --[no-]mac-updating: Enable or disable MAC addresses updating

(enabled by default)\n"

examples/l2fwd/main.c +2 -0
+	       "      When enabled:\n"
+	       "       - The source MAC address is replaced by the TX port MAC

address\n"

examples/l2fwd/main.c +1 -0
+	       "       - The destination MAC address is replaced by

02:00:00:00:00:TX_PORT_ID\n"

examples/l2fwd/main.c +2 -0
+	       "  --portmap: Configure forwarding port pair mapping\n"
+	       "	      Default: alternate port pairs\n\n",
	       prgname);
}
examples/l2fwd/main.c +54 -0
@@ -319,6 +331,61 @@ l2fwd_parse_portmask(const char *portmask)
 	return pm;
 }
 
+static int
+l2fwd_parse_port_pair_config(const char *q_arg) {
+	enum fieldnames {
+		FLD_PORT1 = 0,
+		FLD_PORT2,
+		_NUM_FLD
+	};
+	unsigned long int_fld[_NUM_FLD];
+	const char *p, *p0 = q_arg;
+	char *str_fld[_NUM_FLD];
+	unsigned int size;
+	char s[256];
+	char *end;
+	int i;
+
+	nb_port_pair_params = 0;
+
+	while ((p = strchr(p0, '(')) != NULL) {
+		++p;
+		p0 = strchr(p, ')');
+		if (p0 == NULL)
+			return -1;
+
+		size = p0 - p;
+		if (size >= sizeof(s))
+			return -1;
+
+		memcpy(s, p, size);
+		s[size] = '\0';
+		if (rte_strsplit(s, sizeof(s), str_fld,
+				 _NUM_FLD, ',') != _NUM_FLD)
+			return -1;
+		for (i = 0; i < _NUM_FLD; i++) {
+			errno = 0;
+			int_fld[i] = strtoul(str_fld[i], &end, 0);
+			if (errno != 0 || end == str_fld[i] ||
+			    int_fld[i] >= RTE_MAX_ETHPORTS)
+				return -1;
+		}
+		if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2) {
+			printf("exceeded max number of port pair params:
 %hu\n",
+				nb_port_pair_params);
+			return -1;
+		}
+		port_pair_params_array[nb_port_pair_params].port[0] =
+				(uint16_t)int_fld[FLD_PORT1];
+		port_pair_params_array[nb_port_pair_params].port[1] =
+				(uint16_t)int_fld[FLD_PORT2];
+		++nb_port_pair_params;
+	}
+	port_pair_params = port_pair_params_array;
+	return 0;
+}
+

static unsigned int
l2fwd_parse_nqueue(const char *q_arg)
{

examples/l2fwd/main.c +1 -0
@@ -361,6 +428,7 @@ static const char short_options[] =
 
 #define CMD_LINE_OPT_MAC_UPDATING "mac-updating"
 #define CMD_LINE_OPT_NO_MAC_UPDATING "no-mac-updating"
+#define CMD_LINE_OPT_PORTMAP_CONFIG "portmap"
 
 enum {
 	/* long options mapped to a short option */ @@ -368,11 +436,13 @@
enum {
	/* first long only option value must be >= 256, so that we won't
	 * conflict with short options */
	CMD_LINE_OPT_MIN_NUM = 256,
examples/l2fwd/main.c +1 -0
+	CMD_LINE_OPT_PORTMAP_NUM,

};

static const struct option lgopts[] = {
{ CMD_LINE_OPT_MAC_UPDATING, no_argument, &mac_updating,
1},
{ CMD_LINE_OPT_NO_MAC_UPDATING, no_argument,
&mac_updating, 0},

examples/l2fwd/main.c +1 -0
+	{ CMD_LINE_OPT_PORTMAP_CONFIG, 1, 0,

CMD_LINE_OPT_PORTMAP_NUM},

	{NULL, 0, 0, 0}
};
examples/l2fwd/main.c +2 -0
@@ -386,6 +456,7 @@ l2fwd_parse_args(int argc, char **argv)
 	char *prgname = argv[0];
 
 	argvopt = argv;
+	port_pair_params = NULL;
 
 	while ((opt = getopt_long(argc, argvopt, short_options,
 				  lgopts, &option_index)) != EOF) { @@ -423,7
+494,13 @@ l2fwd_parse_args(int argc, char **argv)
			break;

		/* long options */
examples/l2fwd/main.c +7 -1
-		case 0:
+		case CMD_LINE_OPT_PORTMAP_NUM:
+			ret = l2fwd_parse_port_pair_config(optarg);
+			if (ret) {
+				fprintf(stderr, "Invalid config\n");
+				l2fwd_usage(prgname);
+				return -1;
+			}
			break;

		default:
examples/l2fwd/main.c +42 -0
@@ -440,6 +517,48 @@ l2fwd_parse_args(int argc, char **argv)
 	return ret;
 }
 
+/*
+ * Check port pair config with enabled port mask,
+ * and for valid port pair combinations.
+ */
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port
 mask\n",
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the
 board\n",
+				       portid);
+				return -1;
+			}
+
+			port_pair_mask |= 1 << portid;
+		}
+
+		if (port_pair_config_mask & port_pair_mask) {
+			printf("port %u is used in other port pairs\n", portid);
+			return -1;
+		}
+		port_pair_config_mask |= port_pair_mask;
+	}
+
+	l2fwd_enabled_port_mask &= port_pair_config_mask;
+
+	return 0;
+}
+
 /* Check the link status of all ports in up to 9s, and print them finally */  static

void check_all_ports_link_status(uint32_t port_mask) @@ -555,6 +674,11 @@ main(int argc, char **argv)

	if (nb_ports == 0)
		rte_exit(EXIT_FAILURE, "No Ethernet ports - bye\n");
examples/l2fwd/main.c +5 -0
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
	/* check port mask to possible port mask */
	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
examples/l2fwd/main.c +28 -18
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;
 
-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}
 
 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +

MAX_PKT_BURST +
--
2.17.1

#53 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

In reply to [email protected]

Hi Vamsi & Pavan,

I like this idea, couple of queries

snipped

Quoted 21 lines
examples/l2fwd/main.c +14 -0
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0) {
+				printf("port %u is not enabled in port

mask\n",

examples/l2fwd/main.c +5 -0
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the

board\n",

examples/l2fwd/main.c l2fwd_parse_args(int argc, char **argv) · line 546

Quoted 4 lines
examples/l2fwd/main.c +4 -0
+				       portid);
+				return -1;
+			}
+

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Quoted 3 lines
examples/l2fwd/main.c +3 -0
+			port_pair_mask |= 1 << portid;
+		}
+

snipped

examples/l2fwd/main.c main(int argc, char **argv) · line 691

Quoted 13 lines
examples/l2fwd/main.c +5 -0
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair config\n");
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible (0x%x)\n",
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that port?

Quoted 37 lines
examples/l2fwd/main.c +17 -17
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each port.
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;
 
-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
+				continue;
 
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {

examples/l2fwd/main.c main(int argc, char **argv) · line 721

Quoted 10 lines
examples/l2fwd/main.c +9 -0
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in portmask.\n");
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}

As mentioned above there can ports in mask which might be disabled for port pair. Should not that be skipped rather than setting last port rx-tx loopback?

Quoted 15 lines
examples/l2fwd/main.c +2 -1
 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n", rx_lcore_id,
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +

MAX_PKT_BURST +
--
2.17.1

#54 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Quoted 29 lines

Hi Vamsi & Pavan,

I like this idea, couple of queries

snipped

examples/l2fwd/main.c l2fwd_parse_args(int argc, char **argv)

Quoted 13 lines
examples/l2fwd/main.c +13 -0
+static int
+check_port_pair_config(void)
+{
+	uint32_t port_pair_config_mask = 0;
+	uint32_t port_pair_mask = 0;
+	uint16_t index, i, portid;
+
+	for (index = 0; index < nb_port_pair_params; index++) {
+		port_pair_mask = 0;
+
+		for (i = 0; i < NUM_PORTS; i++)  {
+			portid = port_pair_params[index].port[i];
+			if ((l2fwd_enabled_port_mask & (1 << portid))

== 0) {

Quoted 9 lines
(quoted diff) +1 -0
+				printf("port %u is not enabled in port

mask\n",

examples/l2fwd/main.c +5 -0
+				       portid);
+				return -1;
+			}
+			if (!rte_eth_dev_is_valid_port(portid)) {
+				printf("port %u is not present on the

board\n",

(quoted diff) +1 -0
+				       portid);
Quoted 9 lines

examples/l2fwd/main.c l2fwd_parse_portmask(const char *portmask) · line 388

Quoted 3 lines
examples/l2fwd/main.c +3 -0
+				return -1;
+			}
+

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple examples.

Quoted 18 lines
Quoted 3 lines
examples/l2fwd/main.c +3 -0
+			port_pair_mask |= 1 << portid;
+		}
+

snipped

examples/l2fwd/main.c main(int argc, char **argv)

Quoted 4 lines
examples/l2fwd/main.c +3 -0
+	if (port_pair_params != NULL) {
+		if (check_port_pair_config() < 0)
+			rte_exit(EXIT_FAILURE, "Invalid port pair

config\n");

examples/l2fwd/main.c main(int argc, char **argv)

Quoted 5 lines
examples/l2fwd/main.c +2 -0
+	}
+
 	/* check port mask to possible port mask */
 	if (l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1))
 		rte_exit(EXIT_FAILURE, "Invalid portmask; possible

(0x%x)\n",

Quoted 7 lines

examples/l2fwd/main.c main(int argc, char **argv) · line 691

Quoted 4 lines
examples/l2fwd/main.c +0 -0
@@ -565,26 +689,35 @@ main(int argc, char **argv)
 		l2fwd_dst_ports[portid] = 0;
 	last_port = 0;
 

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that port?

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

Quoted 48 lines

Review main(int argc, char **argv) · line 569

Quoted 3 lines
(quoted diff) +0 -2
-	/*
-	 * Each logical core is assigned a dedicated TX queue on each

port.

Quoted 14 lines
examples/l2fwd/main.c +5 -8
-	 */
-	RTE_ETH_FOREACH_DEV(portid) {
-		/* skip ports that are not enabled */
-		if ((l2fwd_enabled_port_mask & (1 << portid)) == 0)
-			continue;
+	/* populate destination port details */
+	if (port_pair_params != NULL) {
+		uint16_t idx, p;
 
-		if (nb_ports_in_mask % 2) {
-			l2fwd_dst_ports[portid] = last_port;
-			l2fwd_dst_ports[last_port] = portid;
+		for (idx = 0; idx < (nb_port_pair_params << 1); idx++) {
+			p = idx & 1;

examples/l2fwd/main.c main(int argc, char **argv)

Quoted 10 lines
examples/l2fwd/main.c +7 -2
+			portid = port_pair_params[idx >> 1].port[p];
+			l2fwd_dst_ports[portid] =
+				port_pair_params[idx >> 1].port[p ^ 1];
 		}
-		else
-			last_port = portid;
+	} else {
+		RTE_ETH_FOREACH_DEV(portid) {
+			/* skip ports that are not enabled */
+			if ((l2fwd_enabled_port_mask & (1 << portid))

== 0)

Quoted 8 lines
examples/l2fwd/main.c +2 -5
+				continue;
 
-		nb_ports_in_mask++;
-	}
-	if (nb_ports_in_mask % 2) {
-		printf("Notice: odd number of ports in portmask.\n");
-		l2fwd_dst_ports[last_port] = last_port;
+			if (nb_ports_in_mask % 2) {

examples/l2fwd/main.c main(int argc, char **argv)

Quoted 10 lines
examples/l2fwd/main.c +10 -0
+				l2fwd_dst_ports[portid] = last_port;
+				l2fwd_dst_ports[last_port] = portid;
+			} else {
+				last_port = portid;
+			}
+
+			nb_ports_in_mask++;
+		}
+		if (nb_ports_in_mask % 2) {
+			printf("Notice: odd number of ports in

portmask.\n");

Quoted 7 lines

examples/l2fwd/main.c main(int argc, char **argv) · line 721

Quoted 3 lines
examples/l2fwd/main.c +2 -0
+			l2fwd_dst_ports[last_port] = last_port;
+		}
 	}

As mentioned above there can ports in mask which might be disabled for port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and 1x40G Why force the user to explicitly mention 1x40G as port pair of itself in the portpair config?

Quoted 17 lines

examples/l2fwd/main.c main(int argc, char **argv) · line 749

Quoted 9 lines
examples/l2fwd/main.c +1 -1
 	rx_lcore_id = 0;
@@ -613,7 +746,8 @@ main(int argc, char **argv)
 
 		qconf->rx_port_list[qconf->n_rx_port] = portid;
 		qconf->n_rx_port++;
-		printf("Lcore %u: RX port %u\n", rx_lcore_id, portid);
+		printf("Lcore %u: RX port %u TX port %u\n",

rx_lcore_id,

Quoted 7 lines
examples/l2fwd/main.c +1 -0
+		       portid, l2fwd_dst_ports[portid]);
 	}
 
 	nb_mbufs = RTE_MAX(nb_ports * (nb_rxd + nb_txd +

MAX_PKT_BURST +
--
2.17.1

#55 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Hi Pavan,

snipped

Quoted 8 lines
Quoted 6 lines

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple examples.

I believe this patch is for example l2fwd. But you would like to have to updated for all example. I am ok for this.

snipped

Quoted 9 lines
Quoted 3 lines

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that port?

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

snipped

Quoted 8 lines
Quoted 4 lines

As mentioned above there can ports in mask which might be disabled for port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and 1x40G Why force the user to explicitly mention 1x40G as port pair of itself in the portpair config?

I am not sure if I follow your thought, as your current port map only allows 1:1 mapping by struct port_pair_params. This can be to self like (port0:port0),(port1:port1) or (port-0:port-1).

1. But current l2fwd_parse_port_pair_config does not consider the same port mapping as we have hard check for if (nb_port_pair_params >= RTE_MAX_ETHPORTS/2).

2. l2fwd_enabled_port_mask is global variable of user port mask. This can contain both valid and invalid mask. Hence we check l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1).

3. can these scenarios are true if we invoke check_port_pair_config before actual port_mask check.
a. there are only 4 ports, hence possible mask is 0xf.
b. user passes port argument as 0xe
c. check_port_pair_config gets masks for (1,3) as input and populates port_pair_config_mask.
d. As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Note: current speed can be fetched only if the port are started too (in Fortville).

snipped

#56 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Hi Vipin,

Quoted 50 lines

Hi Pavan,

snipped

Quoted 8 lines
Quoted 6 lines

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple

examples.
I believe this patch is for example l2fwd. But you would like to have to updated for all example. I am ok for this.

snipped

Quoted 3 lines
Quoted 3 lines

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that

port?

Quoted 6 lines

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

snipped

Quoted 2 lines
Quoted 2 lines

As mentioned above there can ports in mask which might be

disabled for

Quoted 4 lines
Quoted 2 lines

port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and

1x40G Why

Quoted 1 line

force the user to explicitly mention 1x40G as port pair of itself in the

portpair

Quoted 1 line

config?

I am not sure if I follow your thought, as your current port map only allows 1:1 mapping by struct port_pair_params. This can be to self like (port0:port0),(port1:port1) or (port-0:port-1).

1. But current l2fwd_parse_port_pair_config does not consider the same port mapping as we have hard check for `if (nb_port_pair_params

Quoted 1 line

= RTE_MAX_ETHPORTS/2)`.

2. l2fwd_enabled_port_mask is global variable of user port mask. This can contain both valid and invalid mask. Hence we check l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1).

Quoted 9 lines

3. can these scenarios are true if we invoke check_port_pair_config before actual port_mask check.
a. there are only 4 ports, hence possible mask is 0xf.
b. user passes port argument as 0xe
c. check_port_pair_config gets masks for (1,3) as input and populates port_pair_config_mask.
d. As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

Here user needs to explicitly mention (2,2) for port 2 to be setup else it will be skipped.
If you see check_port_pair_config below we disable the ports that are not Mentioned in portmap.

"
check_port_pair_config(void)
{

<snip>

		port_pair_config_mask |= port_pair_mask;
	}

	l2fwd_enabled_port_mask &= port_pair_config_mask;

	return 0;
}

"

Quoted 7 lines

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Unless the user disables stats using -T 0 option all the prints will be skipped.

Quoted 5 lines

Note: current speed can be fetched only if the port are started too (in Fortville).

snipped

#57 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Bruce, as maintainer of l2fwd example, any opinion about this change?

11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:

Quoted 97 lines

Hi Vipin,

Quoted 50 lines

Hi Pavan,

snipped

Quoted 8 lines
Quoted 6 lines

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple

examples.
I believe this patch is for example l2fwd. But you would like to have to updated for all example. I am ok for this.

snipped

Quoted 3 lines
Quoted 3 lines

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that

port?

Quoted 6 lines

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

snipped

Quoted 2 lines
Quoted 2 lines

As mentioned above there can ports in mask which might be

disabled for

Quoted 4 lines
Quoted 2 lines

port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and

1x40G Why

Quoted 1 line

force the user to explicitly mention 1x40G as port pair of itself in the

portpair

Quoted 1 line

config?

I am not sure if I follow your thought, as your current port map only allows 1:1 mapping by struct port_pair_params. This can be to self like (port0:port0),(port1:port1) or (port-0:port-1).

1. But current l2fwd_parse_port_pair_config does not consider the same port mapping as we have hard check for `if (nb_port_pair_params

Quoted 1 line

= RTE_MAX_ETHPORTS/2)`.

2. l2fwd_enabled_port_mask is global variable of user port mask. This can contain both valid and invalid mask. Hence we check l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1).

Quoted 9 lines

3. can these scenarios are true if we invoke check_port_pair_config before actual port_mask check.
a. there are only 4 ports, hence possible mask is 0xf.
b. user passes port argument as 0xe
c. check_port_pair_config gets masks for (1,3) as input and populates port_pair_config_mask.
d. As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

Here user needs to explicitly mention (2,2) for port 2 to be setup else it will be skipped.
If you see check_port_pair_config below we disable the ports that are not Mentioned in portmap.

"
check_port_pair_config(void)
{

<snip>

		port_pair_config_mask |= port_pair_mask;
	}

	l2fwd_enabled_port_mask &= port_pair_config_mask;

	return 0;
}

"

Quoted 7 lines

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Unless the user disables stats using -T 0 option all the prints will be skipped.

Quoted 5 lines

Note: current speed can be fetched only if the port are started too (in Fortville).

snipped

#58 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Quoted 2 lines · Thomas Monjalon

Bruce, as maintainer of l2fwd example, any opinion about this change?

Assuming all previous discussion on it is resolved, I'm fine with this patch, though I suspect it will only make 20.08 now.

Quoted 103 lines

11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:

Quoted 97 lines

Hi Vipin,

Quoted 50 lines

Hi Pavan,

snipped

Quoted 8 lines
Quoted 6 lines

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple

examples.
I believe this patch is for example l2fwd. But you would like to have to updated for all example. I am ok for this.

snipped

Quoted 3 lines
Quoted 3 lines

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that

port?

Quoted 6 lines

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

snipped

Quoted 2 lines
Quoted 2 lines

As mentioned above there can ports in mask which might be

disabled for

Quoted 4 lines
Quoted 2 lines

port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and

1x40G Why

Quoted 1 line

force the user to explicitly mention 1x40G as port pair of itself in the

portpair

Quoted 1 line

config?

I am not sure if I follow your thought, as your current port map only allows 1:1 mapping by struct port_pair_params. This can be to self like (port0:port0),(port1:port1) or (port-0:port-1).

1. But current l2fwd_parse_port_pair_config does not consider the same port mapping as we have hard check for `if (nb_port_pair_params

Quoted 1 line

= RTE_MAX_ETHPORTS/2)`.

2. l2fwd_enabled_port_mask is global variable of user port mask. This can contain both valid and invalid mask. Hence we check l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1).

Quoted 9 lines

3. can these scenarios are true if we invoke check_port_pair_config before actual port_mask check.
a. there are only 4 ports, hence possible mask is 0xf.
b. user passes port argument as 0xe
c. check_port_pair_config gets masks for (1,3) as input and populates port_pair_config_mask.
d. As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

Here user needs to explicitly mention (2,2) for port 2 to be setup else it will be skipped.
If you see check_port_pair_config below we disable the ports that are not Mentioned in portmap.

"
check_port_pair_config(void)
{

<snip>

		port_pair_config_mask |= port_pair_mask;
	}

	l2fwd_enabled_port_mask &= port_pair_config_mask;

	return 0;
}

"

Quoted 7 lines

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Unless the user disables stats using -T 0 option all the prints will be skipped.

Quoted 5 lines

Note: current speed can be fetched only if the port are started too (in Fortville).

snipped

#59 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

Quoted 8 lines · Bruce Richardson
Quoted 2 lines · Thomas Monjalon

Bruce, as maintainer of l2fwd example, any opinion about this change?

Assuming all previous discussion on it is resolved, I'm fine with this patch, though I suspect it will only make 20.08 now.

Ping for merge.

Quoted 104 lines
Quoted 104 lines

11/05/2020 02:23, Pavan Nikhilesh Bhagavatula:

Quoted 97 lines

Hi Vipin,

Quoted 50 lines

Hi Pavan,

snipped

Quoted 8 lines
Quoted 6 lines

Should we check & warn the user if
1. port speed mismatch
2. on different NUMA
3. port pairs are physical and vdev like tap, and KNI (performance).

Sure, it can be a separate patch as it will be applicable for multiple

examples.
I believe this patch is for example l2fwd. But you would like to have to updated for all example. I am ok for this.

snipped

Quoted 3 lines
Quoted 3 lines

Should not the check_port_pair be after this? If the port is not enabled in port_mask will you skip that pair? or skip RX-TX from that

port?

Quoted 6 lines

We check every port pair against l2fwd_enabled_port_mask in check_port_pair_config()

snipped

Quoted 2 lines
Quoted 2 lines

As mentioned above there can ports in mask which might be

disabled for

Quoted 4 lines
Quoted 2 lines

port pair. Should not that be skipped rather than setting last port rx- tx loopback?

There could be scenarios where user might want to test 2x10G and

1x40G Why

Quoted 1 line

force the user to explicitly mention 1x40G as port pair of itself in the

portpair

Quoted 1 line

config?

I am not sure if I follow your thought, as your current port map only allows 1:1 mapping by struct port_pair_params. This can be to self like (port0:port0),(port1:port1) or (port-0:port-1).

1. But current l2fwd_parse_port_pair_config does not consider the same port mapping as we have hard check for `if (nb_port_pair_params

Quoted 1 line

= RTE_MAX_ETHPORTS/2)`.

2. l2fwd_enabled_port_mask is global variable of user port mask. This can contain both valid and invalid mask. Hence we check l2fwd_enabled_port_mask & ~((1 << nb_ports) - 1).

Quoted 9 lines

3. can these scenarios are true if we invoke check_port_pair_config before actual port_mask check.
a. there are only 4 ports, hence possible mask is 0xf.
b. user passes port argument as 0xe
c. check_port_pair_config gets masks for (1,3) as input and populates port_pair_config_mask.
d. As per the code, port 2 which is valid port and part of user port mask will have lastport (which is port 3)? May be I did understand the logic correct. Can you help me?

Here user needs to explicitly mention (2,2) for port 2 to be setup else it will be skipped.
If you see check_port_pair_config below we disable the ports that are not Mentioned in portmap.

"
check_port_pair_config(void)
{

<snip>

            port_pair_config_mask |= port_pair_mask;
    }

    l2fwd_enabled_port_mask &= port_pair_config_mask;

    return 0;
}

"

Quoted 7 lines

So my concerns are 1) there is no same port mapping, 2) my understanding on lastport logic is not clear and 3) as per the code there is 1:N but 1:1.

Hence there should be sufficient warning to user if port are of wrong speed and NUMA.

Unless the user disables stats using -T 0 option all the prints will be skipped.

Quoted 5 lines

Note: current speed can be fetched only if the port are started too (in Fortville).

snipped

#60 lore raw

Re: [dpdk-dev] [PATCH v4] examples/l2fwd: add cmdline option for forwarding port info

04/07/2020 15:36, Jerin Jacob:

Quoted 4 lines
Quoted 2 lines · Bruce Richardson

On Sun, May 24, 2020 at 06:13:22PM +0200, Thomas Monjalon wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Bruce, as maintainer of l2fwd example, any opinion about this change?

Assuming all previous discussion on it is resolved, I'm fine with this patch, though I suspect it will only make 20.08 now.

Ping for merge.

Applied, thanks

#61 lore raw

Re: [dpdk-dev] [EXT] Re: [PATCH v7] eal: add manual probing option

In reply to Jerin Jacob
Quoted 24 lines
Quoted 18 lines

Should we have a command line option to configure the forwarding rules in the application (2)? I think yes.
Should we implement an application logic to automatically create the best forwarding rules (1)? It would be nice, but anyway, I think we need manual config (2) as a fallback.

Quoted 2 lines

Thoughts? If you think, there is a rework needed in eal then could you enumerate the items for the rework.

Sorry I don't have time to describe dive into EAL probing and enumerate the items to rework.
The most important issues I remind are:
- white/blacklist policy is a mess and should be done in a higher layer
- devargs syntax should allow generic matching (thanks to
class awareness and generic syntax)

Quoted 3 lines

Starting from these 2 items, we could imagine a generic path to disable automatic probing, but I think the l2fwd logic should not rely on probing order anyway.

+ Bruce as l2fwd maintainer.

Quoted 8 lines

Since in any case, l2fwd needs to be updated, We will focus on l2fwd change for v20.05 and leaving the fate of this patch to EAL maintainers. Let us know, Are we are OK with below change in l2fwd as
- Introduce an array-based port lookup table instead of hardcoding to xor based lookup.
- if no argument specified fill dest port as xor of source
- If argument is specified override the lookup table with a user-specified destination port.

I think, this thread can be closed here.

L2fwd change sent as different patch.

Please review if it is for interest.

http://patches.dpdk.org/patch/67722/

Quoted 5 lines
Quoted 5 lines
#62 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

In reply to Gaetan Rivet
Quoted 63 lines · On Thu, 23 Jan 2020 10:58:13 +0100 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

v6: Rebased on master

see http://mails.dpdk.org/archives/dev/2020-January/154178.html
for reference to this version, linking v7 to v5 thread.

v7: Updated author and SoB.

Quoted 11 lines
 doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
 doc/guides/rel_notes/release_20_02.rst     |  9 ++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 +
 lib/librte_eal/common/eal_private.h        |  9 ++++
 lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 ++
 10 files changed, 142 insertions(+)

This patch seems to have been held in limbo for 3 years.

For me, it is ok, but concerned that it opens up a whole scenario of possible usages that may not be tested, and probably don't work. Testing all the possible combinations of probe ordering is a geometric problem.

So if user submits bug then the response would have to be:
Manual probing is an experimental option which may not work.

#63 lore raw

Re: [dpdk-dev] [PATCH v7] eal: add manual probing option

Quoted 78 lines · Stephen Hemminger
Quoted 63 lines · On Thu, 23 Jan 2020 10:58:13 +0100 Gaetan Rivet <[email protected]> wrote:

Add a new EAL option enabling manual probing in the EAL. This command line option will configure the EAL so that buses will not trigger their probe step on their own.

Applications are then expected to hotplug devices as they see fit.

Devices declared on the command line by the user (using -w and --vdev), will be probed using the hotplug API, in the order they are declared.

This has the effect of offering a way for users to control probe order of their devices, for drivers requiring it.

---

haven't heard many opinions on the matter, please shout if you see an issue
with this approach.

@Slava: I have tested rather quickly that it does not break anything,
and that it works as intended for basic cases.
Can you test it further for your use-case and tell me if it works fine?

Beyond the obvious difference between both probe mode, something to keep in mind:
while using -w on invalid devices would not block (PCI) bus probing, it will stop manual probing in its track. All devices need to exist and be valid device IDs.

v2: fixed a few typos, map file (and used Travis to validate).

Slava, are you able to test this patch?

v3: properly fixed the map file (inherited 19.08 instead of 19.05).

Added a function to set the probe manual from the application, without having the user do it from the command line.

Stopped spamming Slava about it, Vamsi was actually the one interested in it!

Standing issue worth chiming in:

Currently manual-probe will cut off probing from all buses. It could be interesting to be able to only cut buses supporting hotplug, given that they are the one able to probe devices afterward.

No real use-case for this right now, so leaving as-is. Might be worth considering in the future.

v4: Rebased on master,
Moved implementation in common EAL,
Used public API within the EAL to set the option, Made the API experimental

v5: added note in the Getting Started Guide.

v6: Rebased on master

see http://mails.dpdk.org/archives/dev/2020-January/154178.html
for reference to this version, linking v7 to v5 thread.

v7: Updated author and SoB.

Quoted 11 lines
 doc/guides/linux_gsg/eal_args.include.rst  | 13 ++++++
 doc/guides/rel_notes/release_20_02.rst     |  9 ++++
 lib/librte_eal/common/eal_common_bus.c     |  6 +++
 lib/librte_eal/common/eal_common_dev.c     | 54 ++++++++++++++++++++++
 lib/librte_eal/common/eal_common_options.c |  8 ++++
 lib/librte_eal/common/eal_internal_cfg.h   |  1 +
 lib/librte_eal/common/eal_options.h        |  2 +
 lib/librte_eal/common/eal_private.h        |  9 ++++
 lib/librte_eal/common/include/rte_eal.h    | 36 +++++++++++++++
 lib/librte_eal/rte_eal_version.map         |  4 ++
 10 files changed, 142 insertions(+)
Quoted 8 lines

This patch seems to have been held in limbo for 3 years.

For me, it is ok, but concerned that it opens up a whole scenario of possible usages that may not be tested, and probably don't work. Testing all the possible combinations of probe ordering is a geometric problem.

So if user submits bug then the response would have to be:
Manual probing is an experimental option which may not work.

Hello Stephen,

I am not pushing for this series anymore.
I wrote it to help other people, I guess they used another way since. If someone needs it, I can take a moment to reanimate it.

I'm still using the PCI bus hack to force manual probing, as well as port hotplug to control strict ordering. I guess at this point this is the stable way of working with DPDK, instead of a proper documented option.

Best regards,

Signature · 1 line
Gaetan Rivet