introduce fail-safe PMD

Other configuration items (i.e. MAC addresses) are stored within rte_eth_dev_data, but not this one.

Signed-off-by Gaetan Rivet <[email protected]>

This device state means that the device is managed externally, by whichever party has set this state (PMD or application).

Note: this new device state is only an information. The related device structure and operators are still valid and can be used normally.

It is however made private by device management helpers within ethdev, making the device invisible to applications.

---
 lib/librte_ether/rte_ethdev.c | 3 ++-
 lib/librte_ether/rte_ethdev.h | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

drivers/net/failsafe/failsafe_eal.c new file +46 -0

+
+static struct rte_eth_dev *
+pci_addr_to_eth_dev(struct rte_pci_addr *addr)
+{
+	uint8_t pid;
+
+	if (addr == NULL)
+		return NULL;
+	for (pid = 0; pid < RTE_MAX_ETHPORTS; pid++) {
+		struct rte_pci_addr *addr2;
+		struct rte_eth_dev *edev;
+
+		edev = &rte_eth_devices[pid];
+		if (edev->device == NULL ||
+		    edev->device->devargs == NULL)
+			continue;
+		addr2 = &edev->device->devargs->pci.addr;
+		if (rte_eal_compare_pci_addr(addr, addr2) == 0)
+			return edev;
+	}
+	return NULL;
+}
+
+static int
+pci_scan_one(struct sub_device *sdev)
+{
+	struct rte_devargs *da;
+	char dirname[PATH_MAX];
+
+	da = &sdev->devargs;
+	snprintf(dirname, sizeof(dirname),
+		"%s/" PCI_PRI_FMT,
+		pci_get_sysfs_path(),
+		da->pci.addr.domain,
+		da->pci.addr.bus,
+		da->pci.addr.devid,
+		da->pci.addr.function);
+	errno = 0;
+	if (rte_eal_pci_parse_sysfs_entry(&sdev->pci_device,
+		dirname, &da->pci.addr) < 0) {
+		if (errno == ENOENT) {
+			DEBUG("Could not scan requested device " PCI_PRI_FMT,
+				da->pci.addr.domain,
+				da->pci.addr.bus,
+				da->pci.addr.devid,
+				da->pci.addr.function);

Quoted 46 lines · On Fri, 3 Mar 2017 16:40:28 +0100 Gaetan Rivet <[email protected]> wrote:

drivers/net/failsafe/failsafe_eal.c new file +46 -0

+
+static struct rte_eth_dev *
+pci_addr_to_eth_dev(struct rte_pci_addr *addr)
+{
+	uint8_t pid;
+
+	if (addr == NULL)
+		return NULL;
+	for (pid = 0; pid < RTE_MAX_ETHPORTS; pid++) {
+		struct rte_pci_addr *addr2;
+		struct rte_eth_dev *edev;
+
+		edev = &rte_eth_devices[pid];
+		if (edev->device == NULL ||
+		    edev->device->devargs == NULL)
+			continue;
+		addr2 = &edev->device->devargs->pci.addr;
+		if (rte_eal_compare_pci_addr(addr, addr2) == 0)
+			return edev;
+	}
+	return NULL;
+}
+
+static int
+pci_scan_one(struct sub_device *sdev)
+{
+	struct rte_devargs *da;
+	char dirname[PATH_MAX];
+
+	da = &sdev->devargs;
+	snprintf(dirname, sizeof(dirname),
+		"%s/" PCI_PRI_FMT,
+		pci_get_sysfs_path(),
+		da->pci.addr.domain,
+		da->pci.addr.bus,
+		da->pci.addr.devid,
+		da->pci.addr.function);
+	errno = 0;
+	if (rte_eal_pci_parse_sysfs_entry(&sdev->pci_device,
+		dirname, &da->pci.addr) < 0) {
+		if (errno == ENOENT) {
+			DEBUG("Could not scan requested device " PCI_PRI_FMT,
+				da->pci.addr.domain,
+				da->pci.addr.bus,
+				da->pci.addr.devid,
+				da->pci.addr.function);

drivers/net/failsafe/failsafe_eal.c line 92

Quoted 10 lines

drivers/net/failsafe/failsafe_eal.c new file +10 -0

+		} else {
+			ERROR("Error while scanning sysfs entry %s",
+					dirname);
+			return -1;
+		}
+	} else {
+		sdev->state = DEV_SCANNED;
+	}
+	return 0;
+}

This needs to be generic and in EAL.
A bigger problem is that it PCI specific and therefore won't work in environments where devices are attached to different busses (SOC and Hyper-V).

Please rework to play well with bus model.

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Each fail-safe instance is configured to run atop one or several devices, with one defined as the preferred device. Hot-plug events are handled on all of them, and Tx is always directed to the preferred device if present or to the next available failover device (Rx is always performed on all devices for simplicity).

Moreover, the configured slaves (preferred or failover) do not need to be present at initialization time and may appear later.

Slaves configuration is continuously synchronized with that of the virtual device, which exposes their common set of capabilities to the application. Failure to apply the current configuration state to a slave for any reason simply reschedules its initialization.

This series depends on the series
[PATCH 0/4] clarify eth_dev state management
[PATCH 0/5] add device removal event

Quoted 26 lines · Gaetan Rivet

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Each fail-safe instance is configured to run atop one or several devices, with one defined as the preferred device. Hot-plug events are handled on all of them, and Tx is always directed to the preferred device if present or to the next available failover device (Rx is always performed on all devices for simplicity).

Moreover, the configured slaves (preferred or failover) do not need to be present at initialization time and may appear later.

Slaves configuration is continuously synchronized with that of the virtual device, which exposes their common set of capabilities to the application. Failure to apply the current configuration state to a slave for any reason simply reschedules its initialization.

Quoted 4 lines

This series depends on the series
[PATCH 0/4] clarify eth_dev state management
[PATCH 0/5] add device removal event

Hi,

this looks an interesting PMD, and I like the wrapper approach. However, why duplicate the functionality of the bonding device in another device driver? Is it not possible to have this driver wrap individual devices and then have the bonding driver use those wrapped devices to present an omni-present device? It seems strange to have support for grouping devices together in two different drivers - one should leverage the other.

Alternatively, should this be merged into the bonding driver or replace it?

Regards,
/Bruce

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Each fail-safe instance is configured to run atop one or several devices, with one defined as the preferred device. Hot-plug events are handled on all of them, and Tx is always directed to the preferred device if present or to the next available failover device (Rx is always performed on all devices for simplicity).

Moreover, the configured slaves (preferred or failover) do not need to be present at initialization time and may appear later.

Slaves configuration is continuously synchronized with that of the virtual device, which exposes their common set of capabilities to the application. Failure to apply the current configuration state to a slave for any reason simply reschedules its initialization.

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Quoted 5 lines · Gaetan Rivet

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

Quoted 7 lines

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Why not just add this feature to the bonding pmd then? A bond is perfectly capable of handling the trivial case of a single underlying device, and adding an option to make the underly slave 'persistent' seem both much simpler in terms of implementation and code size, than adding an entire new pmd, along with its supporting code.

Neil

Quoted 13 lines · Neil Horman

Quoted 12 lines · Gaetan Rivet

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Quoted 8 lines

Why not just add this feature to the bonding pmd then? A bond is perfectly capable of handling the trivial case of a single underlying device, and adding an option to make the underly slave 'persistent' seem both much simpler in terms of implementation and code size, than adding an entire new pmd, along with its supporting code.

Neil

+1
I don't like the idea of having multiple PMDs in DPDK to handle combining multiple other devices into one.

/Bruce

Quoted 20 lines · Bruce Richardson

Quoted 19 lines · Neil Horman

Quoted 5 lines · Gaetan Rivet

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

Quoted 7 lines

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Why not just add this feature to the bonding pmd then? A bond is perfectly capable of handling the trivial case of a single underlying device, and adding an option to make the underly slave 'persistent' seem both much simpler in terms of implementation and code size, than adding an entire new pmd, along with its supporting code.

Quoted 2 lines

Quoted 2 lines

Neil

@Neil
I don't know if you saw my answer to Bruce on the matter [1], it partially adresses your point.

Quoted 4 lines

+1
I don't like the idea of having multiple PMDs in DPDK to handle combining multiple other devices into one.

Quoted 1 line

/Bruce

I understand the concern. Let's first put aside for the moment the link grouping, which is only part of the fail-safe PMD function.

The fail-safe PMD at its core, provides an alternative paradigm, a new proposal for a hot-plug functionality in a lightweight form-factor from a user standpoint.

The central question that I would like to tackle is this: why should we require from our users declaring a bonding device to have hot-plug support?

I took some time to illustrate a few modes of operation:

Fig. 1

   .-------------.
   | application |
   `------.------'
          |
     .----'-----.---------. <------ init, conf, Rx/Tx
     |          |         |
     |      .---|--.------|--. <--- conf, link check, Rx/Tx
     |      |   |  |      |  |
     v      |   v  v      v  v
.---------. | .-------. .------.
| bonding | | | ixgbe | | mlx4 |
`----.----' | `-------' `------'
     |      |
     `------'

Typical link fail-over.

Fig. 2

.-------------.
| application |
`------.------'

        | <-------- init, conf, Rx/Tx
        v
  .-----------.
  | fail-safe |
  `-----.-----'
        |
    .---'----. <--- init, conf, dev check, Rx/Tx
    |        |
    v        v

.-------. .------.
| ixgbe | | mlx4 |
-------' ------'

Typical automatic hot-plug handling with device fail-over.

Fig. 3

   .-------------.
   | application |
   `------.------'
          |
     .----'-----.-------------. <---------- init, conf, Rx/Tx
     |          |             |
     |      .---|--.----------|--. <------- conf, link check, Rx/Tx
     |      |   |  |          |  |
     v      |   v  v          v  v
.---------. | .-----------. .-----------.
| bonding | | | fail-safe | | fail-safe |
`----.----' | `-----.-----' `-----.-----'
     |      |       |             | <------ init, conf, dev check, Rx/Tx
     `------'       v             v
                .-------.      .------.
                | ixgbe |      | mlx4 |
                `-------'      `------'

Combination to provide link fail-over with automatic hot-plug handling.

Fig. 4

   .-------------.
   | application |
   `------.------'
          |
     .----'-----.-------------. <---------- init, conf, Rx/Tx
     |          |             |
     |      .---|--.----------|--. <------- conf, link check, Rx/Tx
     |      |   |  |          |  |
     v      |   v  v          v  v
.---------. | .-----------. .-----------.
| bonding | | | fail-safe | | fail-safe |
`----.----' | `-----.-----' `-----.-----'
     |      |       |             | <------- init, conf, dev check, Rx/Tx
     `------'       |             |
             .------'---.     .---'------.
             |          |     |          |
             v          v     v          v
      .--------. .--------. .--------. .--------.
      | mlx4 1 | | mlx4 2 | | mlx4 1 | | mlx4 2 |
      | port 1 | | port 1 | | port 2 | | port 2 |
      `--------' `--------' `--------' `--------'

Complex use case with link fail-over at port level and automatic hot-plug handling with device fail-over.

1. LSC vs. RMV

A link status change is a valid state for a device. It calls for specific responses, e.g. a link switch in a bonding device, without losing the general configuration of the port.

The removal of a device calls for more than pausing operations and switching an active device. The party responsible for initializing the device should take care of closing it properly. If this party also wants to be able to restore the device if it was plugged back in, it would need be able to initialize it back and reconfigure its previous state.

As we can see that in [Fig. 1], this responsibility lies upon the application.

2. Bonding and link availability

The hot-plug functionality is not a core function of the bonding PMD. It is only interested in knowing if the link is active or not.

Adding the device persistence to the bonding PMD would mean adding the ability to flexibly parse device definitions to cope with plug-ins in evolving busses (PCI hot-plug could mean changing bus addresses), being able to emulate the EAL and the ether layer and to properly store the device configuration. This means formally describing the life of a device in a DPDK application from start to finish.

All of this hot-plug support, in order for the bonding to be aware of the status of some of its links. This seems like scope-creep.

3. Fail-safe and hot-plug support

An attach / detach (pseudo-hotplug) API exists in DPDK. The main problem
of this API is that it does not offer reacting to a device plug-out, only triggering a device detaching from an application. This is a completely different approach from an application standpoint.

The fail-safe PMD offers an out-of-the-box implementation of a newly proposed hot-plug API [2]. It allows a seamless integration to users for device removal support in their applications, without requiring evolutions [Fig. 2]. This flexibility allows it to be used as part of a bond [Fig. 3], [Fig. 4], while current bonding does not allow for detaching devices, even if it were to be considered hot-plug. There is no reason however to require declaring a bonding device to be able to use it, from a user perspective this seems backward.

Both bonding and fail-safe PMDs deal with enslaving devices and acting upon their state changing. The bonding function performs at link level, while the hot-plug function deals with the device itself. I do not think this similarity justify merging both functions. Maintenance would be easier with clear, simpler functions implemented in separate PMDs.

[1]: http://dpdk.org/ml/archives/dev/2017-March/059446.html
[2]: http://dpdk.org/ml/archives/dev/2017-March/059217.html

Signature · 2 lines

Gaëtan Rivet
6WIND

Quoted 4 lines

The central question that I would like to tackle is this: why should we require from our users declaring a bonding device to have hot-plug support?

We'll, strictly speaking, I suppose we don't have to require it. But by that same token, we don't need to do it in a separate PMD either, there are lots of other options.

Quoted 42 lines

I took some time to illustrate a few modes of operation:

Fig. 1

   .-------------.
   | application |
   `------.------'
          |
     .----'-----.---------. <------ init, conf, Rx/Tx
     |          |         |
     |      .---|--.------|--. <--- conf, link check, Rx/Tx
     |      |   |  |      |  |
     v      |   v  v      v  v
.---------. | .-------. .------.
| bonding | | | ixgbe | | mlx4 |
`----.----' | `-------' `------'
     |      |
     `------'

Typical link fail-over.

Fig. 2

.-------------.
| application |
`------.------'

        | <-------- init, conf, Rx/Tx
        v
  .-----------.
  | fail-safe |
  `-----.-----'
        |
    .---'----. <--- init, conf, dev check, Rx/Tx
    |        |
    v        v

.-------. .------.
| ixgbe | | mlx4 |
-------' ------'

Typical automatic hot-plug handling with device fail-over.

Quoted 1 line

[...]

Yes, I think we all understand the purpose of your PMD - its there to provide a placeholder device that can respond to application requests in a sane manner, until such time as real hardware is put in its place via a hot plug/failure. Thats all well and good, I'm saying there are better ways to go about this that can provide the same functionality without having to add an extra 4k lines of code to the project, many of which already exist.

Quoted 13 lines

1. LSC vs. RMV

A link status change is a valid state for a device. It calls for specific responses, e.g. a link switch in a bonding device, without losing the general configuration of the port.

The removal of a device calls for more than pausing operations and switching an active device. The party responsible for initializing the device should take care of closing it properly. If this party also wants to be able to restore the device if it was plugged back in, it would need be able to initialize it back and reconfigure its previous state.

Quoted 3 lines

As we can see that in [Fig. 1], this responsibility lies upon the application.

Again, yes, I think we all see the benefit of centralizing hot plug operations, no one is disagreing with that, its the code/functional duplication that is concerning.

Quoted 5 lines

2. Bonding and link availability

The hot-plug functionality is not a core function of the bonding PMD. It is only interested in knowing if the link is active or not.

Currently, yes. The suggestion was that you augment the bonding driver so that hot plug is a core function of bonding.

Quoted 7 lines

Adding the device persistence to the bonding PMD would mean adding the ability to flexibly parse device definitions to cope with plug-ins in evolving busses (PCI hot-plug could mean changing bus addresses), being able to emulate the EAL and the ether layer and to properly store the device configuration. This means formally describing the life of a device in a DPDK application from start to finish.

Which seems to me to be exactly what your PMD does. I don't see why its fundamentally harder to do that in an existing pmd, than it is in a new one.

My suggestion to you would be to re-architect this in the following way:

1) Augment the null pmd to accept arbitrary parameters. Paremeters which are not explicitly recognized are stored. Create an api call via the ethdev api to retrieve said arguments.

2) Augment the bonding driver so that, upon configuration, null drivers can be added to the bond, and marked as replaceable in some fashion

3) Port your hotplug implementation into the bonding code (or make it a separate library, not sure which yet). On a hot plug event trigger, react by querying the replaceable interfaces to see if the hot plugged nic is meant to replace an existing one. If a match is found, detach the null pmd instance, and add the hot plugged interface.

The intent here is to provide you with the functionality that you have (which is a good feature), without having to support an entire new pmd that replicates a great deal of existing code an functionality.

Quoted 8 lines

Quoted 4 lines

The central question that I would like to tackle is this: why should we require from our users declaring a bonding device to have hot-plug support?

We'll, strictly speaking, I suppose we don't have to require it. But by that same token, we don't need to do it in a separate PMD either, there are lots of other options.

I think I used an ambiguous formulation here. To be sure that we understand each other, what I want to express is that it will certainly seem very strange for a user to declare a bond on a single device, first, and to be expected to do so if they wanted hot-plug support on that particular device.

The bonding here would only be used as a place-holder, which does not really make sense from the user point of view.

I think you understood that, and what I take from your response is that while agreeing you would like to do so differently. I just wanted to be clear.

Quoted 42 lines

Quoted 42 lines

I took some time to illustrate a few modes of operation:

Fig. 1

   .-------------.
   | application |
   `------.------'
          |
     .----'-----.---------. <------ init, conf, Rx/Tx
     |          |         |
     |      .---|--.------|--. <--- conf, link check, Rx/Tx
     |      |   |  |      |  |
     v      |   v  v      v  v
.---------. | .-------. .------.
| bonding | | | ixgbe | | mlx4 |
`----.----' | `-------' `------'
     |      |
     `------'

Typical link fail-over.

Fig. 2

.-------------.
| application |
`------.------'

        | <-------- init, conf, Rx/Tx
        v
  .-----------.
  | fail-safe |
  `-----.-----'
        |
    .---'----. <--- init, conf, dev check, Rx/Tx
    |        |
    v        v

.-------. .------.
| ixgbe | | mlx4 |
-------' ------'

Typical automatic hot-plug handling with device fail-over.

Quoted 9 lines

Quoted 1 line

[...]

Yes, I think we all understand the purpose of your PMD - its there to provide a placeholder device that can respond to application requests in a sane manner, until such time as real hardware is put in its place via a hot plug/failure. Thats all well and good, I'm saying there are better ways to go about this that can provide the same functionality without having to add an extra 4k lines of code to the project, many of which already exist.

Ah, yes, I didn't want to imply that the purpose of this PMD wasn't understood already by many. These figures are there to illustrate some use cases that some users could recognize, and serve as a support for the point made afterward.

The main thing that can be taken from these is the division along the link-level and device-level checking that is done. This explains most of my position. The nature of those checks imply different kind of code, most of which is thus actually not duplicated / would require pretty much the same amount of code to be implemented either as libraries or as part of the bonding PMD. This is the crux of my argument, which I expand upon below.

Quoted 13 lines

Quoted 13 lines

1. LSC vs. RMV

A link status change is a valid state for a device. It calls for specific responses, e.g. a link switch in a bonding device, without losing the general configuration of the port.

The removal of a device calls for more than pausing operations and switching an active device. The party responsible for initializing the device should take care of closing it properly. If this party also wants to be able to restore the device if it was plugged back in, it would need be able to initialize it back and reconfigure its previous state.

Quoted 7 lines

Quoted 3 lines

As we can see that in [Fig. 1], this responsibility lies upon the application.

Again, yes, I think we all see the benefit of centralizing hot plug operations, no one is disagreing with that, its the code/functional duplication that is concerning.

Certainly, I will try to explain why the code is not actually duplicated / why the functions are actually only superficially overlapping.

Quoted 15 lines

Quoted 5 lines

2. Bonding and link availability

The hot-plug functionality is not a core function of the bonding PMD. It is only interested in knowing if the link is active or not.

Currently, yes. The suggestion was that you augment the bonding driver so that hot plug is a core function of bonding.

Quoted 7 lines

Adding the device persistence to the bonding PMD would mean adding the ability to flexibly parse device definitions to cope with plug-ins in evolving busses (PCI hot-plug could mean changing bus addresses), being able to emulate the EAL and the ether layer and to properly store the device configuration. This means formally describing the life of a device in a DPDK application from start to finish.

Quoted 3 lines

Which seems to me to be exactly what your PMD does. I don't see why its fundamentally harder to do that in an existing pmd, than it is in a new one.

Indeed it does. I must emphasize the "formally describe the life of a device". The hot-plug functionality goes beyong the link-level check. The description of a device from a DPDK standpoint is complete in the fail-safe PMD. The state-machine must be able to describe the entire life of a device, from the devargs parsing to its start-up.

We cannot reuse the existing bonding PMD architecture for this. We would have to rewrite the bonding PMD from the ground up for the hot-plug function. Because it is actually a different approach to managing the slaves.

This is what I wanted to illustrate in [Fig. 1] and [Fig. 2]:

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

Quoted 71 lines

Quoted 15 lines

Quoted 5 lines

2. Bonding and link availability

The hot-plug functionality is not a core function of the bonding PMD. It is only interested in knowing if the link is active or not.

Currently, yes. The suggestion was that you augment the bonding driver so that hot plug is a core function of bonding.

Quoted 7 lines

Adding the device persistence to the bonding PMD would mean adding the ability to flexibly parse device definitions to cope with plug-ins in evolving busses (PCI hot-plug could mean changing bus addresses), being able to emulate the EAL and the ether layer and to properly store the device configuration. This means formally describing the life of a device in a DPDK application from start to finish.

Quoted 3 lines

Which seems to me to be exactly what your PMD does. I don't see why its fundamentally harder to do that in an existing pmd, than it is in a new one.

Indeed it does. I must emphasize the "formally describe the life of a device". The hot-plug functionality goes beyong the link-level check. The description of a device from a DPDK standpoint is complete in the fail-safe PMD. The state-machine must be able to describe the entire life of a device, from the devargs parsing to its start-up.

We cannot reuse the existing bonding PMD architecture for this. We would have to rewrite the bonding PMD from the ground up for the hot-plug function. Because it is actually a different approach to managing the slaves.

This is what I wanted to illustrate in [Fig. 1] and [Fig. 2]:

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

Quoted 5 lines

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

If the first solution is indeed not an option, why do you think this second one would be unacceptable for users? If the functionality remains the same, I don't see how it matters much for users which driver provides it or where the code originates.

Despite all the discussion, it still just doesn't make sense to me to have more than one DPDK driver to handle failover - be it link or device. If nothing else, it's going to be awkward to explain to users that if they want fail-over for when a link goes down they have to use driver A, but if they want fail-over when a NIC gets hotplugged they use driver B, and if they want both kinds of failover - which would surely be the expected case - they need to use both drivers. The usability is a problem here.

2017-03-15 03:28, Bruce Richardson:

Quoted 50 lines · Gaëtan Rivet

Quoted 39 lines

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

Quoted 5 lines

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

If the first solution is indeed not an option, why do you think this second one would be unacceptable for users? If the functionality remains the same, I don't see how it matters much for users which driver provides it or where the code originates.

Quoted 8 lines

Despite all the discussion, it still just doesn't make sense to me to have more than one DPDK driver to handle failover - be it link or device. If nothing else, it's going to be awkward to explain to users that if they want fail-over for when a link goes down they have to use driver A, but if they want fail-over when a NIC gets hotplugged they use driver B, and if they want both kinds of failover - which would surely be the expected case - they need to use both drivers. The usability is a problem here.

It seems everybody agrees on the need for the failsafe code. We are just discussing the right place to implement it.

Gaetan, moving this code in the bonding PMD means replacing the bonding API design by the failsafe design, right?
With the failsafe design in the bonding PMD, is it possible to keep other bonding features?

In case we do not have a consensus in the following days, I suggest to add this topic in the next techboard meeting agenda.

Quoted 51 lines · Thomas Monjalon

2017-03-15 03:28, Bruce Richardson:

Quoted 50 lines · Gaëtan Rivet

Quoted 39 lines

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

Quoted 5 lines

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

If the first solution is indeed not an option, why do you think this second one would be unacceptable for users? If the functionality remains the same, I don't see how it matters much for users which driver provides it or where the code originates.

The problem with the second solution is also that bonding is not only a PMD. It exposes its own public API that existing applications rely on, see rte_eth_bond_*() definitions in rte_eth_bond.h.

Although bonding instances can be set up through command-line options, target "users" are mainly applications explicitly written to use it. This must be preserved for no other reason that it hasn't been deprecated.

Also, trying to implement this API for the device failover function would implies a device capture down to the devargs parsing level. This means that a PMD could request taking over a device, messing with the internals of the
EAL: devargs list and busses lists of devices. This seems unacceptable.

The bonding API is thus in conflict with the concept of a device failover in the context of the current DPDK arch.

Quoted 8 lines

Quoted 8 lines

Despite all the discussion, it still just doesn't make sense to me to have more than one DPDK driver to handle failover - be it link or device. If nothing else, it's going to be awkward to explain to users that if they want fail-over for when a link goes down they have to use driver A, but if they want fail-over when a NIC gets hotplugged they use driver B, and if they want both kinds of failover - which would surely be the expected case - they need to use both drivers. The usability is a problem here.

Having both kind of failovers in the same PMD will always lead to the first solution in some form or another.

I am sure we can document all this in a way that does no cause users confusion, with the help of community feedback such as yours.

Perhaps "net_failsafe" is a misnomer? We also thought about "net_persistent" or "net_hotplug". Any other ideas?

It is also possible for me to remove the failover support from this series, only providing deferred hot-plug handling at first. I could then send the failover support as separate patches to better assert that it is a useful, secondary feature that is essentially free to implement.

Quoted 8 lines

It seems everybody agrees on the need for the failsafe code. We are just discussing the right place to implement it.

Gaetan, moving this code in the bonding PMD means replacing the bonding API design by the failsafe design, right?
With the failsafe design in the bonding PMD, is it possible to keep other bonding features?

As seen previously, the bonding API is incompatible with device failover.

Having some features enabled solely for one kind of failover, while having specific code paths for both, seems unecessarily complicated to me ; following suite with my previous points about the first solution.

Quoted 3 lines

In case we do not have a consensus in the following days, I suggest to add this topic in the next techboard meeting agenda.

Regards,

Signature · 2 lines

Gaëtan Rivet
6WIND

Quoted 57 lines · Gaëtan Rivet

Quoted 51 lines · Thomas Monjalon

2017-03-15 03:28, Bruce Richardson:

Quoted 50 lines · Gaëtan Rivet

Quoted 39 lines

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

Quoted 5 lines

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

If the first solution is indeed not an option, why do you think this second one would be unacceptable for users? If the functionality remains the same, I don't see how it matters much for users which driver provides it or where the code originates.

The problem with the second solution is also that bonding is not only a PMD. It exposes its own public API that existing applications rely on, see rte_eth_bond_*() definitions in rte_eth_bond.h.

Quoted 4 lines

Although bonding instances can be set up through command-line options, target "users" are mainly applications explicitly written to use it. This must be preserved for no other reason that it hasn't been deprecated.

I fail to see how either of your points are relevant. The fact that the bonding pmd exposes an api to the application has no bearing on its ability to implement a hot plug function.

Quoted 5 lines

Also, trying to implement this API for the device failover function would implies a device capture down to the devargs parsing level. This means that a PMD could request taking over a device, messing with the internals of the
EAL: devargs list and busses lists of devices. This seems unacceptable.

Why? You just said yourself above that, while there is a devargs interface to the bonding driver, there is also an api, which is the more used method to configure bonding. I'm not sure I agree with that, but I think its beside the point. Your PMD also requires configuration, and it appears necessecary that you do so from the command line (you need to specifically ennumerate the subdevices that you intend to provide failsafe behavior to). I see no reason why such a feature cant' be added to bonding, and the null pmd used as a standin device, should the ennumerated device not yet exist).

To your argument regarding about taking over a device, I don't see how you find that unacceptable, as it is precisely what the bonding driver does today, in the sense that it allows an application to assign a master/slave relationship to devices right now. I see no reason that we can't convey the right and ability for bonding to do that dynamically based on configuration.

Quoted 3 lines

The bonding API is thus in conflict with the concept of a device failover in the context of the current DPDK arch.

I really don't see how you get to this from your argument above.

Quoted 9 lines

Quoted 8 lines

Quoted 8 lines

Despite all the discussion, it still just doesn't make sense to me to have more than one DPDK driver to handle failover - be it link or device. If nothing else, it's going to be awkward to explain to users that if they want fail-over for when a link goes down they have to use driver A, but if they want fail-over when a NIC gets hotplugged they use driver B, and if they want both kinds of failover - which would surely be the expected case - they need to use both drivers. The usability is a problem here.

Quoted 3 lines

Having both kind of failovers in the same PMD will always lead to the first solution in some form or another.

It really isn't because you can model hotplug behavior as a trival form of the failover that bonding does now (i.e. failover between a null device and a preferred real device).

Quoted 11 lines

I am sure we can document all this in a way that does no cause users confusion, with the help of community feedback such as yours.

Perhaps "net_failsafe" is a misnomer? We also thought about "net_persistent" or "net_hotplug". Any other ideas?

It is also possible for me to remove the failover support from this series, only providing deferred hot-plug handling at first. I could then send the failover support as separate patches to better assert that it is a useful, secondary feature that is essentially free to implement.

I think thats solving the wrong problem. I've no issue with the functionality in this patch, its really the implementation that we are all arguing against.

Quoted 9 lines

Quoted 8 lines

It seems everybody agrees on the need for the failsafe code. We are just discussing the right place to implement it.

Gaetan, moving this code in the bonding PMD means replacing the bonding API design by the failsafe design, right?
With the failsafe design in the bonding PMD, is it possible to keep other bonding features?

Quoted 2 lines

As seen previously, the bonding API is incompatible with device failover.

Its not been seen previously, you asserted it to be so, and I certainly disagree with that assertion. I think others might too.

Additionally, its not really in line with this discussion, but in looking at your hotplug detection code, I think somewhat lacking. Currently you seem to implement this with a timer that wakes up and checks for device existance, which is pretty substandard in my mind. Thats going to waste cpu cycles that might lead to packet loss. I'd really prefer to see you augment the eal library with an event handling code (it can tie into udev in linux and kqueue in bsd), and create a generic event hook, that we can use to detect device adds/removes without having to wake up constantly to see if anything has changed.

Quoted 7 lines

Having some features enabled solely for one kind of failover, while having specific code paths for both, seems unecessarily complicated to me ; following suite with my previous points about the first solution.

Quoted 3 lines

In case we do not have a consensus in the following days, I suggest to add this topic in the next techboard meeting agenda.

Quoted 62 lines · Neil Horman

Quoted 61 lines · Gaëtan Rivet

Quoted 51 lines · Thomas Monjalon

2017-03-15 03:28, Bruce Richardson:

Quoted 50 lines · Gaëtan Rivet

Quoted 39 lines

- In the bonding, the init and configuration steps are still the responsibility of the application and no one else. The bonding PMD captures the device, re-applies its configuration upon dev_configure() which is actually re-applying part of the configuration already present
within the slave eth_dev (cf rte_eth_dev_config_restore).

- In the fail-safe, the init and configuration are both the responsibilities of the fail-safe PMD itself, not the application anymore. This handling of these responsibilities in lieu of the application is the whole point of the "deferred hot-plug" support, of proposing a simple implementation to the user.

This change in responsibilities is the bulk of the fail-safe code. It would have to be added as-is to the bonding. Verifying the correctness of the sync of the initialization phase (acceptable states of a device following several events registered by the fail-safe PMD) and the configuration items between the state the application believes it is in and the fail-safe knows it is in, is the bulk of the fail-safe code.

This function is not overlapping with that of the bonding. The reason I did not add this whole architecture to the bonding is that when I tried to do so, I found that I only had two possibilities:

- The current slave handling path is kept, and we only add a new one with additional functionalities: full init and conf handling with extended parsing capabilities.

- The current slave handling is scraped and replaced entirely by the new slave management. The old capturing of existing device is not done anymore.

The first solution is not acceptable, because we effectively end-up with a maintenance nightmare by having to validate two types of slaves with differing capabilities, differing initialization paths and differing configuration code. This is extremely awkward and architecturally unsound. This is essentially the same as having the exact code of the fail-safe as an aside in the bonding, maintening exactly the same breadth of code while having muddier interfaces and organization.

Quoted 5 lines

The second solution is not acceptable, because we are bending the whole existing bonding API to our whim. We could just as well simply rename the fail-safe PMD as bonding, add a few grouping capabilities and call it a day. This is not acceptable for users.

If the first solution is indeed not an option, why do you think this second one would be unacceptable for users? If the functionality remains the same, I don't see how it matters much for users which driver provides it or where the code originates.

The problem with the second solution is also that bonding is not only a PMD. It exposes its own public API that existing applications rely on, see rte_eth_bond_*() definitions in rte_eth_bond.h.

Although bonding instances can be set up through command-line options, target "users" are mainly applications explicitly written to use it. This must be preserved for no other reason that it hasn't been deprecated.

Quoted 4 lines

I fail to see how either of your points are relevant. The fact that the bonding pmd exposes an api to the application has no bearing on its ability to implement a hot plug function.

This depends on the API making sense in the context of the new functionality.

This API offers to add and remove slaves to a bonding and to configure them. In the fail-safe arch, it is not possible to add and remove slaves from the grouping. Doing so would mean adding and removing devices from internal EAL structures.

It is also invalid to try to configure a fail-safe slave. An application only configures a fail-safe device, which will in turn configure its slaves. This separation follows from the nature of a device failover.

As seen previously, the fail-safe PMD handles different responsibilities from the bonding PMD. It is thus necessary to make different assumptions concerning what it can and cannot do with a slave.

Quoted 14 lines

Quoted 5 lines

Also, trying to implement this API for the device failover function would implies a device capture down to the devargs parsing level. This means that a PMD could request taking over a device, messing with the internals of the
EAL: devargs list and busses lists of devices. This seems unacceptable.

Why? You just said yourself above that, while there is a devargs interface to the bonding driver, there is also an api, which is the more used method to configure bonding. I'm not sure I agree with that, but I think its beside the point. Your PMD also requires configuration, and it appears necessecary that you do so from the command line (you need to specifically ennumerate the subdevices that you intend to provide failsafe behavior to). I see no reason why such a feature cant' be added to bonding, and the null pmd used as a standin device, should the ennumerated device not yet exist).

Quoted 6 lines

To your argument regarding about taking over a device, I don't see how you find that unacceptable, as it is precisely what the bonding driver does today, in the sense that it allows an application to assign a master/slave relationship to devices right now. I see no reason that we can't convey the right and ability for bonding to do that dynamically based on configuration.

No, the bonding PMD does not take over a device. It only cares about the ether layer for its link failover. It does not care about parsing parameters of a slave, probing devices, detaching drivers. It does not remove a device from the pci_device_list in the EAL for example.

Doing so would imply exposing private internals structures from the EAL, messing with elements reserved while doing the EAL init. This requires controlling a priority in the device initialization order to create the over-arching ones last (which is a hacky solution). It would wreak havoc with the DPDK arch.

The fail-safe PMD does not rely on the EAL for handling its slaves. This is what I explained before, when I touched upon the differing responsibilities implied by the differences in nature between a link failover and a device failover.

Quoted 5 lines

Quoted 3 lines

The bonding API is thus in conflict with the concept of a device failover in the context of the current DPDK arch.

I really don't see how you get to this from your argument above.

The current DPDK arch does not expose EAL elements to be modified by PMDs, and with good reasons. In this context, it is not possible to handle slaves correctly for a device failover in the bonding PMD,
because the bonding PMD from the get-go expects the EAL to handle its slaves on a device level.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

Quoted 8 lines

Despite all the discussion, it still just doesn't make sense to me to have more than one DPDK driver to handle failover - be it link or device. If nothing else, it's going to be awkward to explain to users that if they want fail-over for when a link goes down they have to use driver A, but if they want fail-over when a NIC gets hotplugged they use driver B, and if they want both kinds of failover - which would surely be the expected case - they need to use both drivers. The usability is a problem here.

Quoted 7 lines

Quoted 3 lines

Having both kind of failovers in the same PMD will always lead to the first solution in some form or another.

It really isn't because you can model hotplug behavior as a trival form of the failover that bonding does now (i.e. failover between a null device and a preferred real device).

The preferred real device still has to be created / destroyed. It still relies on EAL entry points for handling. It still puts additional responsibilities on a PMD. Those responsibilities are expressed in sub layers clearly defined in the fail-safe PMD. You would have to create these sub-layers in some form in the bonding for it to be able to create a preferred real device at some point. This additional way of handling slaves has already been discussed as inducing a messy architecture to the bonding PMD.

Quoted 23 lines

Quoted 11 lines

I am sure we can document all this in a way that does no cause users confusion, with the help of community feedback such as yours.

Perhaps "net_failsafe" is a misnomer? We also thought about "net_persistent" or "net_hotplug". Any other ideas?

It is also possible for me to remove the failover support from this series, only providing deferred hot-plug handling at first. I could then send the failover support as separate patches to better assert that it is a useful, secondary feature that is essentially free to implement.

I think thats solving the wrong problem. I've no issue with the functionality in this patch, its really the implementation that we are all arguing against.

Quoted 9 lines

Quoted 8 lines

It seems everybody agrees on the need for the failsafe code. We are just discussing the right place to implement it.

Gaetan, moving this code in the bonding PMD means replacing the bonding API design by the failsafe design, right?
With the failsafe design in the bonding PMD, is it possible to keep other bonding features?

Quoted 5 lines

Quoted 2 lines

As seen previously, the bonding API is incompatible with device failover.

Its not been seen previously, you asserted it to be so, and I certainly disagree with that assertion. I think others might too.

I also explained at length my assertion. I can certainly expand further if necessary, but you need to point the elements you disagree with.

Quoted 10 lines

Additionally, its not really in line with this discussion, but in looking at your hotplug detection code, I think somewhat lacking. Currently you seem to implement this with a timer that wakes up and checks for device existance, which is pretty substandard in my mind. Thats going to waste cpu cycles that might lead to packet loss. I'd really prefer to see you augment the eal library with an event handling code (it can tie into udev in linux and kqueue in bsd), and create a generic event hook, that we can use to detect device adds/removes without having to wake up constantly to see if anything has changed.

I think it's fine. We can discuss it further once we agree on the form the hot-plug implementation will take in the DPDK.

Quoted 8 lines

Quoted 8 lines

Having some features enabled solely for one kind of failover, while having
specific code paths for both, seems unecessarily complicated to me ; following suite with my previous points about the first solution.

Quoted 3 lines

In case we do not have a consensus in the following days, I suggest to add this topic in the next techboard meeting agenda.

Best regards,

Signature · 2 lines

Gaëtan Rivet
6WIND

There have been some discussions on this new PMD and it will be discussed today in the techboard meeting.

I would like to expose my view and summarize the solutions I have heard. First it is important to remind that everyone agrees on the need for this feature, i.e. masking the hotplug events by maintaining an ethdev object even without real underlying device.

1/
The proposal from Gaetan is to add a failsafe driver with 2 features:
* masking underlying device
* limited and small failover code to switch from a device to another one, with the same centralized configuration
The latter feature makes think to the bonding driver, but it could be kept limited without any intent of implementing real bonding features.

2/
If we really want to merge failsafe and bonding features, we could create a new bonding driver with centralized configuration. The legacy bonding driver let each slave to be configured separately. It is a different model and we should not mix them.
If one is better, it could be deprecated later.

3/
It can be tried to implement the failsafe feature into the bonding driver, as Neil suggests.
However, I am not sure it would work very well or would be easy to use.

4/
We can implement only the failsafe feature as a PMD and use it to wrap the slaves of the bonding driver.
So the order of link would be

	bonding -> failsafe -> real device

In this model, failsafe can have only one slave and do not implement the fail-over feature.

Quoted 28 lines · Thomas Monjalon

There have been some discussions on this new PMD and it will be discussed today in the techboard meeting.

I would like to expose my view and summarize the solutions I have heard. First it is important to remind that everyone agrees on the need for this feature, i.e. masking the hotplug events by maintaining an ethdev object even without real underlying device.

1/
The proposal from Gaetan is to add a failsafe driver with 2 features:
* masking underlying device
* limited and small failover code to switch from a device to another one, with the same centralized configuration
The latter feature makes think to the bonding driver, but it could be kept limited without any intent of implementing real bonding features.

2/
If we really want to merge failsafe and bonding features, we could create a new bonding driver with centralized configuration. The legacy bonding driver let each slave to be configured separately. It is a different model and we should not mix them.
If one is better, it could be deprecated later.

3/
It can be tried to implement the failsafe feature into the bonding driver, as Neil suggests.
However, I am not sure it would work very well or would be easy to use.

Quoted 8 lines

4/
We can implement only the failsafe feature as a PMD and use it to wrap the slaves of the bonding driver.
So the order of link would be

	bonding -> failsafe -> real device

In this model, failsafe can have only one slave and do not implement the fail-over feature.

Tech board decided [1] to "reconsider" the PMD for this release (17.08). So, lets start it J

I think it is good idea to continue on top of above summary, is there a plan to how to proceed?

Thanks,
ferruh

[1]
http://dpdk.org/ml/archives/dev/2017-March/061009.html

This PMD intercepts and manages Ethernet device removal events issued by slave PMDs and re-initializes them transparently when brought back so that existing applications do not need to be modified to benefit from true hot-plugging support.

The stacked PMD approach shares many similarities with the bonding PMD but with a different purpose. While bonding provides the ability to group several links into a single logical device for enhanced throughput and supports fail-over at link level, this one manages the sudden disappearance of the underlying device; it guarantees applications face a valid device in working order at all times.

Each fail-safe instance is configured to run atop one or several devices, with one defined as the preferred device. Hot-plug events are handled on all of them, and Tx is always directed to the preferred device if present or to the next available failover device (Rx is always performed on all devices for simplicity).

Moreover, the configured slaves (preferred or failover) do not need to be present at initialization time and may appear later.

Slaves configuration is continuously synchronized with that of the virtual device, which exposes their common set of capabilities to the application. Failure to apply the current configuration state to a slave for any reason simply reschedules its initialization.

This series depends on the series
[PATCH 0/4] clarify eth_dev state management
[PATCH 0/5] add device removal event

v1 --> v2:

- Wrote documentation
- Fixed commit logs, signed-off-by
- Added LSC event support
- A few minor fixes

Gaetan Rivet (13):
ethdev: save VLAN filter setting
ethdev: add flow API rule copy function
ethdev: add deferred intermediate device state
pci: expose device detach routine
pci: expose parse and probe routines
net/failsafe: add fail-safe PMD
net/failsafe: add plug-in support
net/failsafe: add flexible device definition
net/failsafe: support flow API
net/failsafe: support offload capabilities
net/failsafe: add fast burst functions
net/failsafe: support device removal
net/failsafe: support link status change event