powerpc/eeh: Release EEH device state synchronously

Export rtas_error_rc() so that it can be used by other users of rtas_call() (which is already exported).

---
v3 * New in this version.

 arch/powerpc/include/asm/rtas.h | 1 +
 arch/powerpc/kernel/rtas.c      | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

index 3c1887351c71..7c9e4d3635cf 100644
@@ -379,6 +379,7 @@ extern int rtas_set_rtc_time(struct rtc_time *rtc_time);
 
 extern unsigned int rtas_busy_delay_time(int status);
 extern unsigned int rtas_busy_delay(int status);
+extern int rtas_error_rc(int rtas_rc);
 
 extern int early_init_dt_scan_rtas(unsigned long node,
 		const char *uname, int depth, void *data);

index c5fa251b8950..238bf112d29a 100644
@@ -518,7 +518,7 @@ unsigned int rtas_busy_delay(int status)
 }
 EXPORT_SYMBOL(rtas_busy_delay);
 
-static int rtas_error_rc(int rtas_rc)
+int rtas_error_rc(int rtas_rc)
 {
 	int rc;
 

@@ -546,6 +546,7 @@ static int rtas_error_rc(int rtas_rc)
 	}
 	return rc;
 }
+EXPORT_SYMBOL(rtas_error_rc);
 
 int rtas_get_power_level(int powerdomain, int *level)
 {

2.22.0.216.g00a2a96fc9

Hi everyone,

Here are some fixes and cleanups that have come from other work but that I think stand on their own.

Only one patch ("Release EEH device state synchronously", suggested by Oliver O'Halloran) is a significant change: it moves the cleanup of some EEH device data out of the (possibly asynchronous) device release handler and into the (synchronously called) bus notifier. This is useful for future work as it makes it easier to reason about the lifetimes of EEH structures.

Note that I've left a few WARN_ON_ONCEs in the code because I'm paranoid, but I have not been able to hit them during testing.

Cheers,
Sam.

Notes for v3:
I've tweaked the fix for pseries_eeh_configure_bridge() to return the correct error code (even though it's not used) by calling an already present RTAS function, rtas_error_rc(). However, I had to make another change to export that function and while it does seem like the right thing to do, but I'm concerned it's a bit out of scope for such a small fix.

Notes for v2:

I've dropped both cleanup patches (3/4, 4/4) because that type of cleanup (replacing a call to eeh_rmv_from_parent_pe() with one to eeh_remove_device()) is incorrect: if called during recovery, it will cause edev->pe to remain set when it would have been cleared previously. This would lead to stale information in the edev. I think there should be a way to simplify the code around EEH_PE_KEEP but I'll look at that separately.

Patch set changelog follows:

Patch set v3:
Patch 1/3 (new in this version): powerpc/rtas: Export rtas_error_rc Patch 2/3 (was 1/2): powerpc/eeh: fix pseries_eeh_configure_bridge() Patch 3/3 (was 2/2): powerpc/eeh: Release EEH device state synchronously

Patch set v2:
Patch 1/2: powerpc/eeh: fix pseries_eeh_configure_bridge()
Patch 2/2: powerpc/eeh: Release EEH device state synchronously
- Added comment explaining why the add case can't be handled similarly to the remove case. Dropped (was 3/4) powerpc/eeh: Remove workaround from eeh_add_device_late() Dropped (was 4/4) powerpc/eeh: Clean up edev cleanup for VFs

Patch set v1:
Patch 1/4: powerpc/eeh: fix pseries_eeh_configure_bridge() Patch 2/4: powerpc/eeh: Release EEH device state synchronously Patch 3/4: powerpc/eeh: Remove workaround from eeh_add_device_late() Patch 4/4: powerpc/eeh: Clean up edev cleanup for VFs

Sam Bobroff (3):
powerpc/rtas: Export rtas_error_rc powerpc/eeh: fix pseries_eeh_configure_bridge() powerpc/eeh: Release EEH device state synchronously

 arch/powerpc/include/asm/rtas.h              |  1 +
 arch/powerpc/kernel/eeh.c                    | 31 ++++++++++++++++++++
 arch/powerpc/kernel/pci-hotplug.c            |  2 --
 arch/powerpc/kernel/rtas.c                   |  3 +-
 arch/powerpc/platforms/pseries/eeh_pseries.c |  4 ++-
 5 files changed, 37 insertions(+), 4 deletions(-)

2.22.0.216.g00a2a96fc9

If a device is hot unplgged during EEH recovery, it's possible for the RTAS call to ibm,configure-pe in pseries_eeh_configure() to return parameter error (-3), however negative return values are not checked for and this leads to an infinite loop.

Fix this by correctly bailing out on negative values.

---
 arch/powerpc/platforms/pseries/eeh_pseries.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

index 893ba3f562c4..9ea1c06a78cd 100644
@@ -607,6 +607,8 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
 
 		if (!ret)
 			return ret;
+		if (ret < 0)
+			break;
 
 		/*
 		 * If RTAS returns a delay value that's above 100ms, cut it

@@ -627,7 +629,7 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
 
 	pr_warn("%s: Unable to configure bridge PHB#%x-PE#%x (%d)\n",
 		__func__, pe->phb->global_number, pe->addr, ret);
-	return ret;
+	return rtas_error_rc(ret);
 }
 
 /**

2.22.0.216.g00a2a96fc9

EEH device state is currently removed (by eeh_remove_device()) during the device release handler, which is invoked as the device's reference count drops to zero. This may take some time, or forever, as other threads may hold references.

However, the PCI device state is released synchronously by pci_stop_and_remove_bus_device(). This mismatch causes problems, for example the device may be re-discovered as a new device before the release handler has been called, leaving the PCI and EEH state mismatched.

So instead, call eeh_remove_device() from the bus device removal handlers, which are called synchronously in the removal path.

---
 arch/powerpc/kernel/eeh.c         | 31 +++++++++++++++++++++++++++++++
 arch/powerpc/kernel/pci-hotplug.c |  2 --
 2 files changed, 31 insertions(+), 2 deletions(-)

index 17cb3e9b5697..64361311bc8e 100644
@@ -1106,6 +1106,37 @@ static int eeh_init(void)
 
 core_initcall_sync(eeh_init);
 
+static int eeh_device_notifier(struct notifier_block *nb,
+			       unsigned long action, void *data)
+{
+	struct device *dev = data;
+
+	switch (action) {
+	/*
+	 * Note: It's not possible to perform EEH device addition (i.e.
+	 * {pseries,pnv}_pcibios_bus_add_device()) here because it depends on
+	 * the device's resources, which have not yet been set up.
+	 */
+	case BUS_NOTIFY_DEL_DEVICE:
+		eeh_remove_device(to_pci_dev(dev));
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block eeh_device_nb = {
+	.notifier_call = eeh_device_notifier,
+};
+
+static __init int eeh_set_bus_notifier(void)
+{
+	bus_register_notifier(&pci_bus_type, &eeh_device_nb);
+	return 0;
+}
+arch_initcall(eeh_set_bus_notifier);
+
 /**
  * eeh_add_device_early - Enable EEH for the indicated device node
  * @pdn: PCI device node for which to set up EEH

index d6a67f814983..28e9aa274f64 100644
@@ -57,8 +57,6 @@ void pcibios_release_device(struct pci_dev *dev)
 	struct pci_controller *phb = pci_bus_to_host(dev->bus);
 	struct pci_dn *pdn = pci_get_pdn(dev);
 
-	eeh_remove_device(dev);
-
 	if (phb->controller_ops.release_device)
 		phb->controller_ops.release_device(dev);
 

2.22.0.216.g00a2a96fc9

This will do the right thing for your ibm,configure-pe use case in patch 2, but the -900x => errno translations in rtas_error_rc() appear tailored for the indicator- and sensor-related calls that currently use it. From my reading of PAPR+, the meaning of a -900x RTAS status word depends on the call. For example, -9002 commonly means "not authorized", which we would typically translate to -EPERM, but rtas_error_rc() would translate it to -ENODEV.

Also the semantics of -9001 as a return value seem to vary a bit.

So I don't think rtas_error_rc() should be advertised as a generically useful facility in its current form.

(I have had some thoughts about how firmware/hypervisor call status can be translated to meaningful Linux error values without tedious switch statements, which I'm happy to expand on if anyone is interested, but I don't want to hijack your submission for that discussion.)

Ah, interesting.

I'll do another version as you suggest.

Cheers,
Sam.