Skip to contentSkip to conversation

Asynchronous EEH recovery

RFC PATCH New Ganesh Goudar 2 participants 5 messages Jul 24, 2023 – Jul 26, 2023

Subsystems (8)
PowerPC PCI MM core MGLRU +4 more subsystems
lore
Share & tools
lore
#0 lore raw

v2 cover Asynchronous EEH recovery

Hi,

EEH recovery is currently serialized and these patches shorten the time taken for EEH recovery by making the recovery to run in parallel. The original author of these patches is Sam Bobroff, I have rebased and tested these patches.

On powervm with 64 VFs from same PHB, I see approximately 48% reduction in time taken in EEH recovery.

On powernv with 9 network cards, Where 2 cards installed on one PHB and 1 card on each of the rest of the PHBs, Providing 20 PFs in total. I see approximately 33% reduction in time taken in EEH recovery.

These patches were originally posted as separate RFCs by Sam, And I rebased and posted these patches almost a year back, I stopped pursuing these patches as I was not able test this on powernv, Due to the issues in drivers of cards I was testing this on, Which are now resolved. Since I am re-posting this after long time, Posting this as a fresh RFC, Please comment.

Thanks.

V2:
* Since we now have event list per phb, Have per phb event list lock.
* Appropriate names given to the locks.
* Remove stale comments (few more to be removed).
* Initialize event_id to 0 instead of 1.
* And some cosmetic changes.

Ganesh Goudar (3):
powerpc/eeh: Synchronization for safety powerpc/eeh: Provide a unique ID for each EEH recovery powerpc/eeh: Asynchronous recovery

 arch/powerpc/include/asm/eeh.h               |  13 +-
 arch/powerpc/include/asm/eeh_event.h         |  10 +-
 arch/powerpc/include/asm/pci-bridge.h        |   4 +
 arch/powerpc/include/asm/ppc-pci.h           |   2 +-
 arch/powerpc/kernel/eeh.c                    | 159 +++--
 arch/powerpc/kernel/eeh_driver.c             | 580 +++++++++++++++----
 arch/powerpc/kernel/eeh_event.c              |  75 ++-
 arch/powerpc/kernel/eeh_pe.c                 |  34 +-
 arch/powerpc/platforms/powernv/eeh-powernv.c |  12 +-
 arch/powerpc/platforms/pseries/eeh_pseries.c |   5 +-
 arch/powerpc/platforms/pseries/pci_dlpar.c   |   5 +-
 drivers/pci/hotplug/pnv_php.c                |   5 +-
 drivers/pci/hotplug/rpadlpar_core.c          |   2 +
 drivers/vfio/vfio_iommu_spapr_tce.c          |  10 +-
 include/linux/mmzone.h                       |   2 +-
 15 files changed, 693 insertions(+), 225 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

RFC PATCH v2 1/3 powerpc/eeh: Synchronization for safety

Based on the original work from Sam Bobroff.

There is currently little synchronization between EEH error detection (eeh_dev_check_failure()), EEH error recovery (eeh_handle_{normal,special}_event()) and the PCI subsystem (device addition and removal), and so there are race conditions that lead to crashes (often access to free'd memory or LIST_POISON).

However, a solution must consider:
- EEH error detection can occur in interrupt context, which prevents the use of a mutex.
- EEH recovery may need to sleep, which prevents the use of a spinlock.
- EEH recovery uses PCI operations that may require the PCI rescan/remove lock and/or device lock to be held
- PCI operations may hold the rescan/remove and/or device lock when calling into EEH functions.
- Device driver callbacks may perform arbitrary PCI operations during recovery, including device removal.

In this patch the existing mutex and spinlock are combined with the EEH_PE_RECOVERING flag to provide some assurances that are then used to reduce the race conditions.

The fields to be protected are the ones that provide the structure of the trees of struct eeh_pe that are held for each PHB: the parent pointer and child lists and the list of struct eeh_dev, as well as the pe and pdev pointers within struct eeh_dev.

The existing way of using EEH_PE_RECOVERING is kept and slightly
extended: No struct eeh_pe will be removed while it has the flag set on it. Additionally, when adding new PEs, they are marked EEH_PE_RECOVERING if their parent PE is marked: this allows the recovery thread to assume that all PEs underneath the one it's processing will continue to exist during recovery.

Both the mutex and spinlock are held while any protected field is changed or a PE is deleted, so holding either of them (elsewhere) will keep them stable and safe to access. Additionally, if EEH_PE_RECOVERING is set on a PE then the locks can be released and re-acquired safely, as long as the protected fields aren't used while no locks are held. This is used during recovery to release locks for long sleeps (i.e. during eeh_wait_state() when we may sleep up to 5 minutes), or to maintain lock ordering.

The spinlock is used in error detection (which cannot use a mutex, see above) and also where it's possible that the mutex is already held. The mutex is used in areas that don't have that restriction, and where blocking may be required. Care must be taken when ordering these locks against the PCI rescan/remove lock and the device locks to avoid deadlocking.

---
 arch/powerpc/include/asm/eeh.h               |  12 +-
 arch/powerpc/kernel/eeh.c                    | 112 ++++++--
 arch/powerpc/kernel/eeh_driver.c             | 288 ++++++++++++++-----
 arch/powerpc/kernel/eeh_pe.c                 |  30 +-
 arch/powerpc/platforms/powernv/eeh-powernv.c |  12 +-
 arch/powerpc/platforms/pseries/eeh_pseries.c |   5 +-
 arch/powerpc/platforms/pseries/pci_dlpar.c   |   5 +-
 drivers/pci/hotplug/pnv_php.c                |   5 +-
 drivers/pci/hotplug/rpadlpar_core.c          |   2 +
 drivers/vfio/vfio_iommu_spapr_tce.c          |  10 +-
 10 files changed, 365 insertions(+), 116 deletions(-)
arch/powerpc/include/asm/eeh.h +8 -4
index 514dd056c2c8..95708c801f27 100644
@@ -233,7 +233,7 @@ extern int eeh_subsystem_flags;
 extern u32 eeh_max_freezes;
 extern bool eeh_debugfs_no_recover;
 extern struct eeh_ops *eeh_ops;
-extern raw_spinlock_t confirm_error_lock;
+extern raw_spinlock_t eeh_pe_tree_spinlock;
 
 static inline void eeh_add_flag(int flag)
 {
@@ -257,12 +257,12 @@ static inline bool eeh_enabled(void)
 
 static inline void eeh_serialize_lock(unsigned long *flags)
 {
-	raw_spin_lock_irqsave(&confirm_error_lock, *flags);
+	raw_spin_lock_irqsave(&eeh_pe_tree_spinlock, *flags);
 }
 
 static inline void eeh_serialize_unlock(unsigned long flags)
 {
-	raw_spin_unlock_irqrestore(&confirm_error_lock, flags);
+	raw_spin_unlock_irqrestore(&eeh_pe_tree_spinlock, flags);
 }
 
 static inline bool eeh_state_active(int state)
@@ -271,11 +271,15 @@ static inline bool eeh_state_active(int state)
 	== (EEH_STATE_MMIO_ACTIVE | EEH_STATE_DMA_ACTIVE);
 }
 
+void eeh_recovery_lock(void);
+void eeh_recovery_unlock(void);
+void eeh_recovery_must_be_locked(void);
+
 typedef void (*eeh_edev_traverse_func)(struct eeh_dev *edev, void *flag);
 typedef void *(*eeh_pe_traverse_func)(struct eeh_pe *pe, void *flag);
 void eeh_set_pe_aux_size(int size);
 int eeh_phb_pe_create(struct pci_controller *phb);
-int eeh_wait_state(struct eeh_pe *pe, int max_wait);
+int eeh_wait_state(struct eeh_pe *pe, int max_wait, bool unlock);
 struct eeh_pe *eeh_phb_pe_get(struct pci_controller *phb);
 struct eeh_pe *eeh_pe_next(struct eeh_pe *pe, struct eeh_pe *root);
 struct eeh_pe *eeh_pe_get(struct pci_controller *phb, int pe_no);
arch/powerpc/kernel/eeh.c +81 -31

336 lines not on this page · raw message

arch/powerpc/kernel/eeh_driver.c +218 -70

529 lines not on this page · raw message

arch/powerpc/kernel/eeh_pe.c +28 -2
index e0ce81279624..a457c528113d 100644
@@ -103,8 +103,15 @@ int eeh_phb_pe_create(struct pci_controller *phb)
  *
  * Wait for the state of associated PE. It might take some time
  * to retrieve the PE's state.
+ *
+ * Note that if this function sleeps and unlock is true, the EEH recovery lock
+ * will be released and re-acquired.
+ * It is only safe to do this when the PE has the recovering flag set on it.
+ * This is important because the sleep may be
+ * very long (300 seconds) and device removal will be blocked by the recovery
+ * mutex. See eeh_device_notifier().
  */
-int eeh_wait_state(struct eeh_pe *pe, int max_wait)
+int eeh_wait_state(struct eeh_pe *pe, int max_wait, bool unlock)
 {
 	int ret;
 	int mwait;
@@ -120,6 +127,8 @@ int eeh_wait_state(struct eeh_pe *pe, int max_wait)
 #define EEH_STATE_MIN_WAIT_TIME	(1000)
 #define EEH_STATE_MAX_WAIT_TIME	(300 * 1000)
 
+	WARN_ON_ONCE(unlock && !(pe->state & EEH_PE_RECOVERING));
+
 	while (1) {
 		ret = eeh_ops->get_state(pe, &mwait);
 
@@ -141,8 +150,11 @@ int eeh_wait_state(struct eeh_pe *pe, int max_wait)
 				__func__, mwait);
 			mwait = EEH_STATE_MAX_WAIT_TIME;
 		}
-
+		if (unlock)
+			eeh_recovery_unlock();
 		msleep(min(mwait, max_wait));
+		if (unlock)
+			eeh_recovery_lock();
 		max_wait -= mwait;
 	}
 }
@@ -308,6 +320,7 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 {
 	struct pci_controller *hose = edev->controller;
 	struct eeh_pe *pe, *parent;
+	unsigned long flags;
 
 	/*
 	 * Search the PE has been existing or not according
@@ -315,6 +328,8 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 	 * PE should be composed of PCI bus and its subordinate
 	 * components.
 	 */
+	eeh_recovery_must_be_locked();
+	eeh_serialize_lock(&flags);
 	pe = eeh_pe_get(hose, edev->pe_config_addr);
 	if (pe) {
 		if (pe->type & EEH_PE_INVALID) {
@@ -343,8 +358,10 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 			list_add_tail(&edev->entry, &pe->edevs);
 			eeh_edev_dbg(edev, "Added to bus PE\n");
 		}
+		eeh_serialize_unlock(flags);
 		return 0;
 	}
+	eeh_serialize_unlock(flags);
 
 	/* Create a new EEH PE */
 	if (edev->physfn)
@@ -364,6 +381,7 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 	 * to PHB directly. Otherwise, we have to associate the
 	 * PE with its parent.
 	 */
+	eeh_serialize_lock(&flags);
 	if (!new_pe_parent) {
 		new_pe_parent = eeh_phb_pe_get(hose);
 		if (!new_pe_parent) {
@@ -371,6 +389,7 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 				__func__, hose->global_number);
 			edev->pe = NULL;
 			kfree(pe);
+			eeh_serialize_unlock(flags);
 			return -EEXIST;
 		}
 	}
@@ -385,6 +404,7 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
 	 */
 	list_add_tail(&edev->entry, &pe->edevs);
 	edev->pe = pe;
+	eeh_serialize_unlock(flags);
 	eeh_edev_dbg(edev, "Added to new (parent: PE#%x)\n",
 		     new_pe_parent->addr);
 
@@ -402,13 +422,18 @@ int eeh_pe_tree_insert(struct eeh_dev *edev, struct eeh_pe *new_pe_parent)
  */
 int eeh_pe_tree_remove(struct eeh_dev *edev)
 {
+	unsigned long flags;
 	struct eeh_pe *pe, *parent, *child;
 	bool keep, recover;
 	int cnt;
 
+	/* Both locks are required to make changes */
+	eeh_recovery_must_be_locked();
+	eeh_serialize_lock(&flags);
 	pe = eeh_dev_to_pe(edev);
 	if (!pe) {
 		eeh_edev_dbg(edev, "No PE found for device.\n");
+		eeh_serialize_unlock(flags);
 		return -EEXIST;
 	}
 
@@ -475,6 +500,7 @@ int eeh_pe_tree_remove(struct eeh_dev *edev)
 		pe = parent;
 	}
 
+	eeh_serialize_unlock(flags);
 	return 0;
 }
 
arch/powerpc/platforms/powernv/eeh-powernv.c +10 -2
index a83cb679dd59..12178be1d70a 100644
@@ -72,6 +72,7 @@ static ssize_t pnv_eeh_ei_write(struct file *filp,
 	unsigned long addr, mask;
 	char buf[50];
 	int ret;
+	unsigned long flags;
 
 	if (!eeh_ops || !eeh_ops->err_inject)
 		return -ENXIO;
@@ -86,14 +87,21 @@ static ssize_t pnv_eeh_ei_write(struct file *filp,
 		     &pe_no, &type, &func, &addr, &mask);
 	if (ret != 5)
 		return -EINVAL;
-
+	/*
+	 * Use the spinlock rather than the mutex so that errors can be
+	 * injected during slow recovery operations (for testing).
+	 */
+	eeh_serialize_lock(&flags);
 	/* Retrieve PE */
 	pe = eeh_pe_get(hose, pe_no);
-	if (!pe)
+	if (!pe) {
+		eeh_serialize_unlock(flags);
 		return -ENODEV;
+	}
 
 	/* Do error injection */
 	ret = eeh_ops->err_inject(pe, type, func, addr, mask);
+	eeh_serialize_unlock(flags);
 	return ret < 0 ? ret : count;
 }
 
arch/powerpc/platforms/pseries/eeh_pseries.c +4 -1
index def184da51cf..21e6c04bb6c0 100644
@@ -47,11 +47,13 @@ static void pseries_eeh_init_edev(struct pci_dn *pdn);
 
 static void pseries_pcibios_bus_add_device(struct pci_dev *pdev)
 {
-	struct pci_dn *pdn = pci_get_pdn(pdev);
+	struct pci_dn *pdn;
 
 	if (eeh_has_flag(EEH_FORCE_DISABLED))
 		return;
 
+	eeh_recovery_lock();
+	pdn = pci_get_pdn(pdev);
 	dev_dbg(&pdev->dev, "EEH: Setting up device\n");
 #ifdef CONFIG_PCI_IOV
 	if (pdev->is_virtfn) {
@@ -82,6 +84,7 @@ static void pseries_pcibios_bus_add_device(struct pci_dev *pdev)
 	}
 #endif
 	eeh_probe_device(pdev);
+	eeh_recovery_unlock();
 }
 
 
arch/powerpc/platforms/pseries/pci_dlpar.c +4 -1
index 4ba824568119..495dd9204ee5 100644
@@ -38,8 +38,11 @@ struct pci_controller *init_phb_dynamic(struct device_node *dn)
 	/* Create EEH devices for the PHB */
 	eeh_phb_pe_create(phb);
 
-	if (dn->child)
+	if (dn->child) {
+		eeh_recovery_lock();
 		pseries_eeh_init_edev_recursive(PCI_DN(dn));
+		eeh_recovery_unlock();
+	}
 
 	pcibios_scan_phb(phb);
 	pcibios_finish_adding_to_bus(phb->bus);
drivers/pci/hotplug/pnv_php.c +2 -3
index 881d420637bf..7837c9054c73 100644
@@ -769,7 +769,6 @@ static irqreturn_t pnv_php_interrupt(int irq, void *data)
 	u16 sts, lsts;
 	u8 presence;
 	bool added;
-	unsigned long flags;
 	int ret;
 
 	pcie_capability_read_word(pdev, PCI_EXP_SLTSTA, &sts);
@@ -807,10 +806,10 @@ static irqreturn_t pnv_php_interrupt(int irq, void *data)
 		edev = pchild ? pci_dev_to_eeh_dev(pchild) : NULL;
 		pe = edev ? edev->pe : NULL;
 		if (pe) {
-			eeh_serialize_lock(&flags);
+			eeh_recovery_lock();
 			eeh_pe_mark_isolated(pe);
-			eeh_serialize_unlock(flags);
 			eeh_pe_set_option(pe, EEH_OPT_FREEZE_PE);
+			eeh_recovery_unlock();
 		}
 	}
 
drivers/pci/hotplug/rpadlpar_core.c +2 -0
index 980bb3afd092..1811f24ad151 100644
@@ -141,7 +141,9 @@ static void dlpar_pci_add_bus(struct device_node *dn)
 	struct pci_controller *phb = pdn->phb;
 	struct pci_dev *dev = NULL;
 
+	eeh_recovery_lock();
 	pseries_eeh_init_edev_recursive(pdn);
+	eeh_recovery_unlock();
 
 	/* Add EADS device to PHB bus, adding new entry to bus->devices */
 	dev = of_create_pci_dev(dn, phb->bus, pdn->devfn);
drivers/vfio/vfio_iommu_spapr_tce.c +8 -2
index a94ec6225d31..a8123a94f9b1 100644
@@ -121,6 +121,7 @@ static long tce_iommu_unregister_pages(struct tce_container *container,
 			found = true;
 			break;
 		}
+		eeh_recovery_unlock();
 	}
 
 	if (!found)
@@ -791,6 +792,7 @@ static long vfio_spapr_ioctl_eeh_pe_op(struct iommu_group *group,
 	if (op.argsz < minsz || op.flags)
 		return -EINVAL;
 
+	eeh_recovery_lock();
 	switch (op.op) {
 	case VFIO_EEH_PE_DISABLE:
 		return eeh_pe_set_option(pe, EEH_OPT_DISABLE);
@@ -813,10 +815,14 @@ static long vfio_spapr_ioctl_eeh_pe_op(struct iommu_group *group,
 		return eeh_pe_configure(pe);
 	case VFIO_EEH_PE_INJECT_ERR:
 		minsz = offsetofend(struct vfio_eeh_pe_op, err.mask);
-		if (op.argsz < minsz)
+		if (op.argsz < minsz) {
+			eeh_recovery_unlock();
 			return -EINVAL;
-		if (copy_from_user(&op, (void __user *)arg, minsz))
+		}
+		if (copy_from_user(&op, (void __user *)arg, minsz)) {
+			eeh_recovery_unlock();
 			return -EFAULT;
+		}
 
 		return eeh_pe_inject_err(pe, op.err.type, op.err.func,
 					 op.err.addr, op.err.mask);
Signature · 1 line
2.40.1
#2 lore raw

Re: [RFC PATCH v2 1/3] powerpc/eeh: Synchronization for safety

kernel test robot noticed the following build errors:

Hi Ganesh,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes awilliam-vfio/next awilliam-vfio/for-linus akpm-mm/mm-everything linus/master v6.5-rc3 next-20230726] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Ganesh-Goudar/powerpc-eeh-Synchronization-for-safety/20230724-150512
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20230724070020.239173-2-ganeshgr%40linux.ibm.com
patch subject: [RFC PATCH v2 1/3] powerpc/eeh: Synchronization for safety
config: powerpc-powernv_defconfig (https://download.01.org/0day-ci/archive/20230727/[email protected]/config)
compiler: clang version 17.0.0 (https://github.com/llvm/llvm-project.git 4a5ac14ee968ff0ad5d2cc1ffa0299048db4c88a)
reproduce: (https://download.01.org/0day-ci/archive/20230727/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> arch/powerpc/kernel/eeh_driver.c:310:18: error: no previous prototype for function 'pdev_cache_list_create' [-Werror,-Wmissing-prototypes]
     310 | struct pci_dev **pdev_cache_list_create(struct eeh_pe *root)
         |                  ^
   arch/powerpc/kernel/eeh_driver.c:310:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
     310 | struct pci_dev **pdev_cache_list_create(struct eeh_pe *root)
         | ^
         | static 
>> arch/powerpc/kernel/eeh_driver.c:579:16: error: unused variable 'flags' [-Werror,-Wunused-variable]
     579 |         unsigned long flags;
         |                       ^

2 errors generated.

vim +/pdev_cache_list_create +310 arch/powerpc/kernel/eeh_driver.c

309

 > 310	struct pci_dev **pdev_cache_list_create(struct eeh_pe *root)

311 {
312 struct eeh_pe *pe;
313 struct eeh_dev *edev, *tmp;
314 struct pci_dev **pdevs;
315 int i, n;
316
317 n = 0;
318 eeh_for_each_pe(root, pe) eeh_pe_for_each_dev(pe, edev, tmp) {
319 if (edev->pdev)
320 n++;
321 }
322 pdevs = kmalloc(sizeof(*pdevs) * (n + 1), GFP_KERNEL);
323 if (WARN_ON_ONCE(!pdevs))
324 return NULL;
325 i = 0;
326 eeh_for_each_pe(root, pe) eeh_pe_for_each_dev(pe, edev, tmp) {
327 if (i < n) {
328 get_device(&edev->pdev->dev);
329 pdevs[i++] = edev->pdev;
330 }
331 }
332 if (WARN_ON_ONCE(i < n))
333 n = i;
334 pdevs[n] = NULL; /* terminator */
335 return pdevs;
336 }
337

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#3 lore raw

RFC PATCH v2 2/3 powerpc/eeh: Provide a unique ID for each EEH recovery

In reply to Ganesh Goudar

Based on the original work from Sam Bobroff.

Give a unique ID to each recovery event, to ease log parsing and prepare for parallel recovery.

Also add some new messages with a very simple format that may be useful to log-parsers.

---
 arch/powerpc/include/asm/eeh_event.h |   3 +-
 arch/powerpc/include/asm/ppc-pci.h   |   2 +-
 arch/powerpc/kernel/eeh.c            |  42 +++---
 arch/powerpc/kernel/eeh_driver.c     | 189 +++++++++++++++------------
 arch/powerpc/kernel/eeh_event.c      |  12 +-
 include/linux/mmzone.h               |   2 +-
 6 files changed, 147 insertions(+), 103 deletions(-)
arch/powerpc/include/asm/eeh_event.h +2 -1
index dadde7d52f46..a1fe736bc4cf 100644
@@ -17,13 +17,14 @@
 struct eeh_event {
 	struct list_head	list;	/* to form event queue	*/
 	struct eeh_pe		*pe;	/* EEH PE		*/
+	unsigned int		id;	/* Event ID		*/
 };
 
 int eeh_event_init(void);
 int eeh_send_failure_event(struct eeh_pe *pe);
 int __eeh_send_failure_event(struct eeh_pe *pe);
 void eeh_remove_event(struct eeh_pe *pe, bool force);
-void eeh_handle_normal_event(struct eeh_pe *pe);
+void eeh_handle_normal_event(unsigned int event_id, struct eeh_pe *pe);
 void eeh_handle_special_event(void);
 
 #endif /* __KERNEL__ */
arch/powerpc/include/asm/ppc-pci.h +1 -1
index d9fcff575027..5b82e76dbd19 100644
@@ -40,7 +40,7 @@ extern int rtas_setup_phb(struct pci_controller *phb);
 void eeh_addr_cache_insert_dev(struct pci_dev *dev);
 void eeh_addr_cache_rmv_dev(struct pci_dev *dev);
 struct eeh_dev *eeh_addr_cache_get_dev(unsigned long addr);
-void eeh_slot_error_detail(struct eeh_pe *pe, int severity);
+void eeh_slot_error_detail(unsigned int event_id, struct eeh_pe *pe, int severity);
 int eeh_pci_enable(struct eeh_pe *pe, int function);
 int eeh_pe_reset_full(struct eeh_pe *pe, bool include_passed);
 void eeh_save_bars(struct eeh_dev *edev);
arch/powerpc/kernel/eeh.c +23 -19

122 lines not on this page · raw message

arch/powerpc/kernel/eeh_driver.c +109 -80

477 lines not on this page · raw message

arch/powerpc/kernel/eeh_event.c +11 -1
index c23a454af08a..6c205a77581f 100644
@@ -26,6 +26,9 @@ static DEFINE_SPINLOCK(eeh_eventlist_lock);
 static DECLARE_COMPLETION(eeh_eventlist_event);
 static LIST_HEAD(eeh_eventlist);
 
+/* Event ID 0 is reserved for special events */
+static atomic_t eeh_event_id = ATOMIC_INIT(1);
+
 /**
  * eeh_event_handler - Dispatch EEH events.
  * @dummy - unused
@@ -59,7 +62,7 @@ static int eeh_event_handler(void * dummy)
 
 		/* We might have event without binding PE */
 		if (event->pe)
-			eeh_handle_normal_event(event->pe);
+			eeh_handle_normal_event(event->id, event->pe);
 		else
 			eeh_handle_special_event();
 
@@ -110,6 +113,13 @@ int __eeh_send_failure_event(struct eeh_pe *pe)
 		return -ENOMEM;
 	}
 	event->pe = pe;
+	do {
+		/* Skip over the special value (0) */
+		event->id = (unsigned int)atomic_inc_return(&eeh_event_id);
+	} while (!event->id);
+
+	pr_err("EEH(%u): EVENT=ERROR_DETECTED PHB=%#x PE=%#x\n",
+	      event->id, pe->phb->global_number, pe->addr);
 
 	/*
 	 * Mark the PE as recovering before inserting it in the queue.
include/linux/mmzone.h +1 -1
index 5e50b78d58ea..bd63c828c063 100644
@@ -1739,7 +1739,7 @@ static inline bool movable_only_nodes(nodemask_t *nodes)
 	((1UL << (PFN_SECTION_SHIFT - pageblock_order)) * NR_PAGEBLOCK_BITS)
 
 #if (MAX_ORDER + PAGE_SHIFT) > SECTION_SIZE_BITS
-#error Allocator MAX_ORDER exceeds SECTION_SIZE
+//#error Allocator MAX_ORDER exceeds SECTION_SIZE
 #endif
 
 static inline unsigned long pfn_to_section_nr(unsigned long pfn)
Signature · 1 line
2.40.1
#4 lore raw

RFC PATCH v2 3/3 powerpc/eeh: Asynchronous recovery

In reply to Ganesh Goudar

Based on the original work from Sam Bobroff.

Currently, EEH recovery is entirely serialized and takes place within a single kernel thread. This can cause recovery to take a long time when there are many devices.

To shorten recovery time, this change allows recovery to proceed in parallel in two ways:
- Each PHB is given it's own recovery event queue and can be recovered independently from other PHBs.
- Driver handlers are called in parallel, but with the constraint that handlers higher up (closer to the PHB) in the PE hierarchy must be called before those lower down.

To maintain the constraint, above, the driver handlers are called by traversing the tree of affected PEs from the top, stopping to call handlers (in parallel) when a PE with devices is discovered. When the calls for that PE are complete, traversal continues at each child PE.

---
 arch/powerpc/include/asm/eeh.h        |   1 +
 arch/powerpc/include/asm/eeh_event.h  |   7 +
 arch/powerpc/include/asm/pci-bridge.h |   4 +
 arch/powerpc/kernel/eeh.c             |   5 -
 arch/powerpc/kernel/eeh_driver.c      | 323 +++++++++++++++++++-------
 arch/powerpc/kernel/eeh_event.c       |  69 +++---
 arch/powerpc/kernel/eeh_pe.c          |   4 +
 7 files changed, 294 insertions(+), 119 deletions(-)
arch/powerpc/include/asm/eeh.h +1 -0
index 95708c801f27..a99472635350 100644
@@ -128,6 +128,7 @@ static inline bool eeh_pe_passed(struct eeh_pe *pe)
 #define EEH_DEV_NO_HANDLER	(1 << 8)	/* No error handler	*/
 #define EEH_DEV_SYSFS		(1 << 9)	/* Sysfs created	*/
 #define EEH_DEV_REMOVED		(1 << 10)	/* Removed permanently	*/
+#define EEH_DEV_RECOVERING	(1 << 11)	/* Recovering		*/
 
 struct eeh_dev {
 	int mode;			/* EEH mode			*/
arch/powerpc/include/asm/eeh_event.h +7 -0
index a1fe736bc4cf..b21f49e87b7b 100644
@@ -8,6 +8,8 @@
 #define ASM_POWERPC_EEH_EVENT_H
 #ifdef __KERNEL__
 
+#include <linux/workqueue.h>
+
 /*
  * structure holding pci controller data that describes a
  * change in the isolation status of a PCI slot.  A pointer
@@ -15,16 +17,21 @@
  * callback.
  */
 struct eeh_event {
+	struct work_struct	work;
 	struct list_head	list;	/* to form event queue	*/
 	struct eeh_pe		*pe;	/* EEH PE		*/
 	unsigned int		id;	/* Event ID		*/
 };
 
+extern spinlock_t eeh_eventlist_lock;
+
 int eeh_event_init(void);
+int eeh_phb_event(struct eeh_pe *pe);
 int eeh_send_failure_event(struct eeh_pe *pe);
 int __eeh_send_failure_event(struct eeh_pe *pe);
 void eeh_remove_event(struct eeh_pe *pe, bool force);
 void eeh_handle_normal_event(unsigned int event_id, struct eeh_pe *pe);
+void eeh_handle_normal_event_work(struct work_struct *work);
 void eeh_handle_special_event(void);
 
 #endif /* __KERNEL__ */
arch/powerpc/include/asm/pci-bridge.h +4 -0
index 2aa3a091ef20..61884d9398bf 100644
@@ -138,6 +138,10 @@ struct pci_controller {
 
 	/* iommu_ops support */
 	struct iommu_device	iommu;
+
+	bool eeh_in_progress;
+	struct list_head eeh_eventlist;
+	spinlock_t eeh_eventlist_lock;
 };
 
 /* These are used for config access before all the PCI probing
arch/powerpc/kernel/eeh.c +0 -5
index 733fb290f4b7..12536d892826 100644
@@ -579,11 +579,6 @@ int eeh_dev_check_failure(struct eeh_dev *edev)
 	 * bridges.
 	 */
 	eeh_pe_mark_isolated(pe);
-
-	/* Most EEH events are due to device driver bugs.  Having
-	 * a stack trace will help the device-driver authors figure
-	 * out what happened.  So print that out.
-	 */
 	pr_debug("EEH: %s: Frozen PHB#%x-PE#%x detected\n",
 		__func__, pe->phb->global_number, pe->addr);
 	eeh_send_failure_event(pe);
arch/powerpc/kernel/eeh_driver.c +235 -88

485 lines not on this page · raw message

arch/powerpc/kernel/eeh_event.c +43 -26

108 lines not on this page · raw message

arch/powerpc/kernel/eeh_pe.c +4 -0
index a457c528113d..3bd6e61aac3d 100644
@@ -81,6 +81,10 @@ int eeh_phb_pe_create(struct pci_controller *phb)
 {
 	struct eeh_pe *pe;
 
+	phb->eeh_in_progress = false; /* TODO: Necessary? */
+	INIT_LIST_HEAD(&phb->eeh_eventlist);
+	spin_lock_init(&phb->eeh_eventlist_lock);
+
 	/* Allocate PHB PE */
 	pe = eeh_pe_alloc(phb, EEH_PE_PHB);
 	if (!pe) {
Signature · 1 line
2.40.1