Skip to contentSkip to conversation

error handling and pciehp maintenance

PATCH In review Keith Busch 7 participants 33 messages Sep 19, 2018 – Nov 6, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover error handling and pciehp maintenance

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Signature · 1 line
2.14.4
#2 lore raw

PATCH 2/12 PCI/AER: Covertly inject errors

The aer_inject module was intercepting config requests by overwriting the config accessor operations in the pci_bus. This has several issues.

First, the module was tracking kernel objects unbeknownst to the drivers that own them. The kernel may free those devices, leaving the AER inject module holding stale references and no way to know that happened.

Second, the pci enumeration has child devices inherit the pci_bus ops from its parent bus. Since errors may lead to link resets that trigger re-enuemration, the child devices have now inherited operations that don't know about the devices using them, causing kernel crashes.

Finally, CONFIG_PCI_LOCKLESS_CONFIG doesn't block accessing the pci_bus ops, so it's racing with potential in flight config requests.

It's just a bad idea to muck with kernel objects out from under the drivers depending on them.

This patch uses a more discreet approach to intercept config requests using ftrace for thunking the config space accessors to inject the desired errors, and removing any need for aer_inject to track pci structuresh.

---
 drivers/pci/pcie/Kconfig      |   2 +-
 drivers/pci/pcie/aer_inject.c | 276 +++++++++++++++++++++++-------------------
 2 files changed, 152 insertions(+), 126 deletions(-)
drivers/pci/pcie/Kconfig +1 -1
index 0a1e9d379bc5..87bcf40f415d 100644
@@ -35,7 +35,7 @@ config PCIEAER
 
 config PCIEAER_INJECT
 	tristate "PCI Express error injection support"
-	depends on PCIEAER
+	depends on PCIEAER && DYNAMIC_FTRACE_WITH_REGS
 	default n
 	help
 	  This enables PCI Express Root Port Advanced Error Reporting
drivers/pci/pcie/aer_inject.c +151 -125

370 lines not on this page · raw message

Signature · 1 line
2.14.4
#5 lore raw

PATCH 5/12 PCI/AER: Remove dead code

In reply to Keith Busch

The error recovery callbacks are only run on child devices. A root port is never a child device, so this error resume callback was never invoked.

---
 drivers/pci/pcie/aer.c | 13 -------------
 1 file changed, 13 deletions(-)
drivers/pci/pcie/aer.c +0 -13
index 1318483a080c..a66e33868303 100644
@@ -1545,18 +1545,6 @@ static pci_ers_result_t aer_root_reset(struct pci_dev *dev)
 	return rc ? PCI_ERS_RESULT_DISCONNECT : PCI_ERS_RESULT_RECOVERED;
 }
 
-/**
- * aer_error_resume - clean up corresponding error status bits
- * @dev: pointer to Root Port's pci_dev data structure
- *
- * Invoked by Port Bus driver during nonfatal recovery.
- */
-static void aer_error_resume(struct pci_dev *dev)
-{
-	pci_aer_clear_device_status(dev);
-	pci_cleanup_aer_uncorrect_error_status(dev);
-}
-
 static struct pcie_port_service_driver aerdriver = {
 	.name		= "aer",
 	.port_type	= PCI_EXP_TYPE_ROOT_PORT,
@@ -1564,7 +1552,6 @@ static struct pcie_port_service_driver aerdriver = {
 
 	.probe		= aer_probe,
 	.remove		= aer_remove,
-	.error_resume	= aer_error_resume,
 	.reset_link	= aer_root_reset,
 };
 
Signature · 1 line
2.14.4
#6 lore raw

PATCH 6/12 PCI/AER: Remove error source from aer struct

In reply to Keith Busch

The aer struct was carrying a copy of the error source simply as a temperary variable. This patch removes that from the structure and uses a stack variable for the purpose.

---
 drivers/pci/pcie/aer.c | 33 ++++++++++++++++-----------------
 1 file changed, 16 insertions(+), 17 deletions(-)
drivers/pci/pcie/aer.c +16 -17
index a66e33868303..72f8b4d6be32 100644
@@ -44,7 +44,6 @@ struct aer_rpc {
 	struct pci_dev *rpd;		/* Root Port device */
 	struct work_struct dpc_handler;
 	struct aer_err_source e_sources[AER_ERROR_SOURCES_MAX];
-	struct aer_err_info e_info;
 	unsigned short prod_idx;	/* Error Producer Index */
 	unsigned short cons_idx;	/* Error Consumer Index */
 	int isr;
@@ -1175,7 +1174,7 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 		struct aer_err_source *e_src)
 {
 	struct pci_dev *pdev = rpc->rpd;
-	struct aer_err_info *e_info = &rpc->e_info;
+	struct aer_err_info e_info;
 
 	pci_rootport_aer_stats_incr(pdev, e_src);
 
@@ -1184,36 +1183,36 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 	 * uncorrectable error being logged. Report correctable error first.
 	 */
 	if (e_src->status & PCI_ERR_ROOT_COR_RCV) {
-		e_info->id = ERR_COR_ID(e_src->id);
-		e_info->severity = AER_CORRECTABLE;
+		e_info.id = ERR_COR_ID(e_src->id);
+		e_info.severity = AER_CORRECTABLE;
 
 		if (e_src->status & PCI_ERR_ROOT_MULTI_COR_RCV)
-			e_info->multi_error_valid = 1;
+			e_info.multi_error_valid = 1;
 		else
-			e_info->multi_error_valid = 0;
-		aer_print_port_info(pdev, e_info);
+			e_info.multi_error_valid = 0;
+		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, e_info))
-			aer_process_err_devices(e_info);
+		if (find_source_device(pdev, &e_info))
+			aer_process_err_devices(&e_info);
 	}
 
 	if (e_src->status & PCI_ERR_ROOT_UNCOR_RCV) {
-		e_info->id = ERR_UNCOR_ID(e_src->id);
+		e_info.id = ERR_UNCOR_ID(e_src->id);
 
 		if (e_src->status & PCI_ERR_ROOT_FATAL_RCV)
-			e_info->severity = AER_FATAL;
+			e_info.severity = AER_FATAL;
 		else
-			e_info->severity = AER_NONFATAL;
+			e_info.severity = AER_NONFATAL;
 
 		if (e_src->status & PCI_ERR_ROOT_MULTI_UNCOR_RCV)
-			e_info->multi_error_valid = 1;
+			e_info.multi_error_valid = 1;
 		else
-			e_info->multi_error_valid = 0;
+			e_info.multi_error_valid = 0;
 
-		aer_print_port_info(pdev, e_info);
+		aer_print_port_info(pdev, &e_info);
 
-		if (find_source_device(pdev, e_info))
-			aer_process_err_devices(e_info);
+		if (find_source_device(pdev, &e_info))
+			aer_process_err_devices(&e_info);
 	}
 }
 
Signature · 1 line
2.14.4
#11 lore raw

PATCH 11/12 PCI/AER: Use managed resource allocations

In reply to Keith Busch

This uses the managed device resource allocations for the service data so that the aer driver doesn't need to manage it, further simplifying this driver.

---
 drivers/pci/pcie/aer.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/pci/pcie/aer.c +5 -12
index 1878d9d7760b..7ecad011458d 100644
@@ -1366,11 +1366,7 @@ static void aer_remove(struct pcie_device *dev)
 {
 	struct aer_rpc *rpc = get_service_data(dev);
 
-	if (rpc) {
-		aer_disable_rootport(rpc);
-		kfree(rpc);
-		set_service_data(dev, NULL);
-	}
+	aer_disable_rootport(rpc);
 }
 
 /**
@@ -1383,10 +1379,9 @@ static int aer_probe(struct pcie_device *dev)
 {
 	int status;
 	struct aer_rpc *rpc;
-	struct device *device = &dev->port->device;
+	struct device *device = &dev->device;
 
-	/* Alloc rpc data structure */
-	rpc = kzalloc(sizeof(struct aer_rpc), GFP_KERNEL);
+	rpc = devm_kzalloc(device, sizeof(struct aer_rpc), GFP_KERNEL);
 	if (!rpc) {
 		dev_printk(KERN_DEBUG, device, "alloc AER rpc failed\n");
 		return -ENOMEM;
@@ -1394,13 +1389,11 @@ static int aer_probe(struct pcie_device *dev)
 	rpc->rpd = dev->port;
 	set_service_data(dev, rpc);
 
-	/* Request IRQ ISR */
-	status = request_threaded_irq(dev->irq, aer_irq, aer_isr,
-				      IRQF_SHARED, "aerdrv", dev);
+	status = devm_request_threaded_irq(device, dev->irq, aer_irq, aer_isr,
+					   IRQF_SHARED, "aerdrv", dev);
 	if (status) {
 		dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
 			   dev->irq);
-		aer_remove(dev);
 		return status;
 	}
 
Signature · 1 line
2.14.4
#1 lore raw

PATCH 1/12 PCI: Set PCI bus accessors to noinline

In reply to Keith Busch

The pci bus config accessors could compile inlined with other accessor functions, which makes it so they can't be traced. This patch forces these to never be inlined so that ftrace can hook into these functions.

---
 drivers/pci/access.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/pci/access.c +2 -2
index a3ad2fe185b9..544922f097c0 100644
@@ -33,7 +33,7 @@ DEFINE_RAW_SPINLOCK(pci_lock);
 #endif
 
 #define PCI_OP_READ(size, type, len) \
-int pci_bus_read_config_##size \
+int noinline pci_bus_read_config_##size \
 	(struct pci_bus *bus, unsigned int devfn, int pos, type *value)	\
 {									\
 	int res;							\
@@ -48,7 +48,7 @@ int pci_bus_read_config_##size \
 }
 
 #define PCI_OP_WRITE(size, type, len) \
-int pci_bus_write_config_##size \
+int noinline pci_bus_write_config_##size \
 	(struct pci_bus *bus, unsigned int devfn, int pos, type value)	\
 {									\
 	int res;							\
Signature · 1 line
2.14.4
#3 lore raw

PATCH 3/12 PCI/AER: Reuse existing service device lookup

In reply to Keith Busch

The port services driver already provides a method to find the pcie_device for a service. This patch exports the function and uses it from the aer_inject module and removes the duplicate functionality.

---
 drivers/pci/pcie/aer_inject.c   | 25 ++++---------------------
 drivers/pci/pcie/portdrv_core.c |  1 +
 2 files changed, 5 insertions(+), 21 deletions(-)
drivers/pci/pcie/aer_inject.c +4 -21
index 6f8b96499f44..7d641a543194 100644
@@ -201,32 +201,13 @@ static int aer_inj_write_config(struct pci_bus *bus, unsigned int devfn,
 	return -1;
 }
 
-static int find_aer_device_iter(struct device *device, void *data)
-{
-	struct pcie_device **result = data;
-	struct pcie_device *pcie_dev;
-
-	if (device->bus == &pcie_port_bus_type) {
-		pcie_dev = to_pcie_device(device);
-		if (pcie_dev->service & PCIE_PORT_SERVICE_AER) {
-			*result = pcie_dev;
-			return 1;
-		}
-	}
-	return 0;
-}
-
-static int find_aer_device(struct pci_dev *dev, struct pcie_device **result)
-{
-	return device_for_each_child(&dev->dev, result, find_aer_device_iter);
-}
-
 static int aer_inject(struct aer_error_inj *einj)
 {
 	struct aer_error *err, *rperr;
 	struct aer_error *err_alloc = NULL, *rperr_alloc = NULL;
 	struct pci_dev *dev, *rpdev;
 	struct pcie_device *edev;
+	struct device *device;
 	unsigned long flags;
 	unsigned int devfn = PCI_DEVFN(einj->dev, einj->fn);
 	int pos_cap_err, rp_pos_cap_err;
@@ -355,7 +336,9 @@ static int aer_inject(struct aer_error_inj *einj)
 				       uncor_mask_orig);
 	}
 
-	if (find_aer_device(rpdev, &edev)) {
+	device = pcie_port_find_device(rpdev, PCIE_PORT_SERVICE_AER);
+	if (device) {
+		edev = to_pcie_device(device);
 		if (!get_service_data(edev)) {
 			dev_warn(&edev->device,
 				 "aer_inject: AER service is not initialized\n");
drivers/pci/pcie/portdrv_core.c +1 -0
index 7c37d815229e..5508285a0bff 100644
@@ -466,6 +466,7 @@ struct device *pcie_port_find_device(struct pci_dev *dev,
 	device = pdrvs.dev;
 	return device;
 }
+EXPORT_SYMBOL_GPL(pcie_port_find_device);
 
 /**
  * pcie_port_device_remove - unregister PCI Express port service devices
Signature · 1 line
2.14.4
#4 lore raw

PATCH 4/12 PCI/AER: Abstract AER interrupt handling

In reply to Keith Busch

The aer_inject module was directly calling aer_irq. This required the aer driver export its private irq handler for no other reason than to support error injection. A driver should not have to expose its private interfaces, so this patch uses the irq subsystem to route injection to the aer driver, and makes the aer_irq handler a private interface.

This are additional benefits this provides.

First, directly calling the irq handler bypasses the IRQ subsytem so the injection wasn't really synthesizing what happens if a shared AER interrupt occurs.

The error injection had to provide the callback data directly, which may be racing with a removal that is freeing that structure. The irq subsystem can handle that race.

Finally, if we ever need to modify the aer interrupt handling, for example, to use threaded IRQs for the benefits those provide, abstracting the interface from the error injection will make that easier to modify.

---
 drivers/pci/pcie/aer.c        | 3 +--
 drivers/pci/pcie/aer_inject.c | 5 ++++-
 drivers/pci/pcie/portdrv.h    | 4 ----
 3 files changed, 5 insertions(+), 7 deletions(-)
drivers/pci/pcie/aer.c +1 -2
index 6b59a23568f8..1318483a080c 100644
@@ -1270,7 +1270,7 @@ static void aer_isr(struct work_struct *work)
  *
  * Invoked when Root Port detects AER messages.
  */
-irqreturn_t aer_irq(int irq, void *context)
+static irqreturn_t aer_irq(int irq, void *context)
 {
 	unsigned int status, id;
 	struct pcie_device *pdev = (struct pcie_device *)context;
@@ -1319,7 +1319,6 @@ irqreturn_t aer_irq(int irq, void *context)
 
 	return IRQ_HANDLED;
 }
-EXPORT_SYMBOL_GPL(aer_irq);
 
 static int set_device_error_reporting(struct pci_dev *dev, void *data)
 {
drivers/pci/pcie/aer_inject.c +4 -1
index 7d641a543194..57821bb61c2a 100644
@@ -15,6 +15,7 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/ftrace.h>
+#include <linux/irq.h>
 #include <linux/kallsyms.h>
 #include <linux/kernel.h>
 #include <linux/linkage.h>
@@ -348,7 +349,9 @@ static int aer_inject(struct aer_error_inj *einj)
 		dev_info(&edev->device,
 			 "aer_inject: Injecting errors %08x/%08x into device %s\n",
 			 einj->cor_status, einj->uncor_status, pci_name(dev));
-		aer_irq(-1, edev);
+		local_irq_disable();
+		generic_handle_irq(edev->irq);
+		local_irq_enable();
 	} else {
 		pci_err(rpdev, "aer_inject: AER device not found\n");
 		ret = -ENODEV;
drivers/pci/pcie/portdrv.h +0 -4
index d59afa42fc14..127b8b246437 100644
@@ -123,10 +123,6 @@ static inline int pcie_aer_get_firmware_first(struct pci_dev *pci_dev)
 }
 #endif
 
-#ifdef CONFIG_PCIEAER
-irqreturn_t aer_irq(int irq, void *context);
-#endif
-
 struct pcie_port_service_driver *pcie_port_find_service(struct pci_dev *dev,
 							u32 service);
 struct device *pcie_port_find_device(struct pci_dev *dev, u32 service);
Signature · 1 line
2.14.4
#7 lore raw

PATCH 7/12 PCI/AER: Use kfifo for tracking events

In reply to Keith Busch

The kernel provides a generic fifo implementation, so no need to reinvent that capability in a driver. This patch replaces the aer specific implementation with the kernel provided kfifo. Since the interrupt handler producer and work queue consumer run single threaded, there is no need for additional locking, so this patch removes that lock too.

---
 drivers/pci/pcie/aer.c | 88 +++++++-------------------------------------------
 1 file changed, 11 insertions(+), 77 deletions(-)
drivers/pci/pcie/aer.c +11 -77
index 72f8b4d6be32..813b5dd6fe68 100644
@@ -30,7 +30,7 @@
 #include "../pci.h"
 #include "portdrv.h"
 
-#define AER_ERROR_SOURCES_MAX		100
+#define AER_ERROR_SOURCES_MAX		128
 
 #define AER_MAX_TYPEOF_COR_ERRS		16	/* as per PCI_ERR_COR_STATUS */
 #define AER_MAX_TYPEOF_UNCOR_ERRS	26	/* as per PCI_ERR_UNCOR_STATUS*/
@@ -43,14 +43,8 @@ struct aer_err_source {
 struct aer_rpc {
 	struct pci_dev *rpd;		/* Root Port device */
 	struct work_struct dpc_handler;
-	struct aer_err_source e_sources[AER_ERROR_SOURCES_MAX];
-	unsigned short prod_idx;	/* Error Producer Index */
-	unsigned short cons_idx;	/* Error Consumer Index */
+	DECLARE_KFIFO(aer_fifo, struct aer_err_source, AER_ERROR_SOURCES_MAX);
 	int isr;
-	spinlock_t e_lock;		/*
-					 * Lock access to Error Status/ID Regs
-					 * and error producer/consumer index
-					 */
 	struct mutex rpc_mutex;		/*
 					 * only one thread could do
 					 * recovery on the same
@@ -1216,35 +1210,6 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
 	}
 }
 
-/**
- * get_e_source - retrieve an error source
- * @rpc: pointer to the root port which holds an error
- * @e_src: pointer to store retrieved error source
- *
- * Return 1 if an error source is retrieved, otherwise 0.
- *
- * Invoked by DPC handler to consume an error.
- */
-static int get_e_source(struct aer_rpc *rpc, struct aer_err_source *e_src)
-{
-	unsigned long flags;
-
-	/* Lock access to Root error producer/consumer index */
-	spin_lock_irqsave(&rpc->e_lock, flags);
-	if (rpc->prod_idx == rpc->cons_idx) {
-		spin_unlock_irqrestore(&rpc->e_lock, flags);
-		return 0;
-	}
-
-	*e_src = rpc->e_sources[rpc->cons_idx];
-	rpc->cons_idx++;
-	if (rpc->cons_idx == AER_ERROR_SOURCES_MAX)
-		rpc->cons_idx = 0;
-	spin_unlock_irqrestore(&rpc->e_lock, flags);
-
-	return 1;
-}
-
 /**
  * aer_isr - consume errors detected by root port
  * @work: definition of this work item
@@ -1257,7 +1222,7 @@ static void aer_isr(struct work_struct *work)
 	struct aer_err_source uninitialized_var(e_src);
 
 	mutex_lock(&rpc->rpc_mutex);
-	while (get_e_source(rpc, &e_src))
+	while (kfifo_get(&rpc->aer_fifo, &e_src))
 		aer_isr_one_error(rpc, &e_src);
 	mutex_unlock(&rpc->rpc_mutex);
 }
@@ -1271,51 +1236,23 @@ static void aer_isr(struct work_struct *work)
  */
 static irqreturn_t aer_irq(int irq, void *context)
 {
-	unsigned int status, id;
 	struct pcie_device *pdev = (struct pcie_device *)context;
 	struct aer_rpc *rpc = get_service_data(pdev);
-	int next_prod_idx;
-	unsigned long flags;
-	int pos;
-
-	pos = pdev->port->aer_cap;
-	/*
-	 * Must lock access to Root Error Status Reg, Root Error ID Reg,
-	 * and Root error producer/consumer index
-	 */
-	spin_lock_irqsave(&rpc->e_lock, flags);
+	struct pci_dev *rp = rpc->rpd;
+	struct aer_err_source e_src = {};
+	int pos = rp->aer_cap;
 
-	/* Read error status */
-	pci_read_config_dword(pdev->port, pos + PCI_ERR_ROOT_STATUS, &status);
-	if (!(status & (PCI_ERR_ROOT_UNCOR_RCV|PCI_ERR_ROOT_COR_RCV))) {
-		spin_unlock_irqrestore(&rpc->e_lock, flags);
+	pci_read_config_dword(rp, pos + PCI_ERR_ROOT_STATUS, &e_src.status);
+	if (!(e_src.status & (PCI_ERR_ROOT_UNCOR_RCV|PCI_ERR_ROOT_COR_RCV)))
 		return IRQ_NONE;
-	}
 
-	/* Read error source and clear error status */
-	pci_read_config_dword(pdev->port, pos + PCI_ERR_ROOT_ERR_SRC, &id);
-	pci_write_config_dword(pdev->port, pos + PCI_ERR_ROOT_STATUS, status);
+	pci_read_config_dword(rp, pos + PCI_ERR_ROOT_ERR_SRC, &e_src.id);
+	pci_write_config_dword(rp, pos + PCI_ERR_ROOT_STATUS, e_src.status);
 
-	/* Store error source for later DPC handler */
-	next_prod_idx = rpc->prod_idx + 1;
-	if (next_prod_idx == AER_ERROR_SOURCES_MAX)
-		next_prod_idx = 0;
-	if (next_prod_idx == rpc->cons_idx) {
-		/*
-		 * Error Storm Condition - possibly the same error occurred.
-		 * Drop the error.
-		 */
-		spin_unlock_irqrestore(&rpc->e_lock, flags);
+	if (!kfifo_put(&rpc->aer_fifo, e_src))
 		return IRQ_HANDLED;
-	}
-	rpc->e_sources[rpc->prod_idx].status =  status;
-	rpc->e_sources[rpc->prod_idx].id = id;
-	rpc->prod_idx = next_prod_idx;
-	spin_unlock_irqrestore(&rpc->e_lock, flags);
 
-	/*  Invoke DPC handler */
 	schedule_work(&rpc->dpc_handler);
-
 	return IRQ_HANDLED;
 }
 
@@ -1439,9 +1376,6 @@ static struct aer_rpc *aer_alloc_rpc(struct pcie_device *dev)
 	if (!rpc)
 		return NULL;
 
-	/* Initialize Root lock access, e_lock, to Root Error Status Reg */
-	spin_lock_init(&rpc->e_lock);
-
 	rpc->rpd = dev->port;
 	INIT_WORK(&rpc->dpc_handler, aer_isr);
 	mutex_init(&rpc->rpc_mutex);
Signature · 1 line
2.14.4
#8 lore raw

PATCH 8/12 PCI/AER: Use kfifo helper inserting locked elements

In reply to Keith Busch

This patch uses the recommended kernel API for writing to a concurrently accessed kfifo. No functional change here.

---
 drivers/pci/pcie/aer.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/pci/pcie/aer.c +2 -4
index 813b5dd6fe68..33bbcaa41f65 100644
@@ -1063,7 +1063,6 @@ static DECLARE_WORK(aer_recover_work, aer_recover_work_func);
 void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
 		       int severity, struct aer_capability_regs *aer_regs)
 {
-	unsigned long flags;
 	struct aer_recover_entry entry = {
 		.bus		= bus,
 		.devfn		= devfn,
@@ -1072,13 +1071,12 @@ void aer_recover_queue(int domain, unsigned int bus, unsigned int devfn,
 		.regs		= aer_regs,
 	};
 
-	spin_lock_irqsave(&aer_recover_ring_lock, flags);
-	if (kfifo_put(&aer_recover_ring, entry))
+	if (kfifo_in_spinlocked(&aer_recover_ring, &entry, sizeof(entry),
+				 &aer_recover_ring_lock))
 		schedule_work(&aer_recover_work);
 	else
 		pr_err("AER recover: Buffer overflow when recovering AER for %04x:%02x:%02x:%x\n",
 		       domain, bus, PCI_SLOT(devfn), PCI_FUNC(devfn));
-	spin_unlock_irqrestore(&aer_recover_ring_lock, flags);
 }
 EXPORT_SYMBOL_GPL(aer_recover_queue);
 #endif
Signature · 1 line
2.14.4
#9 lore raw

PATCH 9/12 PCI/AER: Don't read upstream ports below fatal errors

In reply to Keith Busch

The AER driver has never read the config space of an end device that reported a fatal error because the link to that device is considered unreliable.

An ERR_FATAL from upstream port was almost certainly detected that error on its upstream link, so we can't expect to reliably read its config space for the same reason.

---
 drivers/pci/pcie/aer.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/pci/pcie/aer.c +3 -2
index 33bbcaa41f65..41c36916d46c 100644
@@ -1111,8 +1111,9 @@ int aer_get_device_error_info(struct pci_dev *dev, struct aer_err_info *info)
 			&info->mask);
 		if (!(info->status & ~info->mask))
 			return 0;
-	} else if (dev->hdr_type == PCI_HEADER_TYPE_BRIDGE ||
-		info->severity == AER_NONFATAL) {
+	} else if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT ||
+	           pci_pcie_type(dev) == PCI_EXP_TYPE_DOWNSTREAM ||
+		   info->severity == AER_NONFATAL) {
 
 		/* Link is still healthy for IO reads */
 		pci_read_config_dword(dev, pos + PCI_ERR_UNCOR_STATUS,
Signature · 1 line
2.14.4
#10 lore raw

PATCH 10/12 PCI/AER: Use threaded IRQ for bottom half

In reply to Keith Busch

The threaded IRQ is naturally single threaded as desired, so use that to simplify the aer bottom half handler. Since the root port structure is has much less to do now, this patch removes the rpc construction helper routine.

---
 drivers/pci/pcie/aer.c | 62 ++++++++++++--------------------------------------
 1 file changed, 14 insertions(+), 48 deletions(-)
drivers/pci/pcie/aer.c +14 -48
index 41c36916d46c..1878d9d7760b 100644
@@ -42,14 +42,7 @@ struct aer_err_source {
 
 struct aer_rpc {
 	struct pci_dev *rpd;		/* Root Port device */
-	struct work_struct dpc_handler;
 	DECLARE_KFIFO(aer_fifo, struct aer_err_source, AER_ERROR_SOURCES_MAX);
-	int isr;
-	struct mutex rpc_mutex;		/*
-					 * only one thread could do
-					 * recovery on the same
-					 * root port hierarchy
-					 */
 };
 
 /* AER stats for the device */
@@ -1215,15 +1208,18 @@ static void aer_isr_one_error(struct aer_rpc *rpc,
  *
  * Invoked, as DPC, when root port records new detected error
  */
-static void aer_isr(struct work_struct *work)
+static irqreturn_t aer_isr(int irq, void *context)
 {
-	struct aer_rpc *rpc = container_of(work, struct aer_rpc, dpc_handler);
+	struct pcie_device *dev = (struct pcie_device *)context;
+	struct aer_rpc *rpc = get_service_data(dev);
 	struct aer_err_source uninitialized_var(e_src);
 
-	mutex_lock(&rpc->rpc_mutex);
+	if (kfifo_is_empty(&rpc->aer_fifo))
+		return IRQ_NONE;
+
 	while (kfifo_get(&rpc->aer_fifo, &e_src))
 		aer_isr_one_error(rpc, &e_src);
-	mutex_unlock(&rpc->rpc_mutex);
+	return IRQ_HANDLED;
 }
 
 /**
@@ -1251,8 +1247,7 @@ static irqreturn_t aer_irq(int irq, void *context)
 	if (!kfifo_put(&rpc->aer_fifo, e_src))
 		return IRQ_HANDLED;
 
-	schedule_work(&rpc->dpc_handler);
-	return IRQ_HANDLED;
+	return IRQ_WAKE_THREAD;
 }
 
 static int set_device_error_reporting(struct pci_dev *dev, void *data)
@@ -1361,30 +1356,6 @@ static void aer_disable_rootport(struct aer_rpc *rpc)
 	pci_write_config_dword(pdev, pos + PCI_ERR_ROOT_STATUS, reg32);
 }
 
-/**
- * aer_alloc_rpc - allocate Root Port data structure
- * @dev: pointer to the pcie_dev data structure
- *
- * Invoked when Root Port's AER service is loaded.
- */
-static struct aer_rpc *aer_alloc_rpc(struct pcie_device *dev)
-{
-	struct aer_rpc *rpc;
-
-	rpc = kzalloc(sizeof(struct aer_rpc), GFP_KERNEL);
-	if (!rpc)
-		return NULL;
-
-	rpc->rpd = dev->port;
-	INIT_WORK(&rpc->dpc_handler, aer_isr);
-	mutex_init(&rpc->rpc_mutex);
-
-	/* Use PCIe bus function to store rpc into PCIe device */
-	set_service_data(dev, rpc);
-
-	return rpc;
-}
-
 /**
  * aer_remove - clean up resources
  * @dev: pointer to the pcie_dev data structure
@@ -1396,11 +1367,6 @@ static void aer_remove(struct pcie_device *dev)
 	struct aer_rpc *rpc = get_service_data(dev);
 
 	if (rpc) {
-		/* If register interrupt service, it must be free. */
-		if (rpc->isr)
-			free_irq(dev->irq, dev);
-
-		flush_work(&rpc->dpc_handler);
 		aer_disable_rootport(rpc);
 		kfree(rpc);
 		set_service_data(dev, NULL);
@@ -1417,18 +1383,20 @@ static int aer_probe(struct pcie_device *dev)
 {
 	int status;
 	struct aer_rpc *rpc;
-	struct device *device = &dev->port->dev;
+	struct device *device = &dev->port->device;
 
 	/* Alloc rpc data structure */
-	rpc = aer_alloc_rpc(dev);
+	rpc = kzalloc(sizeof(struct aer_rpc), GFP_KERNEL);
 	if (!rpc) {
 		dev_printk(KERN_DEBUG, device, "alloc AER rpc failed\n");
-		aer_remove(dev);
 		return -ENOMEM;
 	}
+	rpc->rpd = dev->port;
+	set_service_data(dev, rpc);
 
 	/* Request IRQ ISR */
-	status = request_irq(dev->irq, aer_irq, IRQF_SHARED, "aerdrv", dev);
+	status = request_threaded_irq(dev->irq, aer_irq, aer_isr,
+				      IRQF_SHARED, "aerdrv", dev);
 	if (status) {
 		dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
 			   dev->irq);
@@ -1436,8 +1404,6 @@ static int aer_probe(struct pcie_device *dev)
 		return status;
 	}
 
-	rpc->isr = 1;
-
 	aer_enable_rootport(rpc);
 	dev_info(device, "AER enabled with IRQ %d\n", dev->irq);
 	return 0;
Signature · 1 line
2.14.4
#17 lore raw

PATCH 12/12 PCI/pciehp: Use device managed allocations

In reply to Keith Busch

All of pciehp's resources are tied to the lifetime of the device it is driving. This patch simplifies the resource tracking by using the device managed resource allocations.

---
 drivers/pci/hotplug/pciehp_core.c | 14 +++---------
 drivers/pci/hotplug/pciehp_hpc.c  | 48 +++++++++++----------------------------
 2 files changed, 16 insertions(+), 46 deletions(-)
drivers/pci/hotplug/pciehp_core.c +3 -11
index 334044814dbe..c9ae89f25e8c 100644
@@ -58,16 +58,16 @@ static int init_slot(struct controller *ctrl)
 	char name[SLOT_NAME_SIZE];
 	int retval = -ENOMEM;
 
-	hotplug = kzalloc(sizeof(*hotplug), GFP_KERNEL);
+	hotplug = devm_kzalloc(&ctrl->pcie->device, sizeof(*hotplug), GFP_KERNEL);
 	if (!hotplug)
 		goto out;
 
-	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	info = devm_kzalloc(&ctrl->pcie->device, sizeof(*info), GFP_KERNEL);
 	if (!info)
 		goto out;
 
 	/* Setup hotplug slot ops */
-	ops = kzalloc(sizeof(*ops), GFP_KERNEL);
+	ops = devm_kzalloc(&ctrl->pcie->device, sizeof(*ops), GFP_KERNEL);
 	if (!ops)
 		goto out;
 
@@ -98,11 +98,6 @@ static int init_slot(struct controller *ctrl)
 	if (retval)
 		ctrl_err(ctrl, "pci_hp_initialize failed: error %d\n", retval);
 out:
-	if (retval) {
-		kfree(ops);
-		kfree(info);
-		kfree(hotplug);
-	}
 	return retval;
 }
 
@@ -111,9 +106,6 @@ static void cleanup_slot(struct controller *ctrl)
 	struct hotplug_slot *hotplug_slot = ctrl->slot->hotplug_slot;
 
 	pci_hp_destroy(hotplug_slot);
-	kfree(hotplug_slot->ops);
-	kfree(hotplug_slot->info);
-	kfree(hotplug_slot);
 }
 
 /*
drivers/pci/hotplug/pciehp_hpc.c +13 -35

106 lines not on this page · raw message

Signature · 1 line
2.14.4
#18 lore raw

Re: [PATCH 12/12] PCI/pciehp: Use device managed allocations

Quoted 6 lines · Keith Busch
drivers/pci/hotplug/pciehp_hpc.c +0 -1
-	ctrl->notification_enabled = 1;
  	return 0;
  }
 
  void pcie_shutdown_notification(struct controller *ctrl)

drivers/pci/hotplug/pciehp_hpc.c int pcie_init_notification(struct controller *ctrl) · line 784

Quoted 10 lines
drivers/pci/hotplug/pciehp_hpc.c +3 -5
  {
-	if (ctrl->notification_enabled) {
-		pcie_disable_notification(ctrl);
-		pciehp_free_irq(ctrl);
-		ctrl->notification_enabled = 0;
-	}
+	pcie_disable_notification(ctrl);
+	if (pciehp_poll_mode)
+		kthread_stop(ctrl->poll_thread);
  }

I think this notification_enabled bit change needs to go to another path. The rest of the change in this file is pretty mechanic changes. Also, are you going to remove the notification_enabled member?

#19 lore raw

Re: [PATCH 12/12] PCI/pciehp: Use device managed allocations

Quoted 11 lines · Sinan Kaya
Quoted 10 lines · Keith Busch
drivers/pci/hotplug/pciehp_hpc.c +0 -6
-	ctrl->notification_enabled = 1;
  	return 0;
  }
  void pcie_shutdown_notification(struct controller *ctrl)
  {
-	if (ctrl->notification_enabled) {
-		pcie_disable_notification(ctrl);
-		pciehp_free_irq(ctrl);
-		ctrl->notification_enabled = 0;
-	}
Quoted 8 lines

drivers/pci/hotplug/pciehp_hpc.c int pcie_init_notification(struct controller *ctrl) · line 784

Quoted 4 lines
drivers/pci/hotplug/pciehp_hpc.c +3 -0
+	pcie_disable_notification(ctrl);
+	if (pciehp_poll_mode)
+		kthread_stop(ctrl->poll_thread);
  }

I think this notification_enabled bit change needs to go to another path. The rest of the change in this file is pretty mechanic changes. Also, are you going to remove the notification_enabled member?

Right, we don't need it, and I should have removed it. There is no path that would call pcie_shutdown_notification if it hadn't been enabled.

#12 lore raw

Re: [PATCH 11/12] PCI/AER: Use managed resource allocations

In reply to Keith Busch

drivers/pci/pcie/aer.c static int aer_probe(struct pcie_device *dev) · line 1398

Quoted 7 lines · Keith Busch

if (status) {
dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
dev->irq);

drivers/pci/pcie/aer.c +0 -1
-		aer_remove(dev);
  		return status;
  	}

Don't we still need to call aer_remove() here?

Old code would call aer_disable_rootport(rpc) via aer_remove() on IRQ allocation failure. We are no longer doing this.

#13 lore raw

Re: [PATCH 11/12] PCI/AER: Use managed resource allocations

Quoted 10 lines · Sinan Kaya

drivers/pci/pcie/aer.c static int aer_probe(struct pcie_device *dev) · line 1398

Quoted 6 lines · Keith Busch

if (status) {
dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
dev->irq);

drivers/pci/pcie/aer.c +0 -1
-		aer_remove(dev);
  		return status;
  	}

Don't we still need to call aer_remove() here?

Quoted 2 lines

Old code would call aer_disable_rootport(rpc) via aer_remove() on IRQ allocation failure. We are no longer doing this.

We need to call aer_disable_rootport only if aer_enable_rootport was called, but that happens *after* irq allocation.

#14 lore raw

Re: [PATCH 11/12] PCI/AER: Use managed resource allocations

Quoted 11 lines · Keith Busch
Quoted 10 lines · Sinan Kaya

drivers/pci/pcie/aer.c static int aer_probe(struct pcie_device *dev) · line 1398

Quoted 6 lines · Keith Busch
   	if (status) {
   		dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
   			   dev->irq);
drivers/pci/pcie/aer.c +0 -1
-		aer_remove(dev);
   		return status;
   	}

Don't we still need to call aer_remove() here?

Quoted 6 lines
Quoted 2 lines

Old code would call aer_disable_rootport(rpc) via aer_remove() on IRQ allocation failure. We are no longer doing this.

We need to call aer_disable_rootport only if aer_enable_rootport was called, but that happens *after* irq allocation.

I see. Thanks for clarification.

#20 lore raw

Re: [PATCH 12/12] PCI/pciehp: Use device managed allocations

In reply to Keith Busch
Quoted 11 lines · Keith Busch
drivers/pci/hotplug/pciehp_core.c +1 -1
@@ -58,16 +58,16 @@ static int init_slot(struct controller *ctrl)
 	char name[SLOT_NAME_SIZE];
 	int retval = -ENOMEM;
 
-	hotplug = kzalloc(sizeof(*hotplug), GFP_KERNEL);
+	hotplug = devm_kzalloc(&ctrl->pcie->device, sizeof(*hotplug), GFP_KERNEL);
 	if (!hotplug)
 		goto out;
 

drivers/pci/hotplug/pciehp_core.c static int init_slot(struct controller *ctrl) · line 72

Quoted 10 lines
drivers/pci/hotplug/pciehp_core.c +2 -2
-	info = kzalloc(sizeof(*info), GFP_KERNEL);
+	info = devm_kzalloc(&ctrl->pcie->device, sizeof(*info), GFP_KERNEL);
 	if (!info)
 		goto out;
 
 	/* Setup hotplug slot ops */
-	ops = kzalloc(sizeof(*ops), GFP_KERNEL);
+	ops = devm_kzalloc(&ctrl->pcie->device, sizeof(*ops), GFP_KERNEL);
 	if (!ops)
 		goto out;

The "hotplug" and "info" allocations are gone on the pci/hotplug branch, so this needs a rebase.

drivers/pci/hotplug/pciehp_core.c static void cleanup_slot(struct controller *ctrl) · line 109

Quoted 8 lines
drivers/pci/hotplug/pciehp_core.c +0 -3
@@ -111,9 +106,6 @@ static void cleanup_slot(struct controller *ctrl)
 	struct hotplug_slot *hotplug_slot = ctrl->slot->hotplug_slot;
 
 	pci_hp_destroy(hotplug_slot);
-	kfree(hotplug_slot->ops);
-	kfree(hotplug_slot->info);
-	kfree(hotplug_slot);
 }

Please replace all invocations of cleanup_slot() with a direct call to pci_hp_destroy() since this is now becoming merely a wrapper function.

drivers/pci/hotplug/pciehp_hpc.c static inline int pciehp_request_irq(struct controller *ctrl) · line 50

Quoted 11 lines
drivers/pci/hotplug/pciehp_hpc.c +3 -2
@@ -45,22 +45,15 @@ static inline int pciehp_request_irq(struct controller *ctrl)
 	}
 
 	/* Installs the interrupt handler */
-	retval = request_threaded_irq(irq, pciehp_isr, pciehp_ist,
-				      IRQF_SHARED, MY_NAME, ctrl);
+	retval = devm_request_threaded_irq(&ctrl->pcie->device, irq, pciehp_isr,
+					   pciehp_ist, IRQF_SHARED, MY_NAME,
+					   ctrl);

While using devm_kzalloc() for the "ops" and "ctrl" allocations is fine, using devm_request_threaded_irq() is *not* because it allows a use-after-free of the hotplug_slot_name():

With your patch the teardown order becomes:
pci_hp_del(&ctrl->hotplug_slot); # After this user space can no longer

                                         # issue enable/disable requests (but
				         # ->reset_slot() is still possible).
  pcie_disable_notification();           # After this, the IRQ thread is no
                                         # longer woken because of slot events.
  pci_hp_destroy(hotplug_slot);          # After this, hotplug_slot_name() must
                                         # no longer be called.
  cancel_delayed_work_sync(&slot->work); # After this, the IRQ thread is no
                                         # longer woken by the button_work.

What can happen here is the button_work gets scheduled before it is canceled, it wakes up the IRQ thread, the IRQ thread brings up or down the slot and prints messages to the log which include the hotplug_slot_name(), leading to a use-after-free.

This cannot happen with what's in v4.19 or on the pci/hotplug branch because the IRQ is released right after the call to pci_hp_del(). If the button_work happens to be scheduled afterwards, it will call irq_wake_thread(ctrl->pcie->irq, ctrl). Now if you look at the implementation of irq_wake_thread(), it iterates over all the actions of the IRQ (in case it's shared by multiple devices) and finds the action which matches the ctrl pointer. But because we've already released the IRQ at that point, it won't find the IRQ action and just return. So irq_wake_thread() essentially becomes a no-op.

The teardown order is very meticulously designed, I went over it dozens of times to ensure it's bullet-proof. The IRQ really has to be released at exactly the right time and using a device-managed IRQ breaks the correct order I'm afraid.

While I had thought of using devm_kzalloc(), I also knew that Bjorn wants to get rid of portdrv and I didn't want to make any changes that might constrain the future design of portdrv. Right now, portdrv creates a sub-device for each service and service drivers bind to it. An alternative design could either be a single driver that binds to ports and handles all services at once (probably wouldn't be all that different to the current situation), or no driver would be bound to ports and the functionality of the port services would be handled within the core. In either case we have the pci_dev of the port to attach the device-managed allocations to, so I suppose using devm_kzalloc() here isn't constraining us all that much, hence seems fine.

Thanks,

Lukas

#21 lore raw

Re: [PATCH 12/12] PCI/pciehp: Use device managed allocations

Quoted 41 lines · Lukas Wunner

drivers/pci/hotplug/pciehp_core.c

Quoted 9 lines · Keith Busch
drivers/pci/hotplug/pciehp_core.c +3 -2
 
 	/* Installs the interrupt handler */
-	retval = request_threaded_irq(irq, pciehp_isr, pciehp_ist,
-				      IRQF_SHARED, MY_NAME, ctrl);
+	retval = devm_request_threaded_irq(&ctrl->pcie->device, irq, pciehp_isr,
+					   pciehp_ist, IRQF_SHARED, MY_NAME,
+					   ctrl);

While using devm_kzalloc() for the "ops" and "ctrl" allocations is fine, using devm_request_threaded_irq() is *not* because it allows a use-after-free of the hotplug_slot_name():

With your patch the teardown order becomes:
pci_hp_del(&ctrl->hotplug_slot); # After this user space can no longer

                                         # issue enable/disable requests (but
                                         # ->reset_slot() is still possible).
  pcie_disable_notification();           # After this, the IRQ thread is no
                                         # longer woken because of slot events.
  pci_hp_destroy(hotplug_slot);          # After this, hotplug_slot_name() must
                                         # no longer be called.
  cancel_delayed_work_sync(&slot->work); # After this, the IRQ thread is no
                                         # longer woken by the button_work.

What can happen here is the button_work gets scheduled before it is canceled, it wakes up the IRQ thread, the IRQ thread brings up or down the slot and prints messages to the log which include the hotplug_slot_name(), leading to a use-after-free.

This cannot happen with what's in v4.19 or on the pci/hotplug branch because the IRQ is released right after the call to pci_hp_del(). If the button_work happens to be scheduled afterwards, it will call irq_wake_thread(ctrl->pcie->irq, ctrl). Now if you look at the implementation of irq_wake_thread(), it iterates over all the actions of the IRQ (in case it's shared by multiple devices) and finds the action which matches the ctrl pointer. But because we've already released the IRQ at that point, it won't find the IRQ action and just return. So irq_wake_thread() essentially becomes a no-op.

Quoted 4 lines

The teardown order is very meticulously designed, I went over it dozens of times to ensure it's bullet-proof. The IRQ really has to be released at exactly the right time and using a device-managed IRQ breaks the correct order I'm afraid.

Thanks for the very detailed analysis!

This seems like a very subtle issue that we're likely to trip over again. I assume most drivers can use device-managed IRQs safely, but it's a problem here because of the extra complexity around hotplug_slot? Is there some way we can make it less subtle, e.g., by moving the slot cleanup into the pci_destroy_dev() path or something?

Is there some value we get from having both struct hotplug_slot and struct pci_slot?

Quoted 4 lines

While I had thought of using devm_kzalloc(), I also knew that Bjorn wants to get rid of portdrv and I didn't want to make any changes that might constrain the future design of portdrv. Right now, portdrv creates a sub-device for each service and service drivers bind to it.

Quoted 8 lines

An alternative design could either be a single driver that binds to ports and handles all services at once (probably wouldn't be all that different to the current situation), or no driver would be bound to ports and the functionality of the port services would be handled within the core. In either case we have the pci_dev of the port to attach the device-managed allocations to, so I suppose using devm_kzalloc() here isn't constraining us all that much, hence seems fine.

I'd *like* to get rid of portdrv, but I don't see a clear path to do that yet, so it might be impractical. One reason I don't like it is that only one driver can bind to a device, so portdrv prevents us from having drivers for device-specific bridge features like performance monitors. For that reason, I'd like to have the architected services handled directly in the core.

Bjorn

#15 lore raw

Re: [PATCH 11/12] PCI/AER: Use managed resource allocations

In reply to Keith Busch
Quoted 3 lines · Keith Busch

This uses the managed device resource allocations for the service data so that the aer driver doesn't need to manage it, further simplifying this driver.

Just be careful (it migh be ok, I haven't audited everything, but I got bitten by something like that in the past) that the devm stuff will get disposed of in two cases:

- The owner device going away (so far so good)

- The owner device's driver being unbound

The latter is something not completely obvious, ie, even if the owner device still has held references, the successful completion of ->remove() on the driver will be followed by a cleanup of the managed stuff.

As I said, it might be ok in the AER case, but you might want to at least keep the set_service_data(dev, NULL) to make sure you don't leave a stale pointer there.

Cheers,
Ben.

Quoted 51 lines
---
 drivers/pci/pcie/aer.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/pci/pcie/aer.c +5 -12
index 1878d9d7760b..7ecad011458d 100644
@@ -1366,11 +1366,7 @@ static void aer_remove(struct pcie_device *dev)
 {
 	struct aer_rpc *rpc = get_service_data(dev);
 
-	if (rpc) {
-		aer_disable_rootport(rpc);
-		kfree(rpc);
-		set_service_data(dev, NULL);
-	}
+	aer_disable_rootport(rpc);
 }
 
 /**
@@ -1383,10 +1379,9 @@ static int aer_probe(struct pcie_device *dev)
 {
 	int status;
 	struct aer_rpc *rpc;
-	struct device *device = &dev->port->device;
+	struct device *device = &dev->device;
 
-	/* Alloc rpc data structure */
-	rpc = kzalloc(sizeof(struct aer_rpc), GFP_KERNEL);
+	rpc = devm_kzalloc(device, sizeof(struct aer_rpc), GFP_KERNEL);
 	if (!rpc) {
 		dev_printk(KERN_DEBUG, device, "alloc AER rpc failed\n");
 		return -ENOMEM;
@@ -1394,13 +1389,11 @@ static int aer_probe(struct pcie_device *dev)
 	rpc->rpd = dev->port;
 	set_service_data(dev, rpc);
 
-	/* Request IRQ ISR */
-	status = request_threaded_irq(dev->irq, aer_irq, aer_isr,
-				      IRQF_SHARED, "aerdrv", dev);
+	status = devm_request_threaded_irq(device, dev->irq, aer_irq, aer_isr,
+					   IRQF_SHARED, "aerdrv", dev);
 	if (status) {
 		dev_printk(KERN_DEBUG, device, "request AER IRQ %d failed\n",
 			   dev->irq);
-		aer_remove(dev);
 		return status;
 	}
 
#22 lore raw

Re: [PATCH 12/12] PCI/pciehp: Use device managed allocations

In reply to Bjorn Helgaas
Quoted 11 lines · Bjorn Helgaas
Quoted 9 lines · Lukas Wunner

This cannot happen with what's in v4.19 or on the pci/hotplug branch because the IRQ is released right after the call to pci_hp_del(). If the button_work happens to be scheduled afterwards, it will call irq_wake_thread(ctrl->pcie->irq, ctrl). Now if you look at the implementation of irq_wake_thread(), it iterates over all the actions of the IRQ (in case it's shared by multiple devices) and finds the action which matches the ctrl pointer. But because we've already released the IRQ at that point, it won't find the IRQ action and just return. So irq_wake_thread() essentially becomes a no-op.

Quoted 5 lines

This seems like a very subtle issue that we're likely to trip over again. I assume most drivers can use device-managed IRQs safely, but it's a problem here because of the extra complexity around hotplug_slot? Is there some way we can make it less subtle, e.g., by moving the slot cleanup into the pci_destroy_dev() path or something?

The extra complexity is caused by the various ways in which events can be triggered (Slot Status register change, sysfs, button_work) plus the deregistration of hotplug_slot. Teardown paths are always hard, I don't really see a better code solution, but I should add more code comments to make it less subtle.

Quoted 2 lines

Is there some value we get from having both struct hotplug_slot and struct pci_slot?

When Alex Chiang introduced struct pci_slot, he thought of migrating struct hotplug_slot into it. See the "migrate over time" code comment in commit f46753c5e354. (It got changed to "move here" by 0aa0f5d1084c.)

On the current pci/hotplug branch, I've embedded struct hotplug_slot in each hotplug driver's struct slot. But that doesn't preclude merging struct hotplug_slot into struct pci_slot, the hotplug drivers would just have to embed a struct pci_slot instead, or struct pci_slot would be embedded in struct hotplug_slot.

TBH the rationale behind all the different structs isn't quite clear to me, e.g. struct pci_slot is also used for non-hotplug slots it seems, see 8344b568f5bd.

(Note, Alex is no longer reachable under his hp.com address, same for his canonical.com address.)

Quoted 13 lines
Quoted 4 lines

While I had thought of using devm_kzalloc(), I also knew that Bjorn wants to get rid of portdrv and I didn't want to make any changes that might constrain the future design of portdrv. Right now, portdrv creates a sub-device for each service and service drivers bind to it.

Quoted 8 lines

An alternative design could either be a single driver that binds to ports and handles all services at once (probably wouldn't be all that different to the current situation), or no driver would be bound to ports and the functionality of the port services would be handled within the core. In either case we have the pci_dev of the port to attach the device-managed allocations to, so I suppose using devm_kzalloc() here isn't constraining us all that much, hence seems fine.

Quoted 6 lines

I'd *like* to get rid of portdrv, but I don't see a clear path to do that yet, so it might be impractical. One reason I don't like it is that only one driver can bind to a device, so portdrv prevents us from having drivers for device-specific bridge features like performance monitors. For that reason, I'd like to have the architected services handled directly in the core.

Hm, well in that case we can't use devm_*() at all, as Benjamin Herrenschmidt has just correctly pointed out, devres_release_all() is called both on driver unbind in __device_release_driver() as well as on device destruction in device_release().

Thus, if the port services are handled in the core to allow a driver to bind to the port and devm_kzalloc() is used by the port services, those allocations will be freed when that driver unbinds, and any further use of the allocations is a use-after-free.

Thanks,

Lukas

#16 lore raw

Re: [PATCH 11/12] PCI/AER: Use managed resource allocations

In reply to Benjamin Herrenschmidt
Quoted 13 lines · Benjamin Herrenschmidt
Quoted 3 lines · Keith Busch

This uses the managed device resource allocations for the service data so that the aer driver doesn't need to manage it, further simplifying this driver.

Just be careful (it migh be ok, I haven't audited everything, but I got bitten by something like that in the past) that the devm stuff will get disposed of in two cases:

- The owner device going away (so far so good)

- The owner device's driver being unbound

Quoted 8 lines

The latter is something not completely obvious, ie, even if the owner device still has held references, the successful completion of ->remove() on the driver will be followed by a cleanup of the managed stuff.

As I said, it might be ok in the AER case, but you might want to at least keep the set_service_data(dev, NULL) to make sure you don't leave a stale pointer there.

Yes, these resource methods should be considered carefully. I think we're okay here, and didn't want to set service data to NULL for a couple reasons:

1. The service data and its device are released together, so the device is already out of scope before it could hold a stale pointer.

2. It is possible the IRQ handler may be invoked after 'remove', but before the managed irq is torn down. Leaving the service data set while it is allocated removes a requirement to check for NULL on each interrupt.

#23 lore raw
In reply to Keith Busch
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

#24 lore raw
Quoted 29 lines · Bjorn Helgaas
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Quoted 5 lines

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

Sounds good, and thanks for applying!

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

#25 lore raw

[+cc arm64 folks, LKML: This conversation is about this patch:

https://lore.kernel.org/linux-pci/20180918235848.26694-3-keith.busch at intel.com

which fixes some PCIe AER error injection bugs, but also makes the error injector dependent on DYNAMIC_FTRACE_WITH_REGS, which not all arches support. Note that this question is only about the error *injection* module used for testing. It doesn't affect AER support itself.]

Quoted 36 lines · Keith Busch
Quoted 29 lines · Bjorn Helgaas
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Quoted 5 lines

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

Quoted 8 lines

Sounds good, and thanks for applying!

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

Oh, indeed, I hadn't noticed this arch dependency. AFAICT, the new DYNAMIC_FTRACE_WITH_REGS dependency means aer_inject will work only on these arches:

  arm		# if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU
  powerpc	# if PPC64 && CPU_LITTLE_ENDIAN
  riscv		# ARCH_RV64I only
  s390
  x86

Notably missing is arm64, which has DYNAMIC_FTRACE but not DYNAMIC_FTRACE_WITH_REGS.

Bjorn

#26 lore raw
Quoted 65 lines · Bjorn Helgaas

[+cc arm64 folks, LKML: This conversation is about this patch:

https://lore.kernel.org/linux-pci/[email protected]

which fixes some PCIe AER error injection bugs, but also makes the error injector dependent on DYNAMIC_FTRACE_WITH_REGS, which not all arches support. Note that this question is only about the error *injection* module used for testing. It doesn't affect AER support itself.]

Quoted 36 lines · Keith Busch
Quoted 29 lines · Bjorn Helgaas
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Quoted 5 lines

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

Quoted 8 lines

Sounds good, and thanks for applying!

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

Oh, indeed, I hadn't noticed this arch dependency. AFAICT, the new DYNAMIC_FTRACE_WITH_REGS dependency means aer_inject will work only on these arches:

  arm		# if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU
  powerpc	# if PPC64 && CPU_LITTLE_ENDIAN
  riscv		# ARCH_RV64I only
  s390
  x86
Quoted 4 lines

Notably missing is arm64, which has DYNAMIC_FTRACE but not DYNAMIC_FTRACE_WITH_REGS.

Bjorn

Looks like the kbuild bot found an ARM kernel config that has DYNAMIC_FTRACE_WITH_REGS set, and then the module can't compile there. I'll need to update this patch regardless, and I think the right thing to do is maintain the "old" way with conditional compiling for any arch specific features.

#27 lore raw
Quoted 69 lines · Keith Busch
Quoted 68 lines · Bjorn Helgaas

[+cc arm64 folks, LKML: This conversation is about this patch:

https://lore.kernel.org/linux-pci/20180918235848.26694-3-keith.busch at intel.com

which fixes some PCIe AER error injection bugs, but also makes the error injector dependent on DYNAMIC_FTRACE_WITH_REGS, which not all arches support. Note that this question is only about the error *injection* module used for testing. It doesn't affect AER support itself.]

Quoted 36 lines · Keith Busch
Quoted 29 lines · Bjorn Helgaas
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Quoted 5 lines

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

Quoted 8 lines

Sounds good, and thanks for applying!

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

Oh, indeed, I hadn't noticed this arch dependency. AFAICT, the new DYNAMIC_FTRACE_WITH_REGS dependency means aer_inject will work only on these arches:

  arm		# if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU
  powerpc	# if PPC64 && CPU_LITTLE_ENDIAN
  riscv		# ARCH_RV64I only
  s390
  x86

Notably missing is arm64, which has DYNAMIC_FTRACE but not DYNAMIC_FTRACE_WITH_REGS.

Quoted 7 lines
Quoted 1 line

Bjorn

Looks like the kbuild bot found an ARM kernel config that has DYNAMIC_FTRACE_WITH_REGS set, and then the module can't compile there. I'll need to update this patch regardless, and I think the right thing to do is maintain the "old" way with conditional compiling for any arch specific features.

Sounds messy, but probably the best route.

I dropped these patches for now:

PCI/AER: Covertly inject errors with ftrace hooks PCI/AER: Reuse existing pcie_port_find_device() interface PCI/AER: Abstract AER interrupt handling

#32 lore raw
In reply to Bjorn Helgaas

Hi Bjorn,

Quoted 21 lines · Bjorn Helgaas

[+cc arm64 folks, LKML: This conversation is about this patch:

https://lore.kernel.org/linux-pci/20180918235848.26694-3-keith.busch at intel.com

which fixes some PCIe AER error injection bugs, but also makes the error injector dependent on DYNAMIC_FTRACE_WITH_REGS, which not all arches support. Note that this question is only about the error *injection* module used for testing. It doesn't affect AER support itself.]

Quoted 6 lines · Keith Busch

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

Oh, indeed, I hadn't noticed this arch dependency. AFAICT, the new DYNAMIC_FTRACE_WITH_REGS dependency means aer_inject will work only on these arches:

Quoted 8 lines
  arm		# if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU
  powerpc	# if PPC64 && CPU_LITTLE_ENDIAN
  riscv		# ARCH_RV64I only
  s390
  x86

Notably missing is arm64, which has DYNAMIC_FTRACE but not DYNAMIC_FTRACE_WITH_REGS.

Thanks for the heads-up here. This feature is currently in development for arm64:

http://lkml.kernel.org/r/20181001141648.1DBED68BDF at newverein.lst.de

but the latest review comments suggest that it's a fair way from ready yet, so I wouldn't hold your breath.

Will

#28 lore raw
In reply to Keith Busch
Quoted 69 lines · Keith Busch
Quoted 68 lines · Bjorn Helgaas

[+cc arm64 folks, LKML: This conversation is about this patch:

https://lore.kernel.org/linux-pci/20180918235848.26694-3-keith.busch at intel.com

which fixes some PCIe AER error injection bugs, but also makes the error injector dependent on DYNAMIC_FTRACE_WITH_REGS, which not all arches support. Note that this question is only about the error *injection* module used for testing. It doesn't affect AER support itself.]

Quoted 36 lines · Keith Busch
Quoted 29 lines · Bjorn Helgaas
Quoted 18 lines · Keith Busch

I ran into a lot of trouble testing error handling, and this series is just trying to simplify some things. The first 4 fix up aer_inject, and the rest are cleanup to make better use of kernel APIs.

Keith Busch (12):
PCI: Set PCI bus accessors to noinline PCI/AER: Covertly inject errors
PCI/AER: Reuse existing service device lookup PCI/AER: Abstract AER interrupt handling PCI/AER: Remove dead code
PCI/AER: Remove error source from aer struct PCI/AER: Use kfifo for tracking events PCI/AER: Use kfifo helper inserting locked elements PCI/AER: Don't read upstream ports below fatal errors PCI/AER: Use threaded IRQ for bottom half PCI/AER: Use managed resource allocations PCI/pciehp: Use device managed allocations

Quoted 9 lines
 drivers/pci/access.c              |   4 +-
 drivers/pci/hotplug/pciehp_core.c |  14 +-
 drivers/pci/hotplug/pciehp_hpc.c  |  48 ++----
 drivers/pci/pcie/Kconfig          |   2 +-
 drivers/pci/pcie/aer.c            | 219 ++++++---------------------
 drivers/pci/pcie/aer_inject.c     | 306 ++++++++++++++++++++------------------
 drivers/pci/pcie/portdrv.h        |   4 -
 drivers/pci/pcie/portdrv_core.c   |   1 +
 8 files changed, 227 insertions(+), 371 deletions(-)
Quoted 5 lines

Thanks a lot for doing this! I applied these to pci/hotplug for v4.20, except for "PCI/AER: Don't read upstream ports below fatal errors", which seems to be already there via another posting, and "PCI/pciehp: Use device managed allocations", which needs a few tweaks.

Quoted 8 lines

Sounds good, and thanks for applying!

In case this went unnoticed, patch 2's aer_inject using ftrace hooks to pci config accessors is really cool and fixes several kernel crashes I encountered, but it may not work on every architecture. I'm not sure how widely aer_inject is used, so maybe there are no concerns with the DYNAMIC_FTRACE_WITH_REGS dependency, but I just want to reemphasize that dependency in case there are valid objections.

Oh, indeed, I hadn't noticed this arch dependency. AFAICT, the new DYNAMIC_FTRACE_WITH_REGS dependency means aer_inject will work only on these arches:

  arm		# if (!XIP_KERNEL) && !CPU_ENDIAN_BE32 && MMU
  powerpc	# if PPC64 && CPU_LITTLE_ENDIAN
  riscv		# ARCH_RV64I only
  s390
  x86

Notably missing is arm64, which has DYNAMIC_FTRACE but not DYNAMIC_FTRACE_WITH_REGS.

Quoted 7 lines
Quoted 1 line

Bjorn

Looks like the kbuild bot found an ARM kernel config that has DYNAMIC_FTRACE_WITH_REGS set, and then the module can't compile there. I'll need to update this patch regardless, and I think the right thing to do is maintain the "old" way with conditional compiling for any arch specific features.

The question is whether we really need to dynamically patch the kernel with ftrace to achieve what that patch does.

Furthermore, it would also be good to report what bugs we are actually fixing, from what you are writing falling back to the current method if !DYNAMIC_FTRACE_WITH_REGS is broken in many ways and I would start with fixing the current behaviour with something that does not depend on arch features that may not even be implemented.

Anyway - please CC me in in the follow up series, I am happy to help you test it.

Thanks,
Lorenzo

#29 lore raw
Quoted 8 lines · Lorenzo Pieralisi

The question is whether we really need to dynamically patch the kernel with ftrace to achieve what that patch does.

Furthermore, it would also be good to report what bugs we are actually fixing, from what you are writing falling back to the current method if !DYNAMIC_FTRACE_WITH_REGS is broken in many ways and I would start with fixing the current behaviour with something that does not depend on arch features that may not even be implemented.

There are two problems with the current method:

1. It may dereference pci_dev after it was freed
2. The pci_dev's children inherit its fake pci_bus's ops on enumeration

Both result in kernel panic.

The dynamic kernel patch just seemed like a cool way to inject errors without messing with the driver's structures. But if there's a more elegant way to do it, I'm all for it.

#30 lore raw
Quoted 10 lines · Keith Busch
Quoted 8 lines · Lorenzo Pieralisi

The question is whether we really need to dynamically patch the kernel with ftrace to achieve what that patch does.

Furthermore, it would also be good to report what bugs we are actually fixing, from what you are writing falling back to the current method if !DYNAMIC_FTRACE_WITH_REGS is broken in many ways and I would start with fixing the current behaviour with something that does not depend on arch features that may not even be implemented.

Quoted 7 lines

There are two problems with the current method:

1. It may dereference pci_dev after it was freed
2. The pci_dev's children inherit its fake pci_bus's ops on enumeration

Both result in kernel panic.

That's my point, current test module is not robust, I wanted to ask if there is a way to fix it that does not depend on arch features, because if there is a dependency that is not met we are still not fixing the current code, using it as a fallback can still create issues.

Quoted 3 lines

The dynamic kernel patch just seemed like a cool way to inject errors without messing with the driver's structures. But if there's a more elegant way to do it, I'm all for it.

If you have a simple reproducer for the bugs I am happy to help you test it (I can also apply arm64 DYNAMIC_FTRACE_WITH_REGS patches and test that new code path if that's the final direction we are taking).

Thanks,
Lorenzo

#31 lore raw
Quoted 3 lines · Lorenzo Pieralisi

If you have a simple reproducer for the bugs I am happy to help you test it (I can also apply arm64 DYNAMIC_FTRACE_WITH_REGS patches and test that new code path if that's the final direction we are taking).

The easiest way to reproduce is load the aer_inject module, inject an error into a bridge, then remove the bridge and rescan.