Skip to contentSkip to conversation

Support managed interrupts for platform devices

PATCH Superseded John Garry 3 participants 11 messages Nov 30, 2020 – Dec 1, 2020

Subsystems (5)
SCSI ACPI Driver core IRQ +1 more subsystems

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

John Garry

author
#0 lore raw

v4 cover Support managed interrupts for platform devices

So far, managed interrupts are only used for PCI MSIs. This series adds platform device support for managed interrupts. Initially this topic was discussed at [0].

The method to enable managed interrupts is to allocate a group of IRQs for the device, and then switch the interrupts to managed - this is done through new function irq_update_affinity_desc().

Function devm_platform_get_irqs_affinity() is added as a helper to manage this work, such that we don't need to export irq_update_affinity_desc() or irq_create_affinity_masks().

In the devm_platform_get_irqs_affinity() release call a new platform method is used to "put" an irq. The reason for this is that per-irq mapping (and irq_desc) needs to be recreated anew for re-probing the LLDD, such that we don't attempt to reconfigure the managed or any other flag for an irq_desc.

For now, the HiSilicon SAS v2 hw driver is switched over. This is used in the D05 dev board.

Performance gain observed for 6x SAS SSDs is ~357K -> 420K IOPs for fio read.

This series is tested based on Marc's "MSI: Track device proxying when allocating MSIs" series, [1].

[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/

Changes since v3:
- Fix genirq change to re-activate interrupt if we have deactivated it
- Remove standalone platform_put_irq(), and combine code into devm_platform_get_irqs_affinity_release()
- Add new inline function in ioport.h rather than making acpi_dev_irqresource_disabled() public

Changes since v2:
- Update genirq change as follows:
- Handle when the irq is started, active, or already managed
- Reject update when CONFIG_GENERIC_IRQ_RESERVATION_MODE is set
- Revamp platform.c API as follows:
- Make it devm type
- Add platform_put_irq() and associated change in ACPI code to allow irq resource to be reset
- Unmap irqs for driver removal
- Change API to accept min and max vectors

John Garry (5):
genirq/affinity: Add irq_update_affinity_desc()
resource: Add irqresource_disabled()
ACPI: Drop acpi_dev_irqresource_disabled()
Driver core: platform: Add devm_platform_get_irqs_affinity()
scsi: hisi_sas: Expose HW queues for v2 hw

 drivers/acpi/resource.c                |  17 +---
 drivers/base/platform.c                | 121 +++++++++++++++++++++++++
 drivers/scsi/hisi_sas/hisi_sas.h       |   4 +
 drivers/scsi/hisi_sas/hisi_sas_main.c  |  11 +++
 drivers/scsi/hisi_sas/hisi_sas_v2_hw.c |  66 +++++++++++---
 include/linux/interrupt.h              |   8 ++
 include/linux/ioport.h                 |   7 ++
 include/linux/platform_device.h        |   6 ++
 kernel/irq/manage.c                    |  63 +++++++++++++
 9 files changed, 278 insertions(+), 25 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v4 1/5 genirq/affinity: Add irq_update_affinity_desc()

Add a function to allow the affinity of an interrupt be switched to managed, such that interrupts allocated for platform devices may be managed.

---
 include/linux/interrupt.h |  8 +++++
 kernel/irq/manage.c       | 63 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)
include/linux/interrupt.h +8 -0
index ee8299eb1f52..870b3251e174 100644
@@ -352,6 +352,8 @@ extern int irq_can_set_affinity(unsigned int irq);
 extern int irq_select_affinity(unsigned int irq);
 
 extern int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m);
+extern int irq_update_affinity_desc(unsigned int irq,
+				    struct irq_affinity_desc *affinity);
 
 extern int
 irq_set_affinity_notifier(unsigned int irq, struct irq_affinity_notify *notify);
@@ -387,6 +389,12 @@ static inline int irq_set_affinity_hint(unsigned int irq,
 	return -EINVAL;
 }
 
+static inline int irq_update_affinity_desc(unsigned int irq,
+					   struct irq_affinity_desc *affinity)
+{
+	return -EINVAL;
+}
+
 static inline int
 irq_set_affinity_notifier(unsigned int irq, struct irq_affinity_notify *notify)
 {
kernel/irq/manage.c +63 -0

69 lines not on this page · raw message

Signature · 1 line
2.26.2
#2 lore raw

Re: [PATCH v4 1/5] genirq/affinity: Add irq_update_affinity_desc()

Quoted 3 lines · John Garry

Add a function to allow the affinity of an interrupt be switched to managed, such that interrupts allocated for platform devices may be managed.

Could you please add a paragraph which explains the limitations of that interface?

Quoted 7 lines

+/**
+ * irq_update_affinity_desc - Update affinity management for an interrupt
+ * @irq: The interrupt number to update
+ * @affinity: Pointer to the affinity descriptor
+ *
+ * This interface can be used to configure the affinity management of
+ * interrupts which have been allocated already.

Same here.

Thanks,

tglx

author
#3 lore raw

Re: [PATCH v4 1/5] genirq/affinity: Add irq_update_affinity_desc()

Hi Thomas,

I'll go with something extra like:

Quoted 7 lines
Quoted 4 lines

Add a function to allow the affinity of an interrupt be switched to managed, such that interrupts allocated for platform devices may be managed.

Could you please add a paragraph which explains the limitations of that interface?

This new interface has certain limitations, and attempts to use it in the following circumstances will fail:
- For when the kernel is configured for generic IRQ reservation mode (in config GENERIC_IRQ_RESERVATION_MODE). The reason being that it could conflict with managed vs. non-managed interrupt accounting there.
- The interrupt is already started, which should not be the case during init
- The interrupt is already configured as managed, which means double init

Quoted 9 lines
Quoted 8 lines

+/**
+ * irq_update_affinity_desc - Update affinity management for an interrupt
+ * @irq: The interrupt number to update
+ * @affinity: Pointer to the affinity descriptor
+ *
+ * This interface can be used to configure the affinity management of
+ * interrupts which have been allocated already.

/*
* There are certain limitations on when it may be used - attempts to
use it for when the kernel is configured for generic IRQ reservation mode (in config GENERIC_IRQ_RESERVATION_MODE) will fail, as it may conflict with managed/non-managed interrupt accounting. In addition, attempts to use it on an interrupt which is already started or which has already been configured as managed will also fail, as these mean invalid init state or double init.

...

Let me know if not good, if not I'll post again soon.

Thanks,
John

#4 lore raw

Re: [PATCH v4 1/5] genirq/affinity: Add irq_update_affinity_desc()

Quoted 9 lines · John Garry

/*
* There are certain limitations on when it may be used - attempts to
use it for when the kernel is configured for generic IRQ reservation mode (in config GENERIC_IRQ_RESERVATION_MODE) will fail, as it may conflict with managed/non-managed interrupt accounting. In addition, attempts to use it on an interrupt which is already started or which has already been configured as managed will also fail, as these mean invalid init state or double init.

Quoted 3 lines

...

Let me know if not good, if not I'll post again soon.

LGTM

author
#5 lore raw

PATCH v4 2/5 resource: Add irqresource_disabled()

In reply to John Garry

Add a common function to set the fields for a irq resource to disabled, which mimics what is done in acpi_dev_irqresource_disabled(), with a view to replace that function.

---
 include/linux/ioport.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/ioport.h +7 -0
index 5135d4b86cd6..f9bf374f9633 100644
@@ -307,6 +307,13 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 struct resource *request_free_mem_region(struct resource *base,
 		unsigned long size, const char *name);
 
+static inline void irqresource_disabled(struct resource *res, u32 irq)
+{
+	res->start = irq;
+	res->end = irq;
+	res->flags = IORESOURCE_IRQ | IORESOURCE_DISABLED | IORESOURCE_UNSET;
+}
+
 #ifdef CONFIG_IO_STRICT_DEVMEM
 void revoke_devmem(struct resource *res);
 #else
Signature · 1 line
2.26.2
maintainer Reviewed-by on v4 2/5
#6 lore raw

Re: [PATCH v4 2/5] resource: Add irqresource_disabled()

Quoted 6 lines · John Garry

Add a common function to set the fields for a irq resource to disabled, which mimics what is done in acpi_dev_irqresource_disabled(), with a view to replace that function.

Quoted 25 lines
---
 include/linux/ioport.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/ioport.h +7 -1
index 5135d4b86cd6..f9bf374f9633 100644
@@ -307,6 +307,13 @@ struct resource *devm_request_free_mem_region(struct device *dev,
 struct resource *request_free_mem_region(struct resource *base,
                unsigned long size, const char *name);
 
+static inline void irqresource_disabled(struct resource *res, u32 irq)
+{
+       res->start = irq;
+       res->end = irq;
+       res->flags = IORESOURCE_IRQ | IORESOURCE_DISABLED | IORESOURCE_UNSET;
+}
+
 #ifdef CONFIG_IO_STRICT_DEVMEM
 void revoke_devmem(struct resource *res);
 #else
--

2.26.2

author
#7 lore raw

PATCH v4 3/5 ACPI: Drop acpi_dev_irqresource_disabled()

In reply to John Garry

The functionality of acpi_dev_irqresource_disabled() is same as in common irqresource_disabled(), so drop acpi_dev_irqresource_disabled() in favour of that function.

---
 drivers/acpi/resource.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/acpi/resource.c +5 -12
index ad04824ca3ba..58203193417e 100644
@@ -380,13 +380,6 @@ unsigned int acpi_dev_get_irq_type(int triggering, int polarity)
 }
 EXPORT_SYMBOL_GPL(acpi_dev_get_irq_type);
 
-static void acpi_dev_irqresource_disabled(struct resource *res, u32 gsi)
-{
-	res->start = gsi;
-	res->end = gsi;
-	res->flags = IORESOURCE_IRQ | IORESOURCE_DISABLED | IORESOURCE_UNSET;
-}
-
 static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
 				     u8 triggering, u8 polarity, u8 shareable,
 				     bool legacy)
@@ -394,7 +387,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
 	int irq, p, t;
 
 	if (!valid_IRQ(gsi)) {
-		acpi_dev_irqresource_disabled(res, gsi);
+		irqresource_disabled(res, gsi);
 		return;
 	}
 
@@ -426,7 +419,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
 		res->start = irq;
 		res->end = irq;
 	} else {
-		acpi_dev_irqresource_disabled(res, gsi);
+		irqresource_disabled(res, gsi);
 	}
 }
 
@@ -463,7 +456,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
 		 */
 		irq = &ares->data.irq;
 		if (index >= irq->interrupt_count) {
-			acpi_dev_irqresource_disabled(res, 0);
+			irqresource_disabled(res, 0);
 			return false;
 		}
 		acpi_dev_get_irqresource(res, irq->interrupts[index],
@@ -473,7 +466,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
 	case ACPI_RESOURCE_TYPE_EXTENDED_IRQ:
 		ext_irq = &ares->data.extended_irq;
 		if (index >= ext_irq->interrupt_count) {
-			acpi_dev_irqresource_disabled(res, 0);
+			irqresource_disabled(res, 0);
 			return false;
 		}
 		if (is_gsi(ext_irq))
@@ -481,7 +474,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
 					 ext_irq->triggering, ext_irq->polarity,
 					 ext_irq->shareable, false);
 		else
-			acpi_dev_irqresource_disabled(res, 0);
+			irqresource_disabled(res, 0);
 		break;
 	default:
 		res->flags = 0;
Signature · 1 line
2.26.2
maintainer Acked-by on v4 3/5
#8 lore raw

Re: [PATCH v4 3/5] ACPI: Drop acpi_dev_irqresource_disabled()

Quoted 6 lines · John Garry

The functionality of acpi_dev_irqresource_disabled() is same as in common irqresource_disabled(), so drop acpi_dev_irqresource_disabled() in favour of that function.

Quoted 70 lines
---
 drivers/acpi/resource.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/acpi/resource.c +5 -13
index ad04824ca3ba..58203193417e 100644
@@ -380,13 +380,6 @@ unsigned int acpi_dev_get_irq_type(int triggering, int polarity)
 }
 EXPORT_SYMBOL_GPL(acpi_dev_get_irq_type);
 
-static void acpi_dev_irqresource_disabled(struct resource *res, u32 gsi)
-{
-       res->start = gsi;
-       res->end = gsi;
-       res->flags = IORESOURCE_IRQ | IORESOURCE_DISABLED | IORESOURCE_UNSET;
-}
-
 static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
                                     u8 triggering, u8 polarity, u8 shareable,
                                     bool legacy)
@@ -394,7 +387,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
        int irq, p, t;
 
        if (!valid_IRQ(gsi)) {
-               acpi_dev_irqresource_disabled(res, gsi);
+               irqresource_disabled(res, gsi);
                return;
        }
 
@@ -426,7 +419,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,
                res->start = irq;
                res->end = irq;
        } else {
-               acpi_dev_irqresource_disabled(res, gsi);
+               irqresource_disabled(res, gsi);
        }
 }
 
@@ -463,7 +456,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
                 */
                irq = &ares->data.irq;
                if (index >= irq->interrupt_count) {
-                       acpi_dev_irqresource_disabled(res, 0);
+                       irqresource_disabled(res, 0);
                        return false;
                }
                acpi_dev_get_irqresource(res, irq->interrupts[index],
@@ -473,7 +466,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
        case ACPI_RESOURCE_TYPE_EXTENDED_IRQ:
                ext_irq = &ares->data.extended_irq;
                if (index >= ext_irq->interrupt_count) {
-                       acpi_dev_irqresource_disabled(res, 0);
+                       irqresource_disabled(res, 0);
                        return false;
                }
                if (is_gsi(ext_irq))
@@ -481,7 +474,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,
                                         ext_irq->triggering, ext_irq->polarity,
                                         ext_irq->shareable, false);
                else
-                       acpi_dev_irqresource_disabled(res, 0);
+                       irqresource_disabled(res, 0);
                break;
        default:
                res->flags = 0;
--

2.26.2

author
#9 lore raw

PATCH v4 4/5 Driver core: platform: Add devm_platform_get_irqs_affinity()

In reply to John Garry

Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.

The function accepts an affinity descriptor pointer, which covers all IRQs expected for the device.

The function is devm class as the only current in-tree user will also use devm method for requesting the interrupts; as such, the function is made as devm as it can ensure ordering of freeing the irq and disposing of the mapping.

---
 drivers/base/platform.c         | 121 ++++++++++++++++++++++++++++++++
 include/linux/platform_device.h |   6 ++
 2 files changed, 127 insertions(+)
drivers/base/platform.c +121 -0
index 88aef93eb4dd..ea8add164b89 100644
@@ -15,6 +15,8 @@
 #include <linux/of_irq.h>
 #include <linux/module.h>
 #include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/ioport.h>
 #include <linux/dma-mapping.h>
 #include <linux/memblock.h>
 #include <linux/err.h>
@@ -289,6 +291,125 @@ int platform_irq_count(struct platform_device *dev)
 }
 EXPORT_SYMBOL_GPL(platform_irq_count);
 
+struct irq_affinity_devres {
+	unsigned int count;
+	unsigned int irq[];
+};
+
+static void platform_disable_acpi_irq(struct platform_device *pdev, int index)
+{
+	struct resource *r;
+
+	r = platform_get_resource(pdev, IORESOURCE_IRQ, index);
+	if (r)
+		irqresource_disabled(r, 0);
+}
+
+static void devm_platform_get_irqs_affinity_release(struct device *dev,
+						    void *res)
+{
+	struct irq_affinity_devres *ptr = res;
+	int i;
+
+	for (i = 0; i < ptr->count; i++) {
+		irq_dispose_mapping(ptr->irq[i]);
+
+		if (has_acpi_companion(dev))
+			platform_disable_acpi_irq(to_platform_device(dev), i);
+	}
+}
+
+/**
+ * devm_platform_get_irqs_affinity - devm method to get a set of IRQs for a
+ *				device using an interrupt affinity descriptor
+ * @dev: platform device pointer
+ * @affd: affinity descriptor
+ * @minvec: minimum count of interrupt vectors
+ * @maxvec: maximum count of interrupt vectors
+ * @irqs: pointer holder for IRQ numbers
+ *
+ * Gets a set of IRQs for a platform device, and updates IRQ afffinty according
+ * to the passed affinity descriptor
+ *
+ * Return: Number of vectors on success, negative error number on failure.
+ */
+int devm_platform_get_irqs_affinity(struct platform_device *dev,
+				    struct irq_affinity *affd,
+				    unsigned int minvec,
+				    unsigned int maxvec,
+				    int **irqs)
+{
+	struct irq_affinity_devres *ptr;
+	struct irq_affinity_desc *desc;
+	size_t size;
+	int i, ret, nvec;
+
+	if (!affd)
+		return -EPERM;
+
+	if (maxvec < minvec)
+		return -ERANGE;
+
+	nvec = platform_irq_count(dev);
+
+	if (nvec < minvec)
+		return -ENOSPC;
+
+	nvec = irq_calc_affinity_vectors(minvec, nvec, affd);
+	if (nvec < minvec)
+		return -ENOSPC;
+
+	if (nvec > maxvec)
+		nvec = maxvec;
+
+	size = sizeof(*ptr) + sizeof(unsigned int) * nvec;
+	ptr = devres_alloc(devm_platform_get_irqs_affinity_release, size,
+			   GFP_KERNEL);
+	if (!ptr)
+		return -ENOMEM;
+
+	ptr->count = nvec;
+
+	for (i = 0; i < nvec; i++) {
+		int irq = platform_get_irq(dev, i);
+		if (irq < 0) {
+			ret = irq;
+			goto err_free_devres;
+		}
+		ptr->irq[i] = irq;
+	}
+
+	desc = irq_create_affinity_masks(nvec, affd);
+	if (!desc) {
+		ret = -ENOMEM;
+		goto err_free_devres;
+	}
+
+	for (i = 0; i < nvec; i++) {
+		ret = irq_update_affinity_desc(ptr->irq[i], &desc[i]);
+		if (ret) {
+			dev_err(&dev->dev, "failed to update irq%d affinity descriptor (%d)\n",
+				ptr->irq[i], ret);
+			goto err_free_desc;
+		}
+	}
+
+	devres_add(&dev->dev, ptr);
+
+	kfree(desc);
+
+	*irqs = ptr->irq;
+
+	return nvec;
+
+err_free_desc:
+	kfree(desc);
+err_free_devres:
+	devres_free(ptr);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(devm_platform_get_irqs_affinity);
+
 /**
  * platform_get_resource_byname - get a resource for a device by name
  * @dev: platform device
include/linux/platform_device.h +6 -0
index 77a2aada106d..4d75633e6735 100644
@@ -15,6 +15,7 @@
 #define PLATFORM_DEVID_NONE	(-1)
 #define PLATFORM_DEVID_AUTO	(-2)
 
+struct irq_affinity;
 struct mfd_cell;
 struct property_entry;
 struct platform_device_id;
@@ -70,6 +71,11 @@ devm_platform_ioremap_resource_byname(struct platform_device *pdev,
 extern int platform_get_irq(struct platform_device *, unsigned int);
 extern int platform_get_irq_optional(struct platform_device *, unsigned int);
 extern int platform_irq_count(struct platform_device *);
+extern int devm_platform_get_irqs_affinity(struct platform_device *dev,
+					   struct irq_affinity *affd,
+					   unsigned int minvec,
+					   unsigned int maxvec,
+					   int **irqs);
 extern struct resource *platform_get_resource_byname(struct platform_device *,
 						     unsigned int,
 						     const char *);
Signature · 1 line
2.26.2
author
#10 lore raw

PATCH v4 5/5 scsi: hisi_sas: Expose HW queues for v2 hw

In reply to John Garry

As a performance enhancement, make the completion queue interrupts managed.

In addition, in commit bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx are offline"), CPU hotplug for MQ devices using managed interrupts is made safe. So expose HW queues to blk-mq to take advantage of this.

Flag Scsi_host.host_tagset is also set to ensure that the HBA is not sent more commands than it can handle. However the driver still does not use request tag for IPTT as there are many HW bugs means that special rules apply for IPTT allocation.

---
 drivers/scsi/hisi_sas/hisi_sas.h       |  4 ++
 drivers/scsi/hisi_sas/hisi_sas_main.c  | 11 +++++
 drivers/scsi/hisi_sas/hisi_sas_v2_hw.c | 66 +++++++++++++++++++++-----
 3 files changed, 68 insertions(+), 13 deletions(-)
drivers/scsi/hisi_sas/hisi_sas.h +4 -0
index a25cfc11c96d..aa67807c5693 100644
@@ -14,6 +14,7 @@
 #include <linux/debugfs.h>
 #include <linux/dmapool.h>
 #include <linux/iopoll.h>
+#include <linux/irq.h>
 #include <linux/lcm.h>
 #include <linux/libata.h>
 #include <linux/mfd/syscon.h>
@@ -312,6 +313,7 @@ enum {
 
 struct hisi_sas_hw {
 	int (*hw_init)(struct hisi_hba *hisi_hba);
+	int (*interrupt_preinit)(struct hisi_hba *hisi_hba);
 	void (*setup_itct)(struct hisi_hba *hisi_hba,
 			   struct hisi_sas_device *device);
 	int (*slot_index_alloc)(struct hisi_hba *hisi_hba,
@@ -418,6 +420,8 @@ struct hisi_hba {
 	u32 refclk_frequency_mhz;
 	u8 sas_addr[SAS_ADDR_SIZE];
 
+	int *irq_map; /* v2 hw */
+
 	int n_phy;
 	spinlock_t lock;
 	struct semaphore sem;
drivers/scsi/hisi_sas/hisi_sas_main.c +11 -0
index c8dd8588f800..624c5ec723fb 100644
@@ -2614,6 +2614,13 @@ static struct Scsi_Host *hisi_sas_shost_alloc(struct platform_device *pdev,
 	return NULL;
 }
 
+static int hisi_sas_interrupt_preinit(struct hisi_hba *hisi_hba)
+{
+	if (hisi_hba->hw->interrupt_preinit)
+		return hisi_hba->hw->interrupt_preinit(hisi_hba);
+	return 0;
+}
+
 int hisi_sas_probe(struct platform_device *pdev,
 		   const struct hisi_sas_hw *hw)
 {
@@ -2671,6 +2678,10 @@ int hisi_sas_probe(struct platform_device *pdev,
 		sha->sas_port[i] = &hisi_hba->port[i].sas_port;
 	}
 
+	rc = hisi_sas_interrupt_preinit(hisi_hba);
+	if (rc)
+		goto err_out_ha;
+
 	rc = scsi_add_host(shost, &pdev->dev);
 	if (rc)
 		goto err_out_ha;
drivers/scsi/hisi_sas/hisi_sas_v2_hw.c +53 -13

127 lines not on this page · raw message

Signature · 1 line
2.26.2