Support managed interrupts for platform devices
Subsystems (5)
v5 cover Support managed interrupts for platform devices
So far, managed interrupts are only used for PCI MSIs. This series adds platform device support for managed interrupts. Initially this topic was discussed at [0].
The method to enable managed interrupts is to allocate a group of IRQs for the device, and then switch the interrupts to managed - this is done through new function irq_update_affinity_desc().
Function devm_platform_get_irqs_affinity() is added as a helper to manage this work, such that we don't need to export irq_update_affinity_desc() or irq_create_affinity_masks().
In the devm_platform_get_irqs_affinity() release call we are required to "put" or unmap the irqs which have been setup. The reason for this is that per-irq mapping (and irq_desc) needs to be recreated anew for re-probing the LLDD, such that we don't attempt to reconfigure the managed or any other flag for an irq_desc.
For now, the HiSilicon SAS v2 hw driver is switched over. This is used in the D05 dev board.
Performance gain observed for 6x SAS SSDs is ~357K -> 420K IOPs for fio read.
This series is tested based on Marc's "MSI: Track device proxying when allocating MSIs" series, [1].
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
Changes since v4:
- Add Rafael's tags (thanks)
- Expand commit message and code comment for genirq change
Changes since v3:
- Fix genirq change to re-activate interrupt if we have deactivated it
- Remove standalone platform_put_irq(), and combine code into devm_platform_get_irqs_affinity_release()
- Add new inline function in ioport.h rather than making acpi_dev_irqresource_disabled() public
Changes since v2:
- Update genirq change as follows:
- Handle when the irq is started, active, or already managed
- Reject update when CONFIG_GENERIC_IRQ_RESERVATION_MODE is set
- Revamp platform.c API as follows:
- Make it devm type
- Add platform_put_irq() and associated change in ACPI code to allow irq resource to be reset
- Unmap irqs for driver removal
- Change API to accept min and max vectors
John Garry (5):
genirq/affinity: Add irq_update_affinity_desc()
resource: Add irqresource_disabled()
ACPI: Drop acpi_dev_irqresource_disabled()
Driver core: platform: Add devm_platform_get_irqs_affinity()
scsi: hisi_sas: Expose HW queues for v2 hw
drivers/acpi/resource.c | 17 +--- drivers/base/platform.c | 121 +++++++++++++++++++++++++ drivers/scsi/hisi_sas/hisi_sas.h | 4 + drivers/scsi/hisi_sas/hisi_sas_main.c | 11 +++ drivers/scsi/hisi_sas/hisi_sas_v2_hw.c | 66 +++++++++++--- include/linux/interrupt.h | 8 ++ include/linux/ioport.h | 7 ++ include/linux/platform_device.h | 6 ++ kernel/irq/manage.c | 70 ++++++++++++++ 9 files changed, 285 insertions(+), 25 deletions(-)
Signature · 1 line
2.26.2
PATCH v5 1/5 genirq/affinity: Add irq_update_affinity_desc()
Add a function to allow the affinity of an interrupt be switched to managed, such that interrupts allocated for platform devices may be managed.
This new interface has certain limitations, and attempts to use it in the following circumstances will fail:
- For when the kernel is configured for generic IRQ reservation mode (in config GENERIC_IRQ_RESERVATION_MODE). The reason being that it could conflict with managed vs. non-managed interrupt accounting.
- The interrupt is already started, which should not be the case during init
- The interrupt is already configured as managed, which means double init
--- include/linux/interrupt.h | 8 +++++ kernel/irq/manage.c | 70 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+)
include/linux/interrupt.h +8 -0
@@ -352,6 +352,8 @@ extern int irq_can_set_affinity(unsigned int irq);extern int irq_select_affinity (unsigned int irq);extern int irq_set_affinity_hint (unsigned int irq,const struct cpumask *m); +extern int irq_update_affinity_desc (unsigned int irq, +struct irq_affinity_desc *affinity);extern int irq_set_affinity_notifier (unsigned int irq,struct irq_affinity_notify *notify);
@@ -387,6 +389,12 @@ static inline int irq_set_affinity_hint(unsigned int irq,return -EINVAL ; } +static inline int irq_update_affinity_desc (unsigned int irq, +struct irq_affinity_desc *affinity) +{ +return -EINVAL ; +} +static inline int irq_set_affinity_notifier (unsigned int irq,struct irq_affinity_notify *notify) {
kernel/irq/manage.c +70 -0
76 lines not on this page · raw message
Signature · 1 line
2.26.2
[irqchip: irq/irqchip-next] genirq/affinity: Add irq_update_affinity_desc()
The following commit has been merged into the irq/irqchip-next branch of irqchip:
The following commit has been merged into the irq/irqchip-next branch of irqchip:
Commit-ID: 1d3aec89286254487df7641c30f1b14ad1d127a5 Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/1d3aec89286254487df7641c30f1b14ad1d127a5 Author: John Garry <[email protected]> AuthorDate: Wed, 02 Dec 2020 18:36:53 +08:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Fri, 11 Dec 2020 14:47:50
genirq/affinity: Add irq_update_affinity_desc()
Add a function to allow the affinity of an interrupt be switched to managed, such that interrupts allocated for platform devices may be managed.
This new interface has certain limitations, and attempts to use it in the following circumstances will fail:
- For when the kernel is configured for generic IRQ reservation mode (in config GENERIC_IRQ_RESERVATION_MODE). The reason being that it could conflict with managed vs. non-managed interrupt accounting.
- The interrupt is already started, which should not be the case during init
- The interrupt is already configured as managed, which means double init
--- include/linux/interrupt.h | 8 ++++- kernel/irq/manage.c | 70 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 78 insertions(+)
include/linux/interrupt.h +8 -0
@@ -352,6 +352,8 @@ extern int irq_can_set_affinity(unsigned int irq);extern int irq_select_affinity (unsigned int irq);extern int irq_set_affinity_hint (unsigned int irq,const struct cpumask *m); +extern int irq_update_affinity_desc (unsigned int irq, +struct irq_affinity_desc *affinity);extern int irq_set_affinity_notifier (unsigned int irq,struct irq_affinity_notify *notify);
@@ -387,6 +389,12 @@ static inline int irq_set_affinity_hint(unsigned int irq,return -EINVAL ; } +static inline int irq_update_affinity_desc (unsigned int irq, +struct irq_affinity_desc *affinity) +{ +return -EINVAL ; +} +static inline int irq_set_affinity_notifier (unsigned int irq,struct irq_affinity_notify *notify) {
kernel/irq/manage.c +70 -0
76 lines not on this page · raw message
PATCH v5 2/5 resource: Add irqresource_disabled()
In reply to John GarryAdd a common function to set the fields for a irq resource to disabled, which mimics what is done in acpi_dev_irqresource_disabled(), with a view to replace that function.
--- include/linux/ioport.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/linux/ioport.h +7 -0
@@ -307,6 +307,13 @@ struct resource *devm_request_free_mem_region(struct device *dev,struct resource *request_free_mem_region (struct resource *base,unsigned long size,const char *name); +static inline void irqresource_disabled (struct resource *res,u32 irq) +{ + res->start = irq; + res->end = irq; + res->flags =IORESOURCE_IRQ |IORESOURCE_DISABLED |IORESOURCE_UNSET ; +} +#ifdef CONFIG_IO_STRICT_DEVMEM void revoke_devmem (struct resource *res);#else
Signature · 1 line
2.26.2
[irqchip: irq/irqchip-next] resource: Add irqresource_disabled()
The following commit has been merged into the irq/irqchip-next branch of irqchip:
The following commit has been merged into the irq/irqchip-next branch of irqchip:
Commit-ID: 9806731db684a475ade1e95d166089b9edbd9da3 Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/9806731db684a475ade1e95d166089b9edbd9da3 Author: John Garry <[email protected]> AuthorDate: Wed, 02 Dec 2020 18:36:54 +08:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Fri, 11 Dec 2020 14:47:50
resource: Add irqresource_disabled()
Add a common function to set the fields for a irq resource to disabled, which mimics what is done in acpi_dev_irqresource_disabled(), with a view to replace that function.
--- include/linux/ioport.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/linux/ioport.h +7 -0
@@ -307,6 +307,13 @@ struct resource *devm_request_free_mem_region(struct device *dev,struct resource *request_free_mem_region (struct resource *base,unsigned long size,const char *name); +static inline void irqresource_disabled (struct resource *res,u32 irq) +{ + res->start = irq; + res->end = irq; + res->flags =IORESOURCE_IRQ |IORESOURCE_DISABLED |IORESOURCE_UNSET ; +} +#ifdef CONFIG_IO_STRICT_DEVMEM void revoke_devmem (struct resource *res);#else
PATCH v5 3/5 ACPI: Drop acpi_dev_irqresource_disabled()
In reply to John GarryThe functionality of acpi_dev_irqresource_disabled() is same as in common irqresource_disabled(), so drop acpi_dev_irqresource_disabled() in favour of that function.
--- drivers/acpi/resource.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/acpi/resource.c +5 -12
@@ -380,13 +380,6 @@ unsigned int acpi_dev_get_irq_type(int triggering, int polarity) }EXPORT_SYMBOL_GPL (acpi_dev_get_irq_type); -static void acpi_dev_irqresource_disabled (struct resource *res,u32 gsi) -{ - res->start = gsi; - res->end = gsi; - res->flags =IORESOURCE_IRQ |IORESOURCE_DISABLED |IORESOURCE_UNSET ; -} -static void acpi_dev_get_irqresource (struct resource *res,u32 gsi,u8 triggering,u8 polarity,u8 shareable,bool legacy)
@@ -394,7 +387,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,int irq, p, t;if (!valid_IRQ (gsi)) { -acpi_dev_irqresource_disabled (res, gsi); +irqresource_disabled (res, gsi);return ; }
@@ -426,7 +419,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi, res->start = irq; res->end = irq; }else { -acpi_dev_irqresource_disabled (res, gsi); +irqresource_disabled (res, gsi); } }
@@ -463,7 +456,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,*/ irq = &ares->data.irq;if (index >= irq->interrupt_count) { -acpi_dev_irqresource_disabled (res,0 ); +irqresource_disabled (res,0 );return false; } acpi_dev_get_irqresource(res, irq->interrupts[index],
@@ -473,7 +466,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,case ACPI_RESOURCE_TYPE_EXTENDED_IRQ : ext_irq = &ares->data.extended_irq;if (index >= ext_irq->interrupt_count) { -acpi_dev_irqresource_disabled (res,0 ); +irqresource_disabled (res,0 );return false; }if (is_gsi (ext_irq))
@@ -481,7 +474,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index, ext_irq->triggering, ext_irq->polarity, ext_irq->shareable, false); else - acpi_dev_irqresource_disabled(res, 0); + irqresource_disabled(res, 0);break ;default : res->flags =0 ;
Signature · 1 line
2.26.2
[irqchip: irq/irqchip-next] ACPI: Drop acpi_dev_irqresource_disabled()
The following commit has been merged into the irq/irqchip-next branch of irqchip:
The following commit has been merged into the irq/irqchip-next branch of irqchip:
Commit-ID: 1c3f69b4543af0aad514c127298e5ea40392575d Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/1c3f69b4543af0aad514c127298e5ea40392575d Author: John Garry <[email protected]> AuthorDate: Wed, 02 Dec 2020 18:36:55 +08:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Fri, 11 Dec 2020 14:47:50
ACPI: Drop acpi_dev_irqresource_disabled()
The functionality of acpi_dev_irqresource_disabled() is same as in common irqresource_disabled(), so drop acpi_dev_irqresource_disabled() in favour of that function.
--- drivers/acpi/resource.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
drivers/acpi/resource.c +5 -12
@@ -380,13 +380,6 @@ unsigned int acpi_dev_get_irq_type(int triggering, int polarity) }EXPORT_SYMBOL_GPL (acpi_dev_get_irq_type); -static void acpi_dev_irqresource_disabled (struct resource *res,u32 gsi) -{ - res->start = gsi; - res->end = gsi; - res->flags =IORESOURCE_IRQ |IORESOURCE_DISABLED |IORESOURCE_UNSET ; -} -static void acpi_dev_get_irqresource (struct resource *res,u32 gsi,u8 triggering,u8 polarity,u8 shareable,bool legacy)
@@ -394,7 +387,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi,int irq, p, t;if (!valid_IRQ (gsi)) { -acpi_dev_irqresource_disabled (res, gsi); +irqresource_disabled (res, gsi);return ; }
@@ -426,7 +419,7 @@ static void acpi_dev_get_irqresource(struct resource *res, u32 gsi, res->start = irq; res->end = irq; }else { -acpi_dev_irqresource_disabled (res, gsi); +irqresource_disabled (res, gsi); } }
@@ -463,7 +456,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,*/ irq = &ares->data.irq;if (index >= irq->interrupt_count) { -acpi_dev_irqresource_disabled (res,0 ); +irqresource_disabled (res,0 );return false; } acpi_dev_get_irqresource(res, irq->interrupts[index],
@@ -473,7 +466,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index,case ACPI_RESOURCE_TYPE_EXTENDED_IRQ : ext_irq = &ares->data.extended_irq;if (index >= ext_irq->interrupt_count) { -acpi_dev_irqresource_disabled (res,0 ); +irqresource_disabled (res,0 );return false; }if (is_gsi (ext_irq))
@@ -481,7 +474,7 @@ bool acpi_dev_resource_interrupt(struct acpi_resource *ares, int index, ext_irq->triggering, ext_irq->polarity, ext_irq->shareable, false); else - acpi_dev_irqresource_disabled(res, 0); + irqresource_disabled(res, 0);break ;default : res->flags =0 ;
PATCH v5 4/5 Driver core: platform: Add devm_platform_get_irqs_affinity()
In reply to John GarryDrivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
The function accepts an affinity descriptor pointer, which covers all IRQs expected for the device.
The function is devm class as the only current in-tree user will also use devm method for requesting the interrupts; as such, the function is made as devm as it can ensure ordering of freeing the irq and disposing of the mapping.
--- drivers/base/platform.c | 121 ++++++++++++++++++++++++++++++++ include/linux/platform_device.h | 6 ++ 2 files changed, 127 insertions(+)
drivers/base/platform.c +121 -0
@@ -15,6 +15,8 @@#include <linux/of_irq.h> #include <linux/module.h> #include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/ioport.h> #include <linux/dma-mapping.h> #include <linux/memblock.h> #include <linux/err.h>
@@ -289,6 +291,125 @@ int platform_irq_count(struct platform_device *dev) }EXPORT_SYMBOL_GPL (platform_irq_count); +struct irq_affinity_devres { +unsigned int count; +unsigned int irq[]; +}; + +static void platform_disable_acpi_irq (struct platform_device *pdev,int index) +{ +struct resource *r; + + r =platform_get_resource (pdev,IORESOURCE_IRQ , index); +if (r) +irqresource_disabled (r,0 ); +} + +static void devm_platform_get_irqs_affinity_release (struct device *dev, +void *res) +{ +struct irq_affinity_devres *ptr = res; +int i; + +for (i =0 ; i < ptr->count; i++) { +irq_dispose_mapping (ptr->irq[i]); + +if (has_acpi_companion (dev)) +platform_disable_acpi_irq (to_platform_device (dev), i); + } +} + +/** +* devm_platform_get_irqs_affinity - devm method to get a set of IRQs for a +* device using an interrupt affinity descriptor +* @dev: platform device pointer +* @affd: affinity descriptor +* @minvec: minimum count of interrupt vectors +* @maxvec: maximum count of interrupt vectors +* @irqs: pointer holder for IRQ numbers +* +* Gets a set of IRQs for a platform device, and updates IRQ afffinty according +* to the passed affinity descriptor +* +* Return: Number of vectors on success, negative error number on failure. +*/ +int devm_platform_get_irqs_affinity (struct platform_device *dev, +struct irq_affinity *affd, +unsigned int minvec, +unsigned int maxvec, +int **irqs) +{ +struct irq_affinity_devres *ptr; +struct irq_affinity_desc *desc; +size_t size; +int i, ret, nvec; + +if (!affd) +return -EPERM ; + +if (maxvec < minvec) +return -ERANGE ; + + nvec =platform_irq_count (dev); + +if (nvec < minvec) +return -ENOSPC ; + + nvec =irq_calc_affinity_vectors (minvec, nvec, affd); +if (nvec < minvec) +return -ENOSPC ; + +if (nvec > maxvec) + nvec = maxvec; + + size =sizeof (*ptr) +sizeof (unsigned int ) * nvec; + ptr =devres_alloc (devm_platform_get_irqs_affinity_release, size, +GFP_KERNEL ); +if (!ptr) +return -ENOMEM ; + + ptr->count = nvec; + +for (i =0 ; i < nvec; i++) { +int irq =platform_get_irq (dev, i); +if (irq <0 ) { + ret = irq; + goto err_free_devres; + } + ptr->irq[i] = irq; + } + + desc =irq_create_affinity_masks (nvec, affd); +if (!desc) { + ret = -ENOMEM ; + goto err_free_devres; + } + +for (i =0 ; i < nvec; i++) { + ret =irq_update_affinity_desc (ptr->irq[i], &desc[i]); +if (ret) { +dev_err (&dev->dev,"failed to update irq%d affinity descriptor (%d)\n" , + ptr->irq[i], ret); + goto err_free_desc; + } + } + +devres_add (&dev->dev, ptr); + +kfree (desc); + + *irqs = ptr->irq; + +return nvec; + +err_free_desc: +kfree (desc); +err_free_devres: +devres_free (ptr); +return ret; +} +EXPORT_SYMBOL_GPL (devm_platform_get_irqs_affinity); +/** * platform_get_resource_byname - get a resource for a device by name * @dev: platform device
include/linux/platform_device.h +6 -0
@@ -15,6 +15,7 @@#define PLATFORM_DEVID_NONE (-1)#define PLATFORM_DEVID_AUTO (-2) +struct irq_affinity ;struct mfd_cell ;struct property_entry ;struct platform_device_id ;
@@ -70,6 +71,11 @@ devm_platform_ioremap_resource_byname(struct platform_device *pdev,extern int platform_get_irq (struct platform_device *,unsigned int );extern int platform_get_irq_optional (struct platform_device *,unsigned int );extern int platform_irq_count (struct platform_device *); +extern int devm_platform_get_irqs_affinity (struct platform_device *dev, +struct irq_affinity *affd, +unsigned int minvec, +unsigned int maxvec, +int **irqs);extern struct resource *platform_get_resource_byname (struct platform_device *,unsigned int ,const char *);
Signature · 1 line
2.26.2
Greg KH
maintainerRe: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 2 lines · John Garry
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
Why would a platform device want all of this? Shouldn't such a device be on a "real" bus instead?
What in-kernel driver needs this complexity? I can't take new apis without a real user in the tree, sorry.
thanks,
greg k-h
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 4 lines · Greg KH
Quoted 2 lines · John Garry
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
Hi Greg,
Quoted 2 lines
Why would a platform device want all of this? Shouldn't such a device be on a "real" bus instead?
For this HW version, the device is on the system bus, directly addressable by the CPU.
Motivation is that I wanted to switch the HW completion queues to use managed interrupts.
Quoted 3 lines
What in-kernel driver needs this complexity? I can't take new apis without a real user in the tree, sorry.
It's in the final patch in the series https://lore.kernel.org/linux-scsi/[email protected]/T/#m0df7e7cd6f0819b99aaeb6b7f8939ef1e17b8a83.
I don't anticipate a huge number of users of this API in future, as most multi-queue devices are PCI devices; so we could do the work of this API in the driver itself, but the preference was not to export genirq functions like irq_update_affinity_desc() or irq_create_affinity_masks(), and rather have a common helper in the core platform code.
Thanks,
John
Greg KH
maintainerRe: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 6 lines · John Garry
Quoted 4 lines · Greg KH
Quoted 2 lines · John Garry
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
Quoted 7 lines
Hi Greg,
Quoted 2 lines
Why would a platform device want all of this? Shouldn't such a device be on a "real" bus instead?
For this HW version, the device is on the system bus, directly addressable by the CPU.
What do you mean by "system bus"?
Quoted 2 lines
Motivation is that I wanted to switch the HW completion queues to use managed interrupts.
Fair enough, seems like overkill for a "platform" bus though :)
Quoted 4 lines
Quoted 2 lines
What in-kernel driver needs this complexity? I can't take new apis without a real user in the tree, sorry.
It's in the final patch in the series https://lore.kernel.org/linux-scsi/[email protected]/T/#m0df7e7cd6f0819b99aaeb6b7f8939ef1e17b8a83.
Ah, I missed that, I thought that was some high-speed scsi thing, not a tiny platform driver...
Quoted 5 lines
I don't anticipate a huge number of users of this API in future, as most multi-queue devices are PCI devices; so we could do the work of this API in the driver itself, but the preference was not to export genirq functions like
irq_update_affinity_desc()orirq_create_affinity_masks(), and rather have a common helper in the core platform code.
Ok, I'd like to have the irq maintainers/developers ack this before taking it in the driver core, as someone is going to have to maintain this crazy thing for forever if it gets merged.
thanks,
greg k-h
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
On 09/12/2020 19:13, Greg KH wrote:
Hi Greg,
Quoted 3 lines
Quoted 2 lines
For this HW version, the device is on the system bus, directly addressable by the CPU.
What do you mean by "system bus"?
Maybe my terminology is wrong, the point is that we have a platform device driver.
Quoted 10 lines
Quoted 3 lines
Motivation is that I wanted to switch the HW completion queues to use managed interrupts.
Fair enough, seems like overkill for a "platform" bus though:)
Quoted 3 lines
Quoted 2 lines
What in-kernel driver needs this complexity? I can't take new apis without a real user in the tree, sorry.
It's in the final patch in the serieshttps://lore.kernel.org/linux-scsi/[email protected]/T/#m0df7e7cd6f0819b99aaeb6b7f8939ef1e17b8a83.
Ah, I missed that, I thought that was some high-speed scsi thing, not a tiny platform driver...
It is actually is a high-speed SCSI thing also, SAS 3.0 :)
Quoted 10 lines
Quoted 6 lines
I don't anticipate a huge number of users of this API in future, as most multi-queue devices are PCI devices; so we could do the work of this API in the driver itself, but the preference was not to export genirq functions like
irq_update_affinity_desc()orirq_create_affinity_masks(), and rather have a common helper in the core platform code.Ok, I'd like to have the irq maintainers/developers ack this before taking it in the driver core, as someone is going to have to maintain this crazy thing for forever if it gets merged.
irq experts are cc'ed and have been very helpful here
So the API mushroomed a bit over time, as I realized that we need to support tearing down the irq mapping, make as devm method, use irq_calc_affinity_vectors(). Not sure how we could factor any of it out to become less of your problem.
Thanks,
John
Marc Zyngier
Acked-byRe: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
In reply to Greg KHQuoted 15 lines · Greg KH
Quoted 6 lines · John Garry
Quoted 4 lines · Greg KH
Quoted 2 lines · John Garry
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
Quoted 7 lines
Hi Greg,
Quoted 2 lines
Why would a platform device want all of this? Shouldn't such a device be on a "real" bus instead?
For this HW version, the device is on the system bus, directly addressable by the CPU.
Quoted 6 lines
What do you mean by "system bus"?
Quoted 2 lines
Motivation is that I wanted to switch the HW completion queues to use managed interrupts.
Fair enough, seems like overkill for a "platform" bus though :)
You should see the box, really... ;-)
Quoted 15 lines
Quoted 5 lines
Quoted 3 lines
What in-kernel driver needs this complexity? I can't take new apis without a real user in the tree, sorry.
It's in the final patch in the series https://lore.kernel.org/linux-scsi/[email protected]/T/#m0df7e7cd6f0819b99aaeb6b7f8939ef1e17b8a83.
Ah, I missed that, I thought that was some high-speed scsi thing, not a tiny platform driver...
Quoted 5 lines
I don't anticipate a huge number of users of this API in future, as most multi-queue devices are PCI devices; so we could do the work of this API in the driver itself, but the preference was not to export genirq functions like
irq_update_affinity_desc()orirq_create_affinity_masks(), and rather have a common helper in the core platform code.
Quoted 3 lines
Ok, I'd like to have the irq maintainers/developers ack this before taking it in the driver core, as someone is going to have to maintain this crazy thing for forever if it gets merged.
I'm actually quite happy with this, and as it turns out, the crazy system that has this SAS thing keeps my backside warm all year long. As long as this machine keeps ticking, I'm happy to help with this.
So if that helps:
We need to work out the merge strategy for the whole lot though, given that it crosses 3 subsystems over two series...
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 5 lines · Marc Zyngier
Quoted 4 lines
Ok, I'd like to have the irq maintainers/developers ack this before taking it in the driver core, as someone is going to have to maintain this crazy thing for forever if it gets merged.
Quoted 7 lines
I'm actually quite happy with this, and as it turns out, the crazy system that has this SAS thing keeps my backside warm all year long. As long as this machine keeps ticking, I'm happy to help with this.
So if that helps:
Acked-by Marc Zyngier <[email protected]>
Cheers
Quoted 3 lines
We need to work out the merge strategy for the whole lot though, given that it crosses 3 subsystems over two series...
Thomas originally suggested taking the genirq change himself and then providing a tag for others to merge:
https://lore.kernel.org/linux-scsi/[email protected]/
Not sure if that still stands. The small ACPI change could go in a cycle after rest merged, but may be best not to split up.
The worst that will happen without Marc's series is that is remove + re-probe the SCSI driver is broken, so I'm happy as long as that ends up in same kernel version somehow:
https://lore.kernel.org/lkml/[email protected]/
Thanks,
John
Greg KH
maintainerRe: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
In reply to Marc ZyngierQuoted 17 lines · Marc Zyngier
Quoted 16 lines · Greg KH
Quoted 14 lines · John Garry
Quoted 4 lines · Greg KH
Quoted 2 lines · John Garry
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
Hi Greg,
Quoted 2 lines
Why would a platform device want all of this? Shouldn't such a device be on a "real" bus instead?
For this HW version, the device is on the system bus, directly addressable
by the CPU.
Quoted 8 lines
Quoted 6 lines
What do you mean by "system bus"?
Quoted 2 lines
Motivation is that I wanted to switch the HW completion queues to use managed interrupts.
Fair enough, seems like overkill for a "platform" bus though :)
You should see the box, really... ;-)
{sigh} why do hardware engineers ignore sane busses...
Anyway, if you all are going to maintain this, no objection from me, it should go through the irq tree.
thanks,
greg k-h
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Hi Greg,
Quoted 1 line
{sigh} why do hardware engineers ignore sane busses...
The next HW version is an integrated PCI endpoint, so there is hope.
Quoted 3 lines
Anyway, if you all are going to maintain this, no objection from me, it should go through the irq tree.
OK, thanks. So this is getting quite late for 5.11, and none of it has seen -next obviously. However, the changes are additive and should only affect a single driver now. I'm talking about this series now, not Marc's companion series.
I just need to hear from Thomas on any merge preference.
Thanks,
John
[irqchip: irq/irqchip-next] driver core: platform: Add devm_platform_get_irqs_affinity()
In reply to John GarryThe following commit has been merged into the irq/irqchip-next branch of irqchip:
The following commit has been merged into the irq/irqchip-next branch of irqchip:
Commit-ID: e15f2fa959f2cce8a05e8e3a596e75d068cd42c5 Gitweb: https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/e15f2fa959f2cce8a05e8e3a596e75d068cd42c5 Author: John Garry <[email protected]> AuthorDate: Wed, 02 Dec 2020 18:36:56 +08:00 Committer: Marc Zyngier <[email protected]> CommitterDate: Fri, 11 Dec 2020 14:47:51
driver core: platform: Add devm_platform_get_irqs_affinity()
Drivers for multi-queue platform devices may also want managed interrupts for handling HW queue completion interrupts, so add support.
The function accepts an affinity descriptor pointer, which covers all IRQs expected for the device.
The function is devm class as the only current in-tree user will also use devm method for requesting the interrupts; as such, the function is made as devm as it can ensure ordering of freeing the irq and disposing of the mapping.
--- drivers/base/platform.c | 121 +++++++++++++++++++++++++++++++- include/linux/platform_device.h | 6 ++- 2 files changed, 127 insertions(+)
drivers/base/platform.c +121 -0
@@ -15,6 +15,8 @@#include <linux/of_irq.h> #include <linux/module.h> #include <linux/init.h> +#include <linux/interrupt.h> +#include <linux/ioport.h> #include <linux/dma-mapping.h> #include <linux/memblock.h> #include <linux/err.h>
@@ -289,6 +291,125 @@ int platform_irq_count(struct platform_device *dev) }EXPORT_SYMBOL_GPL (platform_irq_count); +struct irq_affinity_devres { +unsigned int count; +unsigned int irq[]; +}; + +static void platform_disable_acpi_irq (struct platform_device *pdev,int index) +{ +struct resource *r; + + r =platform_get_resource (pdev,IORESOURCE_IRQ , index); +if (r) +irqresource_disabled (r,0 ); +} + +static void devm_platform_get_irqs_affinity_release (struct device *dev, +void *res) +{ +struct irq_affinity_devres *ptr = res; +int i; + +for (i =0 ; i < ptr->count; i++) { +irq_dispose_mapping (ptr->irq[i]); + +if (has_acpi_companion (dev)) +platform_disable_acpi_irq (to_platform_device (dev), i); + } +} + +/** +* devm_platform_get_irqs_affinity - devm method to get a set of IRQs for a +* device using an interrupt affinity descriptor +* @dev: platform device pointer +* @affd: affinity descriptor +* @minvec: minimum count of interrupt vectors +* @maxvec: maximum count of interrupt vectors +* @irqs: pointer holder for IRQ numbers +* +* Gets a set of IRQs for a platform device, and updates IRQ afffinty according +* to the passed affinity descriptor +* +* Return: Number of vectors on success, negative error number on failure. +*/ +int devm_platform_get_irqs_affinity (struct platform_device *dev, +struct irq_affinity *affd, +unsigned int minvec, +unsigned int maxvec, +int **irqs) +{ +struct irq_affinity_devres *ptr; +struct irq_affinity_desc *desc; +size_t size; +int i, ret, nvec; + +if (!affd) +return -EPERM ; + +if (maxvec < minvec) +return -ERANGE ; + + nvec =platform_irq_count (dev); + +if (nvec < minvec) +return -ENOSPC ; + + nvec =irq_calc_affinity_vectors (minvec, nvec, affd); +if (nvec < minvec) +return -ENOSPC ; + +if (nvec > maxvec) + nvec = maxvec; + + size =sizeof (*ptr) +sizeof (unsigned int ) * nvec; + ptr =devres_alloc (devm_platform_get_irqs_affinity_release, size, +GFP_KERNEL ); +if (!ptr) +return -ENOMEM ; + + ptr->count = nvec; + +for (i =0 ; i < nvec; i++) { +int irq =platform_get_irq (dev, i); +if (irq <0 ) { + ret = irq; + goto err_free_devres; + } + ptr->irq[i] = irq; + } + + desc =irq_create_affinity_masks (nvec, affd); +if (!desc) { + ret = -ENOMEM ; + goto err_free_devres; + } + +for (i =0 ; i < nvec; i++) { + ret =irq_update_affinity_desc (ptr->irq[i], &desc[i]); +if (ret) { +dev_err (&dev->dev,"failed to update irq%d affinity descriptor (%d)\n" , + ptr->irq[i], ret); + goto err_free_desc; + } + } + +devres_add (&dev->dev, ptr); + +kfree (desc); + + *irqs = ptr->irq; + +return nvec; + +err_free_desc: +kfree (desc); +err_free_devres: +devres_free (ptr); +return ret; +} +EXPORT_SYMBOL_GPL (devm_platform_get_irqs_affinity); +/** * platform_get_resource_byname - get a resource for a device by name * @dev: platform device
include/linux/platform_device.h +6 -0
@@ -15,6 +15,7 @@#define PLATFORM_DEVID_NONE (-1)#define PLATFORM_DEVID_AUTO (-2) +struct irq_affinity ;struct mfd_cell ;struct property_entry ;struct platform_device_id ;
@@ -70,6 +71,11 @@ devm_platform_ioremap_resource_byname(struct platform_device *pdev,extern int platform_get_irq (struct platform_device *,unsigned int );extern int platform_get_irq_optional (struct platform_device *,unsigned int );extern int platform_irq_count (struct platform_device *); +extern int devm_platform_get_irqs_affinity (struct platform_device *dev, +struct irq_affinity *affd, +unsigned int minvec, +unsigned int maxvec, +int **irqs);extern struct resource *platform_get_resource_byname (struct platform_device *,unsigned int ,const char *);
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
In reply to John Garrydrivers/base/platform.c int platform_irq_count(struct platform_device *dev) · line 380
Quoted 8 lines · John Garry
drivers/base/platform.c +8
-0+for (i =0 ; i < nvec; i++) { +int irq =platform_get_irq (dev, i); +if (irq <0 ) { + ret = irq; + goto err_free_devres; + } + ptr->irq[i] = irq; + }
(replying to an email from four years ago)
Why does this function call platform_get_irq(dev, i) instead of platform_get_irq(dev, affd->pre_vectors + i)? Is there perhaps something about the hisi_sas driver that I'm missing? I'm asking this because this function would be useful for UFS controller drivers if the affd->pre_vectors offset would be added when calling platform_get_irq().
Thanks,
Bart.
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 10 lines · Bart Van Assche
Quoted 8 lines · John Garry
+ for (i = 0; i < nvec; i++) {
+ int irq =platform_get_irq(dev, i);
+ if (irq < 0) {
+ ret = irq;
+ gotoerr_free_devres;
+ }
+ ptr->irq[i] = irq;
+ }
Quoted 8 lines
(replying to an email from four years ago)
Why does this function call
platform_get_irq(dev, i)instead ofplatform_get_irq(dev, affd->pre_vectors + i)? Is there perhaps something about thehisi_sasdriver that I'm missing? I'm asking this because this function would be useful for UFS controller drivers if theaffd->pre_vectorsoffset would be added when callingplatform_get_irq().
int devm_platform_get_irqs_affinity(struct platform_device *dev,
struct irq_affinity *affd, unsigned int minvec, unsigned int maxvec, int **irqs)
Function devm_platform_get_irqs_affinity() gets the irq number for a total between @minvec and @maxvec interrupts, and fills them into @irqs arg. It does not just get the interrupts for index @minvec to @maxvec only.
For context, as I remember, hisi_sas v2 hw has 128 interrupts lines. Interrupts index [96, 112) are completion queue interrupts, which we want to spread over all CPUs. See interrupt_init_v2_hw() in that driver for how the control interrupts, like phy up/down, are used.
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 26 lines · John Garry
Quoted 10 lines · Bart Van Assche
Quoted 8 lines · John Garry
+ for (i = 0; i < nvec; i++) {
+ int irq =platform_get_irq(dev, i);
+ if (irq < 0) {
+ ret = irq;
+ gotoerr_free_devres;
+ }
+ ptr->irq[i] = irq;
+ }Quoted 8 lines
(replying to an email from four years ago)
Why does this function call
platform_get_irq(dev, i)instead ofplatform_get_irq(dev, affd->pre_vectors + i)? Is there perhaps something about thehisi_sasdriver that I'm missing? I'm asking this because this function would be useful for UFS controller drivers if theaffd->pre_vectorsoffset would be added when callingplatform_get_irq().int devm_platform_get_irqs_affinity(struct
platform_device*dev,
structirq_affinity*affd, unsigned int minvec,
unsigned int maxvec,
int **irqs)
Quoted 3 lines
Function
devm_platform_get_irqs_affinity()gets the irq number for a total between @minvec and @maxvec interrupts, and fills them into @irqs arg. It does not just get the interrupts for index @minvec to @maxvec only.For context, as I remember,
hisi_sasv2 hw has 128 interrupts lines. Interrupts index [96, 112) are completion queue interrupts, which we want to spread over all CPUs. Seeinterrupt_init_v2_hw()in that driver for how the control interrupts, like phy up/down, are used.
Hi John,
In interrupt_init_v2_hw() and also elsewhere in the hisi_sas_v2_hw.c source file I see that the CQ interrupts start at offset 96. However, devm_platform_get_irqs_affinity() passes the arguments 0..(num_cqs-1) to platform_get_irq(). Shouldn't that function pass arguments in the range 96..(96+num_cqs-1) to platform_get_irq() since that is the CQ interrupt range for this storage controller? My understanding is that the devm_platform_get_irqs_affinity() call from hisi_sas_v2_hw.c will affect the affinity of the interrupts 0..(num_cqs-1) instead of the interrupts in the range 96..(96+num_cqs-1). Isn't that wrong?
Thanks,
Bart.
Re: [PATCH v5 4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
Quoted 31 lines · Bart Van Assche
Quoted 30 lines · John Garry
Quoted 10 lines · Bart Van Assche
Quoted 8 lines · John Garry
+ for (i = 0; i < nvec; i++) {
+ int irq =platform_get_irq(dev, i);
+ if (irq < 0) {
+ ret = irq;
+ gotoerr_free_devres;
+ }
+ ptr->irq[i] = irq;
+ }Quoted 8 lines
(replying to an email from four years ago)
Why does this function call
platform_get_irq(dev, i)instead ofplatform_get_irq(dev, affd->pre_vectors + i)? Is there perhaps something about thehisi_sasdriver that I'm missing? I'm asking this because this function would be useful for UFS controller drivers if theaffd->pre_vectorsoffset would be added when callingplatform_get_irq().int devm_platform_get_irqs_affinity(struct
platform_device*dev, structirq_affinity*affd, unsigned int minvec,
unsigned int maxvec,
int **irqs)Function
devm_platform_get_irqs_affinity()gets the irq number for a total between @minvec and @maxvec interrupts, and fills them into @irqs arg. It does not just get the interrupts for index @minvec to
@maxvec only.
Quoted 7 lines
Quoted 1 line
For context, as I remember,
hisi_sasv2 hw has 128 interrupts lines. Interrupts index [96, 112) are completion queue interrupts, which we want to spread over all CPUs. Seeinterrupt_init_v2_hw()in that driver for how the control interrupts, like phy up/down, are used.Hi John,
In
interrupt_init_v2_hw()and also elsewhere in thehisi_sas_v2_hw.csource file I see that the CQ interrupts start at offset 96. However,devm_platform_get_irqs_affinity()passes the arguments 0..(num_cqs-1) to
Total interrupt lines are 128.
dec.{pre, post}_vectors = {96, 16}
which gives
resv=96+16=112
minvec=resv+1 = 113
maxvec=128
Quoted 3 lines
platform_get_irq(). Shouldn't that function pass arguments in the range 96..(96+num_cqs-1) toplatform_get_irq()since that is the CQ interrupt range for this storage controller?
The API is trying to be like pci_alloc_irq_vectors_affinity(), where it accepts a min and max vectors to allocate.
platform_get_irq() actually does more than just do some lookup for an irq number - it also sets up a mapping if it does not already exist.
Furthermore, as I recall, maybe platform msi code or mbigen code or hisi chipset had a restriction that platform_get_irq() had to be called in order for each and every interrupt line. So having a single point to do this made sense. Check interrupt_init_v2_hw() code and comment prior to devm_platform_get_irqs_affinity() existing, like a v5.10 kernel - it called platform_get_irq() for all 128 interrupt lines.
Quoted 5 lines
My understanding is that the
devm_platform_get_irqs_affinity()call fromhisi_sas_v2_hw.cwill affect the affinity of the interrupts 0..(num_cqs-1) instead of the interrupts in the range 96..(96+num_cqs-1). Isn't that wrong?
devm_platform_get_irqs_affinity() will only touch the affinity mapping of the completion queue interrupts, but still call platform_get_irq() for all interrupts
PATCH v5 5/5 scsi: hisi_sas: Expose HW queues for v2 hw
In reply to John GarryAs a performance enhancement, make the completion queue interrupts managed.
In addition, in commit bf0beec0607d ("blk-mq: drain I/O when all CPUs in a hctx are offline"), CPU hotplug for MQ devices using managed interrupts is made safe. So expose HW queues to blk-mq to take advantage of this.
Flag Scsi_host.host_tagset is also set to ensure that the HBA is not sent more commands than it can handle. However the driver still does not use request tag for IPTT as there are many HW bugs means that special rules apply for IPTT allocation.
--- drivers/scsi/hisi_sas/hisi_sas.h | 4 ++ drivers/scsi/hisi_sas/hisi_sas_main.c | 11 +++++ drivers/scsi/hisi_sas/hisi_sas_v2_hw.c | 66 +++++++++++++++++++++----- 3 files changed, 68 insertions(+), 13 deletions(-)
drivers/scsi/hisi_sas/hisi_sas.h +4 -0
@@ -14,6 +14,7 @@#include <linux/debugfs.h> #include <linux/dmapool.h> #include <linux/iopoll.h> +#include <linux/irq.h> #include <linux/lcm.h> #include <linux/libata.h> #include <linux/mfd/syscon.h>
@@ -312,6 +313,7 @@ enum {struct hisi_sas_hw {int (*hw_init)(struct hisi_hba *hisi_hba); +int (*interrupt_preinit)(struct hisi_hba *hisi_hba);void (*setup_itct)(struct hisi_hba *hisi_hba,struct hisi_sas_device *device); int (*slot_index_alloc)(struct hisi_hba *hisi_hba,
@@ -418,6 +420,8 @@ struct hisi_hba {u32 refclk_frequency_mhz;u8 sas_addr[SAS_ADDR_SIZE ]; +int *irq_map;/* v2 hw */ +int n_phy;spinlock_t lock;struct semaphore sem;
drivers/scsi/hisi_sas/hisi_sas_main.c +11 -0
@@ -2614,6 +2614,13 @@ static struct Scsi_Host *hisi_sas_shost_alloc(struct platform_device *pdev,return NULL ; } +static int hisi_sas_interrupt_preinit (struct hisi_hba *hisi_hba) +{ +if (hisi_hba->hw->interrupt_preinit) +return hisi_hba->hw->interrupt_preinit (hisi_hba); +return 0 ; +} +int hisi_sas_probe (struct platform_device *pdev,const struct hisi_sas_hw *hw) {
@@ -2671,6 +2678,10 @@ int hisi_sas_probe(struct platform_device *pdev, sha->sas_port[i] = &hisi_hba->port[i].sas_port; } + rc =hisi_sas_interrupt_preinit (hisi_hba); +if (rc) + goto err_out_ha; + rc =scsi_add_host (shost, &pdev->dev);if (rc) goto err_out_ha;
drivers/scsi/hisi_sas/hisi_sas_v2_hw.c +53 -13
127 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v5 5/5] scsi: hisi_sas: Expose HW queues for v2 hw
Quoted 11 lines · John Garry
As a performance enhancement, make the completion queue interrupts managed.
In addition, in commit
bf0beec0607d("blk-mq: drain I/O when all CPUs in a hctx are offline"), CPU hotplug for MQ devices using managed interrupts is made safe. So expose HW queues to blk-mq to take advantage of this.Flag Scsi_host.host_tagset is also set to ensure that the HBA is not sent more commands than it can handle. However the driver still does not use request tag for IPTT as there are many HW bugs means that special rules apply for IPTT allocation.
Hi Martin, James,
Is there some way in this patch can be merged for 5.11 via the SCSI tree? It has a dependency on the earlier patches in the series, now picked up via irqchip tree. I've seen multiple rounds of SCSI pull requests before, but not sure when we have that or if the tree is rebased for those.
Thanks,
John
Quoted 7 lines
Signed-off-by John Garry <[email protected]>--- drivers/scsi/hisi_sas/hisi_sas.h | 4 ++ drivers/scsi/hisi_sas/hisi_sas_main.c | 11 +++++ drivers/scsi/hisi_sas/hisi_sas_v2_hw.c | 66 +++++++++++++++++++++----- 3 files changed, 68 insertions(+), 13 deletions(-)
Re: [PATCH v5 5/5] scsi: hisi_sas: Expose HW queues for v2 hw
John,
Quoted 5 lines
Is there some way in this patch can be merged for 5.11 via the SCSI tree? It has a dependency on the earlier patches in the series, now picked up via irqchip tree. I've seen multiple rounds of SCSI pull requests before, but not sure when we have that or if the tree is rebased for those.
I'll do a postmerge branch for a few things, including your patch and the megaraid patch.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Re: [PATCH v5 5/5] scsi: hisi_sas: Expose HW queues for v2 hw
Quoted 7 lines · Martin K. Petersen
Quoted 5 lines
Is there some way in this patch can be merged for 5.11 via the SCSI tree? It has a dependency on the earlier patches in the series, now picked up via irqchip tree. I've seen multiple rounds of SCSI pull requests before, but not sure when we have that or if the tree is rebased for those.
I'll do a postmerge branch for a few things, including your patch and the megaraid patch.
Great, thanks.
By a strange coincidence both those patches are doing the same thing.
John
Marc Zyngier
AppliedQuoted 9 lines · John Garry
So far, managed interrupts are only used for PCI MSIs. This series adds platform device support for managed interrupts. Initially this topic was discussed at [0].
The method to enable managed interrupts is to allocate a group of IRQs for the device, and then switch the interrupts to managed - this is done through new function
irq_update_affinity_desc().[...]
Applied to irq/irqchip-next, thanks!
[1/5] genirq/affinity: Add irq_update_affinity_desc()
commit: 1d3aec89286254487df7641c30f1b14ad1d127a5
[2/5] resource: Add irqresource_disabled()
commit: 9806731db684a475ade1e95d166089b9edbd9da3
[3/5] ACPI: Drop acpi_dev_irqresource_disabled()
commit: 1c3f69b4543af0aad514c127298e5ea40392575d
[4/5] Driver core: platform: Add devm_platform_get_irqs_affinity()
commit: e15f2fa959f2cce8a05e8e3a596e75d068cd42c5Please route patch 5 via the SCSI tree when possible.
Cheers,
M.
Signature · 8 lines
Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel