Representing interrupt affinity in devicetree
Subsystems (2)
v2 cover Representing interrupt affinity in devicetree
[This is an updated version of my previous RFC, which can be found at
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/128205.html]
Current devicetree bindings for devices which use cpu-affine shared interrupts assume that interrupts are listed in ascending order of physical cpu id (MPIDR.Aff{2,1,0}). This is problematic for drivers because:
(1) The driver must convert each physical id to a logical id for the purpose of managing interrupts.
(2) In multi-cluster systems the physical ids are not necessarily contiguous, and drivers cannot simply iterate over ids from 0 to NR_CPUS.
(3) It is not possible to specify sets of interrupts which are wired to a subset of cpus (i.e. clusters) not starting at physical id 0, as we can't specify which cpu to start from, and can't skip cpus. This makes it impossible to represent some devices (e.g. cpu PMUs) which may not exist in the first cluster.
(4) Some devices may either be wired with PPIs or SPIs. It is not possible to differentiate the two cases in general from the interrupts list (e.g. when a device has multiple PPIs wired to all cpus).
To represent the general case, we require a mechanism to describe the cpu affinity of a device, and a consistent way to map each interrupt to a cpu (or set of cpus). So far, the only workable mechanism I've been able to come up with is the following:
In addition to the interrupts list we add an optional 'interrupts-affinity' list which describes how to map an interrupt to cpu(s). Each interrupt would have a corresponding element in the interrupts-affinity list.
Each element consists of a pair of cells specifying how to test if a cpu is affine. This test specification consists of a most significant affinity level to match, and values for the MPIDR affinity bits down to this level. Each pair of cells has the form:
< N 0x00AABBCC >
\ \ \ \ \__ MPIDR.Aff0
\ \ \ \___ MPIDR.Aff1
\ \ \____ MPIDR.Aff2
\ \_____ [must be zero]
\________ level-specifierThe level-specifier is in the range [0-3]. When the value is 3, MPIDR affinity levels 2,1,0 are ignored, and thus all cpus are matched. When the value is 0, levels 2,1,0 must all be matched for a cpu to be considered affine. Affinity bits which are not to be matched should be set to zero.
In practice, use of the binding would look something like:
Quoted 9 lines
device at 0 {
compatible = "example,example-affine-device";
interrupts = <0 130 4>,
<0 131 4>,
<1 12 4>;
interrupts-affinity = <0 0x00000000>,
<0 0x00000001>,
<1 0x00000100>;
};To request/free interrupts, drivers would walk over all interrupts, and request the affine cpu(s), which can be automatically mapped to logical ids by common code. This fixes issues 1, 2, and 3. 4 is partially solved, in that we know whether an interrupt is targeted at a single cpu or multiple cpus (i.e. whether it's an SPI or a PPI), but we can't differentiate multiple interrupts targeted at the same cpu. We could solve this with an additional "interrupt-names" property, or something of that sort.
The following patches implement common code that could be used by drivers to deal with this affinity information, allowing drivers to extract a (logical) cpumask specifying the affinity of each interrupt, and additional information specifying whether an interrupt targets multiple cpus.
The patches are based on rmk's for-next branch.
Any thoughts?
Thanks,
Mark.
Mark Rutland (2):
of: add of_property_read_u32_index
ARM: add functions to parse irq affinity from dt
arch/arm/include/asm/dt_irq.h | 9 ++++ arch/arm/kernel/devtree.c | 91 +++++++++++++++++++++++++++++++++++++++++ drivers/of/base.c | 36 ++++++++++++++++ include/linux/of.h | 11 +++++ 4 files changed, 147 insertions(+), 0 deletions(-) create mode 100644 arch/arm/include/asm/dt_irq.h
RFC v2 1/2 of: add of_property_read_u32_index
Currently, we have to pre-allocate an array when reading multiple u32 values from a property using of_property_read_u32_array. This isn't very good for times when we want to iterate over elements of a list of arbitrary size.
This patch adds a function to read a single u32 value from an arbitrary index in a list, making this case easier.
--- drivers/of/base.c | 36 ++++++++++++++++++++++++++++++++++++ include/linux/of.h | 11 +++++++++++ 2 files changed, 47 insertions(+), 0 deletions(-)
drivers/of/base.c +36 -0
@@ -707,6 +707,42 @@ int of_property_read_u32_array(const struct device_node *np,EXPORT_SYMBOL_GPL (of_property_read_u32_array);/** +* of_property_read_u32_index - Find and read a 32 bit integer from a multiple +* integer property. +* +* @np: device node from which the property value is to be read. +* @propname: name of the property to be searched. +* @out_value: pointer to return value, modified only if return value is 0. +* @index: index of the integer in the list of integers. +* +* Search for a property in a device node and read a 32-bit value from +* it. Returns 0 on success, -EINVAL if the property does not exist, +* -ENODATA if property does not have a value, and -EOVERFLOW if the +* index is beyond the range of data. +* +* The out_value is modified only if a valid u32 value can be decoded. +*/ +int of_property_read_u32_index (const struct device_node *np, +const char *propname,u32 *out_value, +int idx) +{ +struct property *prop =of_find_property (np, propname,NULL ); +const __be32 *val; + +if (!prop) +return -EINVAL ; +if (!prop->value) +return -ENODATA ; +if ((idx+1 ) *sizeof (*out_value) > prop->length) +return -EOVERFLOW ; + + val = prop->value; + *out_value =be32_to_cpup (val + idx); +return 0 ; +} +EXPORT_SYMBOL_GPL (of_property_read_u32_index); + +/** * of_property_read_u64 - Find and read a 64 bit integer from a property * @np: device node from which the property value is to be read. * @propname: name of the property to be searched.
include/linux/of.h +11 -0
@@ -227,6 +227,10 @@ extern int of_property_read_u32_array(const struct device_node *np,const char *propname, u32 *out_values, size_t sz); +extern int of_property_read_u32_index (const struct device_node *np, +const char *propname, +u32 *out_value, +int idx);extern int of_property_read_u64 (const struct device_node *np,const char *propname,u64 *out_value);
@@ -371,6 +375,13 @@ static inline int of_property_read_u32_array(const struct device_node *np,return -ENOSYS ; } +static inline int of_property_read_u32_index (const struct device_node *np, +const char *propname, +u32 *out_value,int idx) +{ +return -ENOSYS ; +} +static inline int of_property_read_string (struct device_node *np,const char *propname,const char **out_string)
Signature · 1 line
1.7.0.4
RFC v2 2/2 ARM: add functions to parse irq affinity from dt
In reply to Mark RutlandThis patch adds functions to handle parsing the CPU affinity of interrupts from devicetree, based on an "interrupts-affinity" parameter list. It is assumed that drivers which may optionally use this parameter implement their own fallback behaviour.
The patch adds:
* arm_dt_has_irq_affinity : returns whether or not a device_node has interrupts-affinity information
* arm_dt_affine_irq_is_single : returns whether an interrupt targets a single CPU (i.e. is an SPI) or multiple CPUs (i.e. is a PPI). This test is independent of the number of affine CPUs we can logically map.
* arm_dt_irq_get_affinity : fills a cpumask with the CPUs an interrupt is affine to.
The "interrupts-affinity" property consists of a list of pairs of cells specifying how to test if a CPU is affine. This test specification consists of a most significant affinity level to match, and values for the MPIDR affinity bits down to this level. Each pair of cells has the form:
< N 0x00AABBCC > \ \ \ \ \__ MPIDR.Aff0 \ \ \ \___ MPIDR.Aff1 \ \ \____ MPIDR.Aff2 \ \_____ [must be zero] \________ level-specifier
The level-specifier is in the range [0-3]. When the value is 3, MPIDR affinity levels 2,1,0 are ignored, and all CPUs are matched. Affinity bits which are not to be matched should be set to zero.
--- arch/arm/include/asm/dt_irq.h | 9 ++++ arch/arm/kernel/devtree.c | 91 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+), 0 deletions(-) create mode 100644 arch/arm/include/asm/dt_irq.h
arch/arm/include/asm/dt_irq.h new file +9 -0
@@ -0,0 +1,9 @@ +#ifndef ARM_ASM_DT_IRQ_H +#define ARM_ASM_DT_IRQ_H + +bool arm_dt_has_irq_affinity (struct device_node *node); +bool arm_dt_affine_irq_is_single (struct device_node *node,int idx); +int arm_dt_irq_get_affinity (struct device_node *node,int idx, +cpumask_t *mask); + +#endif /* ARM_ASM_DT_IRQ_H */
arch/arm/kernel/devtree.c +91 -0
103 lines not on this page · raw message
Signature · 1 line
1.7.0.4
v2 cover Representing interrupt affinity in devicetree
In reply to Mark RutlandQuoted 69 lines · Mark Rutland
[This is an updated version of my previous RFC, which can be found at
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/128205.html]Current devicetree bindings for devices which use cpu-affine shared interrupts assume that interrupts are listed in ascending order of physical cpu id (MPIDR.Aff{2,1,0}). This is problematic for drivers because:
(1) The driver must convert each physical id to a logical id for the purpose of managing interrupts.
(2) In multi-cluster systems the physical ids are not necessarily contiguous, and drivers cannot simply iterate over ids from 0 to
NR_CPUS.(3) It is not possible to specify sets of interrupts which are wired to a subset of cpus (i.e. clusters) not starting at physical id 0, as we can't specify which cpu to start from, and can't skip cpus. This makes it impossible to represent some devices (e.g. cpu PMUs) which may not exist in the first cluster.
(4) Some devices may either be wired with PPIs or SPIs. It is not possible to differentiate the two cases in general from the interrupts list (e.g. when a device has multiple PPIs wired to all cpus).
To represent the general case, we require a mechanism to describe the cpu affinity of a device, and a consistent way to map each interrupt to a cpu (or set of cpus). So far, the only workable mechanism I've been able to come up with is the following:
In addition to the interrupts list we add an optional 'interrupts-affinity' list which describes how to map an interrupt to cpu(s). Each interrupt would have a corresponding element in the interrupts-affinity list.
Each element consists of a pair of cells specifying how to test if a cpu is affine. This test specification consists of a most significant affinity level to match, and values for the MPIDR affinity bits down to this level. Each pair of cells has the form:
< N 0x00AABBCC > \ \ \ \ \__ MPIDR.Aff0 \ \ \ \___ MPIDR.Aff1 \ \ \____ MPIDR.Aff2 \ \_____ [must be zero] \________ level-specifierThe level-specifier is in the range [0-3]. When the value is 3, MPIDR affinity levels 2,1,0 are ignored, and thus all cpus are matched. When the value is 0, levels 2,1,0 must all be matched for a cpu to be considered affine. Affinity bits which are not to be matched should be set to zero.
In practice, use of the binding would look something like:
Quoted 9 lines
device at 0 { compatible = "example,example-affine-device"; interrupts = <0 130 4>, <0 131 4>, <1 12 4>; interrupts-affinity = <0 0x00000000>, <0 0x00000001>, <1 0x00000100>; };To request/free interrupts, drivers would walk over all interrupts, and request the affine cpu(s), which can be automatically mapped to logical ids by common code. This fixes issues 1, 2, and 3. 4 is partially solved, in that we know whether an interrupt is targeted at a single cpu or multiple cpus (i.e. whether it's an SPI or a PPI), but we can't differentiate multiple interrupts targeted at the same cpu. We could solve this with an additional "interrupt-names" property, or something of that sort.
Quoted 8 lines
The following patches implement common code that could be used by drivers to deal with this affinity information, allowing drivers to extract a (logical) cpumask specifying the affinity of each interrupt, and additional information specifying whether an interrupt targets multiple cpus.
The patches are based on rmk's for-next branch.
Any thoughts?
Hi Mark,
I could use some more context for how this will be used. Do device drivers need to be aware of which CPU can handle an interrupt for a device, or is it the sort of thing that can be done in the background when an irq is requested? What are some examples of device drivers using this interface.
Part of the reason why I ask is I see a new function for parsing the affinity, but no code that links it into the existing irq infrastructure. Would this need to be called automatically when a DT interrupt is parsed? Or when the irq is requested? Some guidance would help me here.
Thanks,
g.
v2 cover Representing interrupt affinity in devicetree
Quoted 75 lines · Grant Likely
Quoted 74 lines · Mark Rutland
[This is an updated version of my previous RFC, which can be found at
http://lists.infradead.org/pipermail/linux-arm-kernel/2012-October/128205.html]Current devicetree bindings for devices which use cpu-affine shared interrupts assume that interrupts are listed in ascending order of physical cpu id (MPIDR.Aff{2,1,0}). This is problematic for drivers because:
(1) The driver must convert each physical id to a logical id for the purpose of managing interrupts.
(2) In multi-cluster systems the physical ids are not necessarily contiguous, and drivers cannot simply iterate over ids from 0 to
NR_CPUS.(3) It is not possible to specify sets of interrupts which are wired to a subset of cpus (i.e. clusters) not starting at physical id 0, as we can't specify which cpu to start from, and can't skip cpus. This makes it impossible to represent some devices (e.g. cpu PMUs) which may not exist in the first cluster.
(4) Some devices may either be wired with PPIs or SPIs. It is not possible to differentiate the two cases in general from the interrupts list (e.g. when a device has multiple PPIs wired to all cpus).
To represent the general case, we require a mechanism to describe the cpu affinity of a device, and a consistent way to map each interrupt to a cpu (or set of cpus). So far, the only workable mechanism I've been able to come up with is the following:
In addition to the interrupts list we add an optional 'interrupts-affinity' list which describes how to map an interrupt to cpu(s). Each interrupt would have a corresponding element in the interrupts-affinity list.
Each element consists of a pair of cells specifying how to test if a cpu is affine. This test specification consists of a most significant affinity level to match, and values for the MPIDR affinity bits down to this level. Each pair of cells has the form:
< N 0x00AABBCC > \ \ \ \ \__ MPIDR.Aff0 \ \ \ \___ MPIDR.Aff1 \ \ \____ MPIDR.Aff2 \ \_____ [must be zero] \________ level-specifierThe level-specifier is in the range [0-3]. When the value is 3, MPIDR affinity levels 2,1,0 are ignored, and thus all cpus are matched. When the value is 0, levels 2,1,0 must all be matched for a cpu to be considered affine. Affinity bits which are not to be matched should be set to zero.
In practice, use of the binding would look something like:
Quoted 9 lines
device at 0 { compatible = "example,example-affine-device"; interrupts = <0 130 4>, <0 131 4>, <1 12 4>; interrupts-affinity = <0 0x00000000>, <0 0x00000001>, <1 0x00000100>; };To request/free interrupts, drivers would walk over all interrupts, and request the affine cpu(s), which can be automatically mapped to logical ids by common code. This fixes issues 1, 2, and 3. 4 is partially solved, in that we know whether an interrupt is targeted at a single cpu or multiple cpus (i.e. whether it's an SPI or a PPI), but we can't differentiate multiple interrupts targeted at the same cpu. We could solve this with an additional "interrupt-names" property, or something of that sort.
The following patches implement common code that could be used by drivers to deal with this affinity information, allowing drivers to extract a (logical) cpumask specifying the affinity of each interrupt, and additional information specifying whether an interrupt targets multiple cpus.
Quoted 5 lines
Quoted 3 lines
The patches are based on rmk's for-next branch.
Any thoughts?
Hi Mark,
Hi,
Quoted 6 lines
I could use some more context for how this will be used. Do device drivers need to be aware of which CPU can handle an interrupt for a device, or is it the sort of thing that can be done in the background when an irq is requested? What are some examples of device drivers using this interface.
The main users I can think of for this would be PMUs in multi-cluster systems, where we may have differing PMUs in each cluster. The driver for each needs to know the set of CPUs it's handling, and which CPU each interrupt is affine to.
With the above binding scheme, we'd describe the A15x2 A7x3 CoreTile's PMUs something like:
pmu_a15s {
compatible = "arm,cortex-a15-pmu";
interrupts = <0 68 4>,
<0 69 4>;
interrupts-affinity = <0 0x0>,
<0 0x1>;
};pmu_a7s {
compatible = arm,cortex-a7-pmu";
interrupts = <0 128 4>,
<0 129 4>,
<0 130 4>;
interrupts-affinity = <0 0x100>,
<0 0x101>,
<0 0x102>;
};Quoted 6 lines
Part of the reason why I ask is I see a new function for parsing the affinity, but no code that links it into the existing irq infrastructure. Would this need to be called automatically when a DT interrupt is parsed? Or when the irq is requested? Some guidance would help me here.
I envisage that each driver that requires this information would request it manually when parsing the dt, and storing what it needs in an appropriate datastructure. For PMUs this could be something like:
struct irq_affinity {
cpumask_t mask;
int irq;
} interrupt_data[];When setting up or tearing down interrupts, each CPU can find which interrupt(s) it needs, and whether it's a PPI or SPI by iterating over the list and checking against a cpumask, and perform the appropriate steps to setup or teardown.
Other devices may need affinity information (e.g. which CPUs are connected to the ACE ports on CCI) and can be handled by a similar binding.
Thanks,
Mark.
v2 cover Representing interrupt affinity in devicetree
Quoted 22 lines · Mark Rutland
Quoted 5 lines · Grant Likely
I could use some more context for how this will be used. Do device drivers need to be aware of which CPU can handle an interrupt for a device, or is it the sort of thing that can be done in the background when an irq is requested? What are some examples of device drivers using this interface.
The main users I can think of for this would be PMUs in multi-cluster systems, where we may have differing PMUs in each cluster. The driver for each needs to know the set of CPUs it's handling, and which CPU each interrupt is affine to.
With the above binding scheme, we'd describe the A15x2 A7x3 CoreTile's PMUs something like:
pmu_a15s { compatible = "arm,cortex-a15-pmu"; interrupts = <0 68 4>, <0 69 4>; interrupts-affinity = <0 0x0>, <0 0x1>; };
Quoted 9 lines
pmu_a7s {
compatible = arm,cortex-a7-pmu";
interrupts = <0 128 4>,
<0 129 4>,
<0 130 4>;
interrupts-affinity = <0 0x100>,
<0 0x101>,
<0 0x102>;
};That does sound an awful lot like what Lorenzo has been trying to solve. I really do think that you need to coordinate with him.
g.
v2 cover Representing interrupt affinity in devicetree
Quoted 33 lines · Grant Likely
Quoted 22 lines · Mark Rutland
Quoted 5 lines · Grant Likely
I could use some more context for how this will be used. Do device drivers need to be aware of which CPU can handle an interrupt for a device, or is it the sort of thing that can be done in the background when an irq is requested? What are some examples of device drivers using this interface.
The main users I can think of for this would be PMUs in multi-cluster systems, where we may have differing PMUs in each cluster. The driver for each needs to know the set of CPUs it's handling, and which CPU each interrupt is affine to.
With the above binding scheme, we'd describe the A15x2 A7x3 CoreTile's PMUs something like:
pmu_a15s { compatible = "arm,cortex-a15-pmu"; interrupts = <0 68 4>, <0 69 4>; interrupts-affinity = <0 0x0>, <0 0x1>; };Quoted 9 lines
pmu_a7s { compatible = arm,cortex-a7-pmu"; interrupts = <0 128 4>, <0 129 4>, <0 130 4>; interrupts-affinity = <0 0x100>, <0 0x101>, <0 0x102>; };
Quoted 2 lines
That does sound an awful lot like what Lorenzo has been trying to solve. I really do think that you need to coordinate with him.
Indeed it does. Lorenzo and I have been working together on this, and having realised issues with the above approach, we've put a new series together [1] which we both believe has a more workable approach.
Mark.
[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2013-April/162457.html