Using irq-crossbar.c

Hi,

We are trying to write a driver for an interrupt controller (actually more of a crossbar) for an ARM-based SoC.
This IRQ crossbar has 128 inputs and 24 outputs, the outputs are connected directly to the GIC.
The idea is that the GIC handles everything, and just request a mapping from an IRQ number (0...127, from a device's DT entry) into one of its 24 input lines.

By looking at current code (4.7-rc1) there seems to be a driver (drivers/irqchip/irq-crossbar.c) that provides similar functionality.
The driver uses hierarchical irq domains (since commit 783d31863fb8 "irqchip: crossbar: Convert dra7 crossbar to stacked domains") which we believe we don't need because the only controller is the GIC. However the API used previously, register_routable_domain_ops(), was removed with commit a5561c3e845c "irqchip: gic: Get rid of routable domain".

Trying to use the driver with hierarchical domains (after modifications for our SoC), results on the kernel being blocked at some point:

[    0.041524] ThumbEE CPU extension supported.
[    0.041589] Registering SWP/SWPB emulation handler
[    0.052022] Freeing unused kernel memory: 12364K (c029b000 - c0eae000)
[    0.074084] random: dbus-uuidgen urandom read with 0 bits of entropy available

We've put logs on the different domain_ops calls (alloc, free, translate) but they are not called, even if the DT is supposed to tell devices to take interrupts from this controller (*).

Do you have suggestions on what APIs should be used, further reading/examples and/or pointers on how debug this (logs to enable, things to look for, etc.)?

Thanks in advance.
Best regards,

Sebastian

(*):
here's the diff on our DT:

@@ -47,7 +47,7 @@
 
 	soc {
 		compatible = "simple-bus";
-		interrupt-parent = <&irq0>;
+		interrupt-parent = <&irq_mux>;
 		#address-cells = <1>;
 		#size-cells = <1>;
 		ranges;

@@ -75,7 +75,7 @@
 		uart: serial@10700 {
 			compatible = "ralink,rt2880-uart";
 			reg = <0x10700 0x30>;
-			interrupts = <1 IRQ_TYPE_LEVEL_HIGH>;
+			interrupts = <GIC_SPI 1 IRQ_TYPE_LEVEL_HIGH>;
 			clock-frequency = <7372800>;
 			reg-shift = <2>;
 		};

@@ -83,10 +83,11 @@
 		eth0: ethernet@26000 {
 			compatible = "sigma,smp8734-ethernet";
 			reg = <0x26000 0x800>;
-			interrupts = <38 IRQ_TYPE_LEVEL_HIGH>;
+			interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>;
 			clocks = <&clkgen 1>;
 		};
 
+#if 0
 		intc: interrupt-controller@6e000 {
 			compatible = "sigma,smp8642-intc";
 			reg = <0x6e000 0x400>;

@@ -117,5 +118,16 @@
 				interrupts = <GIC_SPI 4 IRQ_TYPE_LEVEL_HIGH>;
 			};
 		};
+#else
+		irq_mux: irq_mux@6f800 {
+			 compatible = "sigma,smp-irq-mux";
+			 reg = <0x6f800 0x400>;
+			 interrupt-controller;
+			 interrupt-parent = <&gic>;
+			 irqs-reserved = <2 3 4 125 126 127>;
+		};
+
+#endif
+
 	};
 };

"Just request a mapping...".

So you need it, but you don't need it? The GIC may be the only interrupt controller with which software interacts when the interrupt occurs, but the crossbar does play a major role in *routing* the interrupt to the right GIC pin.

And every day, I thank $DEITY for having delivered us from this evil. Really. And it wasn't much of an API. It was the son of a hack, bolted on the side of another hack. Unmaintainable, getting in the way. I had much fun slaughtering it! ;-)

Sorry, but that's not much of a log. Anything related to interrupts, maybe?

At all? Nobody is talking to the GIC?

You could start with posting actual logs of an interrupt being requested, as well as perform some basic tracing of the various callbacks into the irqdomain and irqchip layers, all the way down to the interrupt controllers (note the plural). Also, without seeing the code, it is pretty difficult to make any meaningful comment...

	M.

Jazz is not dead. It just smells funny...

Beer and cheese? Beurk! ;-)

I think Sebastian is even more baffled by the DT mess (sorry, intricacies) than I am.

The base file he was referring to is:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/tango4-common.dtsi

Regards.

I don't think you've ever tried (or at least, not with proper cheese and/or beer).

This mess is what has saved us from the apocalypse 5 years ago, and describing a complex system is not easy (what a surprise...). If you just want to apply recipes without understanding the underlying constraints, you're in for a lot of pain.

I know which file that is, it is mentioned in the diff. I was merely trying to point out the glaring mistakes that could be enough for a interrupt controller hierarchy to be completely non-functional:

- Your crossbar doesn't have a #interrupt-cells property. How do you expect the interrupt specifiers to be interpreted?
- You've changed the default interrupt controller to be your crossbar. Which means that all the sub-nodes are inheriting it. Have you checked that this was valid for all of these nodes?

And as it has been pointed out before, you seem to be reusing an existing driver. Do you know for sure that the usage constraints are similar?

	M.

Jazz is not dead. It just smells funny.

The problem with some Linux APIs is that they're logical and obvious to people who've been using them for years. For newcomers, it's not always so obvious.

In this specific instance, the problem statement seems rather simple, on the surface. An interrupt controller, X=0..127 lines in, Y=0..23 lines out (connected to GIC interrupt lines 0..23) and "all" we need is a way to map Xs to Ys.

As a first order approximation, it's enough to map all Xs to 0. And provide a way for the kernel to check the registers containing the bit-vectors indicating which interrupt(s) fired.

For example, the IRQ driver for Tango3/4 calls

irq_find_mapping
generic_handle_irq
irq_desc_get_handler_data
irq_desc_get_chip
chained_irq_enter/chained_irq_exit
irq_setup_alt_chip
irq_get_domain_generic_chip
irq_domain_add_linear
irq_alloc_domain_generic_chips
irq_set_chained_handler
irq_set_handler_data

Taking irq_find_mapping, I see that there's a short comment in kernel/irq/irqdomain.c

/**
* irq_find_mapping() - Find a linux irq from an hw irq number.
* @domain: domain owning this hardware interrupt
* @hwirq: hardware irq number in that domain space */

Is this Doxygen format? Is there a make target to generate some documentation?

Other relevant resources, for my own reference:

https://www.kernel.org/doc/Documentation/IRQ-domain.txt
https://stackoverflow.com/questions/34371352/what-are-linux-irq-domains-why-are-they-needed
https://community.nxp.com/thread/332183

Are there other important kernel documentation?

Only the name of the file was provided, not the path. I was not aware that you already knew where to find it. I made no claim whatsoever on the implementation. In fact, I agree with everything Lennart wrote.

Why do "fundamental" DT properties start with hash?

I'm not sure I follow. All platform interrupts flow into the platform controller. Maybe other platforms have more complex setups, with several cascaded controllers?

Have a nice week end :-)

If that's what your hardware is, then you are taking the wrong approach. The irq-crossbar driver does not do that at all: it has x inputs and y outputs, but connects exactly *one input to one output*. No multiplexing.

And the hierarchical domain infrastructure enforces a similar property:
a Linux interrupt is dealt with at each level of the hierarchy without
multiplexing: the "irq" is the same, while the "hwirq" varies to reflect the "input pin" for a given interrupt controller.

In your particular case, you have an evolved chained interrupt controller, and nothing else.

Try "make help".

Because # is a shorthand for "number of".

Most embedded platforms do.

	M.

Jazz is not dead. It just smells funny.

Connecting one input to one output is possible iff x=y right? (In other words, a bijection.)

Is it possible to support such an "evolved chained intc" through DT only, or does it require a few function calls from driver code?

Documentation targets:
Linux kernel internal documentation in different formats:

  htmldocs        - HTML
  pdfdocs         - PDF
  psdocs          - Postscript
  xmldocs         - XML DocBook
  mandocs         - man pages
  installmandocs  - install man pages generated by mandocs
  cleandocs       - clean all generated DocBook files

My imagination is lacking, I don't see why it needs to be more complex than N platform input lines, and M output lines feeding into the GIC (with M <= N)

(Our previous intc had N=64 and M=3; the new one has N=128 and M=24)

Regards.

It is *always* possible to connect anything to anything else. You were assuming that this particular driver was fitting your particular case, and it is obvious that it is not (iow: the crossbar transformation cannot be surjective).

There is no such thing as "DT only". You will have to do some actual irqchip development.

It is not more complex. It is different.

	M.

Jazz is not dead. It just smells funny...

The way the TI irq crossbar works is:

You have say 200 interrupt sources in the system. You have some coprocessor (say PRU0) that you want to handle UART7, so you tell the
crossbar: Please map the IRQ from UART7 (which might be input 146 on the crossbar) to IRQ 10 on the PRU0 processor. Now whenever the interrupt from UART7 fires, PRU0 will see it as IRQ10. There are no interrupt registers to check in the crossbar, it is purely virtual wires for routing interrupts. It just happens to be an enourmouse and very very flexible set of wires.

If on the other hand you have 128 inputs and 24 outputs and you want all 128 inputs to work, and you have 24 128bit registers in the device to tell you which of the inputs fired to cause a given output to fire, then you in fact have an interrupt controller, not a crossbar. This would be much more similar to how the original x86 design has an i8259 connected with 8 inputs and 1 output to an input on a second i8259 (the output of the secondary chip connects to IRQ2 on the primary). You actually need an irq controller driver to go read the registers to determine which external interrupt fired and to assign a virtual IRQ number so that drivers can register for a given externel pin. Cascaded interrupt controllers like this is perfectly common on lots of systems.

Now of course you could cheat and simply declare that all the inputs that are mixed to a single output are in fact sharing a single interrupt (fine if they are level triggered, could be problematic if edge triggered, depending on how it works exactly). Of course then when an interrupt happens, every driver that handles some device on a given shared interrupt will have the handler called in turn until everyone has had a change to handle the interrupt, at which point everyone should have stopped triggering it and it will clear itself, unless of course another event happened on one of the devices in which case another loop through everyone happens to handle the new events. The loop will continue until the interrupt clears, as long as some of the handlers return that they in fact did do some work to handle the interrupt event. If no one claims to have done something and the interrupt stays, you get the kernel killing the interrupt with a message about "Interrupt foo happened and nobody cared".

I get the impression yours is not as simple as you thought either.

len Sorensen

Hi Lennart,

Thanks for your input, please find my comments below.

Ok, thanks for the information.

Actually we have 128 inputs and 24 outputs, the 24 outputs go straight to the GIC.
The HW block is a many-to-many router.
There are 128 32bit registers which specify, for each of the corresponding 128 inputs, to which of the 24 outputs it would be routed to.

There are 4 32bit registers that can show the RAW status of the 128 inputs, but they do not latch on the inputs.
That's why our understanding is that on Linux terms it is not an interrupt controller, but just a many-to-many mux, the only real interrupt-controller (where one can set if the line is active high or low for example) is the GIC.

Thanks for the background on the i8259 and the cascaded interrupts.
However, our understanding is that it would only be required if more than 24 devices request IRQ lines, in which case, some of them would have to share a single GIC IRQ line, right? Shall we worry about that now?

This is interesting.
We have one interrupt controller already upstream, drivers/irqchip/irq-tango.c, and our understanding is that it dispatches one IRQ at the time, see tangox_dispatch_irqs() function, is that what you are discussing?

Best regards,

Sebastian

Well that does just sound like a mux. But that does mean you either can't use more than 24 inputs at once, or you will be sharing interrupts.

I really hate shared interrutps so I would never design something that way, but it is simpler.

Well if you are sure you never need more than 24 devices registered at once, then it shouldn't be a problem.

That does look like a proper interrupt controller that could be cascaded of another one if needed.

Len Sorensen

If I am not mistaken, the Cortex A9 MPCore GIC has 32 inputs.

So any SoC with more than 32 devices capable of generating IRQs would have to share interrupts, right?

Regards.

Up to 224 SPIs actually, plus 16 PPIs.

	M.

Jazz is not dead. It just smells funny...

No, you simply add another interrupt controller to cascade it. That way every device can still have one interrupt it owns.

One interrupt on the GIC is owned by the second interrupt controller, and its driver determines which actual external interrupt occured.

It is only shared when there is no way to determine which device caused a given interrupt to happen and you have to ask every one of the devices sharing it if they caused it.

Len Sorensen

Hi Marc,

Thanks for your input, please find my comments below.

Nobody said we wanted a recipe, but it would help to have examples of each of the configurations supported.
Indeed, there is effort in writing the DT bindings and even Documentation/IRQ-domain.txt yet I did not see example drivers for the different configurations. Also, it is difficult to know what is the recommended way, since old APIs are kept for a while.

APIs are usually extended/designed around specific cases, so there must be examples, but they are not easy to find.

Thanks for the pointer, actually, after adding "#interrupt-cells = <3>;" I can see the driver getting requests for domain maps and xlates :-)

What do you mean with "valid for all these nodes"? All HW irq lines go to the crossbar.

It is not being reused as is, I just thought (incorrectly apparently) that it was a good example for our case.

Best regards,

Sebastian

OK, we're getting somewhere.

All the interrupt lines *in the system*? Or just those that are in the nodes below the "soc" section?

Thanks,

	M.

Jazz is not dead. It just smells funny...

Hi Marc,

Thanks for your reply, please find my comments below.

My understanding of "hierarchical irq domains" is that it is useful when there are multiple stacked interrupt controllers. Also, the documentation says "the majority of drivers should use the linear map" (as opposed to the hierarchical one).

Maybe the definition of "interrupt controller" could benefit from some clarification, but my understanding is that, in our case, the GIC is the only interrupt controller (that's where the interrupt type must be set active_high/active_low/edge, etc.), in front of it, it happens to be a crossbar, that happens to be programmable, and that can be used to map any 128 line into any of 24 lines of the GIC (actually it is a many-to-many router, without any latching nor edge detection functionality)

Obviously, when the DT says that ethernet driver uses IRQ=120 (for example), the crossbar must be setup to route line 120 to one of the 24 lines going to the GIC. So a minimum of interaction between the GIC driver (and/or the Linux IRQ framework) and the driver programming the crossbar is required, and that's what we are trying to achieve.

Does that makes sense?

That's the last log (it's stuck there) and I was asking how/where to enable more logs to be able to debug this.

Or there are no standardised logs and every person has to come up with its own debug logs?

Ok, thanks.

In the meanwhile, and in case irq-crossbar is not the good example for our case, would it be possible to get some guidance, examples, tips, on how to write a driver like the one described? ie:

[0..127] IRQ inputs => BIG_MUX => [0..23] outputs => [0..23] GIC inputs

BIG_MUX is a many-to-many router:
- 128x32bit registers to setup a route between any of the 128 input (IRQ_dev) and any of the 24 outputs (IRQ_gic)
- 4x32bit registers that read the RAW status of each of the 128 lines (no latching nor edge detection)
not sure how useful is to read such RAW status, because (naive hypothesis follows) Linux's IRQ framework could remember which IRQ_dev lines are routed to which IRQ_gic, and thus when handling IRQ_gic(x), Linux could just ask the drivers tied to it to check for interruptions.

Base code is either 4.7rc1 or 4.4.
The irq-crossbar code is not much different from TI, but you can find it attached.

Anything missing on the diff can be found on the original file:
arch/arm/boot/dts/tango4-common.dtsi

Thanks in advance.
Best regards,

Sebastian

#include <linux/init.h>
#include <linux/irq.h>
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/ioport.h>
#include <linux/io.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/slab.h>

#define IRQ_MUX_INPUT_LINES (128)
#define IRQ_MUX_OUTPUT_LINES (24)
#define IRQ_FREE (-1)
#define IRQ_RESERVED (-2)

/**
* struct tangox_irq_mux : irq mux private driver data
* @lock: spinlock serializing access to @irq_map
* @mux_inputs: inputs (irq lines entering) the mux
* @mux_outputs: outputs (irq lines exiting) the mux (connected to the GIC)
* @irq_map: irq input->output map
* @reg_base: mux base address
*/

struct tangox_irq_mux {
	raw_spinlock_t lock;
	uint mux_inputs;
	uint mux_outputs;
	uint *irq_map;
	void __iomem *reg_base;
};

#define DBGLOG(__format, ...)					\
	do {							\
		pr_info("[%s:%d] %s(): " __format, __FILE__, __LINE__, __FUNCTION__ , ##__VA_ARGS__); \
	} while (0)

static inline u32 intc_readl(int address)

{
	u32 value = readl_relaxed((void __iomem *)address);
	//DBGLOG("read 0x%x @ 0x%x\n", value, address);
	return value;
}

static inline void intc_writel(int value, int address)

{
	//DBGLOG("write 0x%x @ 0x%x\n", value, address);
	writel_relaxed(value, (void __iomem *)address);
}

static inline void tangox_setup_irq_route(struct tangox_irq_mux *irq_mux_context, int irq_in, int irq_out)

{
	u32 value = irq_out;
	u32 offset = (irq_in * 4);
	u32 address = irq_mux_context->reg_base + offset;

	DBGLOG("route irq %2d (@ 0x%08x) => irq %2d\n", irq_in, address, value);

	if (value)
		value |= 0x80000000;

	intc_writel(value, address);
}

static int tangox_allocate_gic_irq(struct irq_domain *domain,
unsigned virq, irq_hw_number_t hwirq)

{
	struct tangox_irq_mux *irq_mux_context = domain->host_data;
	struct irq_fwspec fwspec;
	int i;
	int err;

	DBGLOG("domain 0x%p, virq %d (0x%x) hwirq %d (0x%x)\n", domain, virq, virq, hwirq, hwirq);

	if (!irq_domain_get_of_node(domain->parent))
		return -EINVAL;

	raw_spin_lock(&(irq_mux_context->lock));
	for (i = irq_mux_context->mux_outputs - 1; i >= 0; i--) {
		if (irq_mux_context->irq_map[i] == IRQ_FREE) {
			irq_mux_context->irq_map[i] = hwirq;
			break;
		}
	}
	raw_spin_unlock(&(irq_mux_context->lock));

	if (i < 0)
		return -ENODEV;

	fwspec.fwnode = domain->parent->fwnode;
	fwspec.param_count = 3;
	fwspec.param[0] = 0;	/* SPI */
	fwspec.param[1] = i;
	fwspec.param[2] = IRQ_TYPE_LEVEL_HIGH;

	err = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);
	if (err)
		irq_mux_context->irq_map[i] = IRQ_FREE;
	else
		tangox_setup_irq_route(irq_mux_context, hwirq, i);

	return err;
}

static struct irq_chip mux_chip = {
	.name			= "CBAR",
	.irq_eoi		= irq_chip_eoi_parent,
	.irq_mask		= irq_chip_mask_parent,
	.irq_unmask		= irq_chip_unmask_parent,
	.irq_retrigger		= irq_chip_retrigger_hierarchy,
	.irq_set_type		= irq_chip_set_type_parent,
	.flags			= IRQCHIP_MASK_ON_SUSPEND |
				  IRQCHIP_SKIP_SET_WAKE,

#ifdef CONFIG_SMP
.irq_set_affinity = irq_chip_set_affinity_parent,
#endif
};

/**
* tangox_irq_mux_domain_alloc - map/reserve a mux<->irq connection
* @domain: domain of irq to map
* @virq: virq number
* @nr_irqs: number of irqs to reserve
*
*/
static int tangox_irq_mux_domain_alloc(struct irq_domain *domain,
unsigned int virq, unsigned int nr_irqs, void *data)

{
	struct tangox_irq_mux *irq_mux_context = domain->host_data;
	struct irq_fwspec *fwspec = data;
	irq_hw_number_t hwirq;
	int i;

	DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

	if (fwspec->param_count != 3)
		return -EINVAL;	/* Not GIC compliant */
	if (fwspec->param[0] != 0)
		return -EINVAL;	/* No PPI should point to this domain */

	hwirq = fwspec->param[1];
	if ((hwirq + nr_irqs) > irq_mux_context->mux_inputs)
		return -EINVAL;	/* Can't deal with this */

	for (i = 0; i < nr_irqs; i++) {
		int err = tangox_allocate_gic_irq(domain, virq + i, hwirq + i);

		if (err)
			return err;

		irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
					      &mux_chip, NULL);
	}

	return 0;
}

/**
* tangox_irq_mux_domain_free - unmap/free a mux<->irq connection
* @domain: domain of irq to unmap
* @virq: virq number
* @nr_irqs: number of irqs to free
*
* We do not maintain a use count of total number of map/unmap
* calls for a particular irq to find out if a irq can be really
* unmapped. This is because unmap is called during irq_dispose_mapping(irq),
* after which irq is anyways unusable. So an explicit map has to be called
* after that.
*/
static void tangox_irq_mux_domain_free(struct irq_domain *domain,
unsigned int virq, unsigned int nr_irqs)

{
	int i;
	struct tangox_irq_mux *irq_mux_context = domain->host_data;

	DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

	raw_spin_lock(&(irq_mux_context->lock));
	for (i = 0; i < nr_irqs; i++) {
		struct irq_data *irqdata = irq_domain_get_irq_data(domain, virq + i);

		irq_domain_reset_irq_data(irqdata);
		irq_mux_context->irq_map[irqdata->hwirq] = IRQ_FREE;
		tangox_setup_irq_route(irq_mux_context, 0x0, irqdata->hwirq);
	}
	raw_spin_unlock(&(irq_mux_context->lock));
}

static int tangox_irq_mux_domain_translate(struct irq_domain *domain,
struct irq_fwspec *fwspec, unsigned long *hwirq, unsigned int *type)
{

	DBGLOG("domain 0x%p\n", domain);

	if (is_of_node(fwspec->fwnode)) {
		if (fwspec->param_count != 3)
			return -EINVAL;

		/* No PPI should point to this domain */
		if (fwspec->param[0] != 0)
			return -EINVAL;

		*hwirq = fwspec->param[1];
		*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
		return 0;
	}

	return -EINVAL;
}

static const struct irq_domain_ops tangox_irq_mux_domain_ops = {
.alloc = tangox_irq_mux_domain_alloc,
.free = tangox_irq_mux_domain_free,
.translate = tangox_irq_mux_domain_translate,
};

static int __init tangox_irq_mux_init(struct device_node *node,
struct tangox_irq_mux *irq_mux_context)

{
	int i, per_irq_reg_size, max_in, max_out, offset, entry, entry_size;
	const __be32 *reserved_irq_list;
	int ret = -ENOMEM;

	DBGLOG("init\n");

	if (!irq_mux_context)
		return -ENOMEM;

	irq_mux_context->reg_base = of_iomap(node, 0);
	if (!irq_mux_context->reg_base)
		goto err_exit;

	irq_mux_context->mux_inputs  = IRQ_MUX_INPUT_LINES;
	irq_mux_context->mux_outputs = IRQ_MUX_OUTPUT_LINES;

	max_in  = irq_mux_context->mux_inputs;
	max_out = irq_mux_context->mux_outputs;

	irq_mux_context->irq_map = kcalloc(max_out, sizeof(int), GFP_KERNEL);
	if (!irq_mux_context->irq_map)
		goto err_unmap_base;

	for (i = 0; i < max_out; i++)
		irq_mux_context->irq_map[i] = IRQ_FREE;

	// mark reserved IRQ lines
	reserved_irq_list = of_get_property(node, "irqs-reserved", &entry_size);
	if (reserved_irq_list) {
		entry_size /= sizeof(__be32);

		DBGLOG("setting up %d reserved irqs\n", entry_size);

		for (i = 0; i < entry_size; i++) {
			of_property_read_u32_index(node,
						   "irqs-reserved",
						   i, &entry);
			if (entry >= max_out) {
				pr_err("Invalid reserved entry %d > %d: ignored\n", entry, max_out);
				continue;
			}
			irq_mux_context->irq_map[entry] = IRQ_RESERVED;
		}
	}

	DBGLOG("disabling free IRQs\n");

	// disable free IRQs during initialisation
	for (i = 0; i < max_in; i++) {
		tangox_setup_irq_route(irq_mux_context, i, 0);
	}

	DBGLOG("init backward compatible map\n");

	tangox_setup_irq_route(irq_mux_context, 125, 2);
	tangox_setup_irq_route(irq_mux_context, 126, 3);
	tangox_setup_irq_route(irq_mux_context, 127, 4);

	raw_spin_lock_init(&irq_mux_context->lock);

	return 0;

err_free_map:

	kfree(irq_mux_context->irq_map);
err_unmap_base:
	iounmap(irq_mux_context->reg_base);
err_exit:
	return ret;
}

static int __init tangox_irq_mux_deinit(struct tangox_irq_mux *irq_mux_context)

{
	if (!irq_mux_context)
		return -ENOMEM;

	if (irq_mux_context->reg_base)
		iounmap(irq_mux_context->reg_base);

	if (irq_mux_context->irq_map)
		kfree(irq_mux_context->irq_map);

	kfree(irq_mux_context);

	return 0;
}

static int __init tangox_of_irq_mux_init(struct device_node *node,
struct device_node *parent)

{
	struct tangox_irq_mux *irq_mux_context;
	struct irq_domain *parent_domain, *domain;
	int err;

	DBGLOG("irqv2 begin\n");

	if (!parent) {
		pr_err("%s: no parent, giving up\n", node->full_name);
		return -ENODEV;
	}

	parent_domain = irq_find_host(parent);
	if (!parent_domain) {
		pr_err("%s: unable to obtain parent domain\n", node->full_name);
		return -ENXIO;
	}

	irq_mux_context = kzalloc(sizeof(*irq_mux_context), GFP_KERNEL);

	err = tangox_irq_mux_init(node, irq_mux_context);
	if (err) {
		pr_err("%s: init failed (%d)\n", node->full_name, err);
		tangox_irq_mux_deinit(irq_mux_context);
		return err;
	}

	domain = irq_domain_add_hierarchy(parent_domain, 0,
					  irq_mux_context->mux_inputs,
					  node, &tangox_irq_mux_domain_ops,
					  irq_mux_context);
	if (!domain) {
		pr_err("%s: failed to allocated domain\n", node->full_name);
		tangox_irq_mux_deinit(irq_mux_context);
		return -ENOMEM;
	}

	return 0;

}
IRQCHIP_DECLARE(tangox_intc, "sigma,smp-irq-mux", tangox_of_irq_mux_init);

Sebastian,

The "linear map" to be opposed to the "tree", not to the hierarchy. Hierarchies themselves can be built out most domain type (only the underlying data structure changes).

An interrupt controller is absolutely *anything* that is on the interrupt path, unless it is absolutely transparent, invisible to software, and doesn't require any form of configuration. Your own definition of an interrupt controller is way too restrictive.

Maybe you and Mason should get together and decide what you want to support. Because you seem to have diverging requirements (Mason suggesting the exact opposite over the weekend).

We have basic tracing in the irqdomain layer, and you can then instrument your own driver.

OK, so this is definitely a pure router, and the lack of latch makes it completely unsuitable for a a cascaded interrupt controller. At least, we've managed to establish that this thing will never be able to handle more than 24 devices in a sane way. So let's forget about Mason's idea of cascading everything to a single output line, and let's focus on your initial idea of having something similar to TI's crossbar, which is a much saner approach.

Please post it separately (and inline), the email client I have here makes it hard to review attached patches.

Thanks,

	M.

Jazz is not dead. It just smells funny...

Hi Marc,

Thanks for the clarification.

Well, one could imagine that this many-to-many router could be pre-configured, and thus not require any form of configuration from Linux, yet, somehow the resulting mapping needs to be communicated to Linux, right? What APIs calls should be used?

I see, thanks for the clarification.

IIUC what you refer to, when Mason said that we could route all 128 lines to a single GIC line he was talking about a first order approximation.

I followed this https://www.kernel.org/doc/local/pr_debug.txt for kernel/irq/irqdomain.c

Do you want the whole boot log? Or just the snippets from irqdomain.c and those from our driver?

Ok, so after discussing with some HW engineers, they said that even if this is a pure router and cannot latch by itself, since the devices themselves latch their IRQ output, reading the 4x32bit RAW status registers could work as well, that means that if there are more than 24 devices, some could share IRQs, right?

Two questions then:
a) let's say we need to share some of the IRQs, which APIs should be used?

b) some people have been asking about IRQ affinity, they have not been clear about it, but I suppose maybe they want to redistribute the IRQs. In this case, when using IRQ sharing a device may go from sharing an IRQ line to an exclusive line or viceversa, right? Does Linux handles that on its own or there's some API to call as well?

Ok, I'll post it in a separate email and inline.

Best regards,

Sebastian

Here it goes:

#include <linux/init.h>
#include <linux/irq.h>
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/ioport.h>
#include <linux/io.h>
#include <linux/of_address.h>
#include <linux/of_irq.h>
#include <linux/slab.h>

#define IRQ_MUX_INPUT_LINES (128)
#define IRQ_MUX_OUTPUT_LINES (24)
#define IRQ_FREE (-1)
#define IRQ_RESERVED (-2)

/**
* struct tangox_irq_mux : irq mux private driver data
* @lock: spinlock serializing access to @irq_map
* @mux_inputs: inputs (irq lines entering) the mux
* @mux_outputs: outputs (irq lines exiting) the mux (connected to the GIC)
* @irq_map: irq input->output map
* @reg_base: mux base address
*/

struct tangox_irq_mux {
	raw_spinlock_t lock;
	uint mux_inputs;
	uint mux_outputs;
	uint *irq_map;
	void __iomem *reg_base;
};

#define DBGLOG(__format, ...)					\
	do {							\
		pr_info("[%s:%d] %s(): " __format, __FILE__, __LINE__, __FUNCTION__ , ##__VA_ARGS__); \
	} while (0)

static inline u32 intc_readl(int address)

{
	u32 value = readl_relaxed((void __iomem *)address);
	//DBGLOG("read 0x%x @ 0x%x\n", value, address);
	return value;
}

static inline void intc_writel(int value, int address)

{
	//DBGLOG("write 0x%x @ 0x%x\n", value, address);
	writel_relaxed(value, (void __iomem *)address);
}

static inline void tangox_setup_irq_route(struct tangox_irq_mux *irq_mux_context, int irq_in, int irq_out)

{
	u32 value = irq_out;
	u32 offset = (irq_in * 4);
	u32 address = irq_mux_context->reg_base + offset;

	DBGLOG("route irq %2d (@ 0x%08x) => irq %2d\n", irq_in, address, value);

	if (value)
		value |= 0x80000000;

	intc_writel(value, address);
}

static int tangox_allocate_gic_irq(struct irq_domain *domain,
unsigned virq, irq_hw_number_t hwirq)

{
	struct tangox_irq_mux *irq_mux_context = domain->host_data;
	struct irq_fwspec fwspec;
	int i;
	int err;

	DBGLOG("domain 0x%p, virq %d (0x%x) hwirq %d (0x%x)\n", domain, virq, virq, hwirq, hwirq);

	if (!irq_domain_get_of_node(domain->parent))
		return -EINVAL;

	raw_spin_lock(&(irq_mux_context->lock));
	for (i = irq_mux_context->mux_outputs - 1; i >= 0; i--) {
		if (irq_mux_context->irq_map[i] == IRQ_FREE) {
			irq_mux_context->irq_map[i] = hwirq;
			break;
		}
	}
	raw_spin_unlock(&(irq_mux_context->lock));

	if (i < 0)
		return -ENODEV;

	fwspec.fwnode = domain->parent->fwnode;
	fwspec.param_count = 3;
	fwspec.param[0] = 0;	/* SPI */
	fwspec.param[1] = i;
	fwspec.param[2] = IRQ_TYPE_LEVEL_HIGH;

	err = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);
	if (err)
		irq_mux_context->irq_map[i] = IRQ_FREE;
	else
		tangox_setup_irq_route(irq_mux_context, hwirq, i);

	return err;
}

static struct irq_chip mux_chip = {
	.name			= "CBAR",
	.irq_eoi		= irq_chip_eoi_parent,
	.irq_mask		= irq_chip_mask_parent,
	.irq_unmask		= irq_chip_unmask_parent,
	.irq_retrigger		= irq_chip_retrigger_hierarchy,
	.irq_set_type		= irq_chip_set_type_parent,
	.flags			= IRQCHIP_MASK_ON_SUSPEND |
				  IRQCHIP_SKIP_SET_WAKE,

#ifdef CONFIG_SMP
.irq_set_affinity = irq_chip_set_affinity_parent,
#endif
};

/**
* tangox_irq_mux_domain_alloc - map/reserve a mux<->irq connection
* @domain: domain of irq to map
* @virq: virq number
* @nr_irqs: number of irqs to reserve
*
*/
static int tangox_irq_mux_domain_alloc(struct irq_domain *domain,
unsigned int virq, unsigned int nr_irqs, void *data)

{
	struct tangox_irq_mux *irq_mux_context = domain->host_data;
	struct irq_fwspec *fwspec = data;
	irq_hw_number_t hwirq;
	int i;

	DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

	if (fwspec->param_count != 3)
		return -EINVAL;	/* Not GIC compliant */
	if (fwspec->param[0] != 0)
		return -EINVAL;	/* No PPI should point to this domain */

	hwirq = fwspec->param[1];
	if ((hwirq + nr_irqs) > irq_mux_context->mux_inputs)
		return -EINVAL;	/* Can't deal with this */

	for (i = 0; i < nr_irqs; i++) {
		int err = tangox_allocate_gic_irq(domain, virq + i, hwirq + i);

		if (err)
			return err;

		irq_domain_set_hwirq_and_chip(domain, virq + i, hwirq + i,
					      &mux_chip, NULL);
	}

	return 0;
}

/**
* tangox_irq_mux_domain_free - unmap/free a mux<->irq connection
* @domain: domain of irq to unmap
* @virq: virq number
* @nr_irqs: number of irqs to free
*
* We do not maintain a use count of total number of map/unmap
* calls for a particular irq to find out if a irq can be really
* unmapped. This is because unmap is called during irq_dispose_mapping(irq),
* after which irq is anyways unusable. So an explicit map has to be called
* after that.
*/
static void tangox_irq_mux_domain_free(struct irq_domain *domain,
unsigned int virq, unsigned int nr_irqs)

{
	int i;
	struct tangox_irq_mux *irq_mux_context = domain->host_data;

	DBGLOG("domain 0x%p, virq %d (0x%x) nr_irqs %d\n", domain, virq, virq, nr_irqs);

	raw_spin_lock(&(irq_mux_context->lock));
	for (i = 0; i < nr_irqs; i++) {
		struct irq_data *irqdata = irq_domain_get_irq_data(domain, virq + i);

		irq_domain_reset_irq_data(irqdata);
		irq_mux_context->irq_map[irqdata->hwirq] = IRQ_FREE;
		tangox_setup_irq_route(irq_mux_context, 0x0, irqdata->hwirq);
	}
	raw_spin_unlock(&(irq_mux_context->lock));
}

static int tangox_irq_mux_domain_translate(struct irq_domain *domain,
struct irq_fwspec *fwspec, unsigned long *hwirq, unsigned int *type)
{

	DBGLOG("domain 0x%p\n", domain);

	if (is_of_node(fwspec->fwnode)) {
		if (fwspec->param_count != 3)
			return -EINVAL;

		/* No PPI should point to this domain */
		if (fwspec->param[0] != 0)
			return -EINVAL;

		*hwirq = fwspec->param[1];
		*type = fwspec->param[2] & IRQ_TYPE_SENSE_MASK;
		return 0;
	}

	return -EINVAL;
}

static const struct irq_domain_ops tangox_irq_mux_domain_ops = {
.alloc = tangox_irq_mux_domain_alloc,
.free = tangox_irq_mux_domain_free,
.translate = tangox_irq_mux_domain_translate,
};

static int __init tangox_irq_mux_init(struct device_node *node,
struct tangox_irq_mux *irq_mux_context)

{
	int i, per_irq_reg_size, max_in, max_out, offset, entry, entry_size;
	const __be32 *reserved_irq_list;
	int ret = -ENOMEM;

	DBGLOG("init\n");

	if (!irq_mux_context)
		return -ENOMEM;

	irq_mux_context->reg_base = of_iomap(node, 0);
	if (!irq_mux_context->reg_base)
		goto err_exit;

	irq_mux_context->mux_inputs  = IRQ_MUX_INPUT_LINES;
	irq_mux_context->mux_outputs = IRQ_MUX_OUTPUT_LINES;

	max_in  = irq_mux_context->mux_inputs;
	max_out = irq_mux_context->mux_outputs;

	irq_mux_context->irq_map = kcalloc(max_out, sizeof(int), GFP_KERNEL);
	if (!irq_mux_context->irq_map)
		goto err_unmap_base;

	for (i = 0; i < max_out; i++)
		irq_mux_context->irq_map[i] = IRQ_FREE;

	// mark reserved IRQ lines
	reserved_irq_list = of_get_property(node, "irqs-reserved", &entry_size);
	if (reserved_irq_list) {
		entry_size /= sizeof(__be32);

		DBGLOG("setting up %d reserved irqs\n", entry_size);

		for (i = 0; i < entry_size; i++) {
			of_property_read_u32_index(node,
						   "irqs-reserved",
						   i, &entry);
			if (entry >= max_out) {
				pr_err("Invalid reserved entry %d > %d: ignored\n", entry, max_out);
				continue;
			}
			irq_mux_context->irq_map[entry] = IRQ_RESERVED;
		}
	}

	DBGLOG("disabling free IRQs\n");

	// disable free IRQs during initialisation
	for (i = 0; i < max_in; i++) {
		tangox_setup_irq_route(irq_mux_context, i, 0);
	}

	DBGLOG("init backward compatible map\n");

	tangox_setup_irq_route(irq_mux_context, 125, 2);
	tangox_setup_irq_route(irq_mux_context, 126, 3);
	tangox_setup_irq_route(irq_mux_context, 127, 4);

	raw_spin_lock_init(&irq_mux_context->lock);

	return 0;

err_free_map:

	kfree(irq_mux_context->irq_map);
err_unmap_base:
	iounmap(irq_mux_context->reg_base);
err_exit:
	return ret;
}

static int __init tangox_irq_mux_deinit(struct tangox_irq_mux *irq_mux_context)

{
	if (!irq_mux_context)
		return -ENOMEM;

	if (irq_mux_context->reg_base)
		iounmap(irq_mux_context->reg_base);

	if (irq_mux_context->irq_map)
		kfree(irq_mux_context->irq_map);

	kfree(irq_mux_context);

	return 0;
}

static int __init tangox_of_irq_mux_init(struct device_node *node,
struct device_node *parent)

{
	struct tangox_irq_mux *irq_mux_context;
	struct irq_domain *parent_domain, *domain;
	int err;

	DBGLOG("irqv2 begin\n");

	if (!parent) {
		pr_err("%s: no parent, giving up\n", node->full_name);
		return -ENODEV;
	}

	parent_domain = irq_find_host(parent);
	if (!parent_domain) {
		pr_err("%s: unable to obtain parent domain\n", node->full_name);
		return -ENXIO;
	}

	irq_mux_context = kzalloc(sizeof(*irq_mux_context), GFP_KERNEL);

	err = tangox_irq_mux_init(node, irq_mux_context);
	if (err) {
		pr_err("%s: init failed (%d)\n", node->full_name, err);
		tangox_irq_mux_deinit(irq_mux_context);
		return err;
	}

	domain = irq_domain_add_hierarchy(parent_domain, 0,
					  irq_mux_context->mux_inputs,
					  node, &tangox_irq_mux_domain_ops,
					  irq_mux_context);
	if (!domain) {
		pr_err("%s: failed to allocated domain\n", node->full_name);
		tangox_irq_mux_deinit(irq_mux_context);
		return -ENOMEM;
	}

	return 0;

}
IRQCHIP_DECLARE(tangox_intc, "sigma,smp-irq-mux", tangox_of_irq_mux_init);

Hi Marc,

<snipped code>

I have tested the code, and aside from the missing #interrupt-cells in the DT that you pointed out, it seems it is working (devices using IRQ appear functional), here's some log:

tangox_irq_mux_domain_translate(): domain 0xcf805000 tangox_irq_mux_domain_translate(): hwirq 1 (0x1) type 4 (0x4) tangox_irq_mux_domain_alloc(): domain 0xcf805000, virq 18 (0x12) nr_irqs 1 tangox_allocate_gic_irq(): domain 0xcf805000, virq 18 (0x12) hwirq 1 (0x1) tangox_setup_irq_route(): route irq 1 (@ 0xf006f804) => irq 23 tangox_irq_mux_domain_translate(): domain 0xcf805000 tangox_irq_mux_domain_translate(): hwirq 38 (0x26) type 4 (0x4) tangox_irq_mux_domain_alloc(): domain 0xcf805000, virq 19 (0x13) nr_irqs 1 tangox_allocate_gic_irq(): domain 0xcf805000, virq 19 (0x13) hwirq 38 (0x26) tangox_setup_irq_route(): route irq 38 (@ 0xf006f898) => irq 22 tangox_irq_mux_domain_translate(): domain 0xcf805000 tangox_irq_mux_domain_translate(): hwirq 67 (0x43) type 4 (0x4) tangox_irq_mux_domain_alloc(): domain 0xcf805000, virq 20 (0x14) nr_irqs 1 tangox_allocate_gic_irq(): domain 0xcf805000, virq 20 (0x14) hwirq 67 (0x43) tangox_setup_irq_route(): route irq 67 (@ 0xf006f90c) => irq 21

Since irq-tango_v2.c is similar to irq-crossbar.c from TI (since it is based on it), I was wondering what is the policy or recommendation in such cases? Should I attempt to merge the code (mainly the way to set up the registers) on irq-crossbar.c or should we consider irq-tango_v2.c to live its own life?

NOTE: IMHO, irq-crossbar.c could benefit from clearer names for some DT properties, because "max_irqs" and "max-crossbar-sources" are not straight forward names for "mux_outputs" and "mux_inputs" (respectively)

NOTE2: current irq-tango_v2.c code still has a 24 IRQ limitation since it is not using any API that would allow it to setup IRQ sharing.

Thanks in advance.
Best regards,

Sebastian

[been away for a while, catching up...]

If the HW is significantly different, I'd rather have a separate driver. We can always share some things later on by having a small library of "stuff".

Maybe, but this ship has sailed a long time ago. This is an ABI now, and it is not going to change unless proven to be broken. On the other hand, you can name your own properties as you see fit.

Unless you limit your mux level interrupts only, I cannot see how you could deal with cascaded interrupts. By the time you receive an edge, the line will have dropped, and you won't be able to identify the source interrupt.

Thanks,

	M.

Jazz is not dead. It just smells funny...

Hi Marc,

I'd say it is very similar. Most of the changes I did were done to understand how it worked. However, it may end up being different if we use cascaded interrupts.

Ok.

Yes, cascaded interrupts would be limited to level only.

By the way, did you see my other questions? (copy/pasted here for convenience):

----
Ok, so after discussing with some HW engineers, they said that even if this is a pure router and cannot latch by itself, since the devices themselves latch their IRQ output, reading the 4x32bit RAW status registers could work as well, that means that if there are more than 24 devices, some could share IRQs, right?

Two questions then:
a) let's say we need to share some of the IRQs, which APIs should be used?

b) some people have been asking about IRQ affinity, they have not been clear about it, but I suppose maybe they want to redistribute the IRQs. In this case, when using IRQ sharing a device may go from sharing an IRQ line to an exclusive line or viceversa, right? Does Linux handles that on its own or there's some API to call as well?
----

About a) I did not find any driver that uses irq_domain_add_linear() and irq_domain_add_hierarchy() but maybe I'm not approaching the problem from the right angle.

Best regards,

Sebastian

As mentioned earlier, this only works for level interrupts. If you enforce this, this is OK. I assume that you also have a way to mask these interrupts, right?

The usual irq_set_chained_handler_and_data()/chained_irq_enter()/chained_irq_exit().

You need to implement the .irq_set_affinity in the irqchip. But it hardly makes sense in your particular case:
- if you're using it as a pure router (no multiplexing), the affinity is controlled by the the GIC itself (i.e nothing to do here, except forwarding the request to the underlying irqchip).
- If you're using it as a chained irqchip, then you can't easily migrate a bunch of interrupt, because this is not what userspace expects.

What you could do would be to dedicate an irq line per CPU, and reconfigure the mux when changing the affinity.

A chained interrupt controller cannot be hierarchical. That's pretty fundamental.

Overall, I think you need to settle for a use case (either pure router or chained controller) and implement that. To support both, you'll need two different implementations (basically two drivers in one).

Thanks,

	M.

Jazz is not dead. It just smells funny...

Hi Marc,

Yes, that's what the bit 31 does (see tangox_setup_irq_route() function in the code I sent).

For reference, here's the documentation of the mux:

----
CPU block interrupt interface is now 32bits.
The 24 first interrupt bits are generated from the system interrupts and the 8 msb interrupts are cpu local interrupts :

IRQs [23:0] tango system irqs.
IRQs [27:24] CPU core cross trigger interface interrupt (1 per core). IRQs [31:28] CPU core PMU (performance unit) interrupt (1 per core).

The 24 lsb interrupts are generated through a new interrupt map module that maps the tango 128 interrupts to those 24 interrupts. For each of the 128 input system interrupt, one register is dedicated to program the destination interrupt among the 24 available. The mapper is configured as follows, starting at address (0x6f800) :

offset name            description
0x000  irq_in_0_cfg    "en"=bit[31]; "inv"=bit[16]; "dest"=bits[4:0]
0x004  irq_in_1_cfg    "en"=bit[31]; "inv"=bit[16]; "dest"=bits[4:0]
.
.
.
0x1FC  irq_in_127_cfg  "en"=bit[31]; "inv"=bit[16]; "dest"=bits[4:0]
0x200  soft_irq_cfg    "enable"=bits[15:0]
0x204  soft_irq_map0   "map3"=bits[28:24]; "map2"=bits[20:16]; "map1"=bits[12:8]; "map0"=bits[4:0]
0x208  soft_irq_map1   "map3"=bits[28:24]; "map2"=bits[20:16]; "map1"=bits[12:8]; "map0"=bits[4:0]
0x20C  soft_irq_map2   "map3"=bits[28:24]; "map2"=bits[20:16]; "map1"=bits[12:8]; "map0"=bits[4:0]
0x210  soft_irq_map3   "map3"=bits[28:24]; "map2"=bits[20:16]; "map1"=bits[12:8]; "map0"=bits[4:0]
0x214  soft_irq_set    "set"=bits[15:0]
0x218  soft_irq_clear  "clear"=bits[15:0]
0x21C  read_cpu_irq    "cpu_block_irq"=bits[23:0]
0x220  read_sys_irq0   "system_irq"=bits[31:0]; (irqs: 0->31)
0x224  read_sys_irq1   "system_irq"=bits[31:0]; (irqs: 32->63)
0x228  read_sys_irq2   "system_irq"=bits[31:0]; (irqs: 64->95)
0x22C  read_sys_irq3   "system_irq"=bits[31:0]; (irqs: 96->127)

irq_in_N_cfg : input N mapping :

- dest bits[4:0]    => set destination interrupt among the 24 output interrupts. (if multiple inputs are mapped to the same output, result is an OR of the inputs).
- inv bit[16]       => if set, inverts input interrupt polarity (active at 0).
- en bit[31]        => enable interrupt. Acts like a mask on the input interrupt. 

soft_irq : this module supports up to 16 software interrupts.
- enable bits[15:0] => enable usage of software IRQs (SIRQ), 1 bit per SIRQ.
soft_irq_mapN : For each of the 16 soft IRQ (SIRQ), map them in out IRQ[23:0] vector.
- mapN => 5 bits to select where to connect the SIRQ among the 23 bits output IRQ. (if multiple SIRQ are mapped to the same output IRQ, result is an OR of those signals).
soft_irq_set : 16bits, write 1 bit at one set the corresponding SIRQ. Read returns the software SIRQ vector value.
soft_irq_clear : 16bits, write 1 bit at one clear the corresponding software SIRQ. Read returns the software SIRQ vector value.
read_cpu_irq : 24bits, returns output IRQ value (IRQs connected to the ARM cluster).
read_sys_irqN : 32bits, returns input system IRQ value before mapping.

Ok, thanks.

At first I thought that I could modify tangox_allocate_gic_irq() so that when running out of IRQ lines going to the GIC, it:
- reserved the last line of the GIC with:

irq = fwspec.param[1];
err = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);

- then, it could add an irqchip to the domain with:

err = irq_alloc_domain_generic_chips(domain, 128, 1, node->name, handle_level_irq, 0, 0, 0); for (i = 0; i < 4; i++) {
gc = irq_get_domain_generic_chip(domain, i * 32);
tangox_irq_init_chip(gc, i * IRQ_CTL_HI, i * EDGE_CTL_HI);
}

irq_set_chained_handler(irq, tangox_irq_handler);
irq_set_handler_data(irq, domain);

Not sure if that makes sense (it's hard to get a clear understanding of all these APIs and their possible interactions)

Ok, that confirms my guess from seeing that the .irq_set_affinity callback is set to a framework callback ('irq_chip_set_affinity_parent') in irq-crossbar.c

Ok, so IIUC, when chaining irqchips (like irq-tango.c) it's hard to migrate the interrupts.

Ok, so basically, we'd use only N lines of the GIC (N=number of CPUs, instead of the 24 available) and multiplex the M>N interrupt lines into the N used inputs of the GIC, right? I think this is similar to what Mason had proposed as "first order approximation".

Oh, sorry I must have misunderstood the terminology. I mean, at the end of IRQ-domain.txt it says:

"3) Optionally implement an irq_chip to manage the interrupt controller
hardware."

which I understood as calls to irq_alloc_domain_generic_chips() with a domain allocated with irq_domain_add_linear()

Ok thanks for the advise.
The pure router, while already implemented, seems to have the issue of not being able to handle more than 24 interrupts, and I'm not sure how to handle the sharing. Also I don't see how would the sharing be described in the DT.

With the old driver (irq-tango.c) I can see how the sharing is accomplished, because there were several calls to irq_domain_add_linear/irq_alloc_domain_generic_chips/irq_set_chained_handler/irq_set_handler_data (one for each IRQ line going to the GIC) and then the DT for the device would specify an interrupt-parent + interrupts keys, but I don't see how to do that with the domain_hierarchy or how to describe that on DT.

Best regards,

Sebastian

Well something has to tell the crossbar to map 120 to one of the 24. Of course the driver for the device needs to be told to use the one of the 24 it was mapped to.

As far as I recall the TI irq-crossbar allows you to specify in the DT that you want to use a given input to the crossbar for a given device, and it dynamically allocated a mapping to a free IRQ on the CPU and maps it and the driver is told the correct interrupt to use for that device. Pretty handy. With older versions of the driver you had to setup all the maps manually in the devicetree and manually assign the correct IRQ to the driver, which was more error prone of course and a bit of a hassle.

Len Sorensen

Well irq-crossbar.c seems to be very specific to the crossbar in TI chips which handle lots of inputs and lots of outputs to multiple receivers (not just the GIC) as far as I have understood it.

Also, unless you modified the IRQCHIP_DECLARE at the end, I see nothing in your dtb that would match the driver at all.

And even if that did match, is your crossbar at all register compatible with the TI design?

irq-crossbar might be a good example for how to write such a driver, but I wouldn't have much hope of it being useful as a generic driver, never mind the unfortunate name the source file has. I suspect ti-irq-crossbar.c would have been much more appropriate.

Len Soremsem