Generic DT binding for IPIs

Hi,

This is an attempt to revive a discussion on the right list this time with all the correct people hopefully on CC.

While trying to upstream a driver, Thomas and Marc Zyngier pointed out the need for a generic IPI support in the kernel to allow driver to reserve and send ones. Hopefully my latest RFC patch will help to clarify what's being done.

    https://lkml.org/lkml/2015/10/13/227

We need a generic DT binding support to accompany that to allow a driver to reserve an IPI using this new mechanism.

MarcZ had the following suggestion:

    https://lkml.org/lkml/2015/8/24/628

Which in summary is

    mydevice@f0000000 {
        interrupt-source = <&intc INT_SPEC 2 &inttarg1 &inttarg1>;
    };

    inttarg1: mydevice@f1000000 {
        interrupt-sink = <&intc HWAFFINITY1>;
    };

    inttarg2: cpu@1 {
        interrupt-sink = <&intc HWAFFINITY2>;
    };

interrupt-sink requests to reserve an IPI that it will receive at HWAFFINITY cpumask. interrupt-source will not do any reservation. It will simply connect an IPI reserved by interrupt-sink to the device that will be responsible for generating that IPI. This description should allow connecting any 2 devices. Correct me Marc if I got it wrong please.

I suggested a simplification by assuming that IPIs will only be between host OS and a coprocessor which would gives us this form which I think is easier to deal with

    coprocessor {
             interrupt-source = <&intc INT_SPEC COP_HWAFFINITY>;
             interrupt-sink = <&intc INT_SPEC CPU_HWAFFINITY>;
    }

interrupt-source here reserves an IPI to be sent from host OS to coprocessor at COP_HWAFFINITY. interrupt-sink will reserve an IPI to be received by host OS at CPU_HWAFFINITY. Less generic but I don't know how important it is for host OS to setup IPIs between 2 external coprocessors and whether it should really be doing that.

What do the DT experts think? Any preference or a better suggestion?

I tried to keep this short and simple, please let me know if you need more info or if there's anything that needs more clarification.

Thanks,
Qais

Is there anything more I can do to get more attention about this? I think Marc's suggestion is more generic and future proof, if I send RFC patches for that would this be better?

Quoted 3 lines · Qais Yousef

Is there anything more I can do to get more attention about this? I think Marc's suggestion is more generic and future proof, if I send RFC patches for that would this be better?

Please do.

Hi,

Quoted 7 lines · Jason Cooper

Quoted 4 lines · Qais Yousef

Is there anything more I can do to get more attention about this? I think Marc's suggestion is more generic and future proof, if I send RFC patches for that would this be better?

Quoted 1 line

Please do.

Unfortunately I haven't had a chance to get around writing the patches yet. I came up with a different description though that I thought maybe worth sharing
to see if there's any opinion about it before the actual work being done.

To summarise, the problem I am trying to solve is that we have a type of coprocessors which share the interrupt controller with Linux, hence the IPI mechanism this controller uses. I've been working with Thomas on implementing
a generic API to allocate IPIs for coprocesors and a way for drivers to send these IPIs [1].

To complement this new API, we need a mechanism to describe this in device tree so a driver that wants to allocate an IPI can have this done automatically for it like we handle interrupts.

What I have in mind is:

     coproc {
             ipi-parent = <&gic>;

             ipis = <CPU_VALUE IPI_SPEC>;
             ipi-names = "in";
     };

This will allocate an IPI to go to cpu @CPU_VALUE passing @IPI_SPEC as parameters to the controller. Which means we need a new ipi-cells to define how many cells are in ipis property. Note the new ipi-parent too.

I think this is better than interrupt-sink and interrupt-source [2] as we give the driver the flexibility to give a meaning to what this IPI is. One thing I found confusing about interrupt-source and interrupt-sink is from what perspective are we viewing that, host system or firmware..

ipis property also is similar to interrupts, so using it would be easier (I think).

If we have 2 coprocessors that want to communicate using IPIs that are managed by the host we use ipi-refs property to refer to IPIs defined in another node.

     coproc1 {
             ipis = <CPU1>, <CPU2>, <CPU2>;

             ipi-names = "in", "coproc2data", "coproc2ctrl";

     };

     coproc2 {
             ipi-refs = <&coproc1 "in">, <&coproc1 "coproc2data">, <&coproc1

"corpoc2ctrl">;

Quoted 10 lines · Qais Yousef

Hi,

Quoted 7 lines · Jason Cooper

Quoted 4 lines · Qais Yousef

Is there anything more I can do to get more attention about this? I think Marc's suggestion is more generic and future proof, if I send RFC patches for that would this be better?

Quoted 7 lines

Quoted 1 line

Please do.

Unfortunately I haven't had a chance to get around writing the patches yet. I came up with a different description though that I thought maybe worth sharing
to see if there's any opinion about it before the actual work being done.

I've not given this too much thought, but here's my initial thoughts.

Quoted 21 lines

To summarise, the problem I am trying to solve is that we have a type of coprocessors which share the interrupt controller with Linux, hence the IPI mechanism this controller uses. I've been working with Thomas on implementing
a generic API to allocate IPIs for coprocesors and a way for drivers to send these IPIs [1].

To complement this new API, we need a mechanism to describe this in device tree so a driver that wants to allocate an IPI can have this done automatically for it like we handle interrupts.

What I have in mind is:

     coproc {
             ipi-parent = <&gic>;

             ipis = <CPU_VALUE IPI_SPEC>;
             ipi-names = "in";
     };

Quoted 3 lines

This will allocate an IPI to go to cpu @CPU_VALUE passing @IPI_SPEC as parameters to the controller. Which means we need a new ipi-cells to define how many cells are in ipis property. Note the new ipi-parent too.

These are still interrupts, so I'd prefer to use or extend the interrupt binding if possible.

Quoted 9 lines · Qais Yousef

What I have in mind is:

      coproc {
              ipi-parent = <&gic>;

              ipis = <CPU_VALUE IPI_SPEC>;
              ipi-names = "in";
      };

Quoted 3 lines

This will allocate an IPI to go to cpu @CPU_VALUE passing @IPI_SPEC as parameters to the controller. Which means we need a new ipi-cells to define how many cells are in ipis property. Note the new ipi-parent too.

These are still interrupts, so I'd prefer to use or extend the interrupt binding if possible.

Quoted 9 lines

ipis property also is similar to interrupts, so using it would be easier (I think).

If we have 2 coprocessors that want to communicate using IPIs that are managed by the host we use ipi-refs property to refer to IPIs defined in another node.

      coproc1 {
              ipis = <CPU1>, <CPU2>, <CPU2>;

Don't you need to specify a certain IPI number in addition to which cpu is the target?

I'm thinking the cpu target could be part of the interrupts property flags field or something.

Quoted 2 lines

              ipi-names = "in", "coproc2data", "coproc2ctrl";

-names should be optional in general. So define something that works without them.

Quoted 5 lines

      };

      coproc2 {
              ipi-refs = <&coproc1 "in">, <&coproc1 "coproc2data">, <&coproc1

"corpoc2ctrl">;

This isn't actually parseable. You need a known length of cells after a phandle.

Quoted 14 lines · Rob Herring

Quoted 12 lines · Qais Yousef

What I have in mind is:

     coproc {
             ipi-parent = <&gic>;

             ipis = <CPU_VALUE IPI_SPEC>;
             ipi-names = "in";
     };

This will allocate an IPI to go to cpu @CPU_VALUE passing @IPI_SPEC as parameters to the controller. Which means we need a new ipi-cells to define how many cells are in ipis property. Note the new ipi-parent too.

Quoted 2 lines

These are still interrupts, so I'd prefer to use or extend the interrupt binding if possible.

The IPIs have two properties that are different from a regular interrupts:

1. An IPI is not only received, it could also be sent.

2. The IPI is dynamic. There's an actual allocation from a pool of
available
IPIs happening when we ask for one to be reserved.

The difference might be borderline..

Do you have any rough idea on what a possible extension could look like? Reusing means writing less code, which is always better of course :)

By the way, on MIPS GIC, we can use interrupts property to describe an IPI the host system will receive. But to send one to the coprocessor, we need to define an outgoing IPI.

In this case, the firmware will be hardcoded to send an interrupt to a specific hwirq, so one can then describe it in DT as a regular interrupt to the host system. Hardcoding is not ideal and less portable though.

Quoted 11 lines

Quoted 9 lines

ipis property also is similar to interrupts, so using it would be easier (I think).

If we have 2 coprocessors that want to communicate using IPIs that are managed by the host we use ipi-refs property to refer to IPIs defined in another node.

     coproc1 {
             ipis = <CPU1>, <CPU2>, <CPU2>;

Don't you need to specify a certain IPI number in addition to which cpu is the target?

No. The way IPI reserving works is we just need to specify the target CPU(s).

Quoted 3 lines

I'm thinking the cpu target could be part of the interrupts property flags field or something.

I'll look more at this option.

Quoted 4 lines

Quoted 2 lines

             ipi-names = "in", "coproc2data", "coproc2ctrl";

-names should be optional in general. So define something that works without them.

If it's not specified, then the driver can get the definition by index and it would have to define the order it expects the IPIs in the binding?

Quoted 2 lines

Quoted 2 lines

     };

Quoted 5 lines

Quoted 3 lines

     coproc2 {
             ipi-refs = <&coproc1 "in">, <&coproc1 "coproc2data">, <&coproc1

"corpoc2ctrl">;

This isn't actually parseable. You need a known length of cells after a phandle.

To clarify, what you're saying we can't pass strings, right?

Quoted 4 lines · Qais Yousef

The IPIs have two properties that are different from a regular interrupts:

1. An IPI is not only received, it could also be sent.

Any interrupt is sent by the device, received by an interrupt controller, so this isn't really anything fundamentally different.

Quoted 3 lines

2. The IPI is dynamic. There's an actual allocation from a pool of
available
IPIs happening when we ask for one to be reserved.

It's not really clear to me what that means, and why it requires any particular different information in the device tree.

Quoted 8 lines

The difference might be borderline..

Do you have any rough idea on what a possible extension could look like? Reusing means writing less code, which is always better of course :)

By the way, on MIPS GIC, we can use interrupts property to describe an IPI the host system will receive. But to send one to the coprocessor, we need to define an outgoing IPI.

Ah, ok, so is what you're trying to describe here (from the host OS and CPU point of view) a purely outgoing signal to the coproc?

Quoted 3 lines

In this case, the firmware will be hardcoded to send an interrupt to a specific hwirq, so one can then describe it in DT as a regular interrupt to the host system. Hardcoding is not ideal and less portable though.

Or is the signal that goes to the coproc then somehow being translated into a host interrupt? If that's so you should be able to represent the coproc as an interrupt controller or interrupt nexus.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

      };

      coproc2 {
              ipi-refs = <&coproc1 "in">, <&coproc1 "coproc2data">, <&coproc1

"corpoc2ctrl">;

This isn't actually parseable. You need a known length of cells after a phandle.

To clarify, what you're saying we can't pass strings, right?

So, I'm not entirely sure what point Rob was making. The above certainly isn't valid dts syntax - strings can't appear within the < > construct. But if you make the obvious fix to:

     ipi-refs = <&coproc1>, "in", <&coproc1>, "coproc2data";

then it's certainly a parseable property format. It's kind of clunky mixing integers and strings that way, but it's possible and there are existing bindings using properties in a similar format.

Quoted 7 lines · David Gibson

Quoted 4 lines · Qais Yousef

The IPIs have two properties that are different from a regular interrupts:

1. An IPI is not only received, it could also be sent.

Any interrupt is sent by the device, received by an interrupt controller, so this isn't really anything fundamentally different.

No they're not fundamentally different. It's just the way they're created and used.

Quoted 3 lines

Quoted 3 lines

2. The IPI is dynamic. There's an actual allocation from a pool of
available
IPIs happening when we ask for one to be reserved.

Quoted 2 lines

It's not really clear to me what that means, and why it requires any particular different information in the device tree.

Maybe it would help to look at the new IPI reservation API?

    https://lkml.org/lkml/2015/12/8/249

Quoted 5 lines

Quoted 5 lines

The difference might be borderline..

Do you have any rough idea on what a possible extension could look like? Reusing means writing less code, which is always better of course :)

Quoted 5 lines

Quoted 3 lines

By the way, on MIPS GIC, we can use interrupts property to describe an IPI the host system will receive. But to send one to the coprocessor, we need to define an outgoing IPI.

Ah, ok, so is what you're trying to describe here (from the host OS and CPU point of view) a purely outgoing signal to the coproc?

Yes.

Quoted 7 lines

Quoted 3 lines

In this case, the firmware will be hardcoded to send an interrupt to a specific hwirq, so one can then describe it in DT as a regular interrupt to the host system. Hardcoding is not ideal and less portable though.

Or is the signal that goes to the coproc then somehow being translated into a host interrupt? If that's so you should be able to represent the coproc as an interrupt controller or interrupt nexus.

I'm not sure I understood you completely but no, there's no translation happening. When the IPI is allocated it would be routed
to the coproc. When the host wants to send a signal, it'll use the allocated hwirq value (indirectly via the virq) to write to a register, which will cause an interrupt to be generated at the coproc.

Quoted 8 lines

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

     };

     coproc2 {
             ipi-refs = <&coproc1 "in">, <&coproc1 "coproc2data">, <&coproc1

"corpoc2ctrl">;

This isn't actually parseable. You need a known length of cells after a phandle.

To clarify, what you're saying we can't pass strings, right?

Quoted 9 lines

So, I'm not entirely sure what point Rob was making. The above certainly isn't valid dts syntax - strings can't appear within the < > construct. But if you make the obvious fix to:

    ipi-refs = <&coproc1>, "in", <&coproc1>, "coproc2data";

then it's certainly a parseable property format. It's kind of clunky mixing integers and strings that way, but it's possible and there are existing bindings using properties in a similar format.

Ah OK thanks! I think this form would be handy to get the refs even if we end up reusing the interrupts property to allocate an IPI.

So if reusing the interrupts property is the right thing to do, do you (or anyone else) have a rough idea how this should look like?

Quoted 3 lines

Maybe it would help to look at the new IPI reservation API?

    https://lkml.org/lkml/2015/12/8/249

Hmm.. not as much as I might have hoped.

From the API, it looks like you're reserving IPIs to signal between different CPUs all of which are in Linux. From your description here it sounds like the coproc is a separate specialized thing not running Linux (or at least, not the same Linux instance as the host OS which this device tree is for).

Quoted 3 lines

I'm not sure I understood you completely but no, there's no translation happening. When the IPI is allocated it would be routed
to the coproc. When the host wants to send a signal, it'll use the

Quoted 2 lines

hwirq value (indirectly via the virq) to write to a register, which will cause an interrupt to be generated at the coproc.

Where is this magic register located? In the host cpu? In the coproc? In some special IPI controller?

So as noted above, I'm now sure that 'interrupts' is not the right thing. I'm trying to understand the coprocs and the ipi mechanism a bit better to work out if there is something existing which would make sense, or if we do need something entirely new.

Quoted 11 lines

Quoted 3 lines

Maybe it would help to look at the new IPI reservation API?

    https://lkml.org/lkml/2015/12/8/249

Hmm.. not as much as I might have hoped.

From the API, it looks like you're reserving IPIs to signal between different CPUs all of which are in Linux. From your description here it sounds like the coproc is a separate specialized thing not running Linux (or at least, not the same Linux instance as the host OS which this device tree is for).

No the CPUs aren't only Linux CPUs. It should be any CPU that the interrupt controller can talk to.
That CPU can be one of Linux CPUs, or a coproc.

Quoted 3 lines

Quoted 3 lines

I'm not sure I understood you completely but no, there's no translation happening. When the IPI is allocated it would be routed
to the coproc. When the host wants to send a signal, it'll use the

allocated

Quoted 3 lines

Quoted 2 lines

hwirq value (indirectly via the virq) to write to a register, which will cause an interrupt to be generated at the coproc.

Quoted 2 lines

Where is this magic register located? In the host cpu? In the coproc? In some special IPI controller?

The IPI controller. In my use case, the IPI controller is the same as the interrupt controller used by Linux. Generally they don't have to be the same though.

Quoted 4 lines

So as noted above, I'm now sure that 'interrupts' is not the right thing. I'm trying to understand the coprocs and the ipi mechanism a bit better to work out if there is something existing which would make sense, or if we do need something entirely new.

Let me know if I can help.

Hi,

This is an attempt to revive a discussion on the right list this time with all the correct people hopefully on CC.

While trying to upstream a driver, Thomas and Marc Zyngier pointed out the need for a generic IPI support in the kernel to allow driver to reserve and send ones. Hopefully my latest RFC patch will help to clarify what's being done.

    https://lkml.org/lkml/2015/10/13/227

We need a generic DT binding support to accompany that to allow a driver to reserve an IPI using this new mechanism.

MarcZ had the following suggestion:

    https://lkml.org/lkml/2015/8/24/628

Which in summary is

        mydevice@f0000000 {
                interrupt-source = <&intc INT_SPEC 2 &inttarg1 &inttarg1>;

        };

        inttarg1: mydevice@f1000000 {
                interrupt-sink = <&intc HWAFFINITY1>;

        };

        inttarg2: cpu@1 {
                interrupt-sink = <&intc HWAFFINITY2>;
        };

interrupt-sink requests to reserve an IPI that it will receive at HWAFFINITY cpumask. interrupt-source will not do any reservation. It will simply connect an IPI reserved by interrupt-sink to the device that will be responsible for generating that IPI. This description should allow connecting any 2 devices.
Correct me Marc if I got it wrong please.

I suggested a simplification by assuming that IPIs will only be between host OS and a coprocessor which would gives us this form which I think is easier to deal with

        coprocessor {
             interrupt-source = <&intc INT_SPEC COP_HWAFFINITY>;
             interrupt-sink = <&intc INT_SPEC CPU_HWAFFINITY>;
        }

interrupt-source here reserves an IPI to be sent from host OS to coprocessor at COP_HWAFFINITY. interrupt-sink will reserve an IPI to be received by host OS at CPU_HWAFFINITY. Less generic but I don't know how important it is for host OS to setup IPIs between 2 external coprocessors and whether it should really be doing that.

What do the DT experts think? Any preference or a better suggestion?

Quoted 4 lines · Qais Yousef

Hi,

This is an attempt to revive a discussion on the right list this time with all the correct people hopefully on CC.

devicetree-spec would be more appropriate list for something like this.

Quoted 13 lines

While trying to upstream a driver, Thomas and Marc Zyngier pointed out the need for a generic IPI support in the kernel to allow driver to reserve and send ones. Hopefully my latest RFC patch will help to clarify what's being done.

     https://lkml.org/lkml/2015/10/13/227

We need a generic DT binding support to accompany that to allow a driver to reserve an IPI using this new mechanism.

MarcZ had the following suggestion:

Quoted 6 lines

     https://lkml.org/lkml/2015/8/24/628

Which in summary is

         mydevice@f0000000 {
                 interrupt-source = <&intc INT_SPEC 2 &inttarg1 &inttarg1>;

What is INT_SPEC and "2"? A drawing of the h/w connections and then what the binding looks like would be helpful.

Quoted 5 lines

         };

         inttarg1: mydevice@f1000000 {
                 interrupt-sink = <&intc HWAFFINITY1>;

What is HWAFFINITY1? I want to be able to see if say this value is 1, then the affinity is for cpu0.

Quoted 25 lines

         };

         inttarg2: cpu@1 {
                 interrupt-sink = <&intc HWAFFINITY2>;
         };

interrupt-sink requests to reserve an IPI that it will receive at HWAFFINITY cpumask. interrupt-source will not do any reservation. It will simply connect an IPI reserved by interrupt-sink to the device that will be responsible for generating that IPI. This description should allow connecting any 2 devices.
Correct me Marc if I got it wrong please.

I suggested a simplification by assuming that IPIs will only be between host OS and a coprocessor which would gives us this form which I think is easier to deal with

         coprocessor {
              interrupt-source = <&intc INT_SPEC COP_HWAFFINITY>;
              interrupt-sink = <&intc INT_SPEC CPU_HWAFFINITY>;
         }

Quoted 5 lines

interrupt-source here reserves an IPI to be sent from host OS to coprocessor at COP_HWAFFINITY. interrupt-sink will reserve an IPI to be received by host OS at CPU_HWAFFINITY. Less generic but I don't know how important it is for host OS to setup IPIs between 2 external coprocessors and whether it should really be doing that.

Could we use the existing interrupts binding for interrupt-sink?

Quoted 2 lines

What do the DT experts think? Any preference or a better suggestion?

Depends how you would assign coproc to coproc IPIs in a system. It may be fixed in firmware, or more complex coprocs may read the dtb.