MPIC Bindings and Bindings for AMP Systems

Hi All,

I am currently doing some work on Linux PPC AMP systems (with Hollis, CC'd). We are using device trees to partition resources between the different OSes. To help with this effort, we would like to introduce some new bindings to the MPIC.

Currently, there are no bindings for the MPIC under '.../Documentation/powerpc/dts-bindings/'. I think most folks are following the IEEE 1275 bindings for Open PIC.

It would be nice to check-in a binding for the MPIC that is a superset of the 1275 bindings as a place to document extensions. The already in use property 'protected-sources', for example, could be placed there.

I have included a draft proposal of what this would like. It includes some properties ('no-reset') and a node (for message registers) that we are currently using for our AMP implementation and we would really like to push upstream. If the general idea seems sound, then I will clean up the proposal and submit a patch for the bindings.

Thoughts?

===============

* MPIC Binding

This binding specifies what properties and child nodes must be available on the device tree representation of the MPIC interrupt controller. This binding is a superset of the binding defined for Open PIC in [1].

** Required properties:

- All of the required properties mentioned in [1].

** Optional properties:

- no-reset : the presence of this property indicates that the interrupt controller should not be reset during OS initialization.
- protected-sources : a list of interrupt sources that are not available for use.

** Example:

	mpic: pic@40000 {
		interrupt-controller;
		#address-cells = <0>;
		#interrupt-cells = <2>;
		reg = <0x40000 0x40000>;
		compatible = "chrp,open-pic";
		device_type = "open-pic";
		protected-sources = <0xb1 0xb2>;
		no-reset;
	};

* MPIC Message Registers

This binding specifies how the MPIC message registers implemented on some Freescale platforms for interprocessor communication should be represented.

** Required parent:

- Message register nodes should always been nested under a MPIC node.

** Required properties:

- compatible : compatible list, contains 2 entries, first is "fsl,CHIP-msgr", where CHIP is the processor (p2020, mpc8572, etc ...) and the second is "fsl,mpic-msgr".

- reg : should contain the address and the length of the
message register block.
- cell-index : the index of the message register block.

- interrupts : each one of the interrupts here represents the interrupt line for one message register. These interrupts are routed internally to the MPIC.

NOTE: The 'interrupt-parent' is implicit since message register nodes
are always children of interrupt controller nodes.

** Example:

	mpic: pic@40000 {
		interrupt-controller;
		#address-cells = <0>;
		#interrupt-cells = <2>;
		reg = <0x40000 0x40000>;
		compatible = "chrp,open-pic";
		device_type = "open-pic";
		protected-sources = <0xb1>;

		msgr@1400 {
			compatible = "fsl,p2020-msgr", "fsl,mpic-msgr";
			reg = <0x1400 0x200>;

			cell-index = <0>;
			interrupts = <0xb0 0x2 0xb1 0x2
                                      0xb2 0x2 0xb3 0x2>;
		};

		msgr@2400 {
			compatible = "fsl,p2020-msgr", "fsl,mpic-msgr";
			reg = <0x2400 0x200>;
			cell-index = <1>;
			interrupts = <0xb4 0x2 0xb5 0x2
                                      0xb6 0x2 0xb7 0x2>;
	         };	
	};

* References

[1] PowerPC Microprocessor Common Hardware Reference Platform (CHRP) Binding, Version 1.8, 1998. Published by the Open Firmware Working Group. (http://playground.sun.com/1275/bindings/chrp/chrp1_8a.ps)

Signature · 6 lines

Meador Inge     | meador_inge AT mentor.com
Mentor Embedded | http://www.mentor.com/embedded-software
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev-uLR06cmDAlY/[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev

Hi Meador. Comments below.

g.

You should probably list them here anyway to aid the reader.

What is the use case for the protected-sources property? Wouldn't the irqs simply not be referenced by any device nodes? Documenting the reason for this property would be useful here.

I'd rather see the 'generic' value of mpic-msgr omitted and instead allow new parts to claim compatibility with an older chip. Generic or wildcard compatible values can be troublesome because the meaning has a tendency to change over time.

? This needs some more explanation. cell-index often gets abused as a way to enumerate devices. Typically, the address of the device itself is sufficient to identify the device.

Are these registers memory mapped? If so, then the parent node needs to have a 'ranges' property.

Quoted 3 lines · Grant Likely

Hi Meador. Comments below.

g.

Thanks a lot for the feedback Grant.

Quoted 1 line

You should probably list them here anyway to aid the reader.

Will do.

Quoted 3 lines

What is the use case for the protected-sources property? Wouldn't the irqs simply not be referenced by any device nodes? Documenting the reason for this property would be useful here.

One use case is the original [1]:

Some HW platforms, such as the new cell blades, requires some MPIC sources to be left alone by the operating system. This implements support for a "protected-sources" property in the mpic controller node containing a list of source numbers to be protected against operating system interference.

Our use case is to reserve certain IRQs for certain OSes in an AMP system. As an example, consider that OS 1 has (simplified for discussion):

    mpic: pic@40000 {
        ...
        protected-sources = <0xb1>;

        msgr@41400 {
            interrupts = <0xb0 2 0xb1 2 0xb2 2 0xb3 2>;
        };
    };

and OS 2 has:

    mpic: pic@40000 {
        ...
        protected-sources = <0xb0>;

        msgr@41400 {
            interrupts = <0xb0 2 0xb1 2 0xb2 2 0xb3 2>;
        };
    };

where OS 1 is sent messages through the message register tied to 0xb0 and OS 2 is sent messages through the message register tied to 0xb1.

We can't just remove the IRQ of the _other_ OS from the 'interrupts' property in the message node because we need to know the IRQ in order to talk to the other OS. So, we use protected sources to tell the OS that an IRQ is not available for its own use, while at the same time keeping complete information on all the IRQ mappings for the message registers.

I will update the documentation.

Quoted 4 lines

I'd rather see the 'generic' value of mpic-msgr omitted and instead allow new parts to claim compatibility with an older chip. Generic or wildcard compatible values can be troublesome because the meaning has a tendency to change over time.

I don't completely see the issue here. Do you have a specific example of how generic values like this cause problems?

How do you
see this working in terms of processing the data? It seems like we are going to have to be aware of N values instead of 1, which seems worse.

Quoted 4 lines

? This needs some more explanation. cell-index often gets abused as a way to enumerate devices. Typically, the address of the device itself is sufficient to identify the device.

The message registers typically come in blocks of four memory mapped registers and may not be in contiguous memory (example [3]). The intent of 'cell-index' is to put an ordering on the blocks (so, yes, enumeration). We could order them by address as well I suppose. One less property to worry about :)

Quoted 3 lines

Are these registers memory mapped? If so, then the parent node needs to have a 'ranges' property.

Yes, they are. I will look into adding that.

[1] commit 7fd7218610600b16f6f0af3f9d9353ba0265c09f, http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.36.y.git;a=commit;h=7fd7218610600b16f6f0af3f9d9353ba0265c09f

[2] Documentation/powerpc/dts-bindings/fsl/msi-pic.txt

[3] http://cache.freescale.com/files/32bit/doc/ref_manual/MPC8572ERM.pdf?fr=g

--
Meador Inge | meador_inge AT mentor.com
Mentor Embedded | http://www.mentor.com/embedded-software

Quoted 4 lines · Meador Inge

How do you
see this working in terms of processing the data? It seems like we are going to have to be aware of N values instead of 1, which seems worse.

This argument has been rehashed many times, but it basically comes down to compatible values should ideally be anchored to a real implemented device, not to a family of devices, or to an unversioned specification.

In practise, the implementation doesn't actually look any different except that the 'reference' version specifies a specific implementation instead of a generic name. To use a concrete example, if there are two parts using this MPIC, like the freescale p2040 and p4080, and say for argument that the p2040 was implemented first, then the compatible values would look like:

for the p2040:   compatible = "fsl,p2040-msgr";
for the p4080:   compatible = "fsl,p4080-msgr", "fsl,p2040-msgr";

Quoted 9 lines

Quoted 3 lines

? This needs some more explanation. cell-index often gets abused as a way to enumerate devices. Typically, the address of the device itself is sufficient to identify the device.

The message registers typically come in blocks of four memory mapped registers and may not be in contiguous memory (example [3]). The intent of 'cell-index' is to put an ordering on the blocks (so, yes, enumeration). We could order them by address as well I suppose. One less property to worry about :)

We can't just remove the IRQ of the _other_ OS from the 'interrupts' property in the message node because we need to know the IRQ in order to talk to the other OS. So, we use protected sources to tell the OS that an IRQ is not available for its own use, while at the same time keeping
complete information on all the IRQ mappings for the message registers.

Quoted 10 lines · On Thu, 23 Dec 2010 15:33:25 -0700 Grant Likely<[email protected]>…

Quoted 4 lines · Meador Inge

How do you
see this working in terms of processing the data? It seems like we are going to have to be aware of N values instead of 1, which seems worse.

This argument has been rehashed many times, but it basically comes down to compatible values should ideally be anchored to a real implemented device, not to a family of devices, or to an unversioned specification.

Freescale MPICs do have version numbers (version registers, even). We should put that version (possibly along with a compatible version) in the compatible, though, for blocks such as this which don't include the main MPIC registers and thus the version registers.

Quoted 9 lines

In practise, the implementation doesn't actually look any different except that the 'reference' version specifies a specific implementation instead of a generic name. To use a concrete example, if there are two parts using this MPIC, like the freescale p2040 and p4080, and say for argument that the p2040 was implemented first, then the compatible values would look like:

for the p2040:   compatible = "fsl,p2040-msgr";
for the p4080:   compatible = "fsl,p4080-msgr", "fsl,p2040-msgr";

While I don't think it affected the message unit, p4080 rev 1 has a different version of the MPIC from p4080 rev 2 (4.0 versus 4.1, IIRC).

I don't think "mpic-" should be dropped, whether a specific chip is added or not. "msgr" just seems too generic, and "mpic-" tells the reader where in the chip manual they can find information about it.

Quoted 9 lines

Quoted 9 lines

Quoted 3 lines

? This needs some more explanation. cell-index often gets abused as a way to enumerate devices. Typically, the address of the device itself is sufficient to identify the device.

The message registers typically come in blocks of four memory mapped registers and may not be in contiguous memory (example [3]). The intent of 'cell-index' is to put an ordering on the blocks (so, yes, enumeration). We could order them by address as well I suppose. One less property to worry about :)

But why do we care about ordering them? What's important is just that you use the same one on both the sending partition and the receiving partition.

We need some sort of mapping between a message register and a message register number so that the message registers can be referenced through some sort of API (e.g. 'mpic_msgr_read(0)'). One way to do that would be by putting an order on the registers. Maybe there is a better way, though ...

This argument has been rehashed many times, but it basically comes down to compatible values should ideally be anchored to a real implemented device, not to a family of devices, or to an unversioned specification.

In practise, the implementation doesn't actually look any different except that the 'reference' version specifies a specific implementation instead of a generic name. To use a concrete example, if there are two parts using this MPIC, like the freescale p2040 and p4080, and say for argument that the p2040 was implemented first, then the compatible values would look like:

for the p2040:   compatible = "fsl,p2040-msgr";
for the p4080:   compatible = "fsl,p4080-msgr", "fsl,p2040-msgr";

and the driver could simply bind on "fsl,p2040-msgr" to work with both chips. So, instead of an arbitrary "fsl,mpic-msgr" string, "fsl,p2040-msgr" is used as the baseline value and there is no ambiguity about which part it describes.

If the p4080 is actually subtly different from the p2040, then it would not claim compatibility with the former and the driver can match against either string; adapting its behaviour as necessary.

Quoted 5 lines · On Wed, 5 Jan 2011 15:58:55 -0600 Meador Inge<[email protected]> wrote:

We need some sort of mapping between a message register and a message register number so that the message registers can be referenced through some sort of API (e.g. 'mpic_msgr_read(0)'). One way to do that would be by putting an order on the registers. Maybe there is a better way, though ...

A message register is uniquely identified by a reference to the device tree node, plus a 0-3 index into that node's message registers.

Quoted 8 lines · Scott Wood

Quoted 5 lines · On Wed, 5 Jan 2011 15:58:55 -0600 Meador Inge<[email protected]> wrote:

We need some sort of mapping between a message register and a message register number so that the message registers can be referenced through some sort of API (e.g. 'mpic_msgr_read(0)'). One way to do that would be by putting an order on the registers. Maybe there is a better way, though ...

Quoted 2 lines

A message register is uniquely identified by a reference to the device tree node, plus a 0-3 index into that node's message registers.

Really what we're talking about is software configuration, not hardware description.

We've gone back and forth on representing this information in the device tree, and most recently decided against it. Outside the kernel, a device node reference isn't really practical.

Quoted 9 lines · Hollis

Quoted 8 lines · Scott Wood

Quoted 5 lines · On Wed, 5 Jan 2011 15:58:55 -0600 Meador Inge<[email protected]> wrote:

We need some sort of mapping between a message register and a message register number so that the message registers can be referenced through some sort of API (e.g. 'mpic_msgr_read(0)'). One way to do that would be by putting an order on the registers. Maybe there is a better way, though ...

Quoted 4 lines

Quoted 2 lines

A message register is uniquely identified by a reference to the device tree node, plus a 0-3 index into that node's message registers.

Really what we're talking about is software configuration, not hardware description.

Part of that software configuration involves identifying the hardware being referenced.

Quoted 3 lines

We've gone back and forth on representing this information in the device tree, and most recently decided against it. Outside the kernel, a device node reference isn't really practical.

Global enumeration isn't much fun either. For something like this where it's very unlikely that additional MPIC message units will be added to the system dynamically, it's managable, but it's not a good habit to get into. Look at the pain that's been caused by such assumptions in the i2c subsystem, in kernel interrupt management, etc.

A reference to a node is just a pointer to a software message driver object, which can be obtained from looking up an alias. It's a little less simple than just using a number, but it's not impractical. It also provides a natural place to put a layer of indirection in the code that isolates the upper-layer protocol from the details of what sort of message transport it is using.

Now, if you don't care about this, and want to just use numbers in your application, go ahead. But I don't think that such an assumption should go into the device tree binding. Have the software number the message register banks in increasing physical address order, or based on numbered aliases similar to how U-Boot enumerates ethernet nodes, or something similar.

Quoted 35 lines · Scott Wood

Quoted 9 lines · Hollis

Quoted 8 lines · Scott Wood

Quoted 5 lines · On Wed, 5 Jan 2011 15:58:55 -0600 Meador…

We need some sort of mapping between a message register and a message register number so that the message registers can be referenced through some sort of API (e.g. 'mpic_msgr_read(0)').  One way to do that would be by putting an order on the registers.  Maybe there is a better way, though ...

Quoted 4 lines

Quoted 2 lines

A message register is uniquely identified by a reference to the device tree node, plus a 0-3 index into that node's message registers.

Really what we're talking about is software configuration, not hardware description.

Part of that software configuration involves identifying the hardware being referenced.

Quoted 3 lines

We've gone back and forth on representing this information in the device tree, and most recently decided against it. Outside the kernel, a device node reference isn't really practical.

Global enumeration isn't much fun either.  For something like this where it's very unlikely that additional MPIC message units will be added to the system dynamically, it's managable, but it's not a good habit to get into.  Look at the pain that's been caused by such assumptions in the i2c subsystem, in kernel interrupt management, etc.

A reference to a node is just a pointer to a software message driver object, which can be obtained from looking up an alias.  It's a little less simple than just using a number, but it's not impractical. It also provides a natural place to put a layer of indirection in the code that isolates the upper-layer protocol from the details of what sort of message transport it is using.

Quoted 6 lines

Now, if you don't care about this, and want to just use numbers in your application, go ahead.  But I don't think that such an assumption should go into the device tree binding.  Have the software number the message register banks in increasing physical address order, or based on numbered aliases similar to how U-Boot enumerates ethernet nodes, or something similar.

Using physical addresses doesn't solve the enumeration problem either, but I think it's beside the point: userspace must refer to the device. There is a rich history of userspace *not* walking /proc/device-tree in order to refer to a physical device. Are you suggesting this case is special?

Actually, for a while now the kernel has been moving towards userspace being responsible for device identification. That's what udev is for.
The kernel udev looks at the available information when a device is
registered/bound, and it creates useful symlinks to the dynamically assigned major/minor devices. The rest of userspace doesn't need to know about it; it can simply use the symlinks in /dev, but it is appropriate to let udev figure out the correct naming.

Quoted 7 lines · Grant Likely

Actually, for a while now the kernel has been moving towards userspace being responsible for device identification.  That's what udev is for. The kernel udev looks at the available information when a device is registered/bound, and it creates useful symlinks to the dynamically assigned major/minor devices.  The rest of userspace doesn't need to know about it; it can simply use the symlinks in /dev, but it is appropriate to let udev figure out the correct naming.

Can you point out an example of how this is done for Open Firmware devices currently?

In particular, how are the udev rules supposed to
operate? They get a device path, use that to find additional information in /proc/device-tree (hope that's mounted), and then what? Do we hardcode the block addresses in the rules, for example @1400 and @2400?

Quoted 8 lines · Hollis

Quoted 7 lines · Grant Likely

Actually, for a while now the kernel has been moving towards userspace being responsible for device identification. That's what udev is for.
The kernel udev looks at the available information when a device is
registered/bound, and it creates useful symlinks to the dynamically assigned major/minor devices. The rest of userspace doesn't need to know about it; it can simply use the symlinks in /dev, but it is appropriate to let udev figure out the correct naming.

Quoted 2 lines

Can you point out an example of how this is done for Open Firmware devices currently?

Nope, because while it has been a theoretical concern, in practice the issue hasn't come up on any of the hardware I've worked on or I've seen patches for.

Mostly this is because the device tree
representations are internally self consistent; dependencies/connections are explicitly expressed with phandles or full paths which eliminates any need for enumeration in all the cases I've had to deal with.

Quoted 12 lines · Grant Likely

Quoted 10 lines · Hollis

Quoted 7 lines · Grant Likely

Actually, for a while now the kernel has been moving towards userspace being responsible for device identification. That's what udev is for.
The kernel udev looks at the available information when a device is
registered/bound, and it creates useful symlinks to the dynamically assigned major/minor devices. The rest of userspace doesn't need to know about it; it can simply use the symlinks in /dev, but it is appropriate to let udev figure out the correct naming.

Can you point out an example of how this is done for Open Firmware devices currently?

Quoted 3 lines

Nope, because while it has been a theoretical concern, in practice the issue hasn't come up on any of the hardware I've worked on or I've seen patches for.

That is still this case here: it's a theoretical concern. Don't forget:
we're talking about eight registers in the MPIC. There is no bus, no hotplug, nothing. Really, the only requirement is that somehow we can match the names used in the hardware documentation.

Quoted 5 lines

Mostly this is because the device tree
representations are internally self consistent; dependencies/connections are explicitly expressed with phandles or full paths which eliminates any need for enumeration in all the cases I've had to deal with.

The device tree on every platform contains devices which must be referenced by userspace. That is the problem we're trying to solve -- let's not get distracted by theoretical principles of enumeration.

The closest analogy might be serial devices, which unlike ethernet devices don't have other distinguishing information for udev to interpret. To my surprise, when I reverse the order of the serial nodes in the device tree, the ttyS0/S1 ordering reverses too. This is exactly the problem we were trying to avoid... but it seems nobody has solved it anywhere. :(