iommu/exynos: Fixes and Enhancements of System MMU driver with DT

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt line 58

Quoted 7 lines · On Sunday 27 April 2014 13:07:43 Shaik Ameer Basha wrote:

+- mmu-masters: A phandle to device nodes representing the master for which

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt new file +6 -0

+               the System MMU can provide a translation. Any additional values
+              after the phandle will be ignored because a System MMU never
+              have two or more masters. "#stream-id-cells" specified in the
+              master's node will be also ignored.
+              If more than one phandle is specified, only the first phandle
+              will be treated.

This seems completely backwards: Why would you list the masters for an IOMMU in the IOMMU node?

The master should have a standard property pointing to the IOMMU instead.

We don't have a generic binding for IOMMUs yet it seems, but the time is overdue to make one.

Consider this NAKed until there is a generic binding for IOMMUs that all relevant developers have agreed to.

Quoted 12 lines · Arnd Bergmann

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt line 58

Quoted 7 lines · On Sunday 27 April 2014 13:07:43 Shaik Ameer Basha wrote:

+- mmu-masters: A phandle to device nodes representing the master for which

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt new file +6 -0

+               the System MMU can provide a translation. Any additional values
+              after the phandle will be ignored because a System MMU never
+              have two or more masters. "#stream-id-cells" specified in the
+              master's node will be also ignored.
+              If more than one phandle is specified, only the first phandle
+              will be treated.

This seems completely backwards: Why would you list the masters for an IOMMU in the IOMMU node?

Quoted 7 lines

The master should have a standard property pointing to the IOMMU instead.

We don't have a generic binding for IOMMUs yet it seems, but the time is overdue to make one.

Consider this NAKed until there is a generic binding for IOMMUs that all relevant developers have agreed to.

I'd like to take this opportunity and revive one of the hibernating patch sets that we have for Tegra. The last effort to get things merged was back in January I think. I haven't bothered to look up the reference since it's probably good to start from scratch anyway.

The latest version of the binding that was under discussion back then I think looked something like this:

	device at ... {
		iommus = <&iommu [spec]>[, <&other_iommu [other_spec]>...];
	};

And possibly with a iommu-names property to go along with that. The idea being that a device can be a master on possibly multiple IOMMUs. Using the above it would also be possible to have one device be multiple masters on the same IOMMU.

On Tegra the specifier would be used to encode a memory controller's client ID. One discussion point back at the time was to encode the ID as a bitmask to allow more than a single master per entry. Another solution which I think is a little cleaner and more generic, would be to use one entry per master and use a single cell to encode the client ID. Devices with multiple clients to the same IOMMU could then use multiple entries referencing the same IOMMU.

Quoted 33 lines · On Monday 28 April 2014 12:39:20 Thierry Reding wrote:

Quoted 12 lines · Arnd Bergmann

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt line 58

Quoted 7 lines · On Sunday 27 April 2014 13:07:43 Shaik Ameer Basha wrote:

+- mmu-masters: A phandle to device nodes representing the master for which

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt new file +6 -0

+               the System MMU can provide a translation. Any additional values
+              after the phandle will be ignored because a System MMU never
+              have two or more masters. "#stream-id-cells" specified in the
+              master's node will be also ignored.
+              If more than one phandle is specified, only the first phandle
+              will be treated.

This seems completely backwards: Why would you list the masters for an IOMMU in the IOMMU node?

Quoted 7 lines

The master should have a standard property pointing to the IOMMU instead.

We don't have a generic binding for IOMMUs yet it seems, but the time is overdue to make one.

Consider this NAKed until there is a generic binding for IOMMUs that all relevant developers have agreed to.

I'd like to take this opportunity and revive one of the hibernating patch sets that we have for Tegra. The last effort to get things merged was back in January I think. I haven't bothered to look up the reference since it's probably good to start from scratch anyway.

The latest version of the binding that was under discussion back then I think looked something like this:

	device at ... {
		iommus = <&iommu [spec]>[, <&other_iommu [other_spec]>...];
	};

Quoted 4 lines

And possibly with a iommu-names property to go along with that. The idea being that a device can be a master on possibly multiple IOMMUs. Using the above it would also be possible to have one device be multiple masters on the same IOMMU.

Yes, that seems reasonable. Just one question: How would you represent a device that has multiple masters, with at least one connected to an IOMMU and another one connected to memory directly, without going to the IOMMU?

Quoted 7 lines

On Tegra the specifier would be used to encode a memory controller's client ID. One discussion point back at the time was to encode the ID as a bitmask to allow more than a single master per entry. Another solution which I think is a little cleaner and more generic, would be to use one entry per master and use a single cell to encode the client ID. Devices with multiple clients to the same IOMMU could then use multiple entries referencing the same IOMMU.

I'm not completely following here. Are you talking about the generic binding, or the part that is tegra specific for the specifier?

My first impression is that the generic binding should just allow an arbitrary specifier with a variable #iommu-cells, and leave the format up to the IOMMU driver.

A lot of drivers probably only support one
master, so they can just set #iommu-cells=<0>, others might require IDs that do not fit into one cell.

Quoted 9 lines · Arnd Bergmann

Quoted 4 lines · On Monday 28 April 2014 12:39:20 Thierry Reding wrote:

And possibly with a iommu-names property to go along with that. The idea being that a device can be a master on possibly multiple IOMMUs. Using the above it would also be possible to have one device be multiple masters on the same IOMMU.

Yes, that seems reasonable. Just one question: How would you represent a device that has multiple masters, with at least one connected to an IOMMU and another one connected to memory directly, without going to the IOMMU?

Heh, I don't think I've ever thought about that use-case. I guess I was always assuming that in the absence of an IOMMU the device would simply access memory directly. From what I can tell that's how Tegra works at least. If the IOMMU is not enabled for a given client, that client will access physical memory untranslated.

I suppose if that really must be represented then a global dummy IOMMU could be introduced to help with these cases.

Quoted 8 lines

Quoted 7 lines

On Tegra the specifier would be used to encode a memory controller's client ID. One discussion point back at the time was to encode the ID as a bitmask to allow more than a single master per entry. Another solution which I think is a little cleaner and more generic, would be to use one entry per master and use a single cell to encode the client ID. Devices with multiple clients to the same IOMMU could then use multiple entries referencing the same IOMMU.

Quoted 6 lines

I'm not completely following here. Are you talking about the generic binding, or the part that is tegra specific for the specifier?

My first impression is that the generic binding should just allow an arbitrary specifier with a variable #iommu-cells, and leave the format up to the IOMMU driver.

Yes, I was getting ahead of myself. The idea was to have #iommu-cells and allow the specifier to be IOMMU-specific. On Tegra that would translate to the memory controller client ID, on other devices I suspect something similar might exist, but for the generic binding it should be completely opaque and hence irrelevant.

Really just like any of the other bindings that have foos and #foo-cells properties.

Quoted 3 lines

A lot of drivers probably only support one
master, so they can just set #iommu-cells=<0>, others might require IDs that do not fit into one cell.

You mean "#iommu-cells = <1>" for devices that only require one master?

There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

Quoted 11 lines · On Monday 28 April 2014 13:18:03 Thierry Reding wrote:

Quoted 9 lines · Arnd Bergmann

Quoted 4 lines · On Monday 28 April 2014 12:39:20 Thierry Reding wrote:

And possibly with a iommu-names property to go along with that. The idea being that a device can be a master on possibly multiple IOMMUs. Using the above it would also be possible to have one device be multiple masters on the same IOMMU.

Yes, that seems reasonable. Just one question: How would you represent a device that has multiple masters, with at least one connected to an IOMMU and another one connected to memory directly, without going to the IOMMU?

Quoted 8 lines

Heh, I don't think I've ever thought about that use-case. I guess I was always assuming that in the absence of an IOMMU the device would simply access memory directly. From what I can tell that's how Tegra works at least. If the IOMMU is not enabled for a given client, that client will access physical memory untranslated.

I suppose if that really must be represented then a global dummy IOMMU could be introduced to help with these cases.

It's actually not too uncommon: you can have e.g. the lower 2GB mapped directly from the device address space into the host memory, but have an iommu that translates accesses from some range in the upper 2GB of the 32-bit address space into full 64-bit addresses.

This use case makes no sense if you use the IOMMU for isolation or virtualization, but it gives better performance for lowmem access when the only reason to have the IOMMU is to map highmem addresses.

Quoted 5 lines

Quoted 3 lines

A lot of drivers probably only support one
master, so they can just set #iommu-cells=<0>, others might require IDs that do not fit into one cell.

You mean "#iommu-cells = <1>" for devices that only require one master?

I meant an IOMMU device that acts as the slave for exactly one device, even if that device has multiple master ports.

Quoted 4 lines

There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

One completely open question that I just noticed is how the kernel should deal with the case of multiple IOMMUs attached to one master: the data structures we have assume that we know exactly how to do DMA by setting the per-device dma_map_ops (iommu or not, coherent or not), and by setting a pointer to at most one IOMMU.

Quoted 4 lines · On Monday 28 April 2014 13:18:03 Thierry Reding wrote:

There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

One completely open question that I just noticed is how the kernel should deal with the case of multiple IOMMUs attached to one master: the data structures we have assume that we know exactly how to do DMA by setting the per-device dma_map_ops (iommu or not, coherent or not), and by setting a pointer to at most one IOMMU.

Hi Arnd,

[and thanks Thierry for CCing me -- I have been tangled up with this before :)]

Quoted 37 lines · Arnd Bergmann

Quoted 4 lines · On Monday 28 April 2014 13:18:03 Thierry Reding wrote:

There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

Quoted 7 lines

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

I think this is definitely going in the right direction, but it's not clear to me how the driver for device at 5 knows how to configure the two ports. We're still lacking topology information (unless that's implicit in the ordering of the properties) to say how the mastering capabilities of the device are actually routed and configured.

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

Quoted 26 lines · On Monday 28 April 2014 20:30:56 Will Deacon wrote:

Hi Arnd,

[and thanks Thierry for CCing me -- I have been tangled up with this before :)]

Quoted 20 lines · Arnd Bergmann

Quoted 4 lines · On Monday 28 April 2014 13:18:03 Thierry Reding wrote:

There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

Quoted 5 lines

Quoted 5 lines

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

Quoted 20 lines

Quoted 19 lines

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

Quoted 5 lines

I think this is definitely going in the right direction, but it's not clear to me how the driver for device at 5 knows how to configure the two ports. We're still lacking topology information (unless that's implicit in the ordering of the properties) to say how the mastering capabilities of the device are actually routed and configured.

It would be helpful to have a concrete example of a device that has multiple masters. I have heard people mention this multiple times, and I can understand

how it might be wired up in hardware, but I don't know what it's good for, or who would actually do it.

Quoted 8 lines

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

An IOMMU is really a specialised bridge

, so it may be cleaner to describe
an IOMMU using a real bus node in the DT, if we also define a way to make master/slave linkages explicit where it matters.

The problems of how to describe master/slave linkage, coherency between masters, and how to describe sideband ID information present on the bus are really interrelated.

If we can come up with a consistent description for these things, it should help us to describe IOMMUs, bus mastering peripherals, MSI controllers and complex bridges in a more uniform way, without having to reinvent so much for each binding. That's my hope anyway.

I've been hacking around some proposals on these areas which are a bit different from the approach suggested here -- I'll try to summarise some of it intelligibly and post something tomorrow so that we can discuss.

On Tue, Apr 29, 2014 at 11:16 AM, Dave Martin <[email protected]> wrote:
...

Quoted 1 line

An IOMMU is really a specialised bridge

Is a GART a bridge?

IOMMUs can provide three basic functions:
1) remap address space to reach phys mem ranges that the device is otherwise not capable of accessing (classic 32-bit DMA to reach 64-bit Phys address)

2) implement scatter-gather (page level granularity) so the device doesn't have to

3) provide some level of system protection against "rogue" DMA by forcing everything through a DMA mapping interface and faulting when encountering unmapped DMA transactions.

I summarize IOMMUs as: "participate in the routing of MMIO transactions in the system fabric."
In this sense, IOMMUs are sort of a bridge. Defining what kind of routing they can do (coalesce transactions? remapping MMIO domains?) and which address ranges they route would describe most of that functionality.

This "remapping" of MMIO transaction is also usually asymmetric. Meaning routing of "downstream" transactions *might* be completely different than the routing + remapping of transactions heading upstream. DMA mappings services are designed to handle only the transactions generated (aka "mastered") by a downstream device.

Quoted 3 lines

, so it may be cleaner to describe
an IOMMU using a real bus node in the DT, if we also define a way to make master/slave linkages explicit where it matters.

"where it matters" is a bit vague. Is the goal to just enable DMA mapping services to "do the right thing" for a device that can generate DMA?

Quoted 3 lines

I've been hacking around some proposals on these areas which are a bit different from the approach suggested here -- I'll try to summarise some of it intelligibly and post something tomorrow so that we can discuss.

Are you planning on consolidating Documentation/devicetree/bindings/iommu/ ? Do you care about Documentation/Intel-IOMMU.txt?

Quoted 4 lines

...

Quoted 1 line

An IOMMU is really a specialised bridge

Is a GART a bridge?

Quoted 8 lines

IOMMUs can provide three basic functions:
1) remap address space to reach phys mem ranges that the device is otherwise not capable of accessing (classic 32-bit DMA to reach 64-bit Phys address)

2) implement scatter-gather (page level granularity) so the device doesn't have to

Quoted 3 lines

3) provide some level of system protection against "rogue" DMA by forcing everything through a DMA mapping interface and faulting when encountering unmapped DMA transactions.

[ 4) provide isolation between multiple contexts, typically for purposes
of virtualization]

Quoted 6 lines

I summarize IOMMUs as: "participate in the routing of MMIO transactions in the system fabric."
In this sense, IOMMUs are sort of a bridge. Defining what kind of routing they can do (coalesce transactions? remapping MMIO domains?) and which address ranges they route would describe most of that functionality.

Quoted 5 lines

This "remapping" of MMIO transaction is also usually asymmetric. Meaning routing of "downstream" transactions *might* be completely different than the routing + remapping of transactions heading upstream. DMA mappings services are designed to handle only the transactions generated (aka "mastered") by a downstream device.

For the purposes of the DT binding, we have a 'ranges' property that defines the downstream translation (CPU-to-MMIO) and a 'dma-ranges' property for the opposite address translation (device-to-memory).

Quoted 7 lines

Quoted 3 lines

, so it may be cleaner to describe
an IOMMU using a real bus node in the DT, if we also define a way to make master/slave linkages explicit where it matters.

"where it matters" is a bit vague. Is the goal to just enable DMA mapping services to "do the right thing" for a device that can generate DMA?

Yes. It's very complicated unfortunately, because we have to be able to deal with arbitrary combinations of a lot of oddball cases that can show up in random SoCs:

- device can only do DMA to a limited address range
- DMA is noncoherent and needs manual cache management
- DMA address is at an offset from physical address
- some devices have an IOMMU
- some IOMMUs are shared between devices
- some devices with IOMMU can have multiple simultaneous contexts
- a device may access some memory directly and some other memory through IOMMU
- a device may have DMA access to a bus that is invisible to the CPU
- DMA on some device is only coherent if the IOMMU is enabled
- DMA on some device is only coherent if the IOMMU is disabled
- the IOVA range to an IOMMU is device dependent

Quoted 5 lines

Quoted 3 lines

I've been hacking around some proposals on these areas which are a bit different from the approach suggested here -- I'll try to summarise some of it intelligibly and post something tomorrow so that we can discuss.

Are you planning on consolidating Documentation/devicetree/bindings/iommu/ ?

Quoted 1 line

Do you care about Documentation/Intel-IOMMU.txt?

I think we can ignore the Intel-IOMMU because that is specialized on PCI devices, which we don't normally represent in DT. It is also a special case because the Intel IOMMU is a single-instance device. If it's present and enabled, it will be used by every device. The case we do need to describe is when we don't know which IOMMU is used for which master, or how to configure it.

Quoted 11 lines · On Tuesday 29 April 2014 13:07:54 Grant Grundler wrote:

On Tue, Apr 29, 2014 at 11:16 AM, Dave Martin <[email protected]> wrote:
...

Quoted 1 line

An IOMMU is really a specialised bridge

Is a GART a bridge?

IOMMUs can provide three basic functions:
1) remap address space to reach phys mem ranges that the device is otherwise not capable of accessing (classic 32-bit DMA to reach 64-bit Phys address)

Quoted 6 lines

2) implement scatter-gather (page level granularity) so the device doesn't have to

3) provide some level of system protection against "rogue" DMA by forcing everything through a DMA mapping interface and faulting when encountering unmapped DMA transactions.

[ 4) provide isolation between multiple contexts, typically for purposes
of virtualization]

Quoted 7 lines

I summarize IOMMUs as: "participate in the routing of MMIO transactions in the system fabric."
In this sense, IOMMUs are sort of a bridge. Defining what kind of routing they can do (coalesce transactions? remapping MMIO domains?) and which address ranges they route would describe most of that functionality.

Quoted 5 lines

This "remapping" of MMIO transaction is also usually asymmetric. Meaning routing of "downstream" transactions *might* be completely different than the routing + remapping of transactions heading upstream. DMA mappings services are designed to handle only the transactions generated (aka "mastered") by a downstream device.

For the purposes of the DT binding, we have a 'ranges' property that defines the downstream translation (CPU-to-MMIO) and a 'dma-ranges' property for the opposite address translation (device-to-memory).

Quoted 7 lines

Quoted 3 lines

, so it may be cleaner to describe
an IOMMU using a real bus node in the DT, if we also define a way to make master/slave linkages explicit where it matters.

"where it matters" is a bit vague. Is the goal to just enable DMA mapping services to "do the right thing" for a device that can generate DMA?

Yes. It's very complicated unfortunately, because we have to be able to deal with arbitrary combinations of a lot of oddball cases that can show up in random SoCs:

- device can only do DMA to a limited address range
- DMA is noncoherent and needs manual cache management
- DMA address is at an offset from physical address
- some devices have an IOMMU
- some IOMMUs are shared between devices
- some devices with IOMMU can have multiple simultaneous contexts
- a device may access some memory directly and some other memory through IOMMU

- a device may have DMA access to a bus that is invisible to the CPU
- DMA on some device is only coherent if the IOMMU is enabled
- DMA on some device is only coherent if the IOMMU is disabled
- the IOVA range to an IOMMU is device dependent

Quoted 6 lines

Quoted 3 lines

I've been hacking around some proposals on these areas which are a bit different from the approach suggested here -- I'll try to summarise some of it intelligibly and post something tomorrow so that we can discuss.

Are you planning on consolidating Documentation/devicetree/bindings/iommu/ ? Do you care about Documentation/Intel-IOMMU.txt?

I think we can ignore the Intel-IOMMU because that is specialized on PCI devices, which we don't normally represent in DT. It is also a special case because the Intel IOMMU is a single-instance device. If it's present and enabled, it will be used by every device. The case we do need to describe is when we don't know which IOMMU is used for which master, or how to configure it.

Arnd

Quoted 8 lines

- device can only do DMA to a limited address range
- DMA is noncoherent and needs manual cache management
- DMA address is at an offset from physical address
- some devices have an IOMMU
- some IOMMUs are shared between devices
- some devices with IOMMU can have multiple simultaneous contexts
- a device may access some memory directly and some other memory through IOMMU

Do we need to consider this case?
I don't think a device can have different contexts at the same time. If there such device is in a system, its driver must handle it correctly with different devices descriptors for the different contexts, for example. I mean, if a device has two DMA ports that are in different contexts, they can be treated as different devices which are handed by a driver.

I worry that abstracting everything we can think may make the problem harder.

Quoted 52 lines · Arnd Bergmann

Quoted 11 lines · On Tuesday 29 April 2014 13:07:54 Grant Grundler wrote:

On Tue, Apr 29, 2014 at 11:16 AM, Dave Martin <[email protected]> wrote:
...

Quoted 1 line

An IOMMU is really a specialised bridge

Is a GART a bridge?

IOMMUs can provide three basic functions:
1) remap address space to reach phys mem ranges that the device is otherwise not capable of accessing (classic 32-bit DMA to reach 64-bit Phys address)

Quoted 6 lines

2) implement scatter-gather (page level granularity) so the device doesn't have to

3) provide some level of system protection against "rogue" DMA by forcing everything through a DMA mapping interface and faulting when encountering unmapped DMA transactions.

[ 4) provide isolation between multiple contexts, typically for purposes
of virtualization]

Quoted 7 lines

I summarize IOMMUs as: "participate in the routing of MMIO transactions in the system fabric."
In this sense, IOMMUs are sort of a bridge. Defining what kind of routing they can do (coalesce transactions? remapping MMIO domains?) and which address ranges they route would describe most of that functionality.

Quoted 5 lines

This "remapping" of MMIO transaction is also usually asymmetric. Meaning routing of "downstream" transactions *might* be completely different than the routing + remapping of transactions heading upstream. DMA mappings services are designed to handle only the transactions generated (aka "mastered") by a downstream device.

For the purposes of the DT binding, we have a 'ranges' property that defines the downstream translation (CPU-to-MMIO) and a 'dma-ranges' property for the opposite address translation (device-to-memory).

Quoted 7 lines

Quoted 3 lines

, so it may be cleaner to describe
an IOMMU using a real bus node in the DT, if we also define a way to make master/slave linkages explicit where it matters.

"where it matters" is a bit vague. Is the goal to just enable DMA mapping services to "do the right thing" for a device that can generate DMA?

Yes. It's very complicated unfortunately, because we have to be able to deal with arbitrary combinations of a lot of oddball cases that can show up in random SoCs:

Quoted 7 lines

- device can only do DMA to a limited address range
- DMA is noncoherent and needs manual cache management
- DMA address is at an offset from physical address
- some devices have an IOMMU
- some IOMMUs are shared between devices
- some devices with IOMMU can have multiple simultaneous contexts
- a device may access some memory directly and some other memory through IOMMU

Do we need to consider this case?
I don't think a device can have different contexts at the same time. If there such device is in a system, its driver must handle it correctly with different devices descriptors for the different contexts, for example. I mean, if a device has two DMA ports that are in different contexts, they can be treated as different devices which are handed by a driver.

I worry that abstracting everything we can think may make the problem harder.

Yes. It's very complicated unfortunately, because we have to be able to deal with arbitrary combinations of a lot of oddball cases that can show up in random SoCs:

- a device may have DMA access to a bus that is invisible to the CPU

On 04/29/2014 03:00 PM, Arnd Bergmann wrote:
...

Quoted 3 lines

Yes. It's very complicated unfortunately, because we have to be able to deal with arbitrary combinations of a lot of oddball cases that can show up in random SoCs:

...

Quoted 1 line

- a device may have DMA access to a bus that is invisible to the CPU

The issue is slightly more general than that. It's more that the bus structure "seen" by a device is simply /different/ than that seen by the CPU. I don't think it's a requirement that there be CPU-invisible buses for that to be true.

For example, I could conceive of a HW setup like:

primary CPU bus ----------------------> other devices
| \_________________ /

   |                      \    |
   v                      v    ^
device registers ----> some secondary bus
                          |
                          v
                       memory

Here, all the buses are visible to the CPU, yet the path that transactions take between the buses is simply different to the CPU. More complex situations than the above, while still maintaining that description, are certainly possible.

Quoted 27 lines · Arnd Bergmann

Quoted 24 lines · On Monday 28 April 2014 20:30:56 Will Deacon wrote:

Quoted 11 lines

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

Quoted 7 lines

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

I think this is definitely going in the right direction, but it's not clear to me how the driver for device at 5 knows how to configure the two ports. We're still lacking topology information (unless that's implicit in the ordering of the properties) to say how the mastering capabilities of the device are actually routed and configured.

Quoted 2 lines

It would be helpful to have a concrete example of a device that has multiple masters. I have heard people mention this multiple times, and I can understand

You mean "a device that contains multiple independent bus masters", right? In particular, a device composed of multiple bus masters that do different things or should be handled differently by the interconnect.

There has definitely been talk on the list about real devices that use multiple stream IDs.

I'll ask around for "device-like" examples, but the most obvious example is the IOMMU itself.

Transactions generated by the IOMMU clearly need to be handled differently by the interconnect, compared with transactions translated and forwarded by IOMMU on behalf of its clients.

For example, suppose devices can post MSIs to an interrupt controller via a mailbox accessed through the IOMMU. Suppose also that the IOMMU generates MSIs itself in order to signal management events or faults to a host OS. Linux (as host) will need to configure the interrupt controller separately for the IOMMU and for the IOMMU clients. This means that Linux needs to know which IDs may travel to the interrupt controller for which purpose, and they must be distinct.

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

In the future, it is likely that "HSA"-style GPUs and other high- throughput virtualisable bus mastering devices will have capabilities of this sort, but I don't think there's anything concrete yet.

Quoted 7 lines

how it might be wired up in hardware, but I don't know what it's good for, or who would actually do it.

Quoted 4 lines

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

Quoted 4 lines

Quoted 4 lines

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

Ideally, we would reuse the ePAPR "ranges" concept and describe the way sideband ID signals propagate down the bus hierarchy in a similar way.

On Tuesday 29 April 2014 19:16:02 Dave Martin wrote:

Quoted 7 lines

For example, suppose devices can post MSIs to an interrupt controller via a mailbox accessed through the IOMMU. Suppose also that the IOMMU generates MSIs itself in order to signal management events or faults to a host OS. Linux (as host) will need to configure the interrupt controller separately for the IOMMU and for the IOMMU clients. This means that Linux needs to know which IDs may travel to the interrupt controller for which purpose, and they must be distinct.

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Quoted 5 lines

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

A lot of SoCs have MSI integrated into the PCI root complex, which of course is pointless from MSI perspective, as well as implying that the MSI won't go through the IOMMU.

We have briefly mentioned MSI in the review of the Samsung GH7 PCI support. It's possible that this one can either use the built-in MSI or the one in the GICv2m.

Quoted 3 lines

In the future, it is likely that "HSA"-style GPUs and other high- throughput virtualisable bus mastering devices will have capabilities of this sort, but I don't think there's anything concrete yet.

Wouldn't they just have IOMMUs with multiple contexts?

Quoted 12 lines

Quoted 11 lines

how it might be wired up in hardware, but I don't know what it's good for, or who would actually do it.

Quoted 8 lines

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

Quoted 2 lines

Ideally, we would reuse the ePAPR "ranges" concept and describe the way sideband ID signals propagate down the bus hierarchy in a similar way.

It would be 'dma-ranges'. Unfortunately that would imply that each DMA master is connected to only one IOMMU, which you say is not necessarily the case. The simpler case of a device is only a master on a single IOMMU but can use multiple contexts would however work fine with dma-ranges.

Quoted 1 line · Arnd Bergmann

On Tuesday 29 April 2014 19:16:02 Dave Martin wrote:

[...]

Quoted 8 lines

Quoted 7 lines

For example, suppose devices can post MSIs to an interrupt controller via a mailbox accessed through the IOMMU. Suppose also that the IOMMU generates MSIs itself in order to signal management events or faults to a host OS. Linux (as host) will need to configure the interrupt controller separately for the IOMMU and for the IOMMU clients. This means that Linux needs to know which IDs may travel to the interrupt controller for which purpose, and they must be distinct.

Quoted 4 lines

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Oops, looks like there are a few points I failed to respond to here...

I'm not an expert on PCI -- I'm prepared to believe it works that way.

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Quoted 9 lines

Quoted 4 lines

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

A lot of SoCs have MSI integrated into the PCI root complex, which of course is pointless from MSI perspective, as well as implying that the MSI won't go through the IOMMU.

Quoted 3 lines

We have briefly mentioned MSI in the review of the Samsung GH7 PCI support. It's possible that this one can either use the built-in MSI or the one in the GICv2m.

We are likely to get non-PCI MSIs in future SoC systems too, and there are no standards governing how such systems should look.

Quoted 4 lines

Quoted 3 lines

In the future, it is likely that "HSA"-style GPUs and other high- throughput virtualisable bus mastering devices will have capabilities of this sort, but I don't think there's anything concrete yet.

Quoted 1 line

Wouldn't they just have IOMMUs with multiple contexts?

Who knows? A management component of the GPU that is under exclusive control of the host or hypervisor might be wired up to bypass the IOMMU completely.

I'm not saying this kind of thing definitely will happen, but I can't say confidently that it won't.

Quoted 12 lines

Quoted 12 lines

Quoted 11 lines

how it might be wired up in hardware, but I don't know what it's good for, or who would actually do it.

Quoted 8 lines

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

Quoted 7 lines

Quoted 2 lines

Ideally, we would reuse the ePAPR "ranges" concept and describe the way sideband ID signals propagate down the bus hierarchy in a similar way.

It would be 'dma-ranges'. Unfortunately that would imply that each DMA master is connected to only one IOMMU, which you say is not necessarily the case. The simpler case of a device is only a master on a single IOMMU but can use multiple contexts would however work fine with dma-ranges.

Partly, yes. The concept embodied by "dma-ranges" is correct, but the topological relationship is not: the assumption that a master device always masters onto its parent node doesn't work for non-tree-like topologies.

Quoted 21 lines · On Thursday 01 May 2014 12:15:35 Dave Martin wrote:

Quoted 1 line · Arnd Bergmann

On Tuesday 29 April 2014 19:16:02 Dave Martin wrote:

[...]

Quoted 8 lines

Quoted 7 lines

For example, suppose devices can post MSIs to an interrupt controller via a mailbox accessed through the IOMMU. Suppose also that the IOMMU generates MSIs itself in order to signal management events or faults to a host OS. Linux (as host) will need to configure the interrupt controller separately for the IOMMU and for the IOMMU clients. This means that Linux needs to know which IDs may travel to the interrupt controller for which purpose, and they must be distinct.

Quoted 4 lines

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Oops, looks like there are a few points I failed to respond to here...

Quoted 4 lines

I'm not an expert on PCI -- I'm prepared to believe it works that way.

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Any idea what this is good for? Do we have to use it? It probably doesn't fit very well into the way Linux handles MSIs today.

Quoted 13 lines

Quoted 12 lines

Quoted 4 lines

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

A lot of SoCs have MSI integrated into the PCI root complex, which of course is pointless from MSI perspective, as well as implying that the MSI won't go through the IOMMU.

We have briefly mentioned MSI in the review of the Samsung GH7 PCI support. It's possible that this one can either use the built-in MSI or the one in the GICv2m.

Quoted 2 lines

We are likely to get non-PCI MSIs in future SoC systems too, and there are no standards governing how such systems should look.

I wouldn't call that MSI though -- using the same term in the code can be rather confusing. There are existing SoCs that use message based interrupt notification. We are probably better off modeling those are regular irqchips in Linux and DT, given that they may not be bound by the same constraints as PCI MSI.

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

In the future, it is likely that "HSA"-style GPUs and other high- throughput virtualisable bus mastering devices will have capabilities of this sort, but I don't think there's anything concrete yet.

Quoted 8 lines

Quoted 1 line

Wouldn't they just have IOMMUs with multiple contexts?

Who knows? A management component of the GPU that is under exclusive control of the host or hypervisor might be wired up to bypass the IOMMU completely.

I'm not saying this kind of thing definitely will happen, but I can't say confidently that it won't.

Supporting this case in DT straight away is going to add a major burden. If nobody can say for sure that they are actually going to do it, I'd lean towards assuming that we won't need it and not putting the extra complexity in.

If someone actually needs it later, let's make it their problem for not participating in the design.

Quoted 21 lines

Quoted 20 lines

Quoted 8 lines

Quoted 8 lines

how it might be wired up in hardware, but I don't know what it's good for, or who would actually do it.

Quoted 4 lines

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

Quoted 7 lines

Quoted 4 lines

Quoted 4 lines

That would also allow us to describe ranges of master IDs, which we need for things like PCI RCs on the ARM SMMU. Furthermore, basic transformations of these ranges could also be described like this, although I think Dave (CC'd) has some similar ideas in this area.

Ideally, we would reuse the ePAPR "ranges" concept and describe the way sideband ID signals propagate down the bus hierarchy in a similar way.

It would be 'dma-ranges'. Unfortunately that would imply that each DMA master is connected to only one IOMMU, which you say is not necessarily the case. The simpler case of a device is only a master on a single IOMMU but can use multiple contexts would however work fine with dma-ranges.

Quoted 4 lines

Partly, yes. The concept embodied by "dma-ranges" is correct, but the topological relationship is not: the assumption that a master device always masters onto its parent node doesn't work for non-tree-like topologies.

In almost all cases it will fit. When it doesn't, we can work around it by defining virtual address spaces the way that the PCI binding does. The only major exception that we know we have to handle is IOMMUs.

Quoted 24 lines · Arnd Bergmann

Quoted 23 lines · On Thursday 01 May 2014 12:15:35 Dave Martin wrote:

Quoted 1 line · Arnd Bergmann

On Tuesday 29 April 2014 19:16:02 Dave Martin wrote:

[...]

Quoted 8 lines

Quoted 7 lines

For example, suppose devices can post MSIs to an interrupt controller via a mailbox accessed through the IOMMU. Suppose also that the IOMMU generates MSIs itself in order to signal management events or faults to a host OS. Linux (as host) will need to configure the interrupt controller separately for the IOMMU and for the IOMMU clients. This means that Linux needs to know which IDs may travel to the interrupt controller for which purpose, and they must be distinct.

Quoted 4 lines

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Oops, looks like there are a few points I failed to respond to here...

I'm not an expert on PCI -- I'm prepared to believe it works that way.

Quoted 5 lines

Quoted 2 lines

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Any idea what this is good for? Do we have to use it? It probably doesn't fit very well into the way Linux handles MSIs today.

Marc may be better placed than me to comment on this in detail.

However, I believe it's correct to say that because the GIC is not part of PCI, end-to-end MSI delivery inherently involves a non-PCI step from the PCI RC to the GIC itself.

Thus this is likely to be a fundamental requirement for MSIs on ARM SoCs using GIC, if we want to have a hope of mapping MSIs to VMs efficiently.

Quoted 13 lines

Quoted 13 lines

Quoted 12 lines

Quoted 4 lines

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

A lot of SoCs have MSI integrated into the PCI root complex, which of course is pointless from MSI perspective, as well as implying that the MSI won't go through the IOMMU.

We have briefly mentioned MSI in the review of the Samsung GH7 PCI support. It's possible that this one can either use the built-in MSI or the one in the GICv2m.

Quoted 8 lines

Quoted 2 lines

We are likely to get non-PCI MSIs in future SoC systems too, and there are no standards governing how such systems should look.

I wouldn't call that MSI though -- using the same term in the code can be rather confusing. There are existing SoCs that use message based interrupt notification. We are probably better off modeling those are regular irqchips in Linux and DT, given that they may not be bound by the same constraints as PCI MSI.

We can call it what we like and maybe bury the distinction in irqchip drivers for some fixed-configuration cases, but it's logically the same concept. Naming and subsystem factoring are implementation decisions for Linux.

For full dynamic assignment of pluggable devices or buses to VMs, I'm less sure that we can model that as plain irqchips.

Quoted 16 lines · Dave Martin

Quoted 15 lines · Arnd Bergmann

Quoted 13 lines · On Thursday 01 May 2014 12:15:35 Dave Martin wrote:

Quoted 4 lines · Arnd Bergmann

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Oops, looks like there are a few points I failed to respond to here...

I'm not an expert on PCI -- I'm prepared to believe it works that way.

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Quoted 4 lines

Quoted 2 lines

Any idea what this is good for? Do we have to use it? It probably doesn't fit very well into the way Linux handles MSIs today.

Marc may be better placed than me to comment on this in detail.

As to "fitting Linux", it seems to match what Linux does fairly well (see the kvm-arm64/gicv3 branch in my tree). Not saying that it does it in a very simple way (far from it, actually), but it works.

As to "what it is good for" (and before someone bursts into an Edwin Starr interpretation), it mostly has to do with isolation, and the fact that you may want to let the whole MSI programming to a guest (and yet ensure that the guest cannot generate interrupts that would be assigned to other devices). This is done by sampling the requester-id at the ITS level, and use this information to index a per-device interrupt translation table (I could talk for hours about the concept and its variations, mostly using expletives and possibly a hammer, but I think it is the time for my pink pill).

Quoted 6 lines

However, I believe it's correct to say that because the GIC is not part of PCI, end-to-end MSI delivery inherently involves a non-PCI step from the PCI RC to the GIC itself.

Thus this is likely to be a fundamental requirement for MSIs on ARM SoCs using GIC, if we want to have a hope of mapping MSIs to VMs efficiently.

Indeed. GICv[34] is the ARM way of implementing MSI on SBSA compliant systems (from level 1 onwards, if memory serves well). People are actively building systems with this architecture, and relying on it to provide VM isolation.

Quoted 17 lines · On Thursday 01 May 2014 16:11:48 Marc Zyngier wrote:

Quoted 16 lines · Dave Martin

Quoted 15 lines · Arnd Bergmann

Quoted 13 lines · On Thursday 01 May 2014 12:15:35 Dave Martin wrote:

Quoted 4 lines · Arnd Bergmann

I don't understand. An MSI controller is just an address that acts as a DMA slave for a 4-byte inbound data packet. It has no way of knowing who is sending data, other than by the address or the data sent to it. Are you talking of something else?

Oops, looks like there are a few points I failed to respond to here...

I'm not an expert on PCI -- I'm prepared to believe it works that way.

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

Any idea what this is good for? Do we have to use it? It probably doesn't fit very well into the way Linux handles MSIs today.

Marc may be better placed than me to comment on this in detail.

As to "fitting Linux", it seems to match what Linux does fairly well (see the kvm-arm64/gicv3 branch in my tree). Not saying that it does it in a very simple way (far from it, actually), but it works.

ok.

Quoted 9 lines

As to "what it is good for" (and before someone bursts into an Edwin Starr interpretation), it mostly has to do with isolation, and the fact that you may want to let the whole MSI programming to a guest (and yet ensure that the guest cannot generate interrupts that would be assigned to other devices). This is done by sampling the requester-id at the ITS level, and use this information to index a per-device interrupt translation table (I could talk for hours about the concept and its variations, mostly using expletives and possibly a hammer, but I think it is the time for my pink pill).

So the idea is that you want to give the guest unfiltered access to the PCI config space of an assigned device?

Is that safe?

If the config space access goes through the hypervisor (as I think we always do today), there should be no isse.

Quoted 7 lines

Quoted 6 lines

However, I believe it's correct to say that because the GIC is not part of PCI, end-to-end MSI delivery inherently involves a non-PCI step from the PCI RC to the GIC itself.

Thus this is likely to be a fundamental requirement for MSIs on ARM SoCs using GIC, if we want to have a hope of mapping MSIs to VMs efficiently.

Quoted 4 lines

Indeed. GICv[34] is the ARM way of implementing MSI on SBSA compliant systems (from level 1 onwards, if memory serves well). People are actively building systems with this architecture, and relying on it to provide VM isolation.

I don't mind it being there, but if we don't need it, we shouldn't have to use that isolation feature and be able to just allow any initiator to send an MSI.

Quoted 14 lines · Arnd Bergmann

Quoted 13 lines · On Thursday 01 May 2014 12:15:35 Dave Martin wrote:

Quoted 12 lines

Quoted 4 lines

I'm not sure whether there is actually a SoC today that is MSI-capable and contains an IOMMU, but all the components to build one are out there today. GICv3 is also explicitly designed to support such systems.

A lot of SoCs have MSI integrated into the PCI root complex, which of course is pointless from MSI perspective, as well as implying that the MSI won't go through the IOMMU.

We have briefly mentioned MSI in the review of the Samsung GH7 PCI support. It's possible that this one can either use the built-in MSI or the one in the GICv2m.

Quoted 8 lines

Quoted 2 lines

We are likely to get non-PCI MSIs in future SoC systems too, and there are no standards governing how such systems should look.

I wouldn't call that MSI though -- using the same term in the code can be rather confusing. There are existing SoCs that use message based interrupt notification. We are probably better off modeling those are regular irqchips in Linux and DT, given that they may not be bound by the same constraints as PCI MSI.

We can call it what we like and maybe bury the distinction in irqchip drivers for some fixed-configuration cases, but it's logically the same concept. Naming and subsystem factoring are implementation decisions for Linux.

For full dynamic assignment of pluggable devices or buses to VMs, I'm less sure that we can model that as plain irqchips.

Quoted 7 lines

Supporting this case in DT straight away is going to add a major burden. If nobody can say for sure that they are actually going to do it, I'd lean towards assuming that we won't need it and not putting the extra complexity in.

If someone actually needs it later, let's make it their problem for not participating in the design.

This is a fair point, but there is a difference between the bindings and what kind of wacky configurations a particular version of Linux actually supports.

DT is supposed to be a description of the hardware, not a description of how Linux subsystems are structured, though if the two are not reasonably well aligned that will lead to pain...

The key thing is to make sure the DT bindings are extensible to things that we can reasonably foresee.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

It would be 'dma-ranges'. Unfortunately that would imply that each DMA master is connected to only one IOMMU, which you say is not necessarily the case. The simpler case of a device is only a master on a single IOMMU but can use multiple contexts would however work fine with dma-ranges.

Quoted 8 lines

Quoted 4 lines

Partly, yes. The concept embodied by "dma-ranges" is correct, but the topological relationship is not: the assumption that a master device always masters onto its parent node doesn't work for non-tree-like topologies.

In almost all cases it will fit. When it doesn't, we can work around it by defining virtual address spaces the way that the PCI binding does. The only major exception that we know we have to handle is IOMMUs.

My concern here is that as new exceptions and oddball or complex systems crop up, we will end up repeatedly inventing different bodges to solve essentially the same problem.

Unlike some of the other situations we have to deal with, these are valid hardware configurations rather than quirks or broken systems.

Quoted 2 lines

GICv3 can descriminate between different MSI senders based on ID signals on the bus.

Any idea what this is good for? Do we have to use it? It probably doesn't fit very well into the way Linux handles MSIs today.

Quoted 2 lines

We are likely to get non-PCI MSIs in future SoC systems too, and there are no standards governing how such systems should look.

I wouldn't call that MSI though -- using the same term in the code can be rather confusing. There are existing SoCs that use message based interrupt notification. We are probably better off modeling those are regular irqchips in Linux and DT, given that they may not be bound by the same constraints as PCI MSI.

Quoted 3 lines

Who knows? A management component of the GPU that is under exclusive control of the host or hypervisor might be wired up to bypass the IOMMU completely.

Quoted 4 lines

Partly, yes. The concept embodied by "dma-ranges" is correct, but the topological relationship is not: the assumption that a master device always masters onto its parent node doesn't work for non-tree-like topologies.

In almost all cases it will fit. When it doesn't, we can work around it by defining virtual address spaces the way that the PCI binding does. The only major exception that we know we have to handle is IOMMUs.

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

On Mon, Apr 28, 2014 at 02:05:30PM +0200, Arnd Bergmann wrote:
[...]

Quoted 31 lines

let me clarify by example:

	iommu at 1 {
		compatible = "some,simple-iommu";
		reg = <1>;
		#iommu-cells = <0>; /* supports only one master */
	};

	iommu at 2 {
		compatible = "some,other-iommu";
		reg = <3>;
		#iommu-cells = <1>; /* contains master ID */
	};

	iommu at 3 {
		compatible = "some,windowed-iommu";
		reg = <2>;
		#iommu-cells = <2>; /* contains dma-window */
	};

	device at 4 {
		compatible = "some,ethernet";
		iommus = <&/iommu@1>;
	};

	device at 5 {
		compatible = "some,dmaengine";
		iommus = <&/iommu@2 0x40000000 0x1000000>,
			 <&/iommu@3 0x101>;
	};

Quoted 7 lines

The device at address 4 has a one-one relationship with iommu at 1, so there is no need for any data. device at 5 has two master ports. One is connected to an IOMMU that has a per-device aperture, device at 5 can only issue transfers to the 256MB area at 0x40000000, and the IOMMU will have to put entries for this device into that address. The second master port is connected to iommu at 3, which uses a master ID that gets passed along with each transfer, so that needs to be put into the IOTLBs.

iommu at 3 and the second port of device at 5 seem to match what we need for Tegra (and as I understand also Exynos). Can we settle on this for now so that Hiroshi and Cho can go update their drivers for this binding?

Quoted 3 lines

A variation would be to not use #iommu-cells at all, but provide a #address-cells / #size-cells pair in the IOMMU, and have a translation as we do for dma-ranges. This is probably most flexible.

The remainder of this discussion seems to indicate that #iommu-cells and dma-ranges don't have to be mutually exclusive. For some IOMMUs it might make sense to use both.

In fact perhaps we should require every IOMMU user to also specify a dma-ranges property, even if for some cases the range would be simply the complete physical address space. Perhaps in analogy to the ranges property an empty dma-ranges property could be taken to mean all of the physical address space.

I'm aware that this doesn't cover any of the more exotic cases out there, but the fact is that we have real devices out there that ship with some variations of these simple IOMMUs and I don't think we're doing ourselves a favour by blocking support for these to be added on the hope of merging the perfect solution that covers all use-cases. Patches for Tegra have already been around for close to half a year.

Thierry

On Mon, Apr 28, 2014 at 12:56:03PM +0200, Arnd Bergmann wrote:

Quoted 3 lines

A lot of drivers probably only support one
master, so they can just set #iommu-cells=<0>, others might require IDs that do not fit into one cell.

You mean "#iommu-cells = <1>" for devices that only require one master? There still has to be one cell to specify which master. Unless perhaps if they can be arbitrarily assigned. I guess even if there's a fixed mapping that applies to one SoC generation, it might be good to still employ a specifier and have the mapping in DT for flexibility.

* PGP Signed by an unknown key

Quoted 12 lines · Arnd Bergmann

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt line 58

Quoted 7 lines · On Sunday 27 April 2014 13:07:43 Shaik Ameer Basha wrote:

+- mmu-masters: A phandle to device nodes representing the master for which

Documentation/devicetree/bindings/iommu/samsung,sysmmu.txt new file +6 -0

+               the System MMU can provide a translation. Any additional values
+              after the phandle will be ignored because a System MMU never
+              have two or more masters. "#stream-id-cells" specified in the
+              master's node will be also ignored.
+              If more than one phandle is specified, only the first phandle
+              will be treated.

This seems completely backwards: Why would you list the masters for an IOMMU in the IOMMU node?

Quoted 7 lines

The master should have a standard property pointing to the IOMMU instead.

We don't have a generic binding for IOMMUs yet it seems, but the time is overdue to make one.

Consider this NAKed until there is a generic binding for IOMMUs that all relevant developers have agreed to.

I'd like to take this opportunity and revive one of the hibernating patch sets that we have for Tegra. The last effort to get things merged was back in January I think. I haven't bothered to look up the reference since it's probably good to start from scratch anyway.

The latest version of the binding that was under discussion back then I think looked something like this:

	device at ... {
		iommus = <&iommu [spec]>[, <&other_iommu [other_spec]>...];
	};

And possibly with a iommu-names property to go along with that. The idea being that a device can be a master on possibly multiple IOMMUs. Using the above it would also be possible to have one device be multiple masters on the same IOMMU.

On Tegra the specifier would be used to encode a memory controller's client ID. One discussion point back at the time was to encode the ID as a bitmask to allow more than a single master per entry. Another solution which I think is a little cleaner and more generic, would be to use one entry per master and use a single cell to encode the client ID. Devices with multiple clients to the same IOMMU could then use multiple entries referencing the same IOMMU.

I've added Hiroshi Doyu on Cc since he knows the Tegra IOMMU best. Hiroshi, can you summarize exactly what the proposed bindings were. If my memory serves me well they were mostly along the lines of what Arnd proposes here, and perhaps they are something that can also be used for Exynos.

drivers/iommu/exynos-iommu.c +13 -0

@@ -542,14 +592,41 @@ static int __init exynos_sysmmu_probe(struct platform_device *pdev)
 		}
 	}
 
+	/* Relation between master and System MMU is 1:1. */
+	node = of_parse_phandle(dev->of_node, "mmu-masters", 0);
+	if (node) {
+		struct platform_device *master = of_find_device_by_node(node);
+
+		if (!master) {
+			dev_err(dev, "%s: mmu-master '%s' not found\n",
+				__func__, node->name);
+			return -EINVAL;
+		}
+
+		if (master->dev.archdata.iommu != NULL) {
+			dev_err(dev, "%s: '%s' is master of other MMU\n",

drivers/iommu/exynos-iommu.c +11 -0

+
+static struct notifier_block sysmmu_notifier = {
+	.notifier_call = &sysmmu_hook_driver_register,
+};
+
+static int __init exynos_iommu_prepare(void)
+{
+	static bool registered = false;
+	int ret;
+
+	if (registered)

Quoted 17 lines · On Sunday 27 April 2014 13:07:47 Shaik Ameer Basha wrote:

drivers/iommu/exynos-iommu.c +13 -0

@@ -542,14 +592,41 @@ static int __init exynos_sysmmu_probe(struct platform_device *pdev)
 		}
 	}
 
+	/* Relation between master and System MMU is 1:1. */
+	node = of_parse_phandle(dev->of_node, "mmu-masters", 0);
+	if (node) {
+		struct platform_device *master = of_find_device_by_node(node);
+
+		if (!master) {
+			dev_err(dev, "%s: mmu-master '%s' not found\n",
+				__func__, node->name);
+			return -EINVAL;
+		}
+
+		if (master->dev.archdata.iommu != NULL) {
+			dev_err(dev, "%s: '%s' is master of other MMU\n",

drivers/iommu/exynos-iommu.c static int __init exynos_sysmmu_probe(struct platform_device *pdev) · line 617

Quoted 10 lines

drivers/iommu/exynos-iommu.c +10 -0

+				__func__, node->name);
+			return -EINVAL;
+		}
+
+		/*
+		 * archdata.iommu will be initialized with exynos_iommu_client
+		 * in sysmmu_hook_driver_register().
+		 */
+		master->dev.archdata.iommu = dev;
+	}

I think parsing of generic properties like this shouldn't be done by the individual driver. It belongs into core iommu code and should be done when the devices are probed.

Quoted 16 lines

drivers/iommu/exynos-iommu.c +16 -0

+
+static struct notifier_block sysmmu_notifier = {
+	.notifier_call = &sysmmu_hook_driver_register,
+};
+
+static int __init exynos_iommu_prepare(void)
+{
+	static bool registered = false;
+	int ret;
+
+	if (registered)
+		return 0;
+
+	ret = bus_register_notifier(&platform_bus_type, &sysmmu_notifier);
+	if (!ret)
+		registered = true;

drivers/iommu/exynos-iommu.c err_reg_driver: · line 1128

Quoted 5 lines

drivers/iommu/exynos-iommu.c +5 -0

+	else
+		pr_err("Failed to register sysmmu_notifier\n");
+
+	return ret;
+}

This also means we should not have notifiers. Put it into core code to make sure all IOMMUs are handled the same way.