devicetree: Add generic IOMMU device tree bindings

From: Thierry Reding <[email protected]>

This commit introduces a generic device tree binding for IOMMU devices. Only a very minimal subset is described here, but it is enough to cover the requirements of both the Exynos System MMU and Tegra SMMU as discussed here:

    https://lkml.org/lkml/2014/4/27/346

Signed-off-by Thierry Reding <[email protected]>

---
Changes in v2:
- add notes about "dma-ranges" property (drop note from commit message)
- document priorities of "iommus" property vs. "dma-ranges" property
- drop #iommu-cells in favour of #address-cells and #size-cells
- remove multiple-master device example

 Documentation/devicetree/bindings/iommu/iommu.txt | 167 ++++++++++++++++++++++
 1 file changed, 167 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/iommu.txt

Documentation/devicetree/bindings/iommu/iommu.txt new file +66 -0

new file mode 100644
index 000000000000..6ce759afcc94
@@ -0,0 +1,167 @@
+This document describes the generic device tree binding for IOMMUs and their
+master(s).
+
+
+IOMMU device node:
+==================
+
+An IOMMU can provide the following services:
+
+* Remap address space to allow devices to access physical memory ranges that
+  they otherwise wouldn't be capable of accessing.
+
+  Example: 32-bit DMA to 64-bit physical addresses
+
+* Implement scatter-gather at page level granularity so that the device does
+  not have to.
+
+* Provide system protection against "rogue" DMA by forcing all accesses to go
+  through the IOMMU and faulting when encountering accesses to unmapped
+  address regions.
+
+* Provide address space isolation between multiple contexts.
+
+  Example: Virtualization
+
+Device nodes compatible with this binding represent hardware with some of the
+above capabilities.
+
+IOMMUs can be single-master or multiple-master. Single-master IOMMU devices
+typically have a fixed association to the master device, whereas multiple-
+master IOMMU devices can translate accesses from more than one master.
+
+The device tree node of the IOMMU device's parent bus must contain a valid
+"dma-ranges" property that describes how the physical address space of the
+IOMMU maps to memory. An empty "dma-ranges" property means that there is a
+1:1 mapping from IOMMU to memory.
+
+Required properties:
+--------------------
+- #address-cells: The number of cells in an IOMMU specifier needed to encode
+  an address.
+- #size-cells: The number of cells in an IOMMU specifier needed to represent
+  the length of an address range.
+
+Typical values for the above include:
+- #address-cells = <0>, size-cells = <0>: Single master IOMMU devices are not
+  configurable and therefore no additional information needs to be encoded in
+  the specifier. This may also apply to multiple master IOMMU devices that do
+  not allow the association of masters to be configured.
+- #address-cells = <1>, size-cells = <0>: Multiple master IOMMU devices may
+  need to be configured in order to enable translation for a given master. In
+  such cases the single address cell corresponds to the master device's ID.
+- #address-cells = <2>, size-cells = <2>: Some IOMMU devices allow the DMA
+  window for masters to be configured. The first cell of the address in this
+  may contain the master device's ID for example, while the second cell could
+  contain the start of the DMA window for the given device. The length of the
+  DMA window is specified by two additional cells.
+
+
+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt new file +14 -0

+
+
+Notes:
+======
+
+One possible extension to the above is to use an "iommus" property along with
+a "dma-ranges" property in a bus device node (such as PCI host bridges). This
+can be useful to describe how children on the bus relate to the IOMMU if they
+are not explicitly listed in the device tree (e.g. PCI devices). However, the
+requirements of that use-case haven't been fully determined yet. Implementing
+this is therefore not recommended without further discussion and extension of
+this binding.
+
+

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

This would probably mean that you need both #iommu-cells and #address-cells.

Documentation/devicetree/bindings/iommu/iommu.txt line 81

Quoted 5 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +5 -0

+
+Optional properties:
+--------------------
+- iommu-names: A list of names identifying each entry in the "iommus"
+  property.

Do we really need a name here? I would not expect that you have clearly documented names here from the datasheet like you would for interrupts or clocks, so you'd just be making up names. Sorry, but I'm not a fan of names properties in general.

Quoted 11 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +11 -0

+Multiple-master IOMMU:
+----------------------
+
+       iommu {
+               /* the specifier represents the ID of the master */
+               #address-cells = <1>;
+               #size-cells = <0>;
+       };
+
+       master {
+               /* device has master ID 42 in the IOMMU */

Review

Quoted 2 lines

(quoted diff) +2 -0

+               iommus = <&/iommu 42>;
+       };

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

Quoted 11 lines

Quoted 11 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +11 -0

+Multiple-master IOMMU:
+----------------------
+
+       iommu {
+               /* the specifier represents the ID of the master */
+               #address-cells = <1>;
+               #size-cells = <0>;
+       };
+
+       master {
+               /* device has master ID 42 in the IOMMU */

Quoted 8 lines

Review

Quoted 2 lines

(quoted diff) +2 -0

+               iommus = <&/iommu 42>;
+       };

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

I don't understand the problem. If you have stream IDs 0 through 7, you would have

	master at a {
		...
		iommus = <&smmu 0>;
	};

	master at b {
		...
		iommus = <&smmu 1;
	};

	...

	master at 12 {
		...
		iommus = <&smmu 7;
	};

and you don't need a window at all. Why would you need a mask of some sort?

IIUC the original problem, "a master with 8 streamIDs" means something like below, where some devices have multiple IDs but some have a single. A sinle #address-cells cannot afford those 2 masters at once.

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu 1 2 3>; # 3 IDs
        };

        master at b {
                ...
                iommus = <&smmu 4>;     # 1 ID
        };

Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64 bit) to afford 64 stream IDs so that a single device can hold multiple IDs. If we apply the same bitmap to the above exmaple:

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
        };

        master at b {
                ...
                iommus = <&smmu BIT(4)>;     # ID 4
        };

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Quoted 20 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64 bit) to afford 64 stream IDs so that a single device can hold multiple IDs. If we apply the same bitmap to the above exmaple:

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
        };

        master at b {
                ...
                iommus = <&smmu BIT(4)>;     # ID 4
        };

Quoted 5 lines

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

Quoted 27 lines · Arnd Bergmann

Quoted 20 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64 bit) to afford 64 stream IDs so that a single device can hold multiple IDs. If we apply the same bitmap to the above exmaple:

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
        };

        master at b {
                ...
                iommus = <&smmu BIT(4)>;     # ID 4
        };

Quoted 5 lines

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Quoted 3 lines

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

In general, though, the SMMU will be able to support a large number of stream IDs (say a 16-bit space). The restriction we're interested in here is how many different stream IDs can be emitted by a single master device coming into the SMMU. *That* is a property of the master, not the SMMU.

In the current arm,smmu binding I have a #stream-id-cells property in each master. I can then feed that straight into of_parse_phandle_with_args to enumerate the IDs for that master. The problem with that is we're artificially restricted by MAX_PHANDLE_ARGS.

Quoted 5 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

IIUC the original problem, "a master with 8 streamIDs" means something like below, where some devices have multiple IDs but some have a single. A sinle #address-cells cannot afford those 2 masters at once.

Quoted 15 lines

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu 1 2 3>; # 3 IDs
        };

        master at b {
                ...
                iommus = <&smmu 4>;     # 1 ID
        };

This would not be the usual format really. It should instead be

		iommus = <&smmu 1>, <&smmu 2>, <&smmu 3>;

which can be tedious to type.

Quoted 20 lines

Tegra,SMMU has a similar problem and we have used a fixed size bitmap(64 bit) to afford 64 stream IDs so that a single device can hold multiple IDs. If we apply the same bitmap to the above exmaple:

       iommu {
               /* the specifier represents the ID of the master */
               #address-cells = <1>;
               #size-cells = <0>;
       };

        master at a {
                ...
                iommus = <&smmu (BIT(1) | BIT(2) | BIT(3))>; # IDs 1 2 3
        };

        master at b {
                ...
                iommus = <&smmu BIT(4)>;     # ID 4
        };

Quoted 5 lines

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

Quoted 3 lines

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

That sounds fairly complicated to me. I don't see what that buys us over the clarity and simplicity that the above explicit notation gives us. Is it not more common for a device to have a single master rather than a whole bunch of them?

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Quoted 3 lines

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

That sounds fairly complicated to me. I don't see what that buys us over the clarity and simplicity that the above explicit notation gives us. Is it not more common for a device to have a single master rather than a whole bunch of them?

Quoted 8 lines · Thierry Reding

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Quoted 8 lines

Quoted 3 lines

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

That sounds fairly complicated to me. I don't see what that buys us over the clarity and simplicity that the above explicit notation gives us. Is it not more common for a device to have a single master rather than a whole bunch of them?

I've never seen a device upstream of an ARM SMMU with a single stream-id; they always seem to have a whole bunch of them. Calxeda's SATA controller had 10 IDs, for example, and a PL330 DMA controller tends to have at least 3.

Quoted 18 lines · On Wednesday 04 June 2014 14:56:01 Will Deacon wrote:

Quoted 16 lines · Thierry Reding

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 May 2014 22:29:13 Hiroshi Doyu wrote:

The disadvantage of this is that this limits the max number of streamIDs to support. If # of streamID is increased later more than 64, this format cannot cover any more. You have to predict the max # of streamIDs in advance if steamID is statically assigned.

Well, the iommu specific binding could allow a variable #address-cells. That way, you just need to know the number of stream IDs for that instance of the iommu.

That sounds fairly complicated to me. I don't see what that buys us over the clarity and simplicity that the above explicit notation gives us. Is it not more common for a device to have a single master rather than a whole bunch of them?

Quoted 4 lines

I've never seen a device upstream of an ARM SMMU with a single stream-id; they always seem to have a whole bunch of them. Calxeda's SATA controller had 10 IDs, for example, and a PL330 DMA controller tends to have at least 3.

What are those good for? Would we just always use the first one?

Quoted 25 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

Quoted 1 line

This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window for each device, while there can only be one dma-ranges property in an iommu node.

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt line 81

Quoted 5 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +5 -0

+
+Optional properties:
+--------------------
+- iommu-names: A list of names identifying each entry in the "iommus"
+  property.

Do we really need a name here? I would not expect that you have clearly documented names here from the datasheet like you would for interrupts or clocks, so you'd just be making up names. Sorry, but I'm not a fan of names properties in general.

Good point, this was really overdesign by modeling it after other subsystems that can have a use for names.

Quoted 11 lines

Quoted 11 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +11 -0

+Multiple-master IOMMU:
+----------------------
+
+       iommu {
+               /* the specifier represents the ID of the master */
+               #address-cells = <1>;
+               #size-cells = <0>;
+       };
+
+       master {
+               /* device has master ID 42 in the IOMMU */

Quoted 8 lines

Review

Quoted 2 lines

(quoted diff) +2 -0

+               iommus = <&/iommu 42>;
+       };

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

I don't understand the problem. If you have stream IDs 0 through 7, you would have

        master at a {
                ...
                iommus = <&smmu 0>;
        };

        master at b {
                ...
                iommus = <&smmu 1;
        };

        ...

        master at 12 {
                ...
                iommus = <&smmu 7;
        };

and you don't need a window at all. Why would you need a mask of some sort?

Quoted 26 lines · Arnd Bergmann

Quoted 25 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

Quoted 5 lines

Quoted 1 line

This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window for each device, while there can only be one dma-ranges property in an iommu node.

My suggestion was that you also put the IDs in the dma-ranges as the first cell much as ranges for PCI encodes other information in the first cell. Then you can have an entry for each ID. The downside is another special case like PCI.

The argument for using #address-cells and #size-cells seems to be to align with how ranges work. If that's the route we want to go, then I say we should not stop there and actually use dma-ranges as well. Otherwise, I don't see the advantage over #iommu-cells.

Quoted 20 lines

I don't understand the problem. If you have stream IDs 0 through 7, you would have

        master at a {
                ...
                iommus = <&smmu 0>;
        };

        master at b {
                ...
                iommus = <&smmu 1;
        };

        ...

        master at 12 {
                ...
                iommus = <&smmu 7;
        };

Quoted 2 lines

and you don't need a window at all. Why would you need a mask of some sort?

1 master with 7 IDs like this:

         master at a {
                 ...
                 iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
<&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
         };

If there was any sort of window, then it is almost certainly the same window for each ID.

Quoted 38 lines · On Friday 30 May 2014 14:31:55 Rob Herring wrote:

Quoted 26 lines · Arnd Bergmann

Quoted 25 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

Quoted 5 lines

Quoted 1 line

This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window for each device, while there can only be one dma-ranges property in an iommu node.

My suggestion was that you also put the IDs in the dma-ranges as the first cell much as ranges for PCI encodes other information in the first cell. Then you can have an entry for each ID. The downside is another special case like PCI.

Quoted 4 lines

The argument for using #address-cells and #size-cells seems to be to align with how ranges work. If that's the route we want to go, then I say we should not stop there and actually use dma-ranges as well. Otherwise, I don't see the advantage over #iommu-cells.

I can see how dma-ranges in bus nodes work, it just doesn't seem to have any reasonable meaning in the iommu node itself.

Quoted 26 lines

Quoted 21 lines

I don't understand the problem. If you have stream IDs 0 through 7, you would have

        master at a {
                ...
                iommus = <&smmu 0>;
        };

        master at b {
                ...
                iommus = <&smmu 1;
        };

        ...

        master at 12 {
                ...
                iommus = <&smmu 7;
        };

Quoted 2 lines

and you don't need a window at all. Why would you need a mask of some sort?

1 master with 7 IDs like this:

Quoted 8 lines

         master at a {
                 ...
                 iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
<&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
         };

If there was any sort of window, then it is almost certainly the same window for each ID.

Ok, I see. In that case you'd probably want to have #address-cells = <1> and #size-cells = <1> and give a range of IDs like

	iommus = <&smmu 0 8>;

Do you think that ranges can have a meaningful definition with the ARM SMMU stream IDs?

Quoted 39 lines · Arnd Bergmann

Quoted 38 lines · On Friday 30 May 2014 14:31:55 Rob Herring wrote:

Quoted 26 lines · Arnd Bergmann

Quoted 25 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

Quoted 5 lines

Quoted 1 line

This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window for each device, while there can only be one dma-ranges property in an iommu node.

My suggestion was that you also put the IDs in the dma-ranges as the first cell much as ranges for PCI encodes other information in the first cell. Then you can have an entry for each ID. The downside is another special case like PCI.

Quoted 7 lines

Quoted 4 lines

The argument for using #address-cells and #size-cells seems to be to align with how ranges work. If that's the route we want to go, then I say we should not stop there and actually use dma-ranges as well. Otherwise, I don't see the advantage over #iommu-cells.

I can see how dma-ranges in bus nodes work, it just doesn't seem to have any reasonable meaning in the iommu node itself.

dma-ranges defines a static mapping for mastering through the bus node.

The whole point of an IOMMU is that it maps dynamically, so I agree:
I'm unclear on what dma-ranges should mean in the IOMMU node itself (if anything).

Quoted 26 lines

Quoted 26 lines

Quoted 21 lines

I don't understand the problem. If you have stream IDs 0 through 7, you would have

        master at a {
                ...
                iommus = <&smmu 0>;
        };

        master at b {
                ...
                iommus = <&smmu 1;
        };

        ...

        master at 12 {
                ...
                iommus = <&smmu 7;
        };

Quoted 2 lines

and you don't need a window at all. Why would you need a mask of some sort?

1 master with 7 IDs like this:

Quoted 8 lines

Quoted 8 lines

         master at a {
                 ...
                 iommus = <&smmu 0> <&smmu 1> <&smmu 2> <&smmu 3>
<&smmu 4> <&smmu 5> <&smmu 6> <&smmu 7>;
         };

If there was any sort of window, then it is almost certainly the same window for each ID.

Do we have real examples of using a window *and* an ID? I thought the windowed-IOMMU concept was partly a way of encoding the ID in some real address bits on the bus. If you're doing that, it seems less likely that there is a true "ID" as such (though it is possible).

Quoted 7 lines

Ok, I see. In that case you'd probably want to have #address-cells = <1> and #size-cells = <1> and give a range of IDs like

	iommus = <&smmu 0 8>;

Do you think that ranges can have a meaningful definition with the ARM SMMU stream IDs?

In the strictest sense, no.

But for a large set of sane configurations, this probably works.

Small sets of randomly-assigned IDs can just be enumerated one by one.

We wouldn't be able to describe folding and bit shuffling, but we probably don't want to encourage that anyway.

Quoted 8 lines · Dave Martin

In the strictest sense, no.

But for a large set of sane configurations, this probably works.

Small sets of randomly-assigned IDs can just be enumerated one by one.

We wouldn't be able to describe folding and bit shuffling, but we probably don't want to encourage that anyway.

I'm having some difficulty understanding this. You make it sound like there's a fairly arbitrary number of IDs that the SMMU can handle. So how is the mapping to devices defined? If you say encourage that does make it sound like the assignment of IDs is purely defined by some mechanism in software rather than in hardware. Or they are more or less randomly picked by someone. If that's the case, is that not something that should be dynamically allocated by the kernel rather than put into the device tree?

Quoted 10 lines · Thierry Reding

Quoted 8 lines · Dave Martin

In the strictest sense, no.

But for a large set of sane configurations, this probably works.

Small sets of randomly-assigned IDs can just be enumerated one by one.

We wouldn't be able to describe folding and bit shuffling, but we probably don't want to encourage that anyway.

Quoted 8 lines

I'm having some difficulty understanding this. You make it sound like there's a fairly arbitrary number of IDs that the SMMU can handle. So how is the mapping to devices defined? If you say encourage that does make it sound like the assignment of IDs is purely defined by some mechanism in software rather than in hardware. Or they are more or less randomly picked by someone. If that's the case, is that not something that should be dynamically allocated by the kernel rather than put into the device tree?

The set of StreamIDs that can be generated by a master is fixed in the hardware. The SMMU can then be programmed to map these incoming IDs onto a context ID (or a set of context IDs), which are the IDs used internally by the SMMU to find the page tables etc.

The StreamID -> ContextID mapping is dynamic and controlled by software. The Master -> StreamIDs mapping is fixed in the hardware.

-----Original Message-----
From: iommu-bounces at lists.linux-foundation.org [mailto:iommu- bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
Sent: Wednesday, June 04, 2014 10:12 PM
To: Thierry Reding
Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung- soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 6 lines · Thierry Reding

Quoted 5 lines · Dave Martin

In the strictest sense, no.

But for a large set of sane configurations, this probably works.

Small sets of randomly-assigned IDs can just be enumerated one by

one.

Quoted 4 lines

Quoted 3 lines

We wouldn't be able to describe folding and bit shuffling, but we probably don't want to encourage that anyway.

Quoted 8 lines

I'm having some difficulty understanding this. You make it sound like there's a fairly arbitrary number of IDs that the SMMU can handle. So how is the mapping to devices defined? If you say encourage that does make it sound like the assignment of IDs is purely defined by some mechanism in software rather than in hardware. Or they are more or less randomly picked by someone. If that's the case, is that not something that should be dynamically allocated by the kernel rather than put into the device tree?

The set of StreamIDs that can be generated by a master is fixed in the hardware. The SMMU can then be programmed to map these incoming IDs onto a context ID (or a set of context IDs), which are the IDs used internally by the SMMU to find the page tables etc.

The StreamID -> ContextID mapping is dynamic and controlled by software. The Master -> StreamIDs mapping is fixed in the hardware.

Quoted 7 lines

The set of StreamIDs that can be generated by a master is fixed in the hardware. The SMMU can then be programmed to map these incoming IDs onto a context ID (or a set of context IDs), which are the IDs used internally by the SMMU to find the page tables etc.

The StreamID -> ContextID mapping is dynamic and controlled by software. The Master -> StreamIDs mapping is fixed in the hardware.

The Master -> StreamIDs mapping may not always be fixed in the hardware. At, least in our case we plan to program these via software. PCI devices is one place where this mapping would have to be dynamic (based on the topology i.e. if the devices are connected to a bridge etc). Also, we have a hot plug device architecture where the stream ID is software programmable.

Other than that, based on the isolation requirements (iommu domain) software programmability offers greater flexibility.

-----Original Message-----
From: Will Deacon [mailto:will.deacon at arm.com]
Sent: Monday, June 16, 2014 10:28 AM
To: Sethi Varun-B16395
Cc: Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux- samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org; Yoder Stuart-B08248
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Hi Varun,

Quoted 1 line · Varun Sethi

Quoted 1 line

The set of StreamIDs that can be generated by a master is fixed in

the

Quoted 1 line

Quoted 1 line

hardware. The SMMU can then be programmed to map these incoming IDs

onto

Quoted 1 line

Quoted 1 line

a context ID (or a set of context IDs), which are the IDs used

internally

Quoted 3 lines

Quoted 3 lines

by the SMMU to find the page tables etc.

The StreamID -> ContextID mapping is dynamic and controlled by

software.

Quoted 2 lines

Quoted 1 line

The Master -> StreamIDs mapping is fixed in the hardware.

The Master -> StreamIDs mapping may not always be fixed in the

hardware.

Quoted 1 line

At, least in our case we plan to program these via software. PCI

devices

Quoted 2 lines

is one place where this mapping would have to be dynamic (based on the topology i.e. if the devices are connected to a bridge etc). Also, we

have

Quoted 5 lines

a hot plug device architecture where the stream ID is software programmable.

Other than that, based on the isolation requirements (iommu domain) software programmability offers greater flexibility.

For the sake of consistency, I'd really like to treat the master -> streamIDs mapping as fixed. If that means setting it once during boot in your case, then so be it (ideally in the firmware). That way, we just describe the fixed mapping to the kernel and the driver doesn't have to worry about changing the mapping, especially given that that's likely to be
highly error-prone once the SMMU is in us.

Do you have use-cases where you really need to change these mappings dynamically?

Quoted 2 lines

Do you have use-cases where you really need to change these mappings dynamically?

Yes. In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus. We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
multiple TBUs around. However, this means that the stream ID namespace
is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many more masters than that.

So we look at stream IDs as really corresponding to an 'isolation context' and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables.

So, perhaps by default some/most SMMU masters may have a default stream ID of 0x0 that is used by the host...and that could be represented statically in the device tree.

But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices
are added to it. All the devices in a domain will share
the same stream ID.

So whatever we do, let's please have an architecture flexible enough to allow for this.

Quoted 3 lines · Stuart Yoder

Quoted 2 lines

Do you have use-cases where you really need to change these mappings dynamically?

Quoted 3 lines

Yes. In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus. We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping.

Quoted 9 lines

Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and sprinkling
multiple TBUs around. However, this means that the stream ID namespace
is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many more masters than that.

Quoted 5 lines

So we look at stream IDs as really corresponding to an 'isolation context' and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables.

Ok, this is more compelling.

-----Original Message-----
From: Will Deacon [mailto:will.deacon at arm.com]
Sent: Monday, June 16, 2014 12:04 PM
To: Yoder Stuart-B08248
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Hi Stuart,

Quoted 6 lines · Stuart Yoder

Quoted 2 lines

Do you have use-cases where you really need to change these mappings dynamically?

Yes. In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus. We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping.

Quoted 4 lines

Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and

sprinkling

Quoted 1 line

multiple TBUs around. However, this means that the stream ID

namespace

Quoted 2 lines

is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many

more

Quoted 3 lines

masters than that.

So we look at stream IDs as really corresponding to an 'isolation

context'

Quoted 4 lines

and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables.

Ok, this is more compelling.

Quoted 1 line

So, perhaps by default some/most SMMU masters may have a default stream

ID

Quoted 8 lines

of 0x0 that is used by the host...and that could be represented statically in the device tree.

But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices
are added to it. All the devices in a domain will share
the same stream ID.

Quoted 2 lines

So whatever we do, let's please have an architecture flexible enough to allow for this.

What is the software interface to the logic that assigns the StreamIDs? Is
it part of the SMMU, or a separate device (or set of devices)?

-----Original Message-----
From: Yoder Stuart-B08248
Sent: Tuesday, June 17, 2014 12:24 AM
To: Will Deacon
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 61 lines

-----Original Message-----
From: Will Deacon [mailto:will.deacon at arm.com]
Sent: Monday, June 16, 2014 12:04 PM
To: Yoder Stuart-B08248
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Hi Stuart,

Quoted 6 lines · Stuart Yoder

Quoted 2 lines

Do you have use-cases where you really need to change these mappings dynamically?

Yes. In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus. We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping.

Quoted 5 lines

Another thing to consider-- starting with SMMUv2, as you know, there is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and

sprinkling

Quoted 1 line

multiple TBUs around. However, this means that the stream ID

namespace

Quoted 3 lines

is now global and can be pretty limited. In the SMMU implementation we have there are only 64 stream ID total for our Soc. But we have many

more

Quoted 3 lines

masters than that.

So we look at stream IDs as really corresponding to an 'isolation

context'

Quoted 4 lines

and not to a bus master. An isolation context is the domain you are trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables.

Ok, this is more compelling.

Quoted 2 lines

So, perhaps by default some/most SMMU masters may have a default stream

ID

Quoted 8 lines

of 0x0 that is used by the host...and that could be represented statically in the device tree.

But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices
are added to it. All the devices in a domain will share
the same stream ID.

Quoted 6 lines

Quoted 2 lines

So whatever we do, let's please have an architecture flexible enough to allow for this.

What is the software interface to the logic that assigns the StreamIDs? Is
it part of the SMMU, or a separate device (or set of devices)?

For us at the hardware level there are a few different ways that the streamIDs can be set. It is not part of the SMMU. In the cases where there is simply
1 device to 1 streamID (e.g. USB controller) there is an SoC register where
you just program the stream ID. In the case of PCI, our PCI controller
has a RequesterID-to-streamID table that you set via some PCI controller registers.

The way we generally thought it would work was something like this:
-u-boot/bootloader makes any static streamID allocation if needed,
sets a default streamID (e.g. 0x0) in device and expresses that in the device tree
-device tree would express relationship between devices
(including bus controllers) and the SMMU through mmu-masters property
-u-boot would express the range of unused (or used) streamIDs via a
new
device tree property so the kernel SMMU driver knows what streamIDs
are
free
-in the SMMU driver a different vendor specific 'add_device' callback
could be used to handle our special cases where we need to set/change the stream ID for devices added to a domain

Quoted 8 lines

The way we generally thought it would work was something like this:
-u-boot/bootloader makes any static streamID allocation if needed,
sets a default streamID (e.g. 0x0) in device and expresses that in the device tree
-device tree would express relationship between devices
(including bus controllers) and the SMMU through mmu-masters property

Quoted 8 lines

-u-boot would express the range of unused (or used) streamIDs via a
new
device tree property so the kernel SMMU driver knows what streamIDs
are
free
-in the SMMU driver a different vendor specific 'add_device' callback
could be used to handle our special cases where we need to set/change the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

-----Original Message-----
From: iommu-bounces at lists.linux-foundation.org [mailto:iommu- bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
Sent: Tuesday, June 17, 2014 4:13 PM
To: Sethi Varun-B16395
Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung- soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 6 lines · Varun Sethi

Quoted 6 lines

The way we generally thought it would work was something like this:
-u-boot/bootloader makes any static streamID allocation if needed,
sets a default streamID (e.g. 0x0) in device and expresses that in the device tree
-device tree would express relationship between devices

Quoted 8 lines

Quoted 8 lines

(including bus controllers) and the SMMU through mmu-masters property
-u-boot would express the range of unused (or used) streamIDs via
a new
device tree property so the kernel SMMU driver knows what
streamIDs are
free
-in the SMMU driver a different vendor specific 'add_device'

callback

Quoted 1 line

Quoted 1 line

could be used to handle our special cases where we need to

set/change

Quoted 7 lines

Quoted 1 line

the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

I think for simple masters (i.e. those that have all their StreamIDs under control of one driver), then setting something during attach (or add?) based on the DT could work pretty well. The other case is when we have masters behind a bridge (such as a PCI RC). In this case, it might actually be better to ask the bridge for the IDs and let it sort out the allocation itself. That would also move the RequesterID -> StreamID mapping out of the SMMU code.

What do you think?

-----Original Message-----
From: Sethi Varun-B16395
Sent: Tuesday, June 17, 2014 6:22 AM
To: Will Deacon
Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung- soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 9 lines

-----Original Message-----
From: iommu-bounces at lists.linux-foundation.org [mailto:iommu- bounces at lists.linux-foundation.org] On Behalf Of Will Deacon
Sent: Tuesday, June 17, 2014 4:13 PM
To: Sethi Varun-B16395
Cc: Mark Rutland; devicetree at vger.kernel.org; linux-samsung- soc at vger.kernel.org; Arnd Bergmann; Pawel Moll; Ian Campbell; Grant Grundler; Stephen Warren; Yoder Stuart-B08248; Rob Herring; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Thierry Reding;

Kumar

Quoted 1 line

Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-

arm-

Quoted 8 lines

kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 3 lines · Varun Sethi

Quoted 3 lines

The way we generally thought it would work was something like this:
-u-boot/bootloader makes any static streamID allocation if

needed,

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

sets a default streamID (e.g. 0x0) in device and expresses that in the device tree
-device tree would express relationship between devices
(including bus controllers) and the SMMU through mmu-masters property
-u-boot would express the range of unused (or used) streamIDs

via

Quoted 16 lines

Quoted 5 lines

Quoted 5 lines

a new
device tree property so the kernel SMMU driver knows what
streamIDs are
free
-in the SMMU driver a different vendor specific 'add_device'

callback

Quoted 1 line

Quoted 1 line

could be used to handle our special cases where we need to

set/change

Quoted 7 lines

Quoted 1 line

the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

Quoted 5 lines

I think for simple masters (i.e. those that have all their StreamIDs under control of one driver), then setting something during attach (or add?) based on the DT could work pretty well. The other case is when we have masters behind a bridge (such as a PCI RC). In this case, it might actually be better to ask the bridge for the IDs and let it sort out

the

Quoted 3 lines

allocation itself. That would also move the RequesterID -> StreamID mapping out of the SMMU code.

Quoted 1 line

What do you think?

The PCI bus iommu group creation code would be part of the SMMU driver (it is handled by other IOMMU drivers as well). My understanding is that there would be one is to one correspondence between the requestor ID and the iommu group. May be, we can have an API provided by the PCI bridge (architecture specific) for setting the stream ID.

Quoted 9 lines

Quoted 5 lines

I think for simple masters (i.e. those that have all their StreamIDs under control of one driver), then setting something during attach (or add?) based on the DT could work pretty well. The other case is when we have masters behind a bridge (such as a PCI RC). In this case, it might actually be better to ask the bridge for the IDs and let it sort out

the

Quoted 3 lines

allocation itself. That would also move the RequesterID -> StreamID mapping out of the SMMU code.

Quoted 6 lines

Quoted 1 line

What do you think?

The PCI bus iommu group creation code would be part of the SMMU driver (it is handled by other IOMMU drivers as well). My understanding is that there would be one is to one correspondence between the requestor ID and the iommu group. May be, we can have an API provided by the PCI bridge (architecture specific) for setting the stream ID.

I think Will is suggesting something along those lines-- I think it's a question of where the streamID allocation happens. You could either do something like the following in the SMMU driver when attaching a PCI device:

    id = alloc_stream_id(...);
    pci_set_streamid(pci-dev, id);

    or

    id = pci_get_streamid(pci-dev);

...i.e the PCI RC could allocate (from some TBD
allocator) and set the stream ID itself.

Not sure how big a deal it is to extend PCI RC interfaces for something like that.

-----Original Message-----
From: Sethi Varun-B16395
Sent: Tuesday, June 17, 2014 5:27 AM
To: Yoder Stuart-B08248; Will Deacon
Cc: Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux- samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-
tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 7 lines

-----Original Message-----
From: Yoder Stuart-B08248
Sent: Tuesday, June 17, 2014 12:24 AM
To: Will Deacon
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;

linux-

Quoted 2 lines

kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-

arm-

Quoted 14 lines

kernel at lists.infradead.org
Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 8 lines

-----Original Message-----
From: Will Deacon [mailto:will.deacon at arm.com]
Sent: Monday, June 16, 2014 12:04 PM
To: Yoder Stuart-B08248
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland; devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob

Herring;

Quoted 5 lines

Quoted 5 lines

Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm- kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree bindings

Quoted 7 lines

Quoted 7 lines

Hi Stuart,

Quoted 4 lines · Stuart Yoder

Quoted 2 lines

Do you have use-cases where you really need to change these mappings dynamically?

Yes. In the case of a PCI bus-- you may not know in advance how

many

Quoted 1 line

Quoted 1 line

Quoted 1 line

PCI devices there are until you probe the bus. We have another

FSL

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic transformation from RequesterID to StreamID which corresponds to a fixed mapping.

Quoted 1 line

Another thing to consider-- starting with SMMUv2, as you know,

there

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

is a new distributed architecture with multiple TBUs and a centralized TCU that walks the SMMU page tables. So instead of sprinkling multiple SMMUs all over an SoC you now have the option a 1 central TCU and

sprinkling

Quoted 1 line

multiple TBUs around. However, this means that the stream ID

namespace

Quoted 1 line

is now global and can be pretty limited. In the SMMU

implementation

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

we have there are only 64 stream ID total for our Soc. But we have many

more

Quoted 3 lines

masters than that.

So we look at stream IDs as really corresponding to an 'isolation

context'

Quoted 1 line

and not to a bus master. An isolation context is the domain you

are

Quoted 12 lines

Quoted 12 lines

Quoted 3 lines

trying to isolate with the SMMU. Devices that all belong to the same 'isolation context' can share the same stream ID, since they share the same domain and page tables.

Ok, this is more compelling.

Quoted 2 lines

So, perhaps by default some/most SMMU masters may have a default stream

ID

Quoted 3 lines

of 0x0 that is used by the host...and that could be represented statically in the device tree.

Quoted 6 lines

Quoted 6 lines

Quoted 6 lines

But, we absolutely will need to dynamically set new stream IDs into masters when a new IOMMU 'domain' is created and devices
are added to it. All the devices in a domain will share
the same stream ID.

So whatever we do, let's please have an architecture flexible

enough

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

to allow for this.

What is the software interface to the logic that assigns the

StreamIDs?

Quoted 9 lines

Quoted 2 lines

Is
it part of the SMMU, or a separate device (or set of devices)?

For us at the hardware level there are a few different ways that the streamIDs can be set. It is not part of the SMMU. In the cases where there is simply
1 device to 1 streamID (e.g. USB controller) there is an SoC register where
you just program the stream ID. In the case of PCI, our PCI

controller

Quoted 1 line

has a RequesterID-to-streamID table that you set via some PCI

controller

Quoted 16 lines

registers.

The way we generally thought it would work was something like this:
-u-boot/bootloader makes any static streamID allocation if needed,
sets a default streamID (e.g. 0x0) in device and expresses that in the device tree
-device tree would express relationship between devices
(including bus controllers) and the SMMU through mmu-masters property
-u-boot would express the range of unused (or used) streamIDs via a
new
device tree property so the kernel SMMU driver knows what streamIDs
are
free
-in the SMMU driver a different vendor specific 'add_device'

callback

Quoted 1 line

could be used to handle our special cases where we need to

set/change

Quoted 1 line

the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

Quoted 25 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Quoted 8 lines · Thierry Reding

From: Thierry Reding <[email protected]>

Documentation/devicetree/bindings/iommu/iommu.txt new file +7 -0

+IOMMU master node:
+==================
+
+Devices that access memory through an IOMMU are called masters. A device can
+have multiple master interfaces (to one or more IOMMU devices).
+
+Required properties:

Documentation/devicetree/bindings/iommu/iommu.txt line 76

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +10 -0

+--------------------
+- iommus: A list of phandle and IOMMU specifier pairs that describe the IOMMU
+  master interfaces of the device. One entry in the list describes one master
+  interface of the device.
+
+When an "iommus" property is specified in a device tree node, the IOMMU will
+be used for address translation. If a "dma-ranges" property exists in the
+device's parent node it will be ignored. An exception to this rule is if the
+referenced IOMMU is disabled, in which case the "dma-ranges" property of the
+parent shall take effect.

Just thinking out loud, could you have dma-ranges in the iommu node for the case when the iommu is enabled rather than putting the DMA window information into the iommus property?

Quoted 1 line

This would probably mean that you need both #iommu-cells and #address-cells.

The reason for doing like this was that you may need a different window for each device, while there can only be one dma-ranges property in an iommu node.

Quoted 10 lines

Documentation/devicetree/bindings/iommu/iommu.txt line 81

Quoted 5 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +5 -0

+
+Optional properties:
+--------------------
+- iommu-names: A list of names identifying each entry in the "iommus"
+  property.

Do we really need a name here? I would not expect that you have clearly documented names here from the datasheet like you would for interrupts or clocks, so you'd just be making up names. Sorry, but I'm not a fan of names properties in general.

Good point, this was really overdesign by modeling it after other subsystems that can have a use for names.

Quoted 11 lines

Quoted 11 lines

Documentation/devicetree/bindings/iommu/iommu.txt new file +11 -0

+Multiple-master IOMMU:
+----------------------
+
+       iommu {
+               /* the specifier represents the ID of the master */
+               #address-cells = <1>;
+               #size-cells = <0>;
+       };
+
+       master {
+               /* device has master ID 42 in the IOMMU */

Quoted 8 lines

Review

Quoted 2 lines

(quoted diff) +2 -0

+               iommus = <&/iommu 42>;
+       };

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

I don't understand the problem. If you have stream IDs 0 through 7, you would have

	master at a {
		...
		iommus = <&smmu 0>;
	};

	master at b {
		...
		iommus = <&smmu 1;
	};

	...

	master at 12 {
		...
		iommus = <&smmu 7;
	};

and you don't need a window at all. Why would you need a mask of some sort?

Quoted 27 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

I don't understand the problem. If you have stream IDs 0 through 7, you would have

	master at a {
		...
		iommus = <&smmu 0>;
	};

	master at b {
		...
		iommus = <&smmu 1;
	};

	...

	master at 12 {
		...
		iommus = <&smmu 7;
	};

Quoted 2 lines

and you don't need a window at all. Why would you need a mask of some sort?

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Quoted 37 lines · Olav Haugan

Quoted 27 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 May 2014 08:16:05 Rob Herring wrote:

Presumably the ID would be the streamID on ARM's SMMU. How would a master with 8 streamIDs be described? This is what Calxeda midway has for SATA and I would expect that to be somewhat common. Either you need some ID masking or you'll have lots of duplication when you have windows.

I don't understand the problem. If you have stream IDs 0 through 7, you would have

	master at a {
		...
		iommus = <&smmu 0>;
	};

	master at b {
		...
		iommus = <&smmu 1;
	};

	...

	master at 12 {
		...
		iommus = <&smmu 7;
	};

Quoted 2 lines

and you don't need a window at all. Why would you need a mask of some sort?

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

Quoted 19 lines · Will Deacon

Quoted 6 lines · Olav Haugan

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Quoted 2 lines

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 23 lines · Olav Haugan

Quoted 19 lines · Will Deacon

Quoted 6 lines · Olav Haugan

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

   iommu {
           #address-cells = <2>;
           #size-cells = <0>;
   };

   master at a {
           ...
           iommus = <&iommu StreamID0 MASK0>,
                    <&iommu StreamID1 MASK1>,
                    <&iommu StreamID2 MASK2>;
   };

Quoted 2 lines

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

Quoted 7 lines

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here. I'd be glad to see them revived.

Quoted 9 lines · On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:

Quoted 7 lines · Olav Haugan

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 8 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here. I'd be glad to see them revived.

Note that there are areas where we have in the past decided that dynamic configuration is just too hard for the kernel to do and that we're better off putting the configuration into DT. Pinctrl and clocks are at least partially in that category.

It's always best if you can get the kernel to do things in the ideal way where that is possible, but getting there may be just not worth it.

I have no idea where it should be for SMMU, but it's something to consider:
if you can take reasonable shortcuts by reading parts of the configuration from DT, you may just as well do that.

Quoted 27 lines · Arnd Bergmann

Quoted 9 lines · On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:

Quoted 7 lines · Olav Haugan

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 8 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here. I'd be glad to see them revived.

Note that there are areas where we have in the past decided that dynamic configuration is just too hard for the kernel to do and that we're better off putting the configuration into DT. Pinctrl and clocks are at least partially in that category.

It's always best if you can get the kernel to do things in the ideal way where that is possible, but getting there may be just not worth it.

Quoted 3 lines

I have no idea where it should be for SMMU, but it's something to consider:
if you can take reasonable shortcuts by reading parts of the configuration from DT, you may just as well do that.

I treat this in the same manner as the topology bindings we discussed previously; we should do a best-effort to configure things dynamically and solve corner-cases and quirks as special cases.

Quoted 32 lines · On Wednesday 25 June 2014 10:17:02 Will Deacon wrote:

Quoted 30 lines · Arnd Bergmann

Quoted 9 lines · On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:

Quoted 7 lines · Olav Haugan

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 8 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here. I'd be glad to see them revived.

Note that there are areas where we have in the past decided that dynamic configuration is just too hard for the kernel to do and that we're better off putting the configuration into DT. Pinctrl and clocks are at least partially in that category.

It's always best if you can get the kernel to do things in the ideal way where that is possible, but getting there may be just not worth it.

I have no idea where it should be for SMMU, but it's something to consider:
if you can take reasonable shortcuts by reading parts of the configuration from DT, you may just as well do that.

Quoted 3 lines

I treat this in the same manner as the topology bindings we discussed previously; we should do a best-effort to configure things dynamically and solve corner-cases and quirks as special cases.

I think the situation is a bit different here: It's less about the corner cases for the SMMU, but about the question whether it makes more sense to have the kernel figure out the settings, or have them come from DT all the time.

As I said, I can't tell which approach is best here, but it sounds to me we should either do dynamic configuration and get it right, or hardcode the configuration it all the time if we can't.

Quoted 33 lines · Arnd Bergmann

Quoted 32 lines · On Wednesday 25 June 2014 10:17:02 Will Deacon wrote:

Quoted 30 lines · Arnd Bergmann

Quoted 9 lines · On Tuesday 24 June 2014 19:11:50 Will Deacon wrote:

Quoted 7 lines · Olav Haugan

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 8 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

There have been patches previously posted by Andreas Herrmann helping here. I'd be glad to see them revived.

Note that there are areas where we have in the past decided that dynamic configuration is just too hard for the kernel to do and that we're better off putting the configuration into DT. Pinctrl and clocks are at least partially in that category.

It's always best if you can get the kernel to do things in the ideal way where that is possible, but getting there may be just not worth it.

I have no idea where it should be for SMMU, but it's something to consider:
if you can take reasonable shortcuts by reading parts of the configuration from DT, you may just as well do that.

Quoted 8 lines

Quoted 3 lines

I treat this in the same manner as the topology bindings we discussed previously; we should do a best-effort to configure things dynamically and solve corner-cases and quirks as special cases.

I think the situation is a bit different here: It's less about the corner cases for the SMMU, but about the question whether it makes more sense to have the kernel figure out the settings, or have them come from DT all the time.

But, as far as I can tell, this setting is basically `which bits are constant among this set of IDs'.

Quoted 3 lines

As I said, I can't tell which approach is best here, but it sounds to me we should either do dynamic configuration and get it right, or hardcode the configuration it all the time if we can't.

I disagree. If you have `sensible' StreamID allocations, doing this dynamically should be straight-forward and gives the driver more flexibility (e.g. we then have the option of combining SMR entries for different masters if they are in the same domain). The dynamic approach also lends itself to sanity-checking (it is IMPLEMENTATION DEFINED whether the SMMU detects SMR aliases) and helps with virtualisation (forcing QEMU to generate these masks in a device-tree for a guest using a virtual SMMU interface is very painful).

Quoted 13 lines · On Wednesday 25 June 2014 10:38:25 Will Deacon wrote:

Quoted 4 lines · Arnd Bergmann

I think the situation is a bit different here: It's less about the corner cases for the SMMU, but about the question whether it makes more sense to have the kernel figure out the settings, or have them come from DT all the time.

But, as far as I can tell, this setting is basically `which bits are constant among this set of IDs'.

Quoted 3 lines

As I said, I can't tell which approach is best here, but it sounds to me we should either do dynamic configuration and get it right, or hardcode the configuration it all the time if we can't.

Quoted 7 lines

I disagree. If you have `sensible' StreamID allocations, doing this dynamically should be straight-forward and gives the driver more flexibility (e.g. we then have the option of combining SMR entries for different masters if they are in the same domain). The dynamic approach also lends itself to sanity-checking (it is IMPLEMENTATION DEFINED whether the SMMU detects SMR aliases) and helps with virtualisation (forcing QEMU to generate these masks in a device-tree for a guest using a virtual SMMU interface is very painful).

In which case do you think hardcoding is needed then? I'm still confused why you think we need both, as your arguments seem to all be in favor of dynamic configuration.

So far, I've been avoiding the hardcoding. However, you could potentially build a system with a small number of SMRs (compared to the number of StreamIDs) and allocate the StreamIDs in such a way that I think the dynamic configuration would be NP complete if we require an optimal SMR allocation.

However:

(1) I don't know of a system where this is the case
(2) Not much work has been done on improving the dynamic allocator yet

which is why I'm still favouring dynamic configuration in the driver.

The other thing I forgot to mention earlier is that we need to support device hotplug in the future, so some level of dynamic configuration will always be required.

Quoted 7 lines · On Wednesday 25 June 2014 10:57:36 Will Deacon wrote:

So far, I've been avoiding the hardcoding. However, you could potentially build a system with a small number of SMRs (compared to the number of StreamIDs) and allocate the StreamIDs in such a way that I think the dynamic configuration would be NP complete if we require an optimal SMR allocation.

However:

Quoted 8 lines

(1) I don't know of a system where this is the case
(2) Not much work has been done on improving the dynamic allocator yet

which is why I'm still favouring dynamic configuration in the driver.

The other thing I forgot to mention earlier is that we need to support device hotplug in the future, so some level of dynamic configuration will always be required.

Ok, got it. So we just hope that we can make dynamic configuration work all the time, but if it all fails, then we come up with a hardcoded configuration method.

I guess this could be done similarly to how we handle clocks on a lot of systems: generally these are dynamic, but you have the option to provide hints in the clock controller node about how you expect things to be configured.

For the SMMU that could mean that (if we get into the situation you describe), we add optional properties to the SMMU node itself describing how we expect the SMRs to be used.

Quoted 23 lines · Olav Haugan

Quoted 19 lines · Will Deacon

Quoted 6 lines · Olav Haugan

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Quoted 2 lines

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

Quoted 7 lines

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

Quoted 32 lines · Will Deacon

Quoted 23 lines · Olav Haugan

Quoted 19 lines · Will Deacon

Quoted 6 lines · Olav Haugan

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Quoted 2 lines

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

Quoted 7 lines

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 5 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

I am talking about SMMU_SMRn[MASK] register bits. This is not something that can be dynamically detected at run-time. It is configuration at the same level as the actual StreamIDs.

Quoted 39 lines · Olav Haugan

Quoted 32 lines · Will Deacon

Quoted 23 lines · Olav Haugan

Quoted 19 lines · Will Deacon

Quoted 6 lines · Olav Haugan

We have multiple-master SMMUs and each master emits a variable number of StreamIDs. However, we have to apply a mask (the ARM SMMU spec allows for this) to the StreamIDs due to limited number of StreamID 2 Context Bank entries in the SMMU. If my understanding is correct we would represent this in the DT like this:

Quoted 11 lines

	iommu {
		#address-cells = <2>;
		#size-cells = <0>;
	};

	master at a {
		...
		iommus = <&iommu StreamID0 MASK0>,
			 <&iommu StreamID1 MASK1>,
			 <&iommu StreamID2 MASK2>;
	};

Quoted 2 lines

Stupid question, but why not simply describe the masked IDs? What use does the `raw' ID have to Linux?

Quoted 7 lines

We do describe the masked StreamID (SID) but we need to specify the mask that the SMMU should apply to the incoming SIDs, right?

We have a bus master that emits 43 unique SIDs. However, we have only 40 SMMU_SMRn registers in the SMMU. So we need to mask out some of the incoming SID bits so that the 43 SIDs can match one of 40 entries in the SMR.

Quoted 5 lines

Hmm, so you're talking about stream matching, right? That doesn't belong in the device-tree. I appreciate that the current driver does a terrible job at allocating the SMRs (it's bloody difficult!), but we should try to improve the dynamic behaviour instead of moving configuration of the SMMU out into device-tree, where it's inflexible at best.

Quoted 3 lines

I am talking about SMMU_SMRn[MASK] register bits. This is not something that can be dynamically detected at run-time. It is configuration at the same level as the actual StreamIDs.

Why can't it be dynamically detected? Whilst the StreamIDs are fixed in hardware (from the SMMU architecture perspective), the SMRs are completely programmable. Why doesn't something like Andreas's proposal work for you? The idea there was to find the constant bits among the StreamIDs for a master and create the mask accordingly.

Quoted 6 lines · Will Deacon

Why can't it be dynamically detected? Whilst the StreamIDs are fixed in hardware (from the SMMU architecture perspective), the SMRs are completely programmable. Why doesn't something like Andreas's proposal work for you? The idea there was to find the constant bits among the StreamIDs for a master and create the mask accordingly.

Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the following stream IDs coming from the masters:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x30

To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.

However, I could also have an IOMMU with 2 masters and 9 SMRn slots with the following stream IDs:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x29

Here I would program all SMRn and leave the mask to be 0 for all SMRn's. So how do I detect when to apply a mask or not?

I am not familiar with Andreas's proposal. Do you have a link?

Hi Olav,

Quoted 16 lines · Olav Haugan

Quoted 6 lines · Will Deacon

Why can't it be dynamically detected? Whilst the StreamIDs are fixed in hardware (from the SMMU architecture perspective), the SMRs are completely programmable. Why doesn't something like Andreas's proposal work for you? The idea there was to find the constant bits among the StreamIDs for a master and create the mask accordingly.

Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the following stream IDs coming from the masters:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x30

To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.

Quoted 8 lines

However, I could also have an IOMMU with 2 masters and 9 SMRn slots with the following stream IDs:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x29

Here I would program all SMRn and leave the mask to be 0 for all SMRn's. So how do I detect when to apply a mask or not?

You would aim to use the smallest number of SMRs per master possible. You could probably use:

Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07

            SMR[1].id == 0x28, SMR[1].mask = 0x00

Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00

Quoted 1 line

I am not familiar with Andreas's proposal. Do you have a link?

http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Quoted 19 lines · Will Deacon

Quoted 9 lines · Olav Haugan

Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the following stream IDs coming from the masters:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x30

To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.

Quoted 8 lines

However, I could also have an IOMMU with 2 masters and 9 SMRn slots with the following stream IDs:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x29

Here I would program all SMRn and leave the mask to be 0 for all SMRn's. So how do I detect when to apply a mask or not?

Quoted 7 lines

You would aim to use the smallest number of SMRs per master possible. You could probably use:

Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07

            SMR[1].id == 0x28, SMR[1].mask = 0x00

Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00

So how does an algorithm figure this out in both my examples? The algorithm would have to know about both (all) bus masters and their stream IDs for a specific SMMU. If the algorithm operates on the set of stream IDs for one bus master at a time the algorithm has no way of knowing which bits can be ignored since it doesn't know the value of the other stream IDs for the other bus masters and thus could potentially create a mask that could cause a stream ID to match in two different entries.

Quoted 3 lines

Quoted 1 line

I am not familiar with Andreas's proposal. Do you have a link?

http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Unless I am mistaken the algorithm works on one bus master at a time. I don't think that will work.

Quoted 28 lines · Olav Haugan

Quoted 19 lines · Will Deacon

Quoted 9 lines · Olav Haugan

Lets say I have an IOMMU with 2 masters and 2 SMRn slots with the following stream IDs coming from the masters:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x30

To make this work I would program SMR[0] with StreamID 0x20 and mask 0xF to ignore lower 4 bits. SMR[1] would just be StreamID 0x30 with mask 0x0.

Quoted 8 lines

However, I could also have an IOMMU with 2 masters and 9 SMRn slots with the following stream IDs:

Master 1: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Master 2: 0x29

Here I would program all SMRn and leave the mask to be 0 for all SMRn's. So how do I detect when to apply a mask or not?

Quoted 7 lines

You would aim to use the smallest number of SMRs per master possible. You could probably use:

Master 1: SMR[0].id == 0x20, SMR[0].mask = 0x07

            SMR[1].id == 0x28, SMR[1].mask = 0x00

Master 2: SMR[2].id == 0x29, SMR[2].mask = 0x00

Quoted 8 lines

So how does an algorithm figure this out in both my examples? The algorithm would have to know about both (all) bus masters and their stream IDs for a specific SMMU. If the algorithm operates on the set of stream IDs for one bus master at a time the algorithm has no way of knowing which bits can be ignored since it doesn't know the value of the other stream IDs for the other bus masters and thus could potentially create a mask that could cause a stream ID to match in two different entries.

Complete knowledge of the system topology (i.e. all bus masters) is a requirement for being able to configure the SMMU correctly if you want to guarantee that you don't have SMR aliasing issues.

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

I am not familiar with Andreas's proposal. Do you have a link?

http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Unless I am mistaken the algorithm works on one bus master at a time. I don't think that will work.

IIRC, it works for densely packed SIDs on the master, so it tries to build up power-of-2 sized groups for that master then mops up the rest with individual entries.

Quoted 10 lines · Will Deacon

Quoted 8 lines · Olav Haugan

So how does an algorithm figure this out in both my examples? The algorithm would have to know about both (all) bus masters and their stream IDs for a specific SMMU. If the algorithm operates on the set of stream IDs for one bus master at a time the algorithm has no way of knowing which bits can be ignored since it doesn't know the value of the other stream IDs for the other bus masters and thus could potentially create a mask that could cause a stream ID to match in two different entries.

Quoted 3 lines

Complete knowledge of the system topology (i.e. all bus masters) is a requirement for being able to configure the SMMU correctly if you want to guarantee that you don't have SMR aliasing issues.

So you agree that an algorithm needs to know about all the bus masters/stream IDs for a specific IOMMU before it can figure out the StreamID masks and how many SMRs can be allocated to a specific bus master? Andreas's algorithm does not know about the other bus masters/stream IDs. It operates on one bus master at a time.

Quoted 10 lines

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

I am not familiar with Andreas's proposal. Do you have a link?

http://marc.info/?l=linux-arm-kernel&m=139110598005846&w=2

Unless I am mistaken the algorithm works on one bus master at a time. I don't think that will work.

IIRC, it works for densely packed SIDs on the master, so it tries to build up power-of-2 sized groups for that master then mops up the rest with individual entries.

I ran the algorithm through a few trivial cases:

1)
Stream IDs: 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, 0x28 Number of SMRs: 9

In this case the algorithm decided to set mask to 0 for all entries using up 8 of the SMRs.

2) Same Stream IDs but only 2 SMRs.
The algorithm gave an error saying I did not have enough SMRs.