PCI: of: update endpoint ranges dynamically
v2 cover PCI: of: update endpoint ranges dynamically
A PCI endpoint bus is a devicetree construct that allows a PCI endpoint (function) to have sub-devices defined that are accessible in an SoC via the PCI endpoint's BARs. Such a bus is represented as a devicetree sub-node for a PCI function having the name "pci-ep-bus". There can be one or more pci-ep-bus nodes.
A PCI function with a pci-ep-bus devicetree node must also define "#address-cells", "#size-cells", and "ranges" properties, to specify how endpoint bus addresses are translated to the PCI parent bus.
An endpoint bus address has three cells; the first indicates which of the function's BARs the address is associated with, and the other two specify a 64-bit (2 cell) offset within the BAR's region.
BAR base addresses are determined dynamically by the PCI enumeration process, so generally it's not possible to include them in a static devicetree file. When this addressing scheme was introduced, this was not a problem because the devicetree content was generated dynamically--after booting--based on the information (including BAR addresses) available following PCI enumeration.
It is possible (and in some cases, necessary) to define the devicetree nodes that represent PCI devices ahead of time, in a statically-defined devicetree file. In order to support the PCI endpoint bus model in this case it is necessary to dynamically update the static devicetree so that the BAR base addresses assigned during enumeration are reflected in the endpoint's "ranges" property.
This series implements that dynamic update, leveraging the same code used to create the "ranges" property when PCI_DYNAMIC_OF_NODES is enabled. The first patch makes an argument to of_pci_get_addr_flags() optional. The second patch separates the code that dynamically builds the property value into a helper function, and the last arranges for a statically-defined devicetree node for a PCI endpoint to have its "ranges" property updated (if it includes a "pci-ep-bus" sub-node)..
-Alex
Note: this series is built upon these patches:
https://lore.kernel.org/lkml/[email protected]/
The entire series (based on v7.3-rc2 and including those prerequisites) is available here:
https://github.com/riscstar/linux/tree/outgoing/dynamic_ranges-v2
Between version 1 and version 2:
- Included the first patch (which was previously posted in a different series)
- Modified the last patch so the ranges property is updated only for PCI endpoints having at least one "pci-ep-bus" node
- Rebased on v7.3-rc2 (and the prerequisite series)
Version 1 is available here:
https://lore.kernel.org/lkml/[email protected]/
Alex Elder (3):
PCI: of: make a flags argument optional
PCI: of: introduce of_pci_build_prop_ranges()
PCI: of: introduce of_pci_update_endpoint_node_ranges()
drivers/pci/of.c | 89 ++++++++++++++++++++++--- drivers/pci/of_property.c | 137 +++++++++++++++++++++++++------------- drivers/pci/pci.h | 1 + 3 files changed, 173 insertions(+), 54 deletions(-)
base-commit: 857c3561ea9f2c3107b5c6d830d3b3face159cbc
Signature · 1 line
2.53.0
PATCH v2 1/3 PCI: of: make a flags argument optional
The address of a u32 object is passed to of_pci_get_addr_flags() so it can be filled with the computed flags value.
Allow a null pointer to be passed, so that the validity of the resource's flags can be checked without filling in the flags value.
---
v2: - Moved this patch to this series
drivers/pci/of_property.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
drivers/pci/of_property.c +5 -3
@@ -82,9 +82,11 @@ static int of_pci_get_addr_flags(const struct resource *res, u32 *flags)else return -EINVAL ; - *flags =FIELD_PREP (OF_PCI_ADDR_FIELD_SS , ss); -if (res->flags &IORESOURCE_PREFETCH ) - *flags |=OF_PCI_ADDR_FIELD_PREFETCH ; +if (flags) { + *flags =FIELD_PREP (OF_PCI_ADDR_FIELD_SS , ss); +if (res->flags &IORESOURCE_PREFETCH ) + *flags |=OF_PCI_ADDR_FIELD_PREFETCH ; + }return 0 ; }
Signature · 1 line
2.53.0
PATCH v2 2/3 PCI: of: introduce of_pci_build_prop_ranges()
In reply to Alex ElderMove the bulk of what happens inside of_pci_prop_ranges() into a helper function that builds up the value to be passed to of_changeset_add_prop_u32_array().
This helper will be used in the next commit to build the ranges property for a statically-defined PCI endpoint devicetree node when it has a "pci-ep-bus" sub-node.
--- drivers/pci/of_property.c | 129 +++++++++++++++++++++++++------------- 1 file changed, 86 insertions(+), 43 deletions(-)
drivers/pci/of_property.c +86 -43
@@ -102,57 +102,100 @@ static int of_pci_prop_bus_range(struct pci_dev *pdev, ARRAY_SIZE(bus_range)); } +/* +* Build a "ranges" property value that defines the mapping between +* child and parent PCI address space for each of the given PCI +* device's resources. +* +* Returns a dynamically allocated array of u32 devicetree cells, or +* a null pointer if allocation fails. The cell array is built as a +* of_pci_prop_ranges structure consisting of 8 32-bit cells in host +* byte order. It is suitable for use as the value of a PCI device +* node "ranges" property passed to of_changeset_add_prop_u32_array(). +* The total number of cells in the array is returned in *count. +* +* Caller is responsible for ensuring the returned pointer gets freed. +*/ +static u32 *of_pci_build_prop_ranges (struct pci_dev *pdev,u32 *count) +{ +bool bridge_device =pci_is_bridge (pdev); +struct of_pci_range_entry *entries; +struct of_pci_range_entry *ep; +u32 resource_count =0 ; +struct resource *res; +u32 first; +u32 num; +u32 i; + +if (bridge_device) { + first =PCI_BRIDGE_RESOURCES ; + num =PCI_BRIDGE_RESOURCE_NUM ; + }else { + first =PCI_STD_RESOURCES ; + num =PCI_STD_NUM_BARS ; + } + +/* First count how many resources will get a range property */ + res = &pdev->resource[first]; +for (i =0 ; i < num; i++, res++) +if (resource_size (res) && !of_pci_get_addr_flags (res,NULL )) + resource_count++; + + entries =kzalloc_objs (*entries, resource_count); +if (!entries) +return NULL ; + + ep = entries; + res = &pdev->resource[first]; +for (i =0 ; i < num; i++, res++) { +u64 size =resource_size (res); +u32 flags; + +if (!size ||of_pci_get_addr_flags (res, &flags)) +continue ; + +/* Record the size in the range entry */ + ep->size[0 ] =upper_32_bits (size); + ep->size[1 ] =lower_32_bits (size); + +/* Record the parent bus address from the resource */ +of_pci_set_address (pdev, ep->parent_addr, +pci_bus_address (pdev, first + i), + flags, false); + +/* +* For a bridge device, the child address matches the +* parent address (including its flags cell). For an +* endpoint device, the (flags) cell contains the BAR +* number, and the two address cells are zero. +*/ +if (bridge_device) +memcpy (ep->child_addr, ep->parent_addr, +sizeof (ep->child_addr)); +else + ep->child_addr[0 ] = i; + + ep++; + } + *count = resource_count *sizeof (*ep) /sizeof (u32); + +return (u32 *)entries; +} +static int of_pci_prop_ranges (struct pci_dev *pdev,struct of_changeset *ocs,struct device_node *np) {struct of_pci_range_entry *rp; -struct resource *res; -int i, j, ret; -u32 flags, num; -u64 val64; +u32 *value; +u32 count; +int ret; -if (pci_is_bridge (pdev)) { - num =PCI_BRIDGE_RESOURCE_NUM ; - res = &pdev->resource[PCI_BRIDGE_RESOURCES ]; - }else { - num =PCI_STD_NUM_BARS ; - res = &pdev->resource[PCI_STD_RESOURCES ]; - } - - rp =kzalloc_objs (*rp, num); -if (!rp) + value =of_pci_build_prop_ranges (pdev, &count); +if (!value)return -ENOMEM ; -for (i =0 , j =0 ; j < num; j++) { -if (!resource_size (&res[j])) -continue ; + ret =of_changeset_add_prop_u32_array (ocs, np,"ranges" , value, count); -if (of_pci_get_addr_flags (&res[j], &flags)) -continue ; - - val64 =pci_bus_address (pdev, &res[j] - pdev->resource); -of_pci_set_address (pdev, rp[i].parent_addr, val64, flags, - false); -if (pci_is_bridge (pdev)) { -memcpy (rp[i].child_addr, rp[i].parent_addr, -sizeof (rp[i].child_addr)); - }else { -/* -* For endpoint device, the lower 64-bits of child -* address is always zero. -*/ - rp[i].child_addr[0 ] = j; - } - - val64 =resource_size (&res[j]); - rp[i].size[0 ] =upper_32_bits (val64); - rp[i].size[1 ] =lower_32_bits (val64); - - i++; - } - - ret =of_changeset_add_prop_u32_array (ocs, np,"ranges" , (u32 *)rp, - i *sizeof (*rp) /sizeof (u32));kfree (rp);return ret;
Signature · 1 line
2.53.0
PATCH v2 3/3 PCI: of: introduce of_pci_update_endpoint_node_ranges()
In reply to Alex ElderCommit 407d1a51921e9 ("PCI: Create device tree node for bridge") introduced the PCI_DYNAMIC_OF_NODES Kconfig option, which creates a devicetree node for a PCI bridge as part of pci_bus_add_device(). Its successor commit ae9813db1dc5a ("PCI: Add quirks to generate device tree node for Xilinx Alveo U50") shows how to use a PCI final fixup quirk to also create a devicetree node for a non-bridge PCI device. In both cases, of_pci_make_dev_node() uses an OF changeset to dynamically create a node populated with appropriate properties and apply it to the live devicetree.
The dynamic devicetree node for a PCI device will include a "ranges" property, and a new type of 3-cell address is introduced for use within an endpoint. The endpoint's ranges property will contain a range entry for each of the endpoint's BARs. The "child address" portion of each range will use the BAR number in the "flags" (first) cell in the address. This allows addresses within the endpoint to be expressed relative to whatever address gets assigned to each BAR.
Unfortunately, if a PCI endpoint device had a devicetree node set up statically, its "ranges" property (if present) will be static, and it cannot contain the addresses assigned to the endpoint's BARs during enumeration.
This means that the "BAR number" based addressing scheme doesn't work for PCI endpoints whose devicetree nodes are created statically.
To remedy this, modify of_pci_make_dev_node() to dynamically create a "ranges" property just as is done when the endpoint has a null devicetree node pointer. The device node will be updated to add the new "ranges" property (or replace it if one exists).
This allows "BAR number" addresses to work correctly even when the endpoint's devicetree node is created statically.
---
v2: - Only update ranges property if there is a pci-ep-bus node
drivers/pci/of.c | 89 +++++++++++++++++++++++++++++++++++---- drivers/pci/of_property.c | 2 +- drivers/pci/pci.h | 1 + 3 files changed, 83 insertions(+), 9 deletions(-)
drivers/pci/of.c +81 -8
@@ -742,20 +742,93 @@ void of_pci_remove_node(struct pci_dev *pdev)of_node_put (np); } +/* Returns true if the ranges property was added or updated successfully */ +static bool of_pci_update_endpoint_node_ranges (struct pci_dev *pdev) +{ +struct device_node *np =pci_device_to_OF_node (pdev); +struct property *prop; +u32 *value; +u32 size; + + prop =kzalloc_obj (*prop); +if (!prop) +return false; + + value =of_pci_build_prop_ranges (pdev, &size); +if (!value) { +kfree (prop); +return false; + } + + prop->name ="ranges" ; + prop->length = size *sizeof (u32); + prop->value = value; + +/* The property value needs to be in big-endian byte order */ +while (size--) +cpu_to_be32s (value++); + +/* of_update_property() consumes the allocated property */ +of_update_property (np, prop); + +return true; +} + +/* +* Create a devicetree node for a PCI device. If the device is a bridge +* and it already has a devicetree node, there's nothing further to do. +* If it is a bridge without an existing devicetree node, one is created +* dynamically. +* +* This function can also be called (via PCI quirk) for a PCI endpoint +* (function) that implements a PCI endpoint bus. As with a PCI bridge, +* if the endpoint has no existing devicetree node, one is created +* dynamically. The node will include a ranges property that maps +* BAR-relative addresses in the child to the PCI address ranges +* assigned to the PCI endpoint BARs. +* +* If an endpoint already has a devicetree node, and it includes a +* "pci-ep-bus" sub-node, its ranges property must still be dynamically +* populated so that it can take into account the BAR ranges assigned +* during PCI enumeration. +*/ void of_pci_make_dev_node (struct pci_dev *pdev) { -struct device_node *ppnode, *np =NULL ; +struct device_node *np =pci_device_to_OF_node (pdev); +struct device *dev = &pdev->dev; +struct device_node *ppnode; +struct of_changeset *cset;const char *pci_type; -struct of_changeset *cset;const char *name;int ret; -/* -* If there is already a device tree node linked to this device, -* return immediately. -*/ -if (pci_device_to_OF_node (pdev)) +/* See if the PCI device already has a devicetree node */ +if (np) { +struct device_node *child; + +/* Nothing further needed for a bridge */ +if (pci_is_bridge (pdev)) +return ; + +/* +* We only update the ranges property if the endpoint's +* devicetree node includes a "pci-ep-bus" sub-node. +*/ + child =of_get_child_by_name (np,"pci-ep-bus" ); +if (!child) +return ; +of_node_put (child); + +/* +* Update the ranges property, defining an entry for each +* BAR, mapping BAR offsets to the PCI bus address based +* on the BAR's assigned range. +*/ +if (!of_pci_update_endpoint_node_ranges (pdev)) +dev_err (dev,"failed to update ranges property\n" ); +return ; + }/* Check if there is device tree node for parent device */ if (!pdev->bus->self)
@@ -794,7 +867,7 @@ void of_pci_make_dev_node(struct pci_dev *pdev) np->data = cset; - ret =device_add_of_node (&pdev->dev, np); + ret =device_add_of_node (dev, np);if (ret) goto out_revert_cset;
drivers/pci/of_property.c +1 -1
@@ -116,7 +116,7 @@ static int of_pci_prop_bus_range(struct pci_dev *pdev,* * Caller is responsible for ensuring the returned pointer gets freed. */ -static u32 *of_pci_build_prop_ranges (struct pci_dev *pdev,u32 *count) +u32 *of_pci_build_prop_ranges (struct pci_dev *pdev,u32 *count) {bool bridge_device =pci_is_bridge (pdev);struct of_pci_range_entry *entries;
drivers/pci/pci.h +1 -0
@@ -1318,6 +1318,7 @@ struct of_changeset;#ifdef CONFIG_PCI_DYNAMIC_OF_NODES void of_pci_make_dev_node (struct pci_dev *pdev);void of_pci_remove_node (struct pci_dev *pdev); +u32 *of_pci_build_prop_ranges (struct pci_dev *pdev,u32 *count);int of_pci_add_properties (struct pci_dev *pdev,struct of_changeset *ocs,struct device_node *np);void of_pci_make_host_bridge_node (struct pci_host_bridge *bridge);
Signature · 1 line
2.53.0
Quoted 5 lines · Alex Elder
A PCI endpoint bus is a devicetree construct that allows a PCI endpoint (function) to have sub-devices defined that are accessible in an SoC via the PCI endpoint's BARs. Such a bus is represented as a devicetree sub-node for a PCI function having the name "pci-ep-bus". There can be one or more pci-ep-bus nodes.
Does anyone have any questions about this?
The devicetree pci-ep-bus mechanism defines one or more buses under a PCI endpoint function node (either one per BAR, or one node representing more than one BAR). Being a bus, it depends on ranges properties to translate between the local (BAR) address space and the parent (PCI) address space.
If a devicetree node is predefined (as is the case for the Toshiba TC9564 SoC), it cannot correctly define an endpoint's ranges property, because the PCI address space is determined dynamically during the PCI enumeration process.
This series addresses that by replacing the PCI function's devicetree ranges property at runtime, to reflect each BAR's assigned addresses.
There exist other users of the pci-ep-bus, and in particular the LAN966x driver generates an entire devicetree node to represent the PCI endpoint function--including dynamically generating this ranges property.
This series intentionally reuses the code that generates the ranges property for the "fully dynamic" case used for LAN966x.
This series (or another doing something comparable) is required for the TC9564 SoC to function using pci-ep-bus.
-Alex
Quoted 65 lines
A PCI function with a pci-ep-bus devicetree node must also define "#address-cells", "#size-cells", and "ranges" properties, to specify how endpoint bus addresses are translated to the PCI parent bus.
An endpoint bus address has three cells; the first indicates which of the function's BARs the address is associated with, and the other two specify a 64-bit (2 cell) offset within the BAR's region.
BAR base addresses are determined dynamically by the PCI enumeration process, so generally it's not possible to include them in a static devicetree file. When this addressing scheme was introduced, this was not a problem because the devicetree content was generated dynamically--after booting--based on the information (including BAR addresses) available following PCI enumeration.
It is possible (and in some cases, necessary) to define the devicetree nodes that represent PCI devices ahead of time, in a statically-defined devicetree file. In order to support the PCI endpoint bus model in this case it is necessary to dynamically update the static devicetree so that the BAR base addresses assigned during enumeration are reflected in the endpoint's "ranges" property.
This series implements that dynamic update, leveraging the same code used to create the "ranges" property when
PCI_DYNAMIC_OF_NODESis enabled. The first patch makes an argument toof_pci_get_addr_flags()optional. The second patch separates the code that dynamically builds the property value into a helper function, and the last arranges for a statically-defined devicetree node for a PCI endpoint to have its "ranges" property updated (if it includes a "pci-ep-bus" sub-node)..-Alex
Note: this series is built upon these patches:
https://lore.kernel.org/lkml/[email protected]/The entire series (based on v7.3-rc2 and including those prerequisites) is available here:
https://github.com/riscstar/linux/tree/outgoing/dynamic_ranges-v2Between version 1 and version 2:
- Included the first patch (which was previously posted in a different series)
- Modified the last patch so the ranges property is updated only for PCI endpoints having at least one "pci-ep-bus" node
- Rebased on v7.3-rc2 (and the prerequisite series)Version 1 is available here:
https://lore.kernel.org/lkml/[email protected]/Alex Elder (3):
PCI: of: make a flags argument optional
PCI: of: introduceof_pci_build_prop_ranges()
PCI: of: introduceof_pci_update_endpoint_node_ranges()drivers/pci/of.c | 89 ++++++++++++++++++++++--- drivers/pci/of_property.c | 137 +++++++++++++++++++++++++------------- drivers/pci/pci.h | 1 + 3 files changed, 173 insertions(+), 54 deletions(-)base-commit:
857c3561ea9f2c3107b5c6d830d3b3face159cbc
Quoted 28 lines · Alex Elder
A PCI endpoint bus is a devicetree construct that allows a PCI endpoint (function) to have sub-devices defined that are accessible in an SoC via the PCI endpoint's BARs. Such a bus is represented as a devicetree sub-node for a PCI function having the name "pci-ep-bus". There can be one or more pci-ep-bus nodes.
A PCI function with a pci-ep-bus devicetree node must also define "#address-cells", "#size-cells", and "ranges" properties, to specify how endpoint bus addresses are translated to the PCI parent bus.
An endpoint bus address has three cells; the first indicates which of the function's BARs the address is associated with, and the other two specify a 64-bit (2 cell) offset within the BAR's region.
BAR base addresses are determined dynamically by the PCI enumeration process, so generally it's not possible to include them in a static devicetree file. When this addressing scheme was introduced, this was not a problem because the devicetree content was generated dynamically--after booting--based on the information (including BAR addresses) available following PCI enumeration.
It is possible (and in some cases, necessary) to define the devicetree nodes that represent PCI devices ahead of time, in a statically-defined devicetree file. In order to support the PCI endpoint bus model in this case it is necessary to dynamically update the static devicetree so that the BAR base addresses assigned during enumeration are reflected in the endpoint's "ranges" property.
Quoted 7 lines
This series implements that dynamic update, leveraging the same code used to create the "ranges" property when
PCI_DYNAMIC_OF_NODESis enabled. The first patch makes an argument toof_pci_get_addr_flags()optional. The second patch separates the code that dynamically builds the property value into a helper function, and the last arranges for a statically-defined devicetree node for a PCI endpoint to have its "ranges" property updated (if it includes a "pci-ep-bus" sub-node)..
After some offline discussion, I have decided to slightly change *when* the "ranges" property will be updated.
The Raspberry Pi RP1 device defines (statically) a ranges property within the pci-ep-bus it uses. It maps BAR 1 to a base address 0x40000000, size 0x00400000 range in its parent endpoint's PCI address space.
For the original proposal, the ranges property is replaced for any PCI endpoint devicetree node with a pci-ep-bus subnode.
Instead, I'd like to offer the option of defining a ranges property that does not get replaced. This way, the Raspberry Pi RP1 can continue to work as it does now.
An "empty" (no value) ranges property doesn't work for pci-ep-bus, because it will treat the upper cell of the parent base address as a PCI flags field; pci-ep-bus instead uses that cell as a BAR number.
So I will be posting a new version of this series that differs this way:
- In of_pci_make_dev_node(), if there is no existing device tree node: no change (create a complete node dynamically)
- If there is a preexisting devicetree node but the current device is a PCI bridge: no change (just return)
- If the device node in of_pci_make_dev_node() is not a bridge, but there is no "pci-ep-bus" child node: no change (just return)
- (New) Look up the ranges property in the devicetree node for the PCI endpoint device. If it is not "empty" (i.e., if it has an assigned value, as would the Raspberry Pi RP1): just return (do not change the ranges property value)
- OTHERWISE (empty or perhaps missing ranges property), call of_pci_update_endpoint_node_ranges() to create a new ranges property dynamically, which takes into account the assigned BAR addresses for the PCI endpoint.
-Alex
Quoted 34 lines
-Alex
Note: this series is built upon these patches:
https://lore.kernel.org/lkml/[email protected]/The entire series (based on v7.3-rc2 and including those prerequisites) is available here:
https://github.com/riscstar/linux/tree/outgoing/dynamic_ranges-v2Between version 1 and version 2:
- Included the first patch (which was previously posted in a different series)
- Modified the last patch so the ranges property is updated only for PCI endpoints having at least one "pci-ep-bus" node
- Rebased on v7.3-rc2 (and the prerequisite series)Version 1 is available here:
https://lore.kernel.org/lkml/[email protected]/Alex Elder (3):
PCI: of: make a flags argument optional
PCI: of: introduceof_pci_build_prop_ranges()
PCI: of: introduceof_pci_update_endpoint_node_ranges()drivers/pci/of.c | 89 ++++++++++++++++++++++--- drivers/pci/of_property.c | 137 +++++++++++++++++++++++++------------- drivers/pci/pci.h | 1 + 3 files changed, 173 insertions(+), 54 deletions(-)base-commit:
857c3561ea9f2c3107b5c6d830d3b3face159cbc