Skip to contentSkip to conversation

PCI devices passthrough on Arm

RFC PATCH In review Rahul Singh 7 participants 48 messages Jul 23, 2020 – Jul 28, 2020

lore
Share & tools
lore
#0 lore raw

cover PCI devices passthrough on Arm

Following up on the discussion on PCI devices passthrough support on Arm design proposal.Please feel free to give you feedback.

We are submitting the code that we developed to get the early feedback. PCI passthrough support on ARM is not fully implemented in this patch series for that reason we are not enabling the HAS_PCI and HAS_VPCI flags for ARM.

We will work on design document that we submitted for feedback on mailing list and we will submit the next design document to address all the comments.

This patch series is based on v1 of the design document that we submitted for review. Any comments in the design will be addressed in the later version of the design document and will subsequently implemented in the code in next patch series.

PCI passthrough support is divided into different patches:

Discovering PCI Host Bridge in XEN:
- PCI host bridge discovery in XEN and map the PCI ECAM configuration space to the XEN memory.

Discovering PCI devices:
- In order to support the PCI passthrough, XEN should be aware of the PCI devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then communicate to the XEN via hyper call to add the PCI devices in XEN.

Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config space access.

Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to discover the virtual PCI during guest boot.

This patch series does not implemented the following features. Following features will be implemented in the next version of the patch series.
- MSI support for interrupt.
- ACPI support for PCI host bridge discovery within XEN on ARM.
- SMMU modification to support PCI devices.
- Use already defined config option "pci=[]" in place of new "vpci=ecam" config option to create VPCI bus.
- Map the assigned device PCI BAR values and interrupt to the guest when device is assigned by xl during domain creation.Currently we are using "iomem=[]" config options to map the value to the guest.

Rahul Singh (4):
arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM. xen/arm: Discovering PCI devices and add the PCI devices in XEN. xen/arm: Enable the existing x86 virtual PCI support for ARM. arm/libxl: Emulated PCI device tree node in libxl

 tools/libxl/libxl_arm.c             | 200 ++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl         |   6 +
 tools/xl/xl_parse.c                 |   7 +
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   2 +
 xen/arch/arm/domain.c               |   4 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 +++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/physdev.c              |  42 +++++-
 xen/arch/arm/setup.c                |   2 +
 xen/arch/arm/vpci.c                 | 102 ++++++++++++++
 xen/arch/arm/vpci.h                 |  37 +++++
 xen/drivers/passthrough/pci.c       |   7 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/domain.h        |   5 +
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 xen/include/public/arch-arm.h       |  32 +++++
 20 files changed, 1094 insertions(+), 9 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci.c
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH 1/4 arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49
---
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/setup.c                |   2 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 10 files changed, 654 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci.c
xen/arch/arm/Kconfig +7 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 
 source "arch/Kconfig"
 
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+
 config ACPI
 	bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
 	depends on ARM_64
xen/arch/arm/Makefile +1 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
xen/arch/arm/pci/pci-access.c new file +101 -0

101 lines not on this page · raw message

xen/arch/arm/pci/pci-host-common.c new file +198 -0

198 lines not on this page · raw message

xen/arch/arm/pci/pci-host-generic.c new file +131 -0

131 lines not on this page · raw message

xen/arch/arm/pci/pci.c new file +112 -0

112 lines not on this page · raw message

xen/arch/arm/setup.c +2 -0
index 7968cee47d..2d7f1db44f 100644
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     setup_virt_paging();
 
+    pci_init();
+
     do_initcalls();
 
     /*
xen/include/asm-arm/device.h +4 -3
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@
 enum device_type
 {
     DEV_DT,
+    DEV_PCI,
 };
 
 struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
 enum device_class
 {
     DEVICE_SERIAL,
     DEVICE_IOMMU,
     DEVICE_GIC,
+    DEVICE_PCI,
     /* Use for error */
     DEVICE_UNKNOWN,
 };
xen/include/asm-arm/pci.h +94 -3

101 lines not on this page · raw message

Signature · 1 line
2.17.1
#2 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

+ Jan, Andrew, Roger

Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)

Quoted 9 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49

What is this Change-Id property?

Quoted 146 lines
---
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/setup.c                |   2 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 10 files changed, 654 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci.c
xen/arch/arm/Kconfig +7 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 
 source "arch/Kconfig"
 
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+
 config ACPI
 	bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
 	depends on ARM_64
xen/arch/arm/Makefile +1 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
xen/arch/arm/pci/pci-access.c new file +81 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?

MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.

Quoted 96 lines
xen/arch/arm/pci/pci-access.c new file +9 -0
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +81 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;

xen/arch/arm/pci/pci-host-common.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;

Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?

Quoted 49 lines
xen/arch/arm/pci/pci-host-common.c new file +49 -0
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */
+    cfg = gen_pci_init(dev, ops);
+    if ( !cfg )
+        return -ENOMEM;

In case of errors the allocated bridge is not freed.

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 11 lines
xen/arch/arm/pci/pci-host-common.c new file +11 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

xen/arch/arm/pci/pci-host-common.c line 165

Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Quoted 58 lines
xen/arch/arm/pci/pci-host-common.c new file +32 -0
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).

Maybe pci_config_get_address or something like that?

Quoted 22 lines
xen/arch/arm/pci/pci-host-generic.c new file +22 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);

please use a switch

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {

xen/arch/arm/pci/pci-host-generic.c line 83

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+        *val = ~0;
+        return -ENODEV;
+    }
+
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);

please use a switch

Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5 -0
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {

xen/arch/arm/pci/pci-host-generic.c line 99

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },

spurious blank line

Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31 -0
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +71 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:

xen/arch/arm/pci/pci.c line 81

Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10 -0
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.

Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +0 -0
index 7968cee47d..2d7f1db44f 100644

xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933

Quoted 4 lines
xen/arch/arm/setup.c +1 -0
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     setup_virt_paging();
 
+    pci_init();

pci_init should probably be an initcall

Quoted 67 lines
xen/arch/arm/setup.c +0 -0
     do_initcalls();
 
     /*
xen/include/asm-arm/device.h +4 -3
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@
 enum device_type
 {
     DEV_DT,
+    DEV_PCI,
 };
 
 struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
 enum device_class
 {
     DEVICE_SERIAL,
     DEVICE_IOMMU,
     DEVICE_GIC,
+    DEVICE_PCI,
     /* Use for error */
     DEVICE_UNKNOWN,
 };
xen/include/asm-arm/pci.h +25 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+

xen/include/asm-arm/pci.h line 36

Quoted 10 lines
xen/include/asm-arm/pci.h +9 -0
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

Are you actually using struct device in struct arch_pci_dev? struct device is already part of struct dt_device_node and a pointer to it is stored in bridge->dt_node.

Quoted 15 lines
xen/include/asm-arm/pci.h +15 -0
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;
+    void __iomem        *win;

xen/include/asm-arm/pci.h line 62

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);

I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?

Can we use pci_sbdf_t for the sbdf parameter?

xen/include/asm-arm/pci.h line 73

Quoted 11 lines
xen/include/asm-arm/pci.h +11 -0
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};

Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename pci_ecam_ops to something else, maybe simply "pci_ops".

Quoted 28 lines
xen/include/asm-arm/pci.h +22 -1
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;
 };
 
-#endif /* __X86_PCI_H__ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                struct pci_ecam_ops *ops);
+
+void pci_init(void);
+
+#else   /*!CONFIG_ARM_PCI*/
+struct arch_pci_dev { };
+static inline void  pci_init(void) { }
+#endif  /*!CONFIG_ARM_PCI*/
+#endif /* __ARM_PCI_H__ */
Signature · 1 line
2.17.1
#3 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 10 lines · Stefano Stabellini

+ Jan, Andrew, Roger

Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)

Quoted 3 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

Quoted 7 lines
Quoted 6 lines

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49

What is this Change-Id property?

Gerrit ;)

Quoted 61 lines
Quoted 61 lines
---
  xen/arch/arm/Kconfig                |   7 +
  xen/arch/arm/Makefile               |   1 +
  xen/arch/arm/pci/Makefile           |   4 +
  xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
  xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
  xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
  xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
  xen/arch/arm/setup.c                |   2 +
  xen/include/asm-arm/device.h        |   7 +-
  xen/include/asm-arm/pci.h           |  97 +++++++++++++-
  10 files changed, 654 insertions(+), 6 deletions(-)

create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.c

xen/arch/arm/Kconfig +7 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 
  source "arch/Kconfig"
 
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+
  config ACPI
  	bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
  	depends on ARM_64
xen/arch/arm/Makefile +1 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
  obj-y += platforms/
  endif
  obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
 
  obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
  obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
Quoted 8 lines

xen/arch/arm/pci/pci-access.c line 2

Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.

I think SPDX will fit better in any new code.

Quoted 169 lines
Quoted 79 lines
xen/arch/arm/pci/pci-access.c new file +79 -0
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?

MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.

Quoted 72 lines
xen/arch/arm/pci/pci-access.c new file +9 -0
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +57 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 65

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}

The two functions above seem to deal with the same resources, e.g. cfg->win

and map/unmap. Would it make sense to align those, something like

s/pci_remap_cfgspace/pci_ecam_alloc and pci_ecam_alloc handles cfg->win?

Or anything which makes them look init/fini style?

Quoted 12 lines
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12 -0
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 85

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;

So, if bus start is, for example, 0x10 then we'll end up with bus end at (0x10 + 0xff) > 0xff

which doesn't seem to be what we want

Quoted 6 lines

xen/arch/arm/pci/pci-host-common.c line 91

Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6 -0
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;
Quoted 16 lines

Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?

Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12 -0
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);

I am fine with "win", but can we think of something that tells us that

"win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"

while reading?

Quoted 36 lines
Quoted 36 lines
xen/arch/arm/pci/pci-host-common.c new file +36 -0
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 148

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+    struct pci_config_window *cfg;
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */

Do you expect multiple windows here as the comment says?

Quoted 9 lines

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3 -0
+    cfg = gen_pci_init(dev, ops);
+    if ( !cfg )
+        return -ENOMEM;

In case of errors the allocated bridge is not freed.

Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;

Can we have some sort of dummy ops so we don't have to check for ops != NULL every time

we read/write config? Is it really possible that we have ops set to NULL after we have

the development finished?

Quoted 19 lines

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

Quoted 3 lines

xen/arch/arm/pci/pci-host-common.c line 165

Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Why? Do you expect bridge access in some specific order so ordered

list will make it faster?

Quoted 75 lines
Quoted 60 lines
xen/arch/arm/pci/pci-host-common.c new file +32 -0
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).

Quoted 5 lines

Maybe pci_config_get_address or something like that?

Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;

I am a bit confused of the naming ;)

We already have 2 maps: win -> ECAM base and now sysdata -> cfg.

Can we please have that aligned somehow so it is easier to follow?

Quoted 1 line

xen/arch/arm/pci/pci-host-generic.c line 33

Quoted 1 line
xen/arch/arm/pci/pci-host-generic.c new file +1 -0
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;

We are not checking cfg->ops for NULL, so probably we do not want bridges

with NULL ops as well?

Quoted 253 lines
Quoted 19 lines
xen/arch/arm/pci/pci-host-generic.c new file +19 -0
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);

please use a switch

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {

xen/arch/arm/pci/pci-host-generic.c line 83

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+        *val = ~0;
+        return -ENODEV;
+    }
+
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);

please use a switch

Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5 -0
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {

xen/arch/arm/pci/pci-host-generic.c line 99

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },

spurious blank line

Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31 -0
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +71 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:

xen/arch/arm/pci/pci.c line 81

Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10 -0
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.

Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +0 -0
index 7968cee47d..2d7f1db44f 100644

xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933

Quoted 4 lines
xen/arch/arm/setup.c +1 -0
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
      setup_virt_paging();
 
+    pci_init();

pci_init should probably be an initcall

Quoted 14 lines
xen/arch/arm/setup.c +0 -0
      do_initcalls();
 
      /*
xen/include/asm-arm/device.h +1 -0
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@
  enum device_type
  {
      DEV_DT,
+    DEV_PCI,
  };
 
Quoted 8 lines

xen/include/asm-arm/device.h typedef struct device device_t; · line 30

Quoted 8 lines
xen/include/asm-arm/device.h +0 -3
  struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
  #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)

Didn't we have a patch for that recently or talked about?

Quoted 91 lines
Quoted 45 lines
xen/include/asm-arm/device.h +3 -0
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
  enum device_class
  {
      DEVICE_SERIAL,
      DEVICE_IOMMU,
      DEVICE_GIC,
+    DEVICE_PCI,
      /* Use for error */
      DEVICE_UNKNOWN,
  };
xen/include/asm-arm/pci.h +25 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+

xen/include/asm-arm/pci.h line 36

Quoted 10 lines
xen/include/asm-arm/pci.h +9 -0
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
  struct arch_pci_dev {
+    struct device dev;
+};

Are you actually using struct device in struct arch_pci_dev? struct device is already part of struct dt_device_node and a pointer to it is stored in bridge->dt_node.

Quoted 15 lines
xen/include/asm-arm/pci.h +15 -0
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;
+    void __iomem        *win;

xen/include/asm-arm/pci.h line 62

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);

I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?

Can we use pci_sbdf_t for the sbdf parameter?

Quoted 15 lines

xen/include/asm-arm/pci.h line 73

Quoted 11 lines
xen/include/asm-arm/pci.h +11 -0
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};

Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename pci_ecam_ops to something else, maybe simply "pci_ops".

Yes, please, bear in mind that we are about to work with this code on

non-ECAM HW from the very beginning, so this is something that we would

like to see from the ground up

Quoted 29 lines
Quoted 29 lines
xen/include/asm-arm/pci.h +22 -1
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;
  };
 
-#endif /* __X86_PCI_H__ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                struct pci_ecam_ops *ops);
+
+void pci_init(void);
+
+#else   /*!CONFIG_ARM_PCI*/
+struct arch_pci_dev { };
+static inline void  pci_init(void) { }
+#endif  /*!CONFIG_ARM_PCI*/
+#endif /* __ARM_PCI_H__ */
Signature · 1 line
2.17.1
#4 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Hi,

Quoted 9 lines · Oleksandr Andrushchenko
Quoted 8 lines

xen/arch/arm/pci/pci-access.c line 2

Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.

I think SPDX will fit better in any new code.

While I would love to use SPDX in Xen, there was some push back in the past to use it. So the new code should use the full-blown copyright until there is an agreement to use it.

Quoted 8 lines
Quoted 4 lines

xen/arch/arm/pci/pci-access.c line 3

Quoted 2 lines
xen/arch/arm/pci/pci-access.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Why? Do you expect bridge access in some specific order so ordered

list will make it faster?

Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.

However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).

Cheers,

Signature · 1 line
Julien Grall
#5 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 8 lines · Julien Grall
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

+ list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Why? Do you expect bridge access in some specific order so ordered

list will make it faster?

Quoted 7 lines

Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.

However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).

I was suggesting ordering the list because it is a rather cheap
optimization: it is easy to implement and it typically leads to decent improvements (although it varies on a case by case basis). Something more sophisticated as you mention here would be even better.

#6 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Julien Grall
Quoted 17 lines · Julien Grall

Hi,

Quoted 9 lines · Oleksandr Andrushchenko
Quoted 8 lines

xen/arch/arm/pci/pci-access.c line 2

Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.

I think SPDX will fit better in any new code.

Quoted 1 line

While I would love to use SPDX in Xen, there was some push back in the past to use it. So the new code should use the full-blown copyright until there is an agreement to use it.

Ack. We will use the copyright until we will get the confirmation from the community to use the SPDX in XEN.

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

xen/arch/arm/pci/pci-access.c line 3

Quoted 3 lines
xen/arch/arm/pci/pci-access.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Why? Do you expect bridge access in some specific order so ordered list will make it faster?

Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.

However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).

Yes true every config space access has to scan the host-bridges list to find the corresponding host-bridge. I will try to find better solution so that no need to scan the host-bridge every time for read/write access.

Quoted 4 lines

Cheers,

Signature · 1 line
Julien Grall
#7 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Oleksandr Andrushchenko
Quoted 86 lines · Oleksandr Andrushchenko
Quoted 10 lines · Stefano Stabellini

+ Jan, Andrew, Roger

Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)

Quoted 3 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

Quoted 7 lines
Quoted 6 lines

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49

What is this Change-Id property?

Gerrit ;)

Quoted 62 lines
Quoted 62 lines
---
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/setup.c                |   2 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 10 files changed, 654 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci.c
xen/arch/arm/Kconfig +7 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 
 source "arch/Kconfig"
 
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+
 config ACPI
 	bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
 	depends on ARM_64
xen/arch/arm/Makefile +1 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
xen/arch/arm/pci/pci-access.c +0 -0
Quoted 8 lines
Quoted 7 lines

xen/arch/arm/pci/pci-access.c line 2

Quoted 7 lines

new file mode 100644
index 0000000000..c53ef58336

xen/arch/arm/pci/pci-access.c new file +2 -0
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.

I think SPDX will fit better in any new code.

It will be good if community helps us to decide which license is best for new files.

Quoted 180 lines
Quoted 169 lines
Quoted 79 lines
xen/arch/arm/pci/pci-access.c new file +79 -0
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?

MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.

Quoted 72 lines
xen/arch/arm/pci/pci-access.c new file +9 -0
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +57 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 65

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}

The two functions above seem to deal with the same resources, e.g. cfg->win

Quoted 5 lines

and map/unmap. Would it make sense to align those, something like

s/pci_remap_cfgspace/pci_ecam_alloc and pci_ecam_alloc handles cfg->win?

Or anything which makes them look init/fini style?

Ok will rename the function name.

Quoted 24 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/pci/pci-host-common.c new file +12 -0
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 85

Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;

So, if bus start is, for example, 0x10 then we'll end up with bus end at (0x10 + 0xff) > 0xff

Quoted 1 line

which doesn't seem to be what we want

This is to fix if in device tree bus-ranges property, bus end corresponds to more than 256 bus.

Quoted 24 lines
Quoted 23 lines

xen/arch/arm/pci/pci-host-common.c line 91

Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +6 -0
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;

Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?

Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12 -0
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
Quoted 5 lines

I am fine with "win", but can we think of something that tells us that

"win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"

while reading?

Ack. Will fix.

Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
xen/arch/arm/pci/pci-host-common.c new file +37 -0
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;
Quoted 8 lines
Quoted 7 lines

xen/arch/arm/pci/pci-host-common.c line 148

Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +7 -0
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */

Do you expect multiple windows here as the comment says?

If going forward we want to support 32-bit we have to ioremap the config space for each bus individually, but as of only 64 bit is supported.

Quoted 10 lines
Quoted 9 lines

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3 -0
+    cfg = gen_pci_init(dev, ops)
+    if ( !cfg )
+        return -ENOMEM;

In case of errors the allocated bridge is not freed.

Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
Quoted 5 lines

Can we have some sort of dummy ops so we don't have to check for ops != NULL every time

we read/write config? Is it really possible that we have ops set to NULL after we have

the development finished?

We don’t want to support host-bridges with dump ops. Proper ops has to be setup for the host-bridge to access the read/write. Once we access the config space we are sure that ops is already setup so no need to check for NULL each time.

Quoted 20 lines
Quoted 20 lines

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +8 -0
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

Quoted 7 lines
Quoted 3 lines

xen/arch/arm/pci/pci-host-common.c line 165

Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

Why? Do you expect bridge access in some specific order so ordered

list will make it faster?

As I mentioned in earlier no need to have ordered list as PCI config space access is random.

Quoted 79 lines
Quoted 79 lines
Quoted 61 lines
xen/arch/arm/pci/pci-host-common.c new file +32 -0
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).

Maybe pci_config_get_address or something like that?

Quoted 8 lines
Quoted 2 lines

xen/arch/arm/pci/pci-host-generic.c line 32

Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;

I am a bit confused of the naming ;)

We already have 2 maps: win -> ECAM base and now sysdata -> cfg.

Can we please have that aligned somehow so it is easier to follow?

Ack.

Quoted 6 lines
Quoted 2 lines

xen/arch/arm/pci/pci-host-generic.c line 33

Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +1 -0
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;

We are not checking cfg->ops for NULL, so probably we do not want bridges

with NULL ops as well?

Yes correct we don’t want host-bridge with NULL ops. Do you see any use case to have host-bridge with NULL ops?

Quoted 257 lines
Quoted 257 lines
Quoted 20 lines
xen/arch/arm/pci/pci-host-generic.c new file +19 -0
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);

please use a switch

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {

xen/arch/arm/pci/pci-host-generic.c line 83

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+        *val = ~0;
+        return -ENODEV;
+    }
+
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);

please use a switch

Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5 -0
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {

xen/arch/arm/pci/pci-host-generic.c line 99

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },

spurious blank line

Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31 -0
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +71 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:

xen/arch/arm/pci/pci.c line 81

Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10 -0
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.

Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +0 -0
index 7968cee47d..2d7f1db44f 100644

xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933

Quoted 4 lines
xen/arch/arm/setup.c +1 -0
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     setup_virt_paging();
 
+    pci_init();

pci_init should probably be an initcall

Quoted 17 lines
xen/arch/arm/setup.c +0 -0
     do_initcalls();
 
     /*
xen/include/asm-arm/device.h +1 -0
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@
 enum device_type
 {
     DEV_DT,
+    DEV_PCI,
 };
 
 struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
Quoted 6 lines
Quoted 5 lines

xen/include/asm-arm/device.h typedef struct device device_t; · line 30

Quoted 5 lines
xen/include/asm-arm/device.h +0 -3
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)

Didn't we have a patch for that recently or talked about?

Not sure need to check.

Quoted 107 lines
Quoted 91 lines
Quoted 45 lines
xen/include/asm-arm/device.h +3 -0
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
 enum device_class
 {
     DEVICE_SERIAL,
     DEVICE_IOMMU,
     DEVICE_GIC,
+    DEVICE_PCI,
     /* Use for error */
     DEVICE_UNKNOWN,
 };
xen/include/asm-arm/pci.h +25 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+

xen/include/asm-arm/pci.h line 36

Quoted 10 lines
xen/include/asm-arm/pci.h +9 -0
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

Are you actually using struct device in struct arch_pci_dev? struct device is already part of struct dt_device_node and a pointer to it is stored in bridge->dt_node.

Quoted 15 lines
xen/include/asm-arm/pci.h +15 -0
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;
+    void __iomem        *win;

xen/include/asm-arm/pci.h line 62

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);

I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?

Can we use pci_sbdf_t for the sbdf parameter?

Quoted 15 lines

xen/include/asm-arm/pci.h line 73

Quoted 11 lines
xen/include/asm-arm/pci.h +11 -0
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};

Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename pci_ecam_ops to something else, maybe simply "pci_ops".

Quoted 5 lines

Yes, please, bear in mind that we are about to work with this code on

non-ECAM HW from the very beginning, so this is something that we would

like to see from the ground up

Ok. Will take care of that in next version of the patch.

Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
xen/include/asm-arm/pci.h +22 -1
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;
 };
 
-#endif /* __X86_PCI_H__ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                struct pci_ecam_ops *ops);
+
+void pci_init(void);
+
+#else   /*!CONFIG_ARM_PCI*/
+struct arch_pci_dev { };
+static inline void  pci_init(void) { }
+#endif  /*!CONFIG_ARM_PCI*/
+#endif /* __ARM_PCI_H__ */
Signature · 1 line
2.17.1
#8 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Stefano Stabellini

Hi,

Quoted 15 lines · Stefano Stabellini

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 11 lines
xen/arch/arm/pci/pci-host-common.c new file +11 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

Quoted 5 lines

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

So we definitely need a segment number in hand because this is what the admin will use to assign a PCI device to a guest.

The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.

Looking at the code, I think Linux is using 'segment' as 'domain'. So they should be interchangeably. I am not sure what other OS does though.

Cheers,

Signature · 1 line
Julien Grall
#9 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 17 lines · Julien Grall
Quoted 16 lines · Stefano Stabellini

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +7 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available

in DT\n");

xen/arch/arm/pci/pci-host-common.c new file +4 -0
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

Quoted 8 lines
Quoted 5 lines

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

So we definitely need a segment number in hand because this is what the admin will use to assign a PCI device to a guest.

Yeah

Quoted 2 lines

The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.

As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by _SEG on ACPI systems.

The link above suggests that a Linux domain corresponds to (_SEG, _BBN) where _SEG is the segment and _BBN is the "Bus Number".

I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.

#10 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 9 lines · Stefano Stabellini
Quoted 3 lines · Julien Grall

On 24/07/2020 00:38, Stefano Stabellini wrote:
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.

As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by _SEG on ACPI systems.

Quoted 7 lines

The link above suggests that a Linux domain corresponds to (_SEG, _BBN) where _SEG is the segment and _BBN is the "Bus Number".

I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.

You seem to argue on the name but this doesn't resolve the underlying problem. Indeed, all our external interfaces are expecting a segment number.

If they are not equal, then I fail to see why it would be useful to have this value in Xen. In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

Cheers,

Signature · 1 line
Julien Grall
#11 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 3 lines · Julien Grall

On 24/07/2020 00:38, Stefano Stabellini wrote:
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.

As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by _SEG on ACPI systems.

The link above suggests that a Linux domain corresponds to (_SEG, _BBN) where _SEG is the segment and _BBN is the "Bus Number".

Quoted 7 lines
Quoted 4 lines

I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.

You seem to argue on the name but this doesn't resolve the underlying problem. Indeed, all our external interfaces are expecting a segment number.

Yes, you are right, that is a bigger problem.

Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

#12 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 7 lines
Quoted 3 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

From this discussion, it also looks like there is a mismatch between the implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

Cheers,

[1] https://elixir.bootlin.com/linux/latest/source/arch/arm64/kernel/pci.c#L74

Cheers,

#13 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 23 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 7 lines
Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property. And if we look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.

I do think we need to understand the definitions and the differences. Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.

So maybe it is OK to use domains == segments, but we need to verify this in the specs and also clarify the terminology we use in a doc for our own sanity -- I am hoping that Rahul can come up with a good explanation on the topic :-)

[1] https://stackoverflow.com/questions/49050847/how-is-pci-segmentdomain-related-to-multiple-host-bridgesor-root-bridges
[2] https://wiki.osdev.org/PCI_Express

#14 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 29 lines · Stefano Stabellini
Quoted 27 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

Quoted 6 lines
Quoted 3 lines

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.

My concern is not so much the name of the property, but the definition of it.

AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)

Quoted 5 lines

And if we
look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.

I do think we need to understand the definitions and the differences.

+1

Quoted 8 lines

Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.

So maybe it is OK to use domains == segments, but we need to verify this in the specs and also clarify the terminology we use in a doc for our own sanity -- I am hoping that Rahul can come up with a good explanation on the topic :-)

I am a bit confused.... You were the one arguing that domain == (segment, bus) in this thread. So may I ask why the interpretation wouldn't be valid anymore?

Cheers,

Quoted 2 lines

[1] https://stackoverflow.com/questions/49050847/how-is-pci-segmentdomain-related-to-multiple-host-bridgesor-root-bridges
[2] https://wiki.osdev.org/PCI_Express

#15 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

Quoted 8 lines
Quoted 2 lines

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.

My concern is not so much the name of the property, but the definition of it.

AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)

I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?

Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).

Roger.

#16 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 5 lines

You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.

Thanks for the clarification!

Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

Quoted 8 lines
Quoted 2 lines

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.

My concern is not so much the name of the property, but the definition of it.

AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)

I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?

Quoted 4 lines

Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).

That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.

Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).

The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.

#17 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 71 lines · Stefano Stabellini
Quoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 5 lines

You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.

Thanks for the clarification!

Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

Quoted 8 lines
Quoted 2 lines

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.

My concern is not so much the name of the property, but the definition of it.

AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)

I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?

Quoted 4 lines

Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).

That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.

Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).

Quoted 5 lines

The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.

I don't think the ACPI MCFG spec contains the notion of bridges, it just describes ECAM (or MMCFG) regions, but those could be made up by concatenating different bridge ECAM regions by the firmware itself, so you could AFAICT end up with multiple bridges being aggregated into a single ECAM region, and thus using the same segment number, which seems not possible with the DT spec, where each bridge must get a different segment number?

If you could assign both a segment number and a bus start and end values to a bridge then I think it would be kind of equivalent to ACPI MCFG.

I assume we would never support a system where Xen is getting the hardware description from a DT and the hardware domain is using ACPI (or the other way around)?

If so, I don't think we care that enumeration when using DT is different than when using ACPI, as we can only guarantee consistency when both Xen and the hardware domain use the same source for the hardware description.

If when using DT each bridge has a unique segment number that's fine as long as Xen and the OS agree to not change such values.

Roger.

#18 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 78 lines · Roger Pau Monné
Quoted 71 lines · Stefano Stabellini
Quoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 5 lines

You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.

Thanks for the clarification!

Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

Quoted 8 lines
Quoted 2 lines

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.

My concern is not so much the name of the property, but the definition of it.

AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)

I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?

Quoted 4 lines

Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).

That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.

Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).

Quoted 5 lines

The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.

Quoted 7 lines

I don't think the ACPI MCFG spec contains the notion of bridges, it just describes ECAM (or MMCFG) regions, but those could be made up by concatenating different bridge ECAM regions by the firmware itself, so you could AFAICT end up with multiple bridges being aggregated into a single ECAM region, and thus using the same segment number, which seems not possible with the DT spec, where each bridge must get a different segment number?

Yes, that's my understanding too

Quoted 7 lines

If you could assign both a segment number and a bus start and end values to a bridge then I think it would be kind of equivalent to ACPI MCFG.

I assume we would never support a system where Xen is getting the hardware description from a DT and the hardware domain is using ACPI (or the other way around)?

Yeah, I think it is a good assumption

Quoted 7 lines

If so, I don't think we care that enumeration when using DT is different than when using ACPI, as we can only guarantee consistency when both Xen and the hardware domain use the same source for the hardware description.

If when using DT each bridge has a unique segment number that's fine as long as Xen and the OS agree to not change such values.

I agree

#19 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Stefano Stabellini
Quoted 37 lines · Stefano Stabellini
Quoted 23 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines

If they are not equal, then I fail to see why it would be useful to have this value in Xen.

I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.

AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.

Quoted 6 lines
Quoted 2 lines

In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.

I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.

Given that we have only the domain in hand, how would you enforce that?

Quoted 7 lines
Quoted 1 line

From this discussion, it also looks like there is a mismatch between the

implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.

IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?

I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property. And if we look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.

Quoted 4 lines

I do think we need to understand the definitions and the differences. Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.

If I may, I'd like to put the question a little differently, in the hope for me to understand the actual issue here: On the x86 side, by way of using ACPI, Linux and Xen "naturally" agree on segment numbering (as far as normal devices go; Intel's Volume Management Device concept still needs accommodating so that it would work with Xen). This includes the multiple host bridges case then naturally. How is the Device Tree model different from ACPI?

Jan

#20 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Stefano Stabellini
Quoted 14 lines · Stefano Stabellini

+ Jan, Andrew, Roger

Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)

Quoted 3 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

Quoted 8 lines
Quoted 6 lines

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49

What is this Change-Id property?

Will remove this in next patch series.

Quoted 161 lines
Quoted 148 lines

---

xen/arch/arm/Kconfig                |   7 +
xen/arch/arm/Makefile               |   1 +
xen/arch/arm/pci/Makefile           |   4 +
xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
xen/arch/arm/setup.c                |   2 +
xen/include/asm-arm/device.h        |   7 +-
xen/include/asm-arm/pci.h           |  97 +++++++++++++-
10 files changed, 654 insertions(+), 6 deletions(-)

create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.c

xen/arch/arm/Kconfig +0 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 

source "arch/Kconfig"

xen/arch/arm/Kconfig +7 -0
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+

config ACPI
bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64

xen/arch/arm/Makefile +0 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)

obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/

obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o

xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o
xen/arch/arm/pci/pci-access.c new file +81 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?

Quoted 3 lines

MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.

We have just started to investigate the ACPI support for PCI within XEN on ARM. Once we have understanding how we can support the ACPI we will have a look x86 MMCFG access and will check if we can have a common interface to access the configuration space.

Quoted 104 lines
Quoted 104 lines
xen/arch/arm/pci/pci-access.c new file +9 -0
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +86 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;
+    }
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 91

Quoted 5 lines
xen/arch/arm/pci/pci-host-common.c new file +5 -0
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;

Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?

As of now we are supporting the PCI controllers that have "pci-host-ecam-generic” compatible node in device tree binding.As per my understanding controller that have "pci-host-ecam-generic” compatible property have only one reg property value at index 0.

Yes there are PCI controllers that support ECAM and have more than one "reg" property values.As per my understanding once we support other controller in XEN at that time we have to handle the possibility of multiple addresses and that code is specific to that controller .

Quoted 55 lines
Quoted 55 lines
xen/arch/arm/pci/pci-host-common.c new file +53 -0
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
Quoted 9 lines

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6 -0
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */
+    cfg = gen_pci_init(dev, ops);
+    if ( !cfg )
+        return -ENOMEM;

In case of errors the allocated bridge is not freed.

Will fix this in next version.

Quoted 16 lines

xen/arch/arm/pci/pci-host-common.c line 163

Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +11 -0
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;
+    }
+
+    bridge->segment = (u16)segment;

My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:

Quoted 6 lines

https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html

Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?

As per my understanding linux domain number is an alias to segment number.Device tree binding uses the property domain number and ACPI uses the property segment number. We have just started the investigation on ACPI, once we have more information about MMCFG ACPI table how it works will share the information and will update the patches and design doc accordingly.

Quoted 6 lines

xen/arch/arm/pci/pci-host-common.c line 165

Quoted 2 lines
xen/arch/arm/pci/pci-host-common.c new file +1 -0
+    list_add_tail(&bridge->node, &pci_host_bridges);

It looks like &pci_host_bridges should be an ordered list, ordered by segment number?

As Julien also mentioned access to the PCI config space is random there is no need to have an ordered list.

Quoted 75 lines
Quoted 59 lines
xen/arch/arm/pci/pci-host-common.c new file +32 -0
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).

Quoted 1 line

Maybe pci_config_get_address or something like that?

OK yes will rename the function name.

Quoted 52 lines
Quoted 24 lines
xen/arch/arm/pci/pci-host-generic.c new file +22 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);

please use a switch

Quoted 14 lines
xen/arch/arm/pci/pci-host-generic.c new file +14 -0
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {
+        *val = ~0;
+        return -ENODEV;
+    }
+
Quoted 8 lines

xen/arch/arm/pci/pci-host-generic.c line 83

Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6 -0
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);

please use a switch

Ok.

Quoted 11 lines
Quoted 11 lines
xen/arch/arm/pci/pci-host-generic.c new file +9 -0
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
Quoted 8 lines

xen/arch/arm/pci/pci-host-generic.c line 99

Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6 -0
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },

spurious blank line

Ok will fix this.

Quoted 117 lines
Quoted 117 lines
xen/arch/arm/pci/pci-host-generic.c new file +31 -0
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +78 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
Quoted 9 lines

xen/arch/arm/pci/pci.c line 81

Quoted 3 lines
xen/arch/arm/pci/pci.c new file +3 -0
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.

Ok yes will add it to the documentation.

Quoted 37 lines
Quoted 37 lines
xen/arch/arm/pci/pci.c new file +30 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +0 -0
index 7968cee47d..2d7f1db44f 100644
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
Quoted 5 lines

xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933

Quoted 3 lines
xen/arch/arm/setup.c +1 -0
    setup_virt_paging();
 
+    pci_init();

pci_init should probably be an initcall.

OK.

Quoted 75 lines
Quoted 75 lines
xen/arch/arm/setup.c +0 -0
    do_initcalls();
 
    /*
xen/include/asm-arm/device.h +0 -0
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@

enum device_type

{
    DEV_DT,

+ DEV_PCI,
};

struct dev_archdata {

xen/include/asm-arm/device.h +2 -3
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 

enum device_class
{
DEVICE_SERIAL,
DEVICE_IOMMU,
DEVICE_GIC,
+ DEVICE_PCI,

    /* Use for error */
    DEVICE_UNKNOWN,
};
xen/include/asm-arm/pci.h +31 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
Quoted 8 lines

xen/include/asm-arm/pci.h line 36

Quoted 4 lines
xen/include/asm-arm/pci.h +3 -0
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

Are you actually using struct device in struct arch_pci_dev? struct device is already part of struct dt_device_node and a pointer to it is stored in bridge->dt_node.

Will be using this going forward once we have full PCI passthrough support.

Quoted 31 lines
Quoted 17 lines
xen/include/asm-arm/pci.h +15 -0
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;
+    void __iomem        *win;

xen/include/asm-arm/pci.h line 62

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);

I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?

Quoted 1 line

Can we use pci_sbdf_t for the sbdf parameter?

Ok will fix in next version.

Quoted 18 lines

xen/include/asm-arm/pci.h line 73

Quoted 13 lines
xen/include/asm-arm/pci.h +11 -0
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};

Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename pci_ecam_ops to something else, maybe simply "pci_ops".

Ok will have a look and modify accordingly.

Quoted 29 lines
Quoted 29 lines
xen/include/asm-arm/pci.h +22 -1
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;
 };
 
-#endif /* __X86_PCI_H__ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus);
+
+int pci_host_common_probe(struct dt_device_node *dev,
+                struct pci_ecam_ops *ops);
+
+void pci_init(void);
+
+#else   /*!CONFIG_ARM_PCI*/
+struct arch_pci_dev { };
+static inline void  pci_init(void) { }
+#endif  /*!CONFIG_ARM_PCI*/
+#endif /* __ARM_PCI_H__ */
Signature · 1 line
2.17.1
#21 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Rahul Singh

Hi Rahul,

Quoted 10 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49
Quoted 12 lines
---
  xen/arch/arm/Kconfig                |   7 +
  xen/arch/arm/Makefile               |   1 +
  xen/arch/arm/pci/Makefile           |   4 +
  xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
  xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
  xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
  xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
  xen/arch/arm/setup.c                |   2 +
  xen/include/asm-arm/device.h        |   7 +-
  xen/include/asm-arm/pci.h           |  97 +++++++++++++-
  10 files changed, 654 insertions(+), 6 deletions(-)

As a general comment, I would suggest to split the patch in smaller chunk. This would help the review and also allow to provide more explanation on what is done.

For instance, I think it is possible to a split looking like:
- Add framework to access an hostbridge
- Add support for ECAM
- Add code to initialize the PCI subsystem

There is also some small fixes in this code that probably can move in there own patches.

Cheers,

Signature · 1 line
Julien Grall
#22 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 27 lines · Julien Grall

Hi Rahul,

Quoted 7 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly. As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49
Quoted 12 lines
---
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/setup.c                |   2 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 10 files changed, 654 insertions(+), 6 deletions(-)
Quoted 1 line

As a general comment, I would suggest to split the patch in smaller chunk. This would help the review and also allow to provide more explanation on what is done.

Ok I will split the patches in next version of the patch series.

Quoted 7 lines

For instance, I think it is possible to a split looking like:
- Add framework to access an hostbridge
- Add support for ECAM
- Add code to initialize the PCI subsystem

There is also some small fixes in this code that probably can move in there own patches.

Ack.

Quoted 5 lines

Cheers,

Signature · 1 line
Julien Grall
#23 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Rahul Singh
Quoted 64 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49
---
 xen/arch/arm/Kconfig                |   7 +
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/pci/Makefile           |   4 +
 xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
 xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
 xen/arch/arm/setup.c                |   2 +
 xen/include/asm-arm/device.h        |   7 +-
 xen/include/asm-arm/pci.h           |  97 +++++++++++++-
 10 files changed, 654 insertions(+), 6 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci.c
xen/arch/arm/Kconfig +7 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 
 source "arch/Kconfig"
 
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+
 config ACPI
 	bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT
 	depends on ARM_64
xen/arch/arm/Makefile +1 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)
 obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +0 -0
new file mode 100644
index 0000000000..358508b787

xen/arch/arm/pci/Makefile line 4

Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o

The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?

Quoted 19 lines
xen/arch/arm/pci/pci-access.c new file +13 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License

xen/arch/arm/pci/pci-access.c line 23

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)

Please align with the opening parenthesis (here and everywhere in the patch series).

xen/arch/arm/pci/pci-access.c line 26

Quoted 3 lines
xen/arch/arm/pci/pci-access.c new file +3 -0
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);

You can just set val = ~0. The return type of the pci_conf_readXX helpers will already truncate the value.

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

xen/arch/arm/pci/pci-access.c line 40

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);

There's no need for the uint32_t cast, the sbdf field is already of such type.

Quoted 41 lines
xen/arch/arm/pci/pci-access.c new file +41 -0
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

Please use uintXX_t.

Also, It's nice to add some kind of signals for cscope and friends so they can find the autogenerated functions, ie:

#define pci_conf_read8
#undef pci_conf_read8
#define pci_conf_read16
#undef pci_conf_read16
...

It's tedious but helps future users find where the code is generated.

Quoted 46 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +30 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+

xen/arch/arm/pci/pci-host-common.c line 40

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;

It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.

Quoted 29 lines
xen/arch/arm/pci/pci-host-common.c new file +29 -0
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{

xen/arch/arm/pci/pci-host-common.c line 79

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {

Braces

Quoted 37 lines
xen/arch/arm/pci/pci-host-common.c new file +37 -0
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;
+
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+

xen/arch/arm/pci/pci-host-common.c line 126

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    return cfg;
+
+err_exit_remap:
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)

^ extra space

Quoted 15 lines
xen/arch/arm/pci/pci-host-common.c new file +15 -0
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */
+    cfg = gen_pci_init(dev, ops);
+    if ( !cfg )
+        return -ENOMEM;

You are leaking the allocation of bridge here ...

xen/arch/arm/pci/pci-host-common.c line 160

Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +9 -0
+
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;

... and here.

Quoted 18 lines
xen/arch/arm/pci/pci-host-common.c new file +18 -0
+    }
+
+    bridge->segment = (u16)segment;
+
+    list_add_tail(&bridge->node, &pci_host_bridges);
+
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+

xen/arch/arm/pci/pci-host-common.c line 188

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;

This can be much shorter:

struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)

{
    struct pci_host_bridge *bridge;

    list_for_each_entry( bridge, &pci_host_bridges, node )
        if ( bridge->segment == segment )
            return bridge;

    return NULL;
}

Albeit I'm confused by the fact that you pass a bus number that's completely unused.

Quoted 36 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

You would be better passing a pci_sbdf_t directly here. Also 'where' should be renamed to offset, or reg, and be made unsigned int. AFAICT you will never pass a negative value here.

For sanity you should also assert that the offset falls between the PCI config space used by the device, in order to easily catch with wrong offsets being used.

xen/arch/arm/pci/pci-host-generic.c line 32

Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;

const

xen/arch/arm/pci/pci-host-generic.c line 38

Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6 -0
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;

IMO adding newlines between variable definitions is not helpful, but that's my taste.

Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5 -0
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);

xen/arch/arm/pci/pci-host-generic.c line 53

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);

You can initialize at definition.

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 9 lines
xen/arch/arm/pci/pci-host-generic.c new file +9 -0
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);

Please use a switch, and check against specific values. The default case should be a BUG();. See pci_conf_read from x86 for an example.

Quoted 146 lines
xen/arch/arm/pci/pci-host-generic.c new file +69 -0
+
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {
+        *val = ~0;
+        return -ENODEV;
+    }
+
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);
+
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },
+
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +71 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:

xen/arch/arm/pci/pci.c line 81

Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10 -0
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

You need to introduce the documentation for the parameter at docs/misc/xen-command-line.pandoc

Albeit I'm not sure I like it, why do you need to enable PCI explicitly?

Shouldn't it be discovered automatically and enabled by default?

xen/arch/arm/pci/pci.c line 89

Quoted 7 lines
xen/arch/arm/pci/pci.c new file +7 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;

Just return here, there's not point in having a label to perform a return.

xen/arch/arm/pci/pci.c line 94

Quoted 5 lines
xen/arch/arm/pci/pci.c new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

Also for ACPI you will get called by acpi_boot_init, and likely need to implement a acpi_mmcfg_init or pci_mmcfg_arch_{init,enable}. I'm not sure whether the code in acpi_mmcfg_init could be made shared between both x86 and Arm.

Quoted 49 lines
xen/arch/arm/pci/pci.c new file +18 -0
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +2 -0
index 7968cee47d..2d7f1db44f 100644
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
     setup_virt_paging();
 
+    pci_init();
+
     do_initcalls();
 
     /*
xen/include/asm-arm/device.h +1 -2
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@
 enum device_type
 {
     DEV_DT,
+    DEV_PCI,
 };
 
 struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)

xen/include/asm-arm/device.h typedef struct device device_t; · line 37

Quoted 10 lines
xen/include/asm-arm/device.h +3 -1
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
 enum device_class
 {
     DEVICE_SERIAL,
     DEVICE_IOMMU,
     DEVICE_GIC,
+    DEVICE_PCI,

It seems to be like this wants to be DEVICE_PCI_HOST_BRIDGE or some such, since this is not used to identify all PCI devices, but just bridges?

Quoted 36 lines
xen/include/asm-arm/device.h +0 -0
     /* Use for error */
     DEVICE_UNKNOWN,
 };
xen/include/asm-arm/pci.h +25 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+

xen/include/asm-arm/pci.h line 36

Quoted 10 lines
xen/include/asm-arm/pci.h +9 -0
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

This seems to be completely unused?

Quoted 5 lines
xen/include/asm-arm/pci.h +5 -0
+
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*

xen/include/asm-arm/pci.h line 51

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;

const?

Quoted 23 lines
xen/include/asm-arm/pci.h +23 -0
+    void __iomem        *win;
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};
+

xen/include/asm-arm/pci.h line 84

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;

You seem to introduce a lot of ops structs, yet there's only one implementation the generic ECAM one, and adding such complexity should IMO be done when further implementations are added. Also given this is a fully ECAM compliant bridge you could just use most of the existing logic for x86?

I understand the discovery needs to be different, but x86 MCFG logic should already be capable of handling multiple ECAM regions.

I also agree with Julien that splitting this into separate patch would make it easier to review. For example you can start with the discovery logic, followed by the initialization and the add the accessors to the config space lastly.

Thanks, Roger.

#24 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 5 lines · Roger Pau Monné
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +0 -0
new file mode 100644
index 0000000000..358508b787
Quoted 8 lines

xen/arch/arm/pci/Makefile line 4

Quoted 5 lines
xen/arch/arm/pci/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o

The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?

I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.

Quoted 11 lines

xen/arch/arm/pci/Makefile

Quoted 7 lines
xen/arch/arm/pci/Makefile new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

Quoted 7 lines

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

[...]

Quoted 8 lines

xen/arch/arm/pci/Makefile

Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4 -0
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;

It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.

uint32_t would be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so __be32 is the best choice.

[...]

Quoted 11 lines

xen/arch/arm/pci/Makefile

Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.

Cheers,

Signature · 1 line
Julien Grall
#25 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 16 lines · Julien Grall
Quoted 5 lines · Roger Pau Monné
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +0 -0
new file mode 100644
index 0000000000..358508b787
Quoted 8 lines

xen/arch/arm/pci/Makefile line 4

Quoted 5 lines
xen/arch/arm/pci/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o

The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?

Quoted 3 lines

I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.

Ack. Could the Kconfg be adjusted to not depend on ARM_64? Just stating it's only been tested on Arm64 would be enough IMO.

Quoted 13 lines
Quoted 13 lines

xen/arch/arm/pci/Makefile

Quoted 7 lines
xen/arch/arm/pci/Makefile new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

Quoted 8 lines
Quoted 5 lines

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

I only had an Ack from Jan, so I was missing Intel and AMD Acks, which would now only be Intel since AMD has been absorbed by the x86 maintainers.

Quoted 11 lines

[...]

Quoted 8 lines

xen/arch/arm/pci/Makefile

Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4 -0
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;

It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.

Quoted 3 lines

uint32_t would be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so __be32 is the best choice.

Oh, OK, so this is done to explicitly denote the endianness of a value on the type itself.

Quoted 8 lines

[...]

Quoted 6 lines
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();
Quoted 8 lines
Quoted 5 lines

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.

Ack. In any case I think for ACPI PCI init will get called by acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?

Roger.

#26 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Julien Grall
Quoted 11 lines · Roger Pau Monné

xen/arch/arm/pci/pci.c line 94

Quoted 5 lines
xen/arch/arm/pci/pci.c new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

Quoted 7 lines
Quoted 2 lines

This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.

Ack. In any case I think for ACPI PCI init will get called by acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?

Sorry I got confused and acpi_boot_init is a x86 specific function, wrong context. If Arm is not using that path then maybe it makes sense to init PCI here for ACPI also.

Roger.

#27 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Roger Pau Monné
Quoted 17 lines · Roger Pau Monné
Quoted 16 lines · Julien Grall
Quoted 13 lines · Roger Pau Monné

xen/arch/arm/pci/Makefile line 4

Quoted 10 lines
xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o

The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?

Quoted 6 lines
Quoted 3 lines

I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.

Ack. Could the Kconfg be adjusted to not depend on ARM_64? Just stating it's only been tested on Arm64 would be enough IMO.

We already have an option to select PCI (see CONFIG_HAS_PCI). So I would prefer if we reuse it (possible renamed to CONFIG_PCI) rather than inventing one for Arm specific.

Regarding the dependency, it will depend if it is possible to make it build easily on Arm32. If not, then we will need to keep the ARM_64.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines

xen/arch/arm/pci/Makefile

Quoted 8 lines
xen/arch/arm/pci/Makefile new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

Quoted 6 lines
Quoted 2 lines

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

I only had an Ack from Jan, so I was missing Intel and AMD Acks, which would now only be Intel since AMD has been absorbed by the x86 maintainers.

Ok. So, it should be easier to get it acked now :).

Quoted 12 lines
Quoted 12 lines

[...]

Quoted 8 lines

xen/arch/arm/pci/Makefile

Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4 -0
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;

It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.

Quoted 6 lines
Quoted 3 lines

uint32_t would be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so __be32 is the best choice.

Oh, OK, so this is done to explicitly denote the endianness of a value on the type itself.

That's correct. On Linux, they use sparse to then check the BE and LE fields are not mixed together. We don't have that on Xen, but at least this makes more obvious what we are using.

Quoted 15 lines
Quoted 15 lines

[...]

Quoted 11 lines

xen/arch/arm/pci/Makefile

Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

Quoted 7 lines
Quoted 2 lines

This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.

Ack. In any case I think for ACPI PCI init will get called by acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?

acpi_boot_init() does not exist on Arm. Looking at x86, acpi_mmcfg_init() is not event called from that function.

In general, I would prefer if each subsystem takes care of initialization itself. This makes easier to figure out the difference between ACPI and DT. FWIW, this is inline with majority of the Arm code.

Cheers,

Signature · 1 line
Julien Grall
#28 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Julien Grall
Quoted 14 lines · Julien Grall
Quoted 13 lines · Roger Pau Monné

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

Quoted 8 lines
Quoted 5 lines

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).

Jan

#29 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Hi Jan,

Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 18 lines · Roger Pau Monné

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

Quoted 6 lines
Quoted 2 lines

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).

Looking at the thread, I can see you (relunctantly) acked any components that you are the sole maintainers. Kevin gave his acked for the vtd code and I gave it mine for the common code.

I would suggest to not rehash the argument unless another maintainer agree with your position. It loosk like to me the next step is for Roger (or someone else) to resend the patch so we could collect the missing ack (I think there is only one missing from Andrew).

Cheers,

Signature · 1 line
Julien Grall
#30 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 31 lines · Julien Grall

Hi Jan,

Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 18 lines · Roger Pau Monné

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

Quoted 6 lines
Quoted 2 lines

I forgot about this patch :/. It would be good to revive it. Which acks are you missing?

It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).

Quoted 8 lines

Looking at the thread, I can see you (relunctantly) acked any components that you are the sole maintainers. Kevin gave his acked for the vtd code and I gave it mine for the common code.

I would suggest to not rehash the argument unless another maintainer agree with your position. It loosk like to me the next step is for Roger (or someone else) to resend the patch so we could collect the missing ack (I think there is only one missing from Andrew).

I've rebased and sent the updated patch with the collected Acks.

Roger.

#31 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

In reply to Roger Pau Monné
Quoted 61 lines · Roger Pau Monné
Quoted 56 lines · Rahul Singh

XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.

XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.

As of now "pci-host-ecam-generic" compatible board is supported.

Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49

---

xen/arch/arm/Kconfig                |   7 +
xen/arch/arm/Makefile               |   1 +
xen/arch/arm/pci/Makefile           |   4 +
xen/arch/arm/pci/pci-access.c       | 101 ++++++++++++++
xen/arch/arm/pci/pci-host-common.c  | 198 ++++++++++++++++++++++++++++
xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++
xen/arch/arm/pci/pci.c              | 112 ++++++++++++++++
xen/arch/arm/setup.c                |   2 +
xen/include/asm-arm/device.h        |   7 +-
xen/include/asm-arm/pci.h           |  97 +++++++++++++-
10 files changed, 654 insertions(+), 6 deletions(-)

create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.c

xen/arch/arm/Kconfig +0 -0
index 2777388265..ee13339ae9 100644
@@ -31,6 +31,13 @@ menu "Architecture Features"
 

source "arch/Kconfig"

xen/arch/arm/Kconfig +7 -0
+config ARM_PCI
+	bool "PCI Passthrough Support"
+	depends on ARM_64
+	---help---
+
+	  PCI passthrough support for Xen on ARM64.
+

config ACPI
bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64

xen/arch/arm/Makefile +0 -0
index 7e82b2178c..345cb83eed 100644
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)

obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/

Quoted 15 lines

xen/arch/arm/pci/Makefile line 4

Quoted 12 lines

obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o

xen/arch/arm/pci/Makefile new file +4 -0
new file mode 100644
index 0000000000..358508b787
@@ -0,0 +1,4 @@
+obj-y += pci.o
+obj-y += pci-host-generic.o
+obj-y += pci-host-common.o
+obj-y += pci-access.o

The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?

Ok.As julien also mentioned we tested on ARM64 , We have to test on ARM32 platforms.

Quoted 25 lines
Quoted 25 lines
xen/arch/arm/pci/pci-access.c new file +18 -0
new file mode 100644
index 0000000000..c53ef58336
@@ -0,0 +1,101 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
Quoted 8 lines

xen/arch/arm/pci/pci-access.c line 23

Quoted 5 lines
xen/arch/arm/pci/pci-access.c new file +5 -0
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+
+static uint32_t pci_config_read(pci_sbdf_t sbdf, unsigned int reg,
+                            unsigned int len)

Please align with the opening parenthesis (here and everywhere in the patch series).

Ack.

Quoted 8 lines

xen/arch/arm/pci/pci-access.c line 26

Quoted 4 lines
xen/arch/arm/pci/pci-access.c new file +3 -0
+{
+    int rc;
+    uint32_t val = GENMASK(0, len * 8);

You can just set val = ~0. The return type of the pci_conf_readXX helpers will already truncate the value.

Ack.

Quoted 11 lines

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

Quoted 7 lines

https://patchew.org/Xen/[email protected]/

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

Ok Will integrate that patch once submitted.

Quoted 11 lines

xen/arch/arm/pci/pci-access.c line 40

Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +7 -0
+        return val;
+    }
+
+    if ( unlikely(!bridge->ops->read) )
+        return val;
+
+    rc = bridge->ops->read(bridge, (uint32_t) sbdf.sbdf, reg, len, &val);

There's no need for the uint32_t cast, the sbdf field is already of such type

Ack.

Quoted 57 lines
Quoted 41 lines
xen/arch/arm/pci/pci-access.c new file +41 -0
+    if ( rc )
+        printk(XENLOG_ERR "Failed to read reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+
+    return val;
+}
+
+static void pci_config_write(pci_sbdf_t sbdf, unsigned int reg,
+        unsigned int len, uint32_t val)
+{
+    int rc;
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+        return;
+    }
+
+    if ( unlikely(!bridge->ops->write) )
+        return;
+
+    rc = bridge->ops->write(bridge, (uint32_t) sbdf.sbdf, reg, len, val);
+    if ( rc )
+        printk(XENLOG_ERR "Failed to write reg %#x len %u for "PRI_pci"\n",
+                reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
+}
+
+/*
+ * Wrappers for all PCI configuration access functions.
+ */
+
+#define PCI_OP_WRITE(size, type) \
+    void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \
+{                                               \
+    pci_config_write(sbdf, reg, size / 8, val);     \
+}
+
+#define PCI_OP_READ(size, type) \
+    type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg)  \

xen/arch/arm/pci/pci-access.c line 91

Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+{                                               \
+    return pci_config_read(sbdf, reg, size / 8);     \
+}
+
+PCI_OP_READ(8, u8)
+PCI_OP_READ(16, u16)
+PCI_OP_READ(32, u32)
+PCI_OP_WRITE(8, u8)
+PCI_OP_WRITE(16, u16)
+PCI_OP_WRITE(32, u32)

Please use uintXX_t.

Also, It's nice to add some kind of signals for cscope and friends so they can find the autogenerated functions, ie:

Quoted 7 lines

#define pci_conf_read8
#undef pci_conf_read8
#define pci_conf_read16
#undef pci_conf_read16
...

It's tedious but helps future users find where the code is generated.

Ack.

Quoted 95 lines
Quoted 47 lines
xen/arch/arm/pci/pci-access.c new file +10 -0
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-common.c new file +30 -0
new file mode 100644
index 0000000000..c5f98be698
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+#include <xen/rwlock.h>
+#include <xen/vmap.h>
+

xen/arch/arm/pci/pci-host-common.c line 40

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+/*
+ * List for all the pci host bridges.
+ */
+
+static LIST_HEAD(pci_host_bridges);
+
+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev,
+        struct pci_config_window *cfg)
+{
+    const __be32 *cells;

It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.

Quoted 33 lines
xen/arch/arm/pci/pci-host-common.c new file +33 -0
+    uint32_t len;
+
+    cells = dt_get_property(dev, "bus-range", &len);
+    /* bus-range should at least be 2 cells */
+    if ( !cells || (len < (sizeof(*cells) * 2)) )
+        return false;
+
+    cfg->busn_start = dt_next_cell(1, &cells);
+    cfg->busn_end = dt_next_cell(1, &cells);
+
+    return true;
+}
+
+static inline void __iomem *pci_remap_cfgspace(paddr_t start, size_t len)
+{
+    return ioremap_nocache(start, len);
+}
+
+static void pci_ecam_free(struct pci_config_window *cfg)
+{
+    if ( cfg->win )
+        iounmap(cfg->win);
+
+    xfree(cfg);
+}
+
+static struct pci_config_window *gen_pci_init(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    int err;
+    struct pci_config_window *cfg;
+    paddr_t addr, size;
+
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 79

Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6 -0
+    cfg = xzalloc(struct pci_config_window);
+    if ( !cfg )
+        return NULL;
+
+    err = dt_pci_parse_bus_range(dev, cfg);
+    if ( !err ) {

Braces

Ack

Quoted 41 lines
Quoted 41 lines
xen/arch/arm/pci/pci-host-common.c new file +40 -0
+        cfg->busn_start = 0;
+        cfg->busn_end = 0xff;
+        printk(XENLOG_ERR "No bus range found for pci controller\n");
+    } else {
+        if ( cfg->busn_end > cfg->busn_start + 0xff )
+            cfg->busn_end = cfg->busn_start + 0xff;
+    }
+
+    /* Parse our PCI ecam register address*/
+    err = dt_device_get_address(dev, 0, &addr, &size);
+    if ( err )
+        goto err_exit;
+
+    cfg->phys_addr = addr;
+    cfg->size = size;
+    cfg->ops = ops;
+
+    /*
+     * On 64-bit systems, we do a single ioremap for the whole config space
+     * since we have enough virtual address range available.  On 32-bit, we
+     * ioremap the config space for each bus individually.
+     *
+     * As of now only 64-bit is supported 32-bit is not supported.
+     */
+    cfg->win = pci_remap_cfgspace(cfg->phys_addr, cfg->size);
+    if ( !cfg->win )
+        goto err_exit_remap;
+
+    printk("ECAM at [mem %lx-%lx] for [bus %x-%x] \n",cfg->phys_addr,
+            cfg->phys_addr + cfg->size - 1,cfg->busn_start,cfg->busn_end);
+
+    if ( ops->init ) {
+        err = ops->init(cfg);
+        if (err)
+            goto err_exit;
+    }
+
+    return cfg;
+
+err_exit_remap:
Quoted 8 lines

xen/arch/arm/pci/pci-host-common.c line 126

Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +7 -0
+    printk(XENLOG_ERR "ECAM ioremap failed\n");
+err_exit:
+    pci_ecam_free(cfg);
+    return NULL;
+}
+
+static struct pci_host_bridge * pci_alloc_host_bridge(void)

^ extra space

Ack.

Quoted 82 lines
Quoted 15 lines
xen/arch/arm/pci/pci-host-common.c new file +15 -0
+{
+    struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge);
+
+    if ( !bridge )
+        return NULL;
+
+    INIT_LIST_HEAD(&bridge->node);
+    return bridge;
+}
+
+int pci_host_common_probe(struct dt_device_node *dev,
+        struct pci_ecam_ops *ops)
+{
+    struct pci_host_bridge *bridge;
+    struct pci_config_window *cfg;

xen/arch/arm/pci/pci-host-common.c line 151

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    u32 segment;
+
+    bridge = pci_alloc_host_bridge();
+    if ( !bridge )
+        return -ENOMEM;
+
+    /* Parse and map our Configuration Space windows */
+    cfg = gen_pci_init(dev, ops);
+    if ( !cfg )
+        return -ENOMEM;

You are leaking the allocation of bridge here ...

xen/arch/arm/pci/pci-host-common.c line 160

Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +9 -0
+
+    bridge->dt_node = dev;
+    bridge->sysdata = cfg;
+    bridge->ops = &ops->pci_ops;
+
+    if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) )
+    {
+        printk(XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n");
+        return -ENODEV;

... and here.

Quoted 18 lines
xen/arch/arm/pci/pci-host-common.c new file +18 -0
+    }
+
+    bridge->segment = (u16)segment;
+
+    list_add_tail(&bridge->node, &pci_host_bridges);
+
+    return 0;
+}
+
+/*
+ * This function will lookup an hostbridge based on the segment and bus
+ * number.
+ */
+struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
+{
+    struct pci_host_bridge *bridge;
+    bool found = false;
+

xen/arch/arm/pci/pci-host-common.c line 188

Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+    list_for_each_entry( bridge, &pci_host_bridges, node )
+    {
+        if ( bridge->segment != segment )
+            continue;
+
+        found = true;
+        break;
+    }
+
+    return (found) ? bridge : NULL;

This can be much shorter:

struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)

{
   struct pci_host_bridge *bridge;

   list_for_each_entry( bridge, &pci_host_bridges, node )
       if ( bridge->segment == segment )
           return bridge;

   return NULL;
}
Quoted 2 lines

Albeit I'm confused by the fact that you pass a bus number that's completely unused.

Ok Will fix.

Quoted 52 lines
Quoted 37 lines
xen/arch/arm/pci/pci-host-common.c new file +10 -0
+}
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci-host-generic.c new file +20 -0
new file mode 100644
index 0000000000..cd67b3dec6
@@ -0,0 +1,131 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+

xen/arch/arm/pci/pci-host-generic.c line 30

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10 -0
+#include <asm/device.h>
+#include <asm/io.h>
+#include <xen/pci.h>
+#include <asm/pci.h>
+
+/*
+ * Function to get the config space base.
+ */
+static void __iomem *pci_config_base(struct pci_host_bridge *bridge,
+        uint32_t sbdf, int where)

You would be better passing a pci_sbdf_t directly here. Also 'where' should be renamed to offset, or reg, and be made unsigned int. AFAICT you will never pass a negative value here.

Quoted 3 lines

For sanity you should also assert that the offset falls between the PCI config space used by the device, in order to easily catch with wrong offsets being used.

Ack.

Quoted 5 lines

xen/arch/arm/pci/pci-host-generic.c line 32

Quoted 3 lines
xen/arch/arm/pci/pci-host-generic.c new file +2 -0
+{
+    struct pci_config_window *cfg = bridge->sysdata;

const

Ack.

Quoted 10 lines

xen/arch/arm/pci/pci-host-generic.c line 38

Quoted 7 lines
xen/arch/arm/pci/pci-host-generic.c new file +6 -0
+    unsigned int devfn_shift = cfg->ops->bus_shift - 8;
+
+    pci_sbdf_t sbdf_t = (pci_sbdf_t) sbdf ;
+
+    unsigned int busn = sbdf_t.bus;
+    void __iomem *base;

IMO adding newlines between variable definitions is not helpful, but that's my taste.

Ack.

Quoted 10 lines
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +9 -0
+
+    if ( busn < cfg->busn_start || busn > cfg->busn_end )
+        return NULL;
+
+    base = cfg->win + (busn << cfg->ops->bus_shift);
+
+    return base + (PCI_DEVFN(sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where;
+}
+
Quoted 8 lines

xen/arch/arm/pci/pci-host-generic.c line 53

Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6 -0
+int pci_ecam_config_write(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);

You can initialize at definition.

Ack.

Quoted 11 lines

xen/arch/arm/pci/pci-host-generic.c line 62

Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +9 -0
+    if ( !addr )
+        return -ENODEV;
+
+    if ( size == 1 )
+        writeb(val, addr);
+    else if ( size == 2 )
+        writew(val, addr);
+    else
+        writel(val, addr);
Quoted 2 lines

Please use a switch, and check against specific values. The default case should be a BUG();. See pci_conf_read from x86 for an example.

Ack.

Quoted 161 lines
Quoted 147 lines
xen/arch/arm/pci/pci-host-generic.c new file +69 -0
+
+    return 0;
+}
+
+int pci_ecam_config_read(struct pci_host_bridge *bridge, uint32_t sbdf,
+        int where, int size, u32 *val)
+{
+    void __iomem *addr;
+
+    addr = pci_config_base(bridge, sbdf, where);
+    if ( !addr ) {
+        *val = ~0;
+        return -ENODEV;
+    }
+
+    if ( size == 1 )
+        *val = readb(addr);
+    else if ( size == 2 )
+        *val = readw(addr);
+    else
+        *val = readl(addr);
+
+    return 0;
+}
+
+/* ECAM ops */
+struct pci_ecam_ops pci_generic_ecam_ops = {
+    .bus_shift  = 20,
+    .pci_ops    = {
+        .read       = pci_ecam_config_read,
+        .write      = pci_ecam_config_write,
+    }
+};
+
+static const struct dt_device_match gen_pci_dt_match[] = {
+    { .compatible = "pci-host-ecam-generic",
+      .data =       &pci_generic_ecam_ops },
+
+    { },
+};
+
+static int gen_pci_dt_init(struct dt_device_node *dev, const void *data)
+{
+    const struct dt_device_match *of_id;
+    struct pci_ecam_ops *ops;
+
+    of_id = dt_match_node(gen_pci_dt_match, dev->dev.of_node);
+    ops = (struct pci_ecam_ops *) of_id->data;
+
+    printk(XENLOG_INFO "Found PCI host bridge %s compatible:%s \n",
+            dt_node_full_name(dev), of_id->compatible);
+
+    return pci_host_common_probe(dev, ops);
+}
+
+DT_DEVICE_START(pci_gen, "PCI HOST GENERIC", DEVICE_PCI)
+.dt_match = gen_pci_dt_match,
+.init = gen_pci_dt_init,
+DT_DEVICE_END
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/pci/pci.c new file +71 -0
new file mode 100644
index 0000000000..f8cbb99591
@@ -0,0 +1,112 @@
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/acpi.h>
+#include <xen/device_tree.h>
+#include <xen/errno.h>
+#include <xen/init.h>
+#include <xen/pci.h>
+#include <xen/param.h>
+
+static int __init dt_pci_init(void)
+{
+    struct dt_device_node *np;
+    int rc;
+
+    dt_for_each_device_node(dt_host, np)
+    {
+        rc = device_init(np, DEVICE_PCI, NULL);
+        if( !rc )
+            continue;
+        /*
+         * Ignore the following error codes:
+         *   - EBADF: Indicate the current is not an pci
+         *   - ENODEV: The pci device is not present or cannot be used by
+         *     Xen.
+         */
+        else if ( rc != -EBADF && rc != -ENODEV )
+        {
+            printk(XENLOG_ERR "No driver found in XEN or driver init error.\n");
+            return rc;
+        }
+    }
+
+    return 0;
+}
+
+#ifdef CONFIG_ACPI
+static void __init acpi_pci_init(void)
+{
+    printk(XENLOG_ERR "ACPI pci init not supported \n");
+    return;
+}
+#else
+static inline void __init acpi_pci_init(void) { }
+#endif
+
+static bool __initdata param_pci_enable;
+static int __init parse_pci_param(const char *arg)
+{
+    if ( !arg )
+    {
+        param_pci_enable = false;
+        return 0;
+    }
+
+    switch ( parse_bool(arg, NULL) )
+    {
+        case 0:

xen/arch/arm/pci/pci.c line 81

Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10 -0
+            param_pci_enable = false;
+            return 0;
+        case 1:
+            param_pci_enable = true;
+            return 0;
+    }
+
+    return -EINVAL;
+}
+custom_param("pci", parse_pci_param);

You need to introduce the documentation for the parameter at docs/misc/xen-command-line.pandoc

Quoted 4 lines

Albeit I'm not sure I like it, why do you need to enable PCI explicitly?

Shouldn't it be discovered automatically and enabled by default?

Ack.

Quoted 11 lines

xen/arch/arm/pci/pci.c line 89

Quoted 8 lines
xen/arch/arm/pci/pci.c new file +7 -0
+void __init pci_init(void)
+{
+    /*
+     * Enable PCI when has been enabled explicitly (pci=on)
+     */
+    if ( !param_pci_enable)
+        goto disable;

Just return here, there's not point in having a label to perform a return.

Ack.

Quoted 114 lines

xen/arch/arm/pci/pci.c line 94

Quoted 6 lines
xen/arch/arm/pci/pci.c new file +5 -0
+
+    if ( acpi_disabled )
+        dt_pci_init();
+    else
+        acpi_pci_init();

Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?

This if .. else seems fragile.

Also for ACPI you will get called by acpi_boot_init, and likely need to implement a acpi_mmcfg_init or pci_mmcfg_arch_{init,enable}. I'm not sure whether the code in acpi_mmcfg_init could be made shared between both x86 and Arm.

Quoted 53 lines
xen/arch/arm/pci/pci.c new file +18 -0
+
+#ifdef CONFIG_HAS_PCI
+    pci_segments_init();
+#endif
+
+disable:
+    return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/arch/arm/setup.c +2 -0
index 7968cee47d..2d7f1db44f 100644
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
 
    setup_virt_paging();
 
+    pci_init();
+
    do_initcalls();
 
    /*
xen/include/asm-arm/device.h +0 -0
index ee7cff2d44..28f8049cfd 100644
@@ -4,6 +4,7 @@

enum device_type

{
    DEV_DT,

+ DEV_PCI,
};

struct dev_archdata {

xen/include/asm-arm/device.h +2 -3
@@ -25,15 +26,15 @@ typedef struct device device_t;
 
 #include <xen/device_tree.h>
 
-/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */
-#define dev_is_pci(dev) ((void)(dev), 0)
-#define dev_is_dt(dev)  ((dev->type == DEV_DT)
+#define dev_is_pci(dev) (dev->type == DEV_PCI)
+#define dev_is_dt(dev)  (dev->type == DEV_DT)
 
Quoted 6 lines

enum device_class
{
DEVICE_SERIAL,
DEVICE_IOMMU,
DEVICE_GIC,
+ DEVICE_PCI,

It seems to be like this wants to be DEVICE_PCI_HOST_BRIDGE or some such, since this is not used to identify all PCI devices, but just bridges?

Quoted 33 lines
    /* Use for error */
    DEVICE_UNKNOWN,
};
xen/include/asm-arm/pci.h +22 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
Quoted 15 lines

xen/include/asm-arm/pci.h line 36

Quoted 13 lines
xen/include/asm-arm/pci.h +12 -0
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

This seems to be completely unused?

Will be using going forward.

Quoted 59 lines
Quoted 6 lines
xen/include/asm-arm/pci.h +5 -0
+
+#define PRI_pci "%04x:%02x:%02x.%u"
+#define pci_to_dev(pcidev) (&(pcidev)->arch.dev)
+
+/*

xen/include/asm-arm/pci.h line 51

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+ * struct to hold the mappings of a config space window. This
+ * is expected to be used as sysdata for PCI controllers that
+ * use ECAM.
+ */
+struct pci_config_window {
+    paddr_t     phys_addr;
+    paddr_t     size;
+    uint8_t     busn_start;
+    uint8_t     busn_end;
+    struct pci_ecam_ops     *ops;

const?

Quoted 23 lines
xen/include/asm-arm/pci.h +23 -0
+    void __iomem        *win;
+};
+
+/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */
+struct pci_host_bridge;
+
+struct pci_ops {
+    int (*read)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 *val);
+    int (*write)(struct pci_host_bridge *bridge,
+                    uint32_t sbdf, int where, int size, u32 val);
+};
+
+/*
+ * struct to hold pci ops and bus shift of the config window
+ * for a PCI controller.
+ */
+struct pci_ecam_ops {
+    unsigned int            bus_shift;
+    struct pci_ops          pci_ops;
+    int             (*init)(struct pci_config_window *);
+};
+

xen/include/asm-arm/pci.h line 84

Quoted 10 lines
xen/include/asm-arm/pci.h +10 -0
+/*
+ * struct to hold pci host bridge information
+ * for a PCI controller.
+ */
+struct pci_host_bridge {
+    struct dt_device_node *dt_node;  /* Pointer to the associated DT node */
+    struct list_head node;           /* Node in list of host bridges */
+    uint16_t segment;                /* Segment number */
+    void *sysdata;                   /* Pointer to the config space window*/
+    const struct pci_ops *ops;

You seem to introduce a lot of ops structs, yet there's only one implementation the generic ECAM one, and adding such complexity should IMO be done when further implementations are added. Also given this is a fully ECAM compliant bridge you could just use most of the existing logic for x86?

Quoted 7 lines

I understand the discovery needs to be different, but x86 MCFG logic should already be capable of handling multiple ECAM regions.

I also agree with Julien that splitting this into separate patch would make it easier to review. For example you can start with the discovery logic, followed by the initialization and the add the accessors to the config space lastly.

Ok . Will spilt the patches in the next version of the patch series.

Quoted 2 lines

Thanks, Roger.

#32 lore raw

Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

Quoted 18 lines · Rahul Singh
Quoted 18 lines · Roger Pau Monné

xen/arch/arm/pci/pci-access.c line 33

Quoted 7 lines · Rahul Singh
xen/arch/arm/pci/pci-access.c new file +7 -0
+
+    struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus);
+
+    if ( unlikely(!bridge) )
+    {
+        printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n",
+                sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);

I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:

https://patchew.org/Xen/[email protected]/

Quoted 7 lines
Quoted 5 lines

It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).

Ok Will integrate that patch once submitted.

I've posted an updated version to the list yesterday:

https://lore.kernel.org/xen-devel/[email protected]/

Quoted 46 lines
Quoted 30 lines
Quoted 30 lines
xen/include/asm-arm/pci.h +22 -2
index de13359f65..94fd00360a 100644
@@ -1,7 +1,98 @@
-#ifndef __X86_PCI_H__
-#define __X86_PCI_H__
+/*
+ * Copyright (C) 2020 Arm Ltd.
+ *
+ * Based on Linux drivers/pci/ecam.c
+ * Copyright 2016 Broadcom.
+ *
+ * Based on Linux drivers/pci/controller/pci-host-common.c
+ * Based on Linux drivers/pci/controller/pci-host-generic.c
+ * Copyright (C) 2014 ARM Limited Will Deacon <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://secure-web.cisco.com/1YIShdE6aU0WIMsRjhIYYGdRo_H_wyOVeLqk0ITHI7i_FMIXuMEmV9Y6lR76qCBB4XHss0ba_WMfQa0mLGD37bM1S3AFMBVC1WUEF0LmbURPWld2kU2zuyOamUxqnYwy4ZAv4N2EUJy7HCgDYn2s183YCQ-5YBc98vH8TO-tfBgEShkKVzAm23AgzIqikuKN_BIhcQhAbsjypr9ffeNZDsSX2jC8ClqegEPXFsCWJldrASGgWoR16rMjI-INfPNy55m9nGP5UmHIYWBmTpnHLjXtqCkruENixh20vPWIUROhcjjtzVBrR8d-Q5HnJy0hSR97WdlUSAryfxeH-8VxcPw/http%3A%2F%2Fwww.gnu.org%2Flicenses%2F>.
+ */
 
Quoted 15 lines

xen/include/asm-arm/pci.h line 36

Quoted 13 lines
xen/include/asm-arm/pci.h +12 -0
+#ifndef __ARM_PCI_H__
+#define __ARM_PCI_H__
+
+#include <xen/pci.h>
+#include <xen/device_tree.h>
+#include <asm/device.h>
+
+#ifdef CONFIG_ARM_PCI
+
+/* Arch pci dev struct */
 struct arch_pci_dev {
+    struct device dev;
+};

This seems to be completely unused?

Quoted 1 line

Will be using going forward.

Please introduce it when it's required then.

Thanks, Roger.

#33 lore raw

RFC PATCH 2/4 xen/arm: Discovering PCI devices and add the PCI devices in XEN.

In reply to Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id Ie87e19741689503b4b62da911c8dc2ee318584ac
---
 xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +39 -3
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }
+
+    return ret;
 }
 
 /*
Signature · 1 line
2.17.1
#34 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

Quoted 46 lines
---
 xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +27 -3
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

xen/common/sysctl.c:do_sysctl

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Jan, Andrew, Roger, any opinions?

#35 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

Quoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

Quoted 17 lines
---
  xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
  1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +2 -1
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
  #include <xen/errno.h>
  #include <xen/sched.h>
  #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
Quoted 9 lines

xen/arch/arm/physdev.c line 21

Quoted 9 lines
xen/arch/arm/physdev.c +5 -2
  int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
  {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI

In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".

Is this still valid?

Quoted 36 lines
Quoted 20 lines
xen/arch/arm/physdev.c +20 -0
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

xen/common/sysctl.c:do_sysctl

Quoted 9 lines

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Jan, Andrew, Roger, any opinions?

I think we can also have a look at [1] by Julien. That implementation,

IMO, had some thoughts on making Arm/x86 code common where possible

[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci

#36 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

Hi,

Quoted 84 lines · Oleksandr Andrushchenko
Quoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

Quoted 17 lines
---
   xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
   1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +2 -1
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
   #include <xen/errno.h>
   #include <xen/sched.h>
   #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
Quoted 9 lines

xen/arch/arm/physdev.c line 21

Quoted 9 lines
xen/arch/arm/physdev.c +5 -2
   int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
   {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI

In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".

Is this still valid?

Quoted 42 lines
Quoted 20 lines
xen/arch/arm/physdev.c +20 -0
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

xen/common/sysctl.c:do_sysctl

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Quoted 6 lines
Quoted 3 lines

Jan, Andrew, Roger, any opinions?

I think we can also have a look at [1] by Julien. That implementation,

IMO, had some thoughts on making Arm/x86 code common where possible

There are some ideas on how I would like to see the split, although they need some cleanup. :)

In particular, I was expecting some preparatory work to get the existing PCI code non-x86 specific.

Regarding the hypercall Stefano mentionned, I think it should be possible to make it common if we abstract the NUMA node lookup part.

Cheers,

Quoted 4 lines

[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci

Signature · 1 line
Julien Grall
#37 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

In reply to Oleksandr Andrushchenko
Quoted 44 lines · Oleksandr Andrushchenko
Quoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

Quoted 17 lines
---
 xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +2 -1
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
Quoted 9 lines

xen/arch/arm/physdev.c line 21

Quoted 9 lines
xen/arch/arm/physdev.c +5 -2
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI

In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".

Quoted 1 line

Is this still valid?

Yes right we are not enabling it because full support of PCI passthrough is not implemented and tested.

Quoted 43 lines
Quoted 43 lines
Quoted 21 lines
xen/arch/arm/physdev.c +20 -0
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

xen/common/sysctl.c:do_sysctl

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Quoted 6 lines
Quoted 3 lines

Jan, Andrew, Roger, any opinions?

I think we can also have a look at [1] by Julien. That implementation,

IMO, had some thoughts on making Arm/x86 code common where possible

Ok. Thanks for the pointer. We will have a look.

Quoted 3 lines

[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci

#38 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

In reply to Stefano Stabellini
Quoted 73 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

Quoted 46 lines
---
 xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +27 -3
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

xen/common/sysctl.c:do_sysctl

Quoted 7 lines

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Jan, Andrew, Roger, any opinions?

Oh, physdev ops don't have a common entry point, it's all per-arch. Since Arm has no physdev ops at all, I think we should start by adding a common do_physdev_op and move PHYSDEVOP_pci_device_add into it, leaving the rest of x86 operations as arch_do_physdev_op.

Thanks, Roger.

#39 lore raw

Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.

In reply to Stefano Stabellini

Sorry for the late reply.

Quoted 10 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh

Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.

Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac

Same question about Change-Id

I think by-mistake Gerrit Change-id is added in the patch series. I will remove the Change-Id in next version of the patch.

Quoted 63 lines
Quoted 48 lines

---
xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++---

1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +27 -3
index e91355fe22..274720f98a 100644
@@ -9,12 +9,48 @@
 #include <xen/errno.h>
 #include <xen/sched.h>
 #include <asm/hypercall.h>
-
+#include <xen/guest_access.h>
+#include <xsm/xsm.h>
 
 int do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
-    gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
-    return -ENOSYS;
+    int ret = 0;
+
+    switch ( cmd )
+    {
+#ifdef CONFIG_HAS_PCI
+        case PHYSDEVOP_pci_device_add:
+            {
+                struct physdev_pci_device_add add;
+                struct pci_dev_info pdev_info;
+                nodeid_t node = NUMA_NO_NODE;
+
+                ret = -EFAULT;
+                if ( copy_from_guest(&add, arg, 1) != 0 )
+                    break;
+
+                pdev_info.is_extfn = !!(add.flags & XEN_PCI_DEV_EXTFN);
+                if ( add.flags & XEN_PCI_DEV_VIRTFN )
+                {
+                    pdev_info.is_virtfn = 1;
+                    pdev_info.physfn.bus = add.physfn.bus;
+                    pdev_info.physfn.devfn = add.physfn.devfn;
+                }
+                else
+                    pdev_info.is_virtfn = 0;
+

xen/arch/arm/physdev.c line 51

Quoted 10 lines
xen/arch/arm/physdev.c +10 -0
+                ret = pci_add_device(add.seg, add.bus, add.devfn,
+                                &pdev_info, node);
+
+                break;
+            }
+#endif
+        default:
+            gdprintk(XENLOG_DEBUG, "PHYSDEVOP cmd=%d: not implemented\n", cmd);
+            ret = -ENOSYS;
+    }

I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:

Quoted 7 lines

xen/common/sysctl.c:do_sysctl

and

xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl

Ok sure I will check if we can create a common entry for ARM and x86 for do_physdev_op().

Quoted 1 line

Jan, Andrew, Roger, any opinions?

#40 lore raw

RFC PATCH 3/4 xen/arm: Enable the existing x86 virtual PCI support for ARM.

In reply to Rahul Singh

The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.

A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.

VPCI MSI support is disable for ARM as it is not tested on ARM.

Change-Id I5501db2781f8064640403fecce53713091cd9ab4
---
 xen/arch/arm/Makefile         |   1 +
 xen/arch/arm/domain.c         |   4 ++
 xen/arch/arm/vpci.c           | 102 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpci.h           |  37 ++++++++++++
 xen/drivers/passthrough/pci.c |   7 +++
 xen/include/asm-arm/domain.h  |   5 ++
 xen/include/public/arch-arm.h |   4 ++
 7 files changed, 160 insertions(+)
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
xen/arch/arm/Makefile +1 -0
index 345cb83eed..5a23ec5cc0 100644
@@ -7,6 +7,7 @@ obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
 obj-$(CONFIG_ARM_PCI) += pci/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/domain.c +4 -0
index 31169326b2..23098ffd02 100644
@@ -39,6 +39,7 @@
 #include <asm/vtimer.h>
 
 #include "vuart.h"
+#include "vpci.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
     if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
         goto fail;
 
+    if ( (rc = domain_vpci_init(d)) != 0 )
+        goto fail;
+
     return 0;
 
 fail:
xen/arch/arm/vpci.c new file +102 -0

102 lines not on this page · raw message

xen/arch/arm/vpci.h new file +37 -0
new file mode 100644
index 0000000000..20dce1f4c4
@@ -0,0 +1,37 @@
+/*
+ * xen/arch/arm/vpci.h
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+    return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/drivers/passthrough/pci.c +7 -0
index 5846978890..28511eb641 100644
@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     else
         iommu_enable_device(pdev);
 
+#ifdef CONFIG_ARM
+    ret = vpci_add_handlers(pdev);
+    if ( ret ) {
+        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
+        goto out;
+    }
+#endif
     pci_enable_acs(pdev);
 
 out:
xen/include/asm-arm/domain.h +5 -0
index 4e2f582006..ad70610226 100644
@@ -34,6 +34,11 @@ enum domain_type {
 /* The hardware domain has always its memory direct mapped. */
 #define is_domain_direct_mapped(d) ((d) == hardware_domain)
 
+/* For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) (true)
+
 struct vtimer {
     struct vcpu *v;
     int irq;
xen/include/public/arch-arm.h +4 -0
index c365b1b39e..7364a07362 100644
@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
 #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
 #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
 
+/* VPCI ECAM mappings */
+#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
+
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
Signature · 1 line
2.17.1
#41 lore raw

Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.

Quoted 5 lines · Rahul Singh

The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.

Quoted 8 lines

A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.

VPCI MSI support is disable for ARM as it is not tested on ARM.

Change-Id: I5501db2781f8064640403fecce53713091cd9ab4

Same question

Quoted 119 lines
---
 xen/arch/arm/Makefile         |   1 +
 xen/arch/arm/domain.c         |   4 ++
 xen/arch/arm/vpci.c           | 102 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpci.h           |  37 ++++++++++++
 xen/drivers/passthrough/pci.c |   7 +++
 xen/include/asm-arm/domain.h  |   5 ++
 xen/include/public/arch-arm.h |   4 ++
 7 files changed, 160 insertions(+)
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
xen/arch/arm/Makefile +1 -0
index 345cb83eed..5a23ec5cc0 100644
@@ -7,6 +7,7 @@ obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
 obj-$(CONFIG_ARM_PCI) += pci/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/domain.c +4 -0
index 31169326b2..23098ffd02 100644
@@ -39,6 +39,7 @@
 #include <asm/vtimer.h>
 
 #include "vuart.h"
+#include "vpci.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
     if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
         goto fail;
 
+    if ( (rc = domain_vpci_init(d)) != 0 )
+        goto fail;
+
     return 0;
 
 fail:
xen/arch/arm/vpci.c new file +66 -0
new file mode 100644
index 0000000000..49e473ab0d
@@ -0,0 +1,102 @@
+/*
+ * xen/arch/arm/vpci.c
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * Based on arch/x86/hvm/io.c
+ * Copyright (c) 2004, Intel Corporation.
+ * Copyright (c) 2005, International Business Machines Corporation.
+ * Copyright (c) 2008, Citrix Systems, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+#include <asm/mmio.h>
+
+/* Do some sanity checks. */
+static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)
+{
+    /* Check access size. */
+    if ( len != 1 && len != 2 && len != 4 && len != 8 )
+        return false;
+
+    /* Check that access is size aligned. */
+    if ( (reg & (len - 1)) )
+        return false;
+
+    return true;
+}
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+        register_t *r, void *priv)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = 0;
+    unsigned int size = 1U << info->dabt.size;
+
+    sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    data = vpci_read(sbdf, reg, size);
+
+    memcpy(r, &data, size);
+
+    return 1;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+        register_t r, void *priv)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = r;
+    unsigned int size = 1U << info->dabt.size;
+

xen/arch/arm/vpci.c line 76

Quoted 10 lines
xen/arch/arm/vpci.c new file +10 -0
+    sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    vpci_write(sbdf, reg, size, data);
+
+    return 1;
+}

I wonder if we could share vpci_mmcfg_read/write. Again, it is OK if we can't, or if it is not worth the effort. just want to make sure we thought about it :-)

Quoted 91 lines
xen/arch/arm/vpci.c new file +25 -0
+static const struct mmio_handler_ops vpci_mmio_handler = {
+    .read  = vpci_mmio_read,
+    .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)
+{
+    if ( !has_vpci(d) || is_hardware_domain(d) )
+        return 0;
+
+    register_mmio_handler(d, &vpci_mmio_handler,
+            GUEST_VPCI_ECAM_BASE,GUEST_VPCI_ECAM_SIZE,NULL);
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
xen/arch/arm/vpci.h new file +37 -0
new file mode 100644
index 0000000000..20dce1f4c4
@@ -0,0 +1,37 @@
+/*
+ * xen/arch/arm/vpci.h
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+    return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/drivers/passthrough/pci.c +7 -0
index 5846978890..28511eb641 100644
@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     else
         iommu_enable_device(pdev);
 
+#ifdef CONFIG_ARM
+    ret = vpci_add_handlers(pdev);
+    if ( ret ) {
+        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
+        goto out;
+    }
+#endif
     pci_enable_acs(pdev);
 
 out:
xen/include/asm-arm/domain.h +0 -0
index 4e2f582006..ad70610226 100644

xen/include/asm-arm/domain.h enum domain_type { · line 40

Quoted 7 lines
xen/include/asm-arm/domain.h +4 -0
@@ -34,6 +34,11 @@ enum domain_type {
 /* The hardware domain has always its memory direct mapped. */
 #define is_domain_direct_mapped(d) ((d) == hardware_domain)
 
+/* For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) (true)

As mentioned we could make this configurabled based on the presence of pci=[] or something similar in device tree for dom0less guests.

Quoted 8 lines
xen/include/asm-arm/domain.h +0 -0
 struct vtimer {
     struct vcpu *v;
     int irq;
xen/include/public/arch-arm.h +0 -0
index c365b1b39e..7364a07362 100644

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 427

Quoted 6 lines
xen/include/public/arch-arm.h +3 -0
@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;
 #define GUEST_PL011_BASE    xen_mk_ullong(0x22000000)
 #define GUEST_PL011_SIZE    xen_mk_ullong(0x00001000)
 
+/* VPCI ECAM mappings */
+#define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
+#define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)

Is 256MB in size part of the ECAM standard?

Quoted 6 lines
xen/include/public/arch-arm.h +0 -0
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
Signature · 1 line
2.17.1
#42 lore raw

Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.

In reply to Rahul Singh
Quoted 11 lines · Rahul Singh

The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.

A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.

VPCI MSI support is disable for ARM as it is not tested on ARM.

I'm not seeing anything on this patch that would disable vPCI MSI support?

Quoted 70 lines
Change-Id I5501db2781f8064640403fecce53713091cd9ab4
---
 xen/arch/arm/Makefile         |   1 +
 xen/arch/arm/domain.c         |   4 ++
 xen/arch/arm/vpci.c           | 102 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/vpci.h           |  37 ++++++++++++
 xen/drivers/passthrough/pci.c |   7 +++
 xen/include/asm-arm/domain.h  |   5 ++
 xen/include/public/arch-arm.h |   4 ++
 7 files changed, 160 insertions(+)
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
xen/arch/arm/Makefile +1 -0
index 345cb83eed..5a23ec5cc0 100644
@@ -7,6 +7,7 @@ obj-y += platforms/
 endif
 obj-$(CONFIG_TEE) += tee/
 obj-$(CONFIG_ARM_PCI) += pci/
+obj-$(CONFIG_HAS_VPCI) += vpci.o
 
 obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
 obj-y += bootfdt.init.o
xen/arch/arm/domain.c +4 -0
index 31169326b2..23098ffd02 100644
@@ -39,6 +39,7 @@
 #include <asm/vtimer.h>
 
 #include "vuart.h"
+#include "vpci.h"
 
 DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
 
@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,
     if ( is_hardware_domain(d) && (rc = domain_vuart_init(d)) )
         goto fail;
 
+    if ( (rc = domain_vpci_init(d)) != 0 )
+        goto fail;
+
     return 0;
 
 fail:
xen/arch/arm/vpci.c new file +15 -0
new file mode 100644
index 0000000000..49e473ab0d
@@ -0,0 +1,102 @@
+/*
+ * xen/arch/arm/vpci.c
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * Based on arch/x86/hvm/io.c
+ * Copyright (c) 2004, Intel Corporation.
+ * Copyright (c) 2005, International Business Machines Corporation.
+ * Copyright (c) 2008, Citrix Systems, Inc.
+ *
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *

xen/arch/arm/vpci.c line 25

Quoted 10 lines
xen/arch/arm/vpci.c new file +10 -0
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+#include <xen/sched.h>
+#include <asm/mmio.h>
+
+/* Do some sanity checks. */
+static bool vpci_mmio_access_allowed(unsigned int reg, unsigned int len)

This is just a copy of vpci_access_allowed from x86, I think you should consider moving the function to common vpci code and just share it between x86 and Arm?

Quoted 40 lines
xen/arch/arm/vpci.c new file +40 -0
+{
+    /* Check access size. */
+    if ( len != 1 && len != 2 && len != 4 && len != 8 )
+        return false;
+
+    /* Check that access is size aligned. */
+    if ( (reg & (len - 1)) )
+        return false;
+
+    return true;
+}
+
+static int vpci_mmio_read(struct vcpu *v, mmio_info_t *info,
+        register_t *r, void *priv)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = 0;
+    unsigned int size = 1U << info->dabt.size;
+
+    sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    data = vpci_read(sbdf, reg, size);
+
+    memcpy(r, &data, size);
+
+    return 1;
+}
+
+static int vpci_mmio_write(struct vcpu *v, mmio_info_t *info,
+        register_t r, void *priv)
+{
+    unsigned int reg;
+    pci_sbdf_t sbdf;
+    uint32_t data = r;
+    unsigned int size = 1U << info->dabt.size;

xen/arch/arm/vpci.c line 75

Quoted 10 lines
xen/arch/arm/vpci.c new file +10 -0
+
+    sbdf.bdf = (((info->gpa) & 0x0ffff000) >> 12);
+    reg = (((info->gpa) & 0x00000ffc) | (info->gpa & 3));
+
+    if ( !vpci_mmio_access_allowed(reg, size) )
+        return 1;
+
+    vpci_write(sbdf, reg, size, data);
+
+    return 1;

Both functions will only return 1 always, so can likely drop the return value completely?

xen/arch/arm/vpci.c line 83

Quoted 8 lines
xen/arch/arm/vpci.c new file +8 -0
+}
+
+static const struct mmio_handler_ops vpci_mmio_handler = {
+    .read  = vpci_mmio_read,
+    .write = vpci_mmio_write,
+};
+
+int domain_vpci_init(struct domain *d)

FWIW, I think you can drop the domain_ prefix, vPCI is always tied to a domain.

xen/arch/arm/vpci.c line 85

Quoted 2 lines
xen/arch/arm/vpci.c new file +2 -0
+{
+    if ( !has_vpci(d) || is_hardware_domain(d) )

I wouldn't add a hardware domain exception here, and just make sure the VPCI flag is not set for the hardware domain on Arm if you don't want to use it there.

xen/arch/arm/vpci.c line 89

Quoted 4 lines
xen/arch/arm/vpci.c new file +4 -0
+        return 0;
+
+    register_mmio_handler(d, &vpci_mmio_handler,
+            GUEST_VPCI_ECAM_BASE,GUEST_VPCI_ECAM_SIZE,NULL);

Missing spaces, and proper indentation.

xen/arch/arm/vpci.c line 91

Quoted 2 lines
xen/arch/arm/vpci.c new file +2 -0
+
+    return 0;

Doesn't seem like domain_vpci_init can fail, so you can likely skip the return value?

Quoted 59 lines
xen/arch/arm/vpci.c new file +11 -0
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
+
xen/arch/arm/vpci.h new file +37 -0
new file mode 100644
index 0000000000..20dce1f4c4
@@ -0,0 +1,37 @@
+/*
+ * xen/arch/arm/vpci.h
+ * Copyright (c) 2020 Arm Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __ARCH_ARM_VPCI_H__
+#define __ARCH_ARM_VPCI_H__
+
+#ifdef CONFIG_HAS_VPCI
+int domain_vpci_init(struct domain *d);
+#else
+static inline int domain_vpci_init(struct domain *d)
+{
+    return 0;
+}
+#endif
+
+#endif /* __ARCH_ARM_VPCI_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/drivers/passthrough/pci.c +0 -0
index 5846978890..28511eb641 100644

xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 808

Quoted 5 lines
xen/drivers/passthrough/pci.c +2 -0
@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,
     else
         iommu_enable_device(pdev);
 
+#ifdef CONFIG_ARM
+    ret = vpci_add_handlers(pdev);

Don't you need to drop the __hwdom_init from that function? Or else it might be freed by the time dom0 calls pci_add_device?

Quoted 13 lines
xen/drivers/passthrough/pci.c +5 -0
+    if ( ret ) {
+        printk(XENLOG_ERR "setup of vPCI for failed: %d\n",ret);
+        goto out;
+    }
+#endif
     pci_enable_acs(pdev);
 
 out:
xen/include/asm-arm/domain.h +0 -0
index 4e2f582006..ad70610226 100644

xen/include/asm-arm/domain.h enum domain_type { · line 40

Quoted 7 lines
xen/include/asm-arm/domain.h +4 -0
@@ -34,6 +34,11 @@ enum domain_type {
 /* The hardware domain has always its memory direct mapped. */
 #define is_domain_direct_mapped(d) ((d) == hardware_domain)
 
+/* For X86 VPCI is enabled and tested for PVH DOM0 only but
+ * for ARM we enable support VPCI for guest domain also.
+ */
+#define has_vpci(d) (true)

Urg, that's kind of unconditional, couldn't you pass a flag for user-space in order to signal whether vPCI should be enabled? There's no point on enabling it if the domain doesn't support PCI passthrough, or if there are no PCI devices on the system.

Roger.

#43 lore raw

RFC PATCH 4/4 arm/libxl: Emulated PCI device tree node in libxl

In reply to Rahul Singh

libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.

We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.

A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).

Change-Id I47d39cbe8184de2226f174644df9790ecc610ccd
---
 tools/libxl/libxl_arm.c       | 200 ++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl   |   6 +
 tools/xl/xl_parse.c           |   7 ++
 xen/include/public/arch-arm.h |  28 +++++
 4 files changed, 241 insertions(+)
tools/libxl/libxl_arm.c +200 -0
index 34f8a29056..84568e9dc9 100644
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
     return fdt_property(fdt, "reg", regs, sizeof(regs));
 }
 
+static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t bus_range[num_cells];
+    be32 *cells = &bus_range[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
+}
+
+static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t interrupt_map_mask[num_cells];
+    be32 *cells = &interrupt_map_mask[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
+                                sizeof(interrupt_map_mask));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+        unsigned vpci_addr_cells,
+        unsigned cpu_addr_cells,
+        unsigned vpci_size_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
+    be32 *cells = &regs[0];
+    int i;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the memory bit field */
+        arg = va_arg(ap, uint64_t);
+        set_cell(&cells, 1, arg);
+
+        /* Set the vpci bus address */
+        arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, 2 , arg);
+
+        /* Set the cpu bus address where vpci address is mapped */
+        arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, cpu_addr_cells, arg);
+
+        /* Set the vpci size requested */
+        arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, vpci_size_cells,arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
+static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
+        unsigned child_unit_addr_cells,
+        unsigned child_interrupt_specifier_cells,
+        unsigned parent_unit_addr_cells,
+        unsigned parent_interrupt_specifier_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
+            child_interrupt_specifier_cells + parent_unit_addr_cells
+            + parent_interrupt_specifier_cells + 1)];
+    be32 *cells = &interrupt_map[0];
+    int i,j;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the child unit address*/
+        for (j = 0 ; j < child_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the child interrupt specifier*/
+        for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the interrupt-parent*/
+        set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
+
+        /* Set the parent unit address*/
+        for (j = 0 ; j < parent_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the parent interrupt specifier*/
+        for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map", interrupt_map,
+                                sizeof(interrupt_map));
+}
+
 static int make_root_properties(libxl__gc *gc,
                                 const libxl_version_info *vers,
                                 void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
     return 0;
 }
 
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+        const struct arch_info *ainfo,
+        struct xc_dom_image *dom)
+{
+    int res;
+    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+    res = fdt_begin_node(fdt, name);
+    if (res) return res;
+
+    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "device_type", "pci");
+    if (res) return res;
+
+    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+    if (res) return res;
+
+    res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "linux,pci-domain", 0);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#address-cells", 3);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#size-cells", 2);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#interrupt-cells", 1);
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "status", "okay");
+    if (res) return res;
+
+    res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
+        GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
+        3,
+        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
+        GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
+        GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
+        GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
+    if (res) return res;
+
+    res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);
+    if (res) return res;
+
+    /*
+     * Legacy interrupt is forced and assigned to the guest.
+     * This will be removed once we have implementation for MSI support.
+     *
+     */
+    res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+            4,
+            0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
+    if (res) return res;
+
+    res = fdt_end_node(fdt);
+    if (res) return res;
+
+    return 0;
+}
+
 static const struct arch_info *get_arch_info(libxl__gc *gc,
                                              const struct xc_dom_image *dom)
 {
@@ -962,6 +1159,9 @@ next_resize:
         if (info->tee == LIBXL_TEE_TYPE_OPTEE)
             FDT( make_optee_node(gc, fdt) );
 
+        if (info->arch_arm.vpci == LIBXL_VPCI_TYPE_ECAM)
+            FDT( make_vpci_node(gc, fdt, ainfo, dom) );
+
         if (pfdt)
             FDT( copy_partial_fdt(gc, fdt, pfdt) );
 
tools/libxl/libxl_types.idl +6 -0
index 9d3f05f399..d493637705 100644
@@ -257,6 +257,11 @@ libxl_vuart_type = Enumeration("vuart_type", [
     (1, "sbsa_uart"),
     ])
 
+libxl_vpci_type = Enumeration("vpci_type", [
+    (0, "unknown"),
+    (1, "ecam"),
+    ])
+
 libxl_vkb_backend = Enumeration("vkb_backend", [
     (0, "UNKNOWN"),
     (1, "QEMU"),
@@ -640,6 +645,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
 
     ("arch_arm", Struct(None, [("gic_version", libxl_gic_version),
                                ("vuart", libxl_vuart_type),
+                               ("vpci", libxl_vpci_type),
                               ])),
     # Alternate p2m is not bound to any architecture or guest type, as it is
     # supported by x86 HVM and ARM support is planned.
tools/xl/xl_parse.c +7 -0
index 61b4ef7b7e..58b7e6f56a 100644
@@ -1386,6 +1386,13 @@ void parse_config_data(const char *config_source,
         }
     }
 
+    if (!xlu_cfg_get_string(config, "vpci", &buf, 0)) {
+        if (libxl_vpci_type_from_string(buf, &b_info->arch_arm.vpci)) {
+            fprintf(stderr, "ERROR: invalid value \"%s\" for \"vpci\"\n",
+                    buf);
+            exit(1);
+        }
+    }
     parse_vnuma_config(config, b_info);
 
     /* Set max_memkb to target_memkb and max_vcpus to avail_vcpus if
xen/include/public/arch-arm.h +28 -0
index 7364a07362..4e19c62948 100644
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
 #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
 #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
 
+#define GUEST_PCI_ADDRESS_CELLS 3
+#define GUEST_PCI_SIZE_CELLS 2
+
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_ADDR_TYPE_IO           xen_mk_ullong(0x01000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR  xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_CPU_ADDR           xen_mk_ullong(0x04020000)
+#define GUEST_VPCI_IO_CPU_ADDR            xen_mk_ullong(0xC0200800)
+
+/*
+ * This is hardcoded values for the real PCI physical addresses.
+ * This will be removed once we read the real PCI-PCIe physical
+ * addresses form the config space and map to the guest memory map
+ * when assigning the device to guest via VPCI.
+ *
+ */
+#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR  xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_PCI_ADDR           xen_mk_ullong(0x50000000)
+#define GUEST_VPCI_IO_PCI_ADDR            xen_mk_ullong(0x00000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x08000000)
+#define GUEST_VPCI_IO_SIZE                xen_mk_ullong(0x00800000)
+
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
Signature · 1 line
2.17.1
#44 lore raw

Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl

Quoted 8 lines · Rahul Singh

libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.

We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.

Quoted 5 lines

A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).

Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd

Same question

Quoted 189 lines
---
 tools/libxl/libxl_arm.c       | 200 ++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_types.idl   |   6 +
 tools/xl/xl_parse.c           |   7 ++
 xen/include/public/arch-arm.h |  28 +++++
 4 files changed, 241 insertions(+)
tools/libxl/libxl_arm.c +166 -0
index 34f8a29056..84568e9dc9 100644
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
     return fdt_property(fdt, "reg", regs, sizeof(regs));
 }
 
+static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t bus_range[num_cells];
+    be32 *cells = &bus_range[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
+}
+
+static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t interrupt_map_mask[num_cells];
+    be32 *cells = &interrupt_map_mask[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
+                                sizeof(interrupt_map_mask));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+        unsigned vpci_addr_cells,
+        unsigned cpu_addr_cells,
+        unsigned vpci_size_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
+    be32 *cells = &regs[0];
+    int i;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the memory bit field */
+        arg = va_arg(ap, uint64_t);
+        set_cell(&cells, 1, arg);
+
+        /* Set the vpci bus address */
+        arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, 2 , arg);
+
+        /* Set the cpu bus address where vpci address is mapped */
+        arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, cpu_addr_cells, arg);
+
+        /* Set the vpci size requested */
+        arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, vpci_size_cells,arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
+static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
+        unsigned child_unit_addr_cells,
+        unsigned child_interrupt_specifier_cells,
+        unsigned parent_unit_addr_cells,
+        unsigned parent_interrupt_specifier_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
+            child_interrupt_specifier_cells + parent_unit_addr_cells
+            + parent_interrupt_specifier_cells + 1)];
+    be32 *cells = &interrupt_map[0];
+    int i,j;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the child unit address*/
+        for (j = 0 ; j < child_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the child interrupt specifier*/
+        for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the interrupt-parent*/
+        set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
+
+        /* Set the parent unit address*/
+        for (j = 0 ; j < parent_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the parent interrupt specifier*/
+        for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map", interrupt_map,
+                                sizeof(interrupt_map));
+}
+
 static int make_root_properties(libxl__gc *gc,
                                 const libxl_version_info *vers,
                                 void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
     return 0;
 }
 
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+        const struct arch_info *ainfo,
+        struct xc_dom_image *dom)
+{
+    int res;
+    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+    res = fdt_begin_node(fdt, name);
+    if (res) return res;
+
+    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "device_type", "pci");
+    if (res) return res;
+
+    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+    if (res) return res;
+
+    res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "linux,pci-domain", 0);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#address-cells", 3);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#size-cells", 2);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#interrupt-cells", 1);
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "status", "okay");
+    if (res) return res;
+
+    res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
+        GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837

Quoted 10 lines
tools/libxl/libxl_arm.c +10 -0
+        3,
+        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
+        GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
+        GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
+        GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
+    if (res) return res;
+
+    res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);

it would make sense to separate out child_unit_addr_cells and child_interrupt_specifier_cells here like we do below with fdt_property_vpci_interrupt_map

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850

Quoted 13 lines
tools/libxl/libxl_arm.c +13 -0
+    if (res) return res;
+
+    /*
+     * Legacy interrupt is forced and assigned to the guest.
+     * This will be removed once we have implementation for MSI support.
+     *
+     */
+    res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+            4,
+            0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);

The 4 interrupt allocated for this need to be defined in xen/include/public/arch-arm.h as well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.

Quoted 11 lines
tools/libxl/libxl_arm.c +8 -0
+    if (res) return res;
+
+    res = fdt_end_node(fdt);
+    if (res) return res;
+
+    return 0;
+}
+
 static const struct arch_info *get_arch_info(libxl__gc *gc,
                                              const struct xc_dom_image *dom)
 {

[...]

Quoted 8 lines
xen/include/public/arch-arm.h +0 -0
index 7364a07362..4e19c62948 100644
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
 #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
 #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
 

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438

Quoted 10 lines
xen/include/public/arch-arm.h +10 -0
+#define GUEST_PCI_ADDRESS_CELLS 3
+#define GUEST_PCI_SIZE_CELLS 2
+
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_ADDR_TYPE_IO           xen_mk_ullong(0x01000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR  xen_mk_ullong(0x4000000000)

It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440

Quoted 2 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_MEM_CPU_ADDR           xen_mk_ullong(0x04020000)
+#define GUEST_VPCI_IO_CPU_ADDR            xen_mk_ullong(0xC0200800)

0xC0200800 looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454

Quoted 13 lines
xen/include/public/arch-arm.h +13 -0
+/*
+ * This is hardcoded values for the real PCI physical addresses.
+ * This will be removed once we read the real PCI-PCIe physical
+ * addresses form the config space and map to the guest memory map
+ * when assigning the device to guest via VPCI.
+ *
+ */
+#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR  xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_PCI_ADDR           xen_mk_ullong(0x50000000)
+#define GUEST_VPCI_IO_PCI_ADDR            xen_mk_ullong(0x00000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x08000000)

How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?

If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?

Quoted 5 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_IO_SIZE                xen_mk_ullong(0x00800000)
+
 /*
  * 16MB == 4096 pages reserved for guest to use as a region to map its
  * grant table in.
#45 lore raw

Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl

Quoted 234 lines · Stefano Stabellini
Quoted 8 lines · Rahul Singh

libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.

We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.

Quoted 5 lines

A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).

Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd

Same question

Quoted 189 lines
---
  tools/libxl/libxl_arm.c       | 200 ++++++++++++++++++++++++++++++++++
  tools/libxl/libxl_types.idl   |   6 +
  tools/xl/xl_parse.c           |   7 ++
  xen/include/public/arch-arm.h |  28 +++++
  4 files changed, 241 insertions(+)
tools/libxl/libxl_arm.c +166 -0
index 34f8a29056..84568e9dc9 100644
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
      return fdt_property(fdt, "reg", regs, sizeof(regs));
  }
 
+static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t bus_range[num_cells];
+    be32 *cells = &bus_range[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
+}
+
+static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t interrupt_map_mask[num_cells];
+    be32 *cells = &interrupt_map_mask[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
+                                sizeof(interrupt_map_mask));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+        unsigned vpci_addr_cells,
+        unsigned cpu_addr_cells,
+        unsigned vpci_size_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
+    be32 *cells = &regs[0];
+    int i;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the memory bit field */
+        arg = va_arg(ap, uint64_t);
+        set_cell(&cells, 1, arg);
+
+        /* Set the vpci bus address */
+        arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, 2 , arg);
+
+        /* Set the cpu bus address where vpci address is mapped */
+        arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, cpu_addr_cells, arg);
+
+        /* Set the vpci size requested */
+        arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, vpci_size_cells,arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
+static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
+        unsigned child_unit_addr_cells,
+        unsigned child_interrupt_specifier_cells,
+        unsigned parent_unit_addr_cells,
+        unsigned parent_interrupt_specifier_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
+            child_interrupt_specifier_cells + parent_unit_addr_cells
+            + parent_interrupt_specifier_cells + 1)];
+    be32 *cells = &interrupt_map[0];
+    int i,j;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the child unit address*/
+        for (j = 0 ; j < child_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the child interrupt specifier*/
+        for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the interrupt-parent*/
+        set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
+
+        /* Set the parent unit address*/
+        for (j = 0 ; j < parent_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the parent interrupt specifier*/
+        for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map", interrupt_map,
+                                sizeof(interrupt_map));
+}
+
  static int make_root_properties(libxl__gc *gc,
                                  const libxl_version_info *vers,
                                  void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
      return 0;
  }
 
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+        const struct arch_info *ainfo,
+        struct xc_dom_image *dom)
+{
+    int res;
+    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+    res = fdt_begin_node(fdt, name);
+    if (res) return res;
+
+    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "device_type", "pci");
+    if (res) return res;
+
+    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+    if (res) return res;
+
+    res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "linux,pci-domain", 0);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#address-cells", 3);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#size-cells", 2);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#interrupt-cells", 1);
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "status", "okay");
+    if (res) return res;
+
+    res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
+        GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837

Quoted 10 lines
tools/libxl/libxl_arm.c +10 -0
+        3,
+        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
+        GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
+        GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
+        GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
+    if (res) return res;
+
+    res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);

it would make sense to separate out child_unit_addr_cells and child_interrupt_specifier_cells here like we do below with fdt_property_vpci_interrupt_map

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850

Quoted 13 lines
tools/libxl/libxl_arm.c +13 -0
+    if (res) return res;
+
+    /*
+     * Legacy interrupt is forced and assigned to the guest.
+     * This will be removed once we have implementation for MSI support.
+     *
+     */
+    res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+            4,
+            0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
Quoted 4 lines

The 4 interrupt allocated for this need to be defined in xen/include/public/arch-arm.h as well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.

I think this is more about shared interrupts support complexity

Quoted 71 lines
Quoted 13 lines
tools/libxl/libxl_arm.c +8 -0
+    if (res) return res;
+
+    res = fdt_end_node(fdt);
+    if (res) return res;
+
+    return 0;
+}
+
  static const struct arch_info *get_arch_info(libxl__gc *gc,
                                               const struct xc_dom_image *dom)
  {

[...]

Quoted 8 lines
xen/include/public/arch-arm.h +0 -0
index 7364a07362..4e19c62948 100644
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
  #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
  #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
 

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438

Quoted 10 lines
xen/include/public/arch-arm.h +10 -0
+#define GUEST_PCI_ADDRESS_CELLS 3
+#define GUEST_PCI_SIZE_CELLS 2
+
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_ADDR_TYPE_IO           xen_mk_ullong(0x01000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR  xen_mk_ullong(0x4000000000)

It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440

Quoted 2 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_MEM_CPU_ADDR           xen_mk_ullong(0x04020000)
+#define GUEST_VPCI_IO_CPU_ADDR            xen_mk_ullong(0xC0200800)

0xC0200800 looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454

Quoted 13 lines
xen/include/public/arch-arm.h +13 -0
+/*
+ * This is hardcoded values for the real PCI physical addresses.
+ * This will be removed once we read the real PCI-PCIe physical
+ * addresses form the config space and map to the guest memory map
+ * when assigning the device to guest via VPCI.
+ *
+ */
+#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR  xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_PCI_ADDR           xen_mk_ullong(0x50000000)
+#define GUEST_VPCI_IO_PCI_ADDR            xen_mk_ullong(0x00000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x08000000)

How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?

If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?

Quoted 5 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_IO_SIZE                xen_mk_ullong(0x00800000)
+
  /*
   * 16MB == 4096 pages reserved for guest to use as a region to map its
   * grant table in.
#46 lore raw

Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl

In reply to Stefano Stabellini

Hi,

Quoted 15 lines · Stefano Stabellini

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850

Quoted 13 lines · Rahul Singh
tools/libxl/libxl_arm.c +13 -0
+    if (res) return res;
+
+    /*
+     * Legacy interrupt is forced and assigned to the guest.
+     * This will be removed once we have implementation for MSI support.
+     *
+     */
+    res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+            4,
+            0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
Quoted 3 lines

The 4 interrupt allocated for this need to be defined in xen/include/public/arch-arm.h as well. Also, why would we want to get rid of the legacy interrupts completely?

With legacy interrupts, there are a few cases to take into account:
1) Two PCI devices from the same hostbridge are assigned to
different cases. As SPIs (used for legacy interrupts) can only be routed to one guest, we would need to now be able to share them. This raises the question when to EOI the physical interrupts. AFAICT, Linux has some code to deal with it using a timer if it takes too long.

2) Two PCI devices from two distinct hostbridge are assigned to the
same virtual hostbridge. Legacy interrupts would need to be virtual and we would possibly need to merge multiple physical interrupts into one virtual.

3) A mix of virtual and physical PCI devices are assigned to the
same host bridges. Same as 2) legacy interrupts would need to be virtual.

Given the complexity of handling interrupts legacy and the fact they will largely be slower compare to MSIs, I would rather focus on MSIs at first.

We can then decide to implement legacy interrupts if there is a real need.

Quoted 2 lines

It would be possible to still
find device or software that rely on them.

PCI devices without MSI support is getting extremely rare. There are actually Arm HW that doesn't support legacy interrupts at all (such as Thunder-X). I can't tell for the software.

Signature · 1 line
Julien Grall
#47 lore raw

Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl

In reply to Stefano Stabellini
Quoted 13 lines · Stefano Stabellini
Quoted 8 lines · Rahul Singh

libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.

We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.

Quoted 7 lines
Quoted 5 lines

A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).

Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd

Same question

I will remove the change-id in the next version.

Quoted 197 lines
Quoted 197 lines

---

tools/libxl/libxl_arm.c       | 200 ++++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl   |   6 +
tools/xl/xl_parse.c           |   7 ++
xen/include/public/arch-arm.h |  28 +++++
4 files changed, 241 insertions(+)
tools/libxl/libxl_arm.c +172 -0
index 34f8a29056..84568e9dc9 100644
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,
    return fdt_property(fdt, "reg", regs, sizeof(regs));
 }
 
+static int fdt_property_vpci_bus_range(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t bus_range[num_cells];
+    be32 *cells = &bus_range[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "bus-range", bus_range, sizeof(bus_range));
+}
+
+static int fdt_property_vpci_interrupt_map_mask(libxl__gc *gc, void *fdt,
+        unsigned num_cells, ...)
+{
+    uint32_t interrupt_map_mask[num_cells];
+    be32 *cells = &interrupt_map_mask[0];
+    int i;
+    va_list ap;
+    uint32_t arg;
+
+    va_start(ap, num_cells);
+    for (i = 0 ; i < num_cells; i++) {
+        arg = va_arg(ap, uint32_t);
+        set_cell(&cells, 1, arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map-mask", interrupt_map_mask,
+                                sizeof(interrupt_map_mask));
+}
+
+static int fdt_property_vpci_ranges(libxl__gc *gc, void *fdt,
+        unsigned vpci_addr_cells,
+        unsigned cpu_addr_cells,
+        unsigned vpci_size_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)];
+    be32 *cells = &regs[0];
+    int i;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the memory bit field */
+        arg = va_arg(ap, uint64_t);
+        set_cell(&cells, 1, arg);
+
+        /* Set the vpci bus address */
+        arg = vpci_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, 2 , arg);
+
+        /* Set the cpu bus address where vpci address is mapped */
+        arg = cpu_addr_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, cpu_addr_cells, arg);
+
+        /* Set the vpci size requested */
+        arg = vpci_size_cells ? va_arg(ap, uint64_t) : 0;
+        set_cell(&cells, vpci_size_cells,arg);
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "ranges", regs, sizeof(regs));
+}
+
+static int fdt_property_vpci_interrupt_map(libxl__gc *gc, void *fdt,
+        unsigned child_unit_addr_cells,
+        unsigned child_interrupt_specifier_cells,
+        unsigned parent_unit_addr_cells,
+        unsigned parent_interrupt_specifier_cells,
+        unsigned num_regs, ...)
+{
+    uint32_t interrupt_map[num_regs * (child_unit_addr_cells +
+            child_interrupt_specifier_cells + parent_unit_addr_cells
+            + parent_interrupt_specifier_cells + 1)];
+    be32 *cells = &interrupt_map[0];
+    int i,j;
+    va_list ap;
+    uint64_t arg;
+
+    va_start(ap, num_regs);
+    for (i = 0 ; i < num_regs; i++) {
+        /* Set the child unit address*/
+        for (j = 0 ; j < child_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the child interrupt specifier*/
+        for (j = 0 ; j < child_interrupt_specifier_cells ; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the interrupt-parent*/
+        set_cell(&cells, 1 , GUEST_PHANDLE_GIC);
+
+        /* Set the parent unit address*/
+        for (j = 0 ; j < parent_unit_addr_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+
+        /* Set the parent interrupt specifier*/
+        for (j = 0 ; j < parent_interrupt_specifier_cells; j++) {
+            arg = va_arg(ap, uint32_t);
+            set_cell(&cells, 1 , arg);
+        }
+    }
+    va_end(ap);
+
+    return fdt_property(fdt, "interrupt-map", interrupt_map,
+                                sizeof(interrupt_map));
+}
+
 static int make_root_properties(libxl__gc *gc,
                                const libxl_version_info *vers,
                                void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,
    return 0;
 }
 
+static int make_vpci_node(libxl__gc *gc, void *fdt,
+        const struct arch_info *ainfo,
+        struct xc_dom_image *dom)
+{
+    int res;
+    const uint64_t vpci_ecam_base = GUEST_VPCI_ECAM_BASE;
+    const uint64_t vpci_ecam_size = GUEST_VPCI_ECAM_SIZE;
+    const char *name = GCSPRINTF("pcie@%"PRIx64, vpci_ecam_base);
+
+    res = fdt_begin_node(fdt, name);
+    if (res) return res;
+
+    res = fdt_property_compat(gc, fdt, 1, "pci-host-ecam-generic");
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "device_type", "pci");
+    if (res) return res;
+
+    res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,
+            GUEST_ROOT_SIZE_CELLS, 1, vpci_ecam_base, vpci_ecam_size);
+    if (res) return res;
+
+    res = fdt_property_vpci_bus_range(gc, fdt, 2, 0,255);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "linux,pci-domain", 0);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#address-cells", 3);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#size-cells", 2);
+    if (res) return res;
+
+    res = fdt_property_cell(fdt, "#interrupt-cells", 1);
+    if (res) return res;
+
+    res = fdt_property_string(fdt, "status", "okay");
+    if (res) return res;
+
+    res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS,
+        GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
+        3,
+        GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR,
+        GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR,
+        GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE,
+        GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
Quoted 8 lines

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837

Quoted 4 lines
tools/libxl/libxl_arm.c +4 -0
+        GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE);
+    if (res) return res;
+
+    res = fdt_property_vpci_interrupt_map_mask(gc, fdt, 4, 0, 0, 0, 7);

it would make sense to separate out child_unit_addr_cells and child_interrupt_specifier_cells here like we do below with fdt_property_vpci_interrupt_map

Ok will fix.

Quoted 16 lines
Quoted 5 lines
tools/libxl/libxl_arm.c +3 -0
+    if (res) return res;
+
+    /*

tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850

Quoted 10 lines
tools/libxl/libxl_arm.c +10 -0
+     * Legacy interrupt is forced and assigned to the guest.
+     * This will be removed once we have implementation for MSI support.
+     *
+     */
+    res = fdt_property_vpci_interrupt_map(gc, fdt, 3, 1, 0, 3,
+            4,
+            0, 0, 0, 1, 0, 136, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 2, 0, 137, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 3, 0, 138, DT_IRQ_TYPE_LEVEL_HIGH,
+            0, 0, 0, 4, 0, 139, DT_IRQ_TYPE_LEVEL_HIGH);
Quoted 5 lines

The 4 interrupt allocated for this need to be defined in xen/include/public/arch-arm.h as well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.

Ok will fix that.
Regarding legacy interrupt we have just tested on one of the board don’t know how it will work on other boards. We will mostly support MSI and will see if we have to support the legacy interrupt going forward.

Quoted 24 lines
Quoted 12 lines
tools/libxl/libxl_arm.c +8 -0
+    if (res) return res;
+
+    res = fdt_end_node(fdt);
+    if (res) return res;
+
+    return 0;
+}
+
 static const struct arch_info *get_arch_info(libxl__gc *gc,
                                             const struct xc_dom_image *dom)
 {

[...]

Quoted 8 lines
xen/include/public/arch-arm.h +0 -0
index 7364a07362..4e19c62948 100644
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;
 #define GUEST_VPCI_ECAM_BASE    xen_mk_ullong(0x10000000)
 #define GUEST_VPCI_ECAM_SIZE    xen_mk_ullong(0x10000000)
 
Quoted 12 lines

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438

Quoted 10 lines
xen/include/public/arch-arm.h +10 -0
+#define GUEST_PCI_ADDRESS_CELLS 3
+#define GUEST_PCI_SIZE_CELLS 2
+
+/* PCI-PCIe memory space types */
+#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000)
+#define GUEST_VPCI_ADDR_TYPE_MEM          xen_mk_ullong(0x02000000)
+#define GUEST_VPCI_ADDR_TYPE_IO           xen_mk_ullong(0x01000000)
+
+/* Guest PCI-PCIe memory space where config space and BAR will be available.*/
+#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR  xen_mk_ullong(0x4000000000)

It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?

Ok yes will fix that and will define the address ranges for guest once we finalised the VPCI topology for the guest. We are currently investigating if we want to follow the hardware topology for the guest or we will create a different virtual topology for the guest independent of the hw topology.

Quoted 8 lines

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440

Quoted 4 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_MEM_CPU_ADDR           xen_mk_ullong(0x04020000)
+#define GUEST_VPCI_IO_CPU_ADDR            xen_mk_ullong(0xC0200800)

0xC0200800 looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?

Same comment above.

Quoted 19 lines
Quoted 4 lines
xen/include/public/arch-arm.h +3 -0
+/*
+ * This is hardcoded values for the real PCI physical addresses.
+ * This will be removed once we read the real PCI-PCIe physical

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454

Quoted 10 lines
xen/include/public/arch-arm.h +10 -0
+ * addresses form the config space and map to the guest memory map
+ * when assigning the device to guest via VPCI.
+ *
+ */
+#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR  xen_mk_ullong(0x4000000000)
+#define GUEST_VPCI_MEM_PCI_ADDR           xen_mk_ullong(0x50000000)
+#define GUEST_VPCI_IO_PCI_ADDR            xen_mk_ullong(0x00000000)
+
+#define GUEST_VPCI_PREFETCH_MEM_SIZE      xen_mk_ullong(0x100000000)
+#define GUEST_VPCI_MEM_SIZE               xen_mk_ullong(0x08000000)

How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?

Quoted 2 lines

If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?

Same comments above.

Quoted 9 lines
Quoted 9 lines
xen/include/public/arch-arm.h +2 -0
+#define GUEST_VPCI_IO_SIZE                xen_mk_ullong(0x00800000)
+
 /*
 * 16MB == 4096 pages reserved for guest to use as a region to map its
 * grant table in.