PCI devices passthrough on Arm
RFC PATCH 3/4 xen/arm: Enable the existing x86 virtual PCI support for ARM.
In reply to Rahul SinghThe existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.
A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.
VPCI MSI support is disable for ARM as it is not tested on ARM.
--- xen/arch/arm/Makefile | 1 + xen/arch/arm/domain.c | 4 ++ xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++ xen/arch/arm/vpci.h | 37 ++++++++++++ xen/drivers/passthrough/pci.c | 7 +++ xen/include/asm-arm/domain.h | 5 ++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 160 insertions(+) create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.h
xen/arch/arm/Makefile +1 -0
@@ -7,6 +7,7 @@ obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ obj-$(CONFIG_ARM_PCI) += pci/ +obj-$(CONFIG_HAS_VPCI) += vpci.o obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.o
xen/arch/arm/domain.c +4 -0
@@ -39,6 +39,7 @@#include <asm/vtimer.h> #include "vuart.h" +#include "vpci.h" DEFINE_PER_CPU(struct vcpu *, curr_vcpu);
@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,if (is_hardware_domain (d) && (rc =domain_vuart_init (d)) ) goto fail; +if ( (rc =domain_vpci_init (d)) !=0 ) + goto fail; +return 0 ; fail:
xen/arch/arm/vpci.c new file +102 -0
102 lines not on this page · raw message
xen/arch/arm/vpci.h new file +37 -0
@@ -0,0 +1,37 @@ +/* +* xen/arch/arm/vpci.h +* Copyright (c) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#ifndef __ARCH_ARM_VPCI_H__ +#define __ARCH_ARM_VPCI_H__ + +#ifdef CONFIG_HAS_VPCI +int domain_vpci_init (struct domain *d); +#else +static inline int domain_vpci_init (struct domain *d) +{ +return 0 ; +} +#endif + +#endif /* __ARCH_ARM_VPCI_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/
xen/drivers/passthrough/pci.c +7 -0
@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,else iommu_enable_device (pdev); +#ifdef CONFIG_ARM + ret =vpci_add_handlers (pdev); +if ( ret ) { +printk (XENLOG_ERR "setup of vPCI for failed: %d\n" ,ret); + goto out; + } +#endif pci_enable_acs (pdev); out:
xen/include/asm-arm/domain.h +5 -0
@@ -34,6 +34,11 @@ enum domain_type {/* The hardware domain has always its memory direct mapped. */ #define is_domain_direct_mapped (d) ((d) == hardware_domain) +/* For X86 VPCI is enabled and tested for PVH DOM0 only but +* for ARM we enable support VPCI for guest domain also. +*/ +#define has_vpci (d) (true) +struct vtimer {struct vcpu *v;int irq;
xen/include/public/arch-arm.h +4 -0
@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;#define GUEST_PL011_BASE xen_mk_ullong(0x22000000)#define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) +/* VPCI ECAM mappings */ +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) +/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in.
Signature · 1 line
2.17.1
RFC PATCH 4/4 arm/libxl: Emulated PCI device tree node in libxl
In reply to Rahul Singhlibxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.
We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.
A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).
--- tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + tools/xl/xl_parse.c | 7 ++ xen/include/public/arch-arm.h | 28 +++++ 4 files changed, 241 insertions(+)
tools/libxl/libxl_arm.c +200 -0
@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,return fdt_property (fdt,"reg" , regs,sizeof (regs)); } +static int fdt_property_vpci_bus_range (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t bus_range[num_cells]; +be32 *cells = &bus_range[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"bus-range" , bus_range,sizeof (bus_range)); +} + +static int fdt_property_vpci_interrupt_map_mask (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t interrupt_map_mask[num_cells]; +be32 *cells = &interrupt_map_mask[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map-mask" , interrupt_map_mask, +sizeof (interrupt_map_mask)); +} + +static int fdt_property_vpci_ranges (libxl__gc *gc,void *fdt, +unsigned vpci_addr_cells, +unsigned cpu_addr_cells, +unsigned vpci_size_cells, +unsigned num_regs, ...) +{ +uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)]; +be32 *cells = ®s[0 ]; +int i; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the memory bit field */ + arg =va_arg (ap, uint64_t); +set_cell (&cells,1 , arg); + +/* Set the vpci bus address */ + arg = vpci_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells,2 , arg); + +/* Set the cpu bus address where vpci address is mapped */ + arg = cpu_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, cpu_addr_cells, arg); + +/* Set the vpci size requested */ + arg = vpci_size_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, vpci_size_cells,arg); + } +va_end (ap); + +return fdt_property (fdt,"ranges" , regs,sizeof (regs)); +} + +static int fdt_property_vpci_interrupt_map (libxl__gc *gc,void *fdt, +unsigned child_unit_addr_cells, +unsigned child_interrupt_specifier_cells, +unsigned parent_unit_addr_cells, +unsigned parent_interrupt_specifier_cells, +unsigned num_regs, ...) +{ +uint32_t interrupt_map[num_regs * (child_unit_addr_cells + + child_interrupt_specifier_cells + parent_unit_addr_cells + + parent_interrupt_specifier_cells +1 )]; +be32 *cells = &interrupt_map[0 ]; +int i,j; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the child unit address*/ +for (j =0 ; j < child_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the child interrupt specifier*/ +for (j =0 ; j < child_interrupt_specifier_cells ; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the interrupt-parent*/ +set_cell (&cells,1 ,GUEST_PHANDLE_GIC ); + +/* Set the parent unit address*/ +for (j =0 ; j < parent_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the parent interrupt specifier*/ +for (j =0 ; j < parent_interrupt_specifier_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map" , interrupt_map, +sizeof (interrupt_map)); +} +static int make_root_properties (libxl__gc *gc,const libxl_version_info *vers,void *fdt)
@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,return 0 ; } +static int make_vpci_node (libxl__gc *gc,void *fdt, +const struct arch_info *ainfo, +struct xc_dom_image *dom) +{ +int res; +const uint64_t vpci_ecam_base =GUEST_VPCI_ECAM_BASE ; +const uint64_t vpci_ecam_size =GUEST_VPCI_ECAM_SIZE ; +const char *name =GCSPRINTF ("pcie@%" PRIx64, vpci_ecam_base); + + res =fdt_begin_node (fdt, name); +if (res)return res; + + res =fdt_property_compat (gc, fdt,1 ,"pci-host-ecam-generic" ); +if (res)return res; + + res =fdt_property_string (fdt,"device_type" ,"pci" ); +if (res)return res; + + res =fdt_property_regs (gc, fdt,GUEST_ROOT_ADDRESS_CELLS , +GUEST_ROOT_SIZE_CELLS ,1 , vpci_ecam_base, vpci_ecam_size); +if (res)return res; + + res =fdt_property_vpci_bus_range (gc, fdt,2 ,0 ,255 ); +if (res)return res; + + res =fdt_property_cell (fdt,"linux,pci-domain" ,0 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#address-cells" ,3 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#size-cells" ,2 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#interrupt-cells" ,1 ); +if (res)return res; + + res =fdt_property_string (fdt,"status" ,"okay" ); +if (res)return res; + + res =fdt_property_vpci_ranges (gc, fdt,GUEST_PCI_ADDRESS_CELLS , +GUEST_ROOT_ADDRESS_CELLS ,GUEST_PCI_SIZE_CELLS , +3 , +GUEST_VPCI_ADDR_TYPE_MEM ,GUEST_VPCI_MEM_PCI_ADDR , +GUEST_VPCI_MEM_CPU_ADDR ,GUEST_VPCI_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM ,GUEST_VPCI_PREFETCH_MEM_PCI_ADDR , +GUEST_VPCI_PREFETCH_MEM_CPU_ADDR ,GUEST_VPCI_PREFETCH_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_IO ,GUEST_VPCI_IO_PCI_ADDR , +GUEST_VPCI_IO_CPU_ADDR ,GUEST_VPCI_IO_SIZE ); +if (res)return res; + + res =fdt_property_vpci_interrupt_map_mask (gc, fdt,4 ,0 ,0 ,0 ,7 ); +if (res)return res; + +/* +* Legacy interrupt is forced and assigned to the guest. +* This will be removed once we have implementation for MSI support. +* +*/ + res =fdt_property_vpci_interrupt_map (gc, fdt,3 ,1 ,0 ,3 , +4 , +0 ,0 ,0 ,1 ,0 ,136 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,2 ,0 ,137 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,3 ,0 ,138 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,4 ,0 ,139 ,DT_IRQ_TYPE_LEVEL_HIGH ); +if (res)return res; + + res =fdt_end_node (fdt); +if (res)return res; + +return 0 ; +} +static const struct arch_info *get_arch_info (libxl__gc *gc,const struct xc_dom_image *dom) {
@@ -962,6 +1159,9 @@ next_resize:if (info->tee ==LIBXL_TEE_TYPE_OPTEE )FDT (make_optee_node (gc, fdt) ); +if (info->arch_arm.vpci ==LIBXL_VPCI_TYPE_ECAM ) +FDT (make_vpci_node (gc, fdt, ainfo, dom) ); +if (pfdt)FDT (copy_partial_fdt (gc, fdt, pfdt) );
tools/libxl/libxl_types.idl +6 -0
@@ -257,6 +257,11 @@ libxl_vuart_type = Enumeration("vuart_type", [ (1, "sbsa_uart"), ]) +libxl_vpci_type = Enumeration("vpci_type", [ + (0, "unknown"), + (1, "ecam"), + ]) + libxl_vkb_backend = Enumeration("vkb_backend", [ (0, "UNKNOWN"), (1, "QEMU"),
@@ -640,6 +645,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("arch_arm", Struct(None, [("gic_version", libxl_gic_version), ("vuart", libxl_vuart_type), + ("vpci", libxl_vpci_type), ])), # Alternate p2m is not bound to any architecture or guest type, as it is # supported by x86 HVM and ARM support is planned.
tools/xl/xl_parse.c +7 -0
@@ -1386,6 +1386,13 @@ void parse_config_data(const char *config_source, } } +if (!xlu_cfg_get_string (config,"vpci" , &buf,0 )) { +if (libxl_vpci_type_from_string (buf, &b_info->arch_arm.vpci)) { +fprintf (stderr,"ERROR: invalid value \"%s\" for \"vpci\"\n" , + buf); +exit (1 ); + } + }parse_vnuma_config (config, b_info);/* Set max_memkb to target_memkb and max_vcpus to avail_vcpus if
xen/include/public/arch-arm.h +28 -0
@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000) +#define GUEST_PCI_ADDRESS_CELLS 3 +#define GUEST_PCI_SIZE_CELLS 2 + +/* PCI-PCIe memory space types */ +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000) + +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000) +#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000) +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800) + +/* +* This is hardcoded values for the real PCI physical addresses. +* This will be removed once we read the real PCI-PCIe physical +* addresses form the config space and map to the guest memory map +* when assigning the device to guest via VPCI. +* +*/ +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000) +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000) +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000) + +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000) +#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000) +/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in.
Signature · 1 line
2.17.1
RFC PATCH 2/4 xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Rahul SinghHardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call PHYSDEVOP_pci_device_add to add the PCI devices in XEN.
--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)
xen/arch/arm/physdev.c +39 -3
@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI +case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; + + ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + +break ; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + } + +return ret; }/*
Signature · 1 line
2.17.1
RFC PATCH 1/4 arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Rahul SinghXEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.c
xen/arch/arm/Kconfig +7 -0
@@ -31,6 +31,13 @@ menu "Architecture Features" source "arch/Kconfig" +config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. + config ACPI bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64
xen/arch/arm/Makefile +1 -0
@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_ARM_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.o
xen/arch/arm/pci/Makefile new file +4 -0
@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.o
xen/arch/arm/pci/pci-access.c new file +101 -0
101 lines not on this page · raw message
xen/arch/arm/pci/pci-host-common.c new file +198 -0
198 lines not on this page · raw message
xen/arch/arm/pci/pci-host-generic.c new file +131 -0
131 lines not on this page · raw message
xen/arch/arm/pci/pci.c new file +112 -0
112 lines not on this page · raw message
xen/arch/arm/setup.c +2 -0
@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init (); +do_initcalls ();/*
xen/include/asm-arm/device.h +4 -3
@@ -4,6 +4,7 @@enum device_type {DEV_DT , +DEV_PCI , };struct dev_archdata {
@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT) +#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum device_class {DEVICE_SERIAL ,DEVICE_IOMMU ,DEVICE_GIC , +DEVICE_PCI ,/* Use for error */ DEVICE_UNKNOWN , };
xen/include/asm-arm/pci.h +94 -3
101 lines not on this page · raw message
Signature · 1 line
2.17.1
cover PCI devices passthrough on Arm
Following up on the discussion on PCI devices passthrough support on Arm design proposal.Please feel free to give you feedback.
We are submitting the code that we developed to get the early feedback. PCI passthrough support on ARM is not fully implemented in this patch series for that reason we are not enabling the HAS_PCI and HAS_VPCI flags for ARM.
We will work on design document that we submitted for feedback on mailing list and we will submit the next design document to address all the comments.
This patch series is based on v1 of the design document that we submitted for review. Any comments in the design will be addressed in the later version of the design document and will subsequently implemented in the code in next patch series.
PCI passthrough support is divided into different patches:
Discovering PCI Host Bridge in XEN:
- PCI host bridge discovery in XEN and map the PCI ECAM configuration space to the XEN memory.
Discovering PCI devices:
- In order to support the PCI passthrough, XEN should be aware of the PCI devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then communicate to the XEN via hyper call to add the PCI devices in XEN.
Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config space access.
Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to discover the virtual PCI during guest boot.
This patch series does not implemented the following features. Following features will be implemented in the next version of the patch series.
- MSI support for interrupt.
- ACPI support for PCI host bridge discovery within XEN on ARM.
- SMMU modification to support PCI devices.
- Use already defined config option "pci=[]" in place of new "vpci=ecam" config option to create VPCI bus.
- Map the assigned device PCI BAR values and interrupt to the guest when device is assigned by xl during domain creation.Currently we are using "iomem=[]" config options to map the value to the guest.
Rahul Singh (4):
arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM. xen/arm: Discovering PCI devices and add the PCI devices in XEN. xen/arm: Enable the existing x86 virtual PCI support for ARM. arm/libxl: Emulated PCI device tree node in libxl
tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + tools/xl/xl_parse.c | 7 + xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 2 + xen/arch/arm/domain.c | 4 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 +++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/physdev.c | 42 +++++- xen/arch/arm/setup.c | 2 + xen/arch/arm/vpci.c | 102 ++++++++++++++ xen/arch/arm/vpci.h | 37 +++++ xen/drivers/passthrough/pci.c | 7 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/domain.h | 5 + xen/include/asm-arm/pci.h | 97 +++++++++++++- xen/include/public/arch-arm.h | 32 +++++ 20 files changed, 1094 insertions(+), 9 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.c create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.h
Signature · 1 line
2.17.1
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Rahul SinghQuoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
Quoted 46 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +27
-3@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI +case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +
xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }
I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl
Jan, Andrew, Roger, any opinions?
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Rahul Singh+ Jan, Andrew, Roger
Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)
Quoted 9 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
What is this Change-Id property?
Quoted 146 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +7
-0@@ -31,6 +31,13 @@ menu "Architecture Features" source "arch/Kconfig" +config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. + config ACPI bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64xen/arch/arm/Makefile +1
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_ARM_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oxen/arch/arm/pci/pci-access.c new file +81
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len) +{ +int rc; +uint32_t val =GENMASK (0 , len *8 ); + +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val); +if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)
This looks like a subset of xen/arch/x86/x86_64/mmconfig_64.c ?
MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.
Quoted 96 lines
xen/arch/arm/pci/pci-access.c new file +9
-0+/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +81
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> + +/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells; +uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +} + +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +} + +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +int err; +struct pci_config_window *cfg; +paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; + + err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) { + cfg->busn_start =0 ; + cfg->busn_end =0xff ;
xen/arch/arm/pci/pci-host-common.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ; + } + +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit;
Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?
Quoted 49 lines
xen/arch/arm/pci/pci-host-common.c new file +49
-0+ cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size); +if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } + +return cfg; + +err_exit_remap: +printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void ) +{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge; +struct pci_config_window *cfg;
xen/arch/arm/pci/pci-host-common.c line 151
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge ) +return -ENOMEM ; + +/* Parse and map our Configuration Space windows */ + cfg =gen_pci_init (dev, ops); +if ( !cfg ) +return -ENOMEM ;
In case of errors the allocated bridge is not freed.
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 11 lines
xen/arch/arm/pci/pci-host-common.c new file +11
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ; + } + + bridge->segment = (u16 )segment;
My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
xen/arch/arm/pci/pci-host-common.c line 165
Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);
It looks like &pci_host_bridges should be an ordered list, ordered by segment number?
Quoted 58 lines
xen/arch/arm/pci/pci-host-common.c new file +32
-0+return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; + +list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ; +} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +
xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)
I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).
Maybe pci_config_get_address or something like that?
Quoted 22 lines
xen/arch/arm/pci/pci-host-generic.c new file +22
-0+{ +struct pci_config_window *cfg = bridge->sysdata; +unsigned int devfn_shift = cfg->ops->bus_shift -8 ; + +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base; + +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift); + +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; +
xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);
please use a switch
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) {
xen/arch/arm/pci/pci-host-generic.c line 83
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ *val = ~0 ; +return -ENODEV ; + } + +if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr);
please use a switch
Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5
-0+return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = {
xen/arch/arm/pci/pci-host-generic.c line 99
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ .bus_shift = 20, + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write, + } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops },
spurious blank line
Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31
-0+ { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +71
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 :
xen/arch/arm/pci/pci.c line 81
Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10
-0+ param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } + +return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);
When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.
Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable; + +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init (); + +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +0
-0
xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933
Quoted 4 lines
xen/arch/arm/setup.c +1
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init ();
pci_init should probably be an initcall
Quoted 67 lines
xen/arch/arm/setup.c +0
-0do_initcalls ();/* xen/include/asm-arm/device.h +4
-3@@ -4,6 +4,7 @@enum device_type {DEV_DT , +DEV_PCI , };struct dev_archdata {@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT) +#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum device_class {DEVICE_SERIAL ,DEVICE_IOMMU ,DEVICE_GIC , +DEVICE_PCI ,/* Use for error */ DEVICE_UNKNOWN , };xen/include/asm-arm/pci.h +25
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ +
xen/include/asm-arm/pci.h line 36
Quoted 10 lines
xen/include/asm-arm/pci.h +9
-0+#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};
Are you actually using struct device in struct arch_pci_dev? struct device is already part of struct dt_device_node and a pointer to it is stored in bridge->dt_node.
Quoted 15 lines
xen/include/asm-arm/pci.h +15
-0+#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/* +* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops; +void __iomem *win;
xen/include/asm-arm/pci.h line 62
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val);
I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?
Can we use pci_sbdf_t for the sbdf parameter?
xen/include/asm-arm/pci.h line 73
Quoted 11 lines
xen/include/asm-arm/pci.h +11
-0+}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +};
Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename pci_ecam_ops to something else, maybe simply "pci_ops".
Quoted 28 lines
xen/include/asm-arm/pci.h +22
-1+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops; }; -#endif /* __X86_PCI_H__ */ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus); + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops); + +void pci_init (void ); + +#else /*!CONFIG_ARM_PCI*/ +struct arch_pci_dev { }; +static inline void pci_init (void ) { } +#endif /*!CONFIG_ARM_PCI*/ +#endif /* __ARM_PCI_H__ */ Signature · 1 line
2.17.1
Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.
In reply to Rahul SinghQuoted 5 lines · Rahul Singh
The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.
Quoted 8 lines
A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.
VPCI MSI support is disable for ARM as it is not tested on ARM.
Change-Id: I5501db2781f8064640403fecce53713091cd9ab4
Same question
Quoted 119 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/domain.c | 4 ++ xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++ xen/arch/arm/vpci.h | 37 ++++++++++++ xen/drivers/passthrough/pci.c | 7 +++ xen/include/asm-arm/domain.h | 5 ++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 160 insertions(+) create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.hxen/arch/arm/Makefile +1
-0@@ -7,6 +7,7 @@ obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ obj-$(CONFIG_ARM_PCI) += pci/ +obj-$(CONFIG_HAS_VPCI) += vpci.o obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/domain.c +4
-0@@ -39,6 +39,7 @@#include <asm/vtimer.h> #include "vuart.h" +#include "vpci.h" DEFINE_PER_CPU(struct vcpu *, curr_vcpu);@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,if (is_hardware_domain (d) && (rc =domain_vuart_init (d)) ) goto fail; +if ( (rc =domain_vpci_init (d)) !=0 ) + goto fail; +return 0 ; fail:xen/arch/arm/vpci.c new file +66
-0@@ -0,0 +1,102 @@ +/* +* xen/arch/arm/vpci.c +* Copyright (c) 2020 Arm Ltd. +* +* Based on arch/x86/hvm/io.c +* Copyright (c) 2004, Intel Corporation. +* Copyright (c) 2005, International Business Machines Corporation. +* Copyright (c) 2008, Citrix Systems, Inc. +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ +#include <xen/sched.h> +#include <asm/mmio.h> + +/* Do some sanity checks. */ +static bool vpci_mmio_access_allowed (unsigned int reg,unsigned int len) +{ +/* Check access size. */ +if ( len !=1 && len !=2 && len !=4 && len !=8 ) +return false; + +/* Check that access is size aligned. */ +if ( (reg & (len -1 )) ) +return false; + +return true; +} + +static int vpci_mmio_read (struct vcpu *v,mmio_info_t *info, +register_t *r,void *priv) +{ +unsigned int reg; +pci_sbdf_t sbdf; +uint32_t data =0 ; +unsigned int size =1U << info->dabt.size; + + sbdf.bdf = (((info->gpa) &0x0ffff000 ) >>12 ); + reg = (((info->gpa) &0x00000ffc ) | (info->gpa &3 )); + +if ( !vpci_mmio_access_allowed (reg, size) ) +return 1 ; + + data =vpci_read (sbdf, reg, size); + +memcpy (r, &data, size); + +return 1 ; +} + +static int vpci_mmio_write (struct vcpu *v,mmio_info_t *info, +register_t r,void *priv) +{ +unsigned int reg; +pci_sbdf_t sbdf; +uint32_t data = r; +unsigned int size =1U << info->dabt.size; +
xen/arch/arm/vpci.c line 76
Quoted 10 lines
xen/arch/arm/vpci.c new file +10
-0+ sbdf.bdf = (((info->gpa) &0x0ffff000 ) >>12 ); + reg = (((info->gpa) &0x00000ffc ) | (info->gpa &3 )); + +if ( !vpci_mmio_access_allowed (reg, size) ) +return 1 ; + +vpci_write (sbdf, reg, size, data); + +return 1 ; +}
I wonder if we could share vpci_mmcfg_read/write. Again, it is OK if we can't, or if it is not worth the effort. just want to make sure we thought about it :-)
Quoted 91 lines
xen/arch/arm/vpci.c new file +25
-0+static const struct mmio_handler_ops vpci_mmio_handler = { + .read = vpci_mmio_read, + .write = vpci_mmio_write, +}; + +int domain_vpci_init (struct domain *d) +{ +if ( !has_vpci (d) ||is_hardware_domain (d) ) +return 0 ; + +register_mmio_handler (d, &vpci_mmio_handler, +GUEST_VPCI_ECAM_BASE ,GUEST_VPCI_ECAM_SIZE ,NULL ); + +return 0 ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ +xen/arch/arm/vpci.h new file +37
-0@@ -0,0 +1,37 @@ +/* +* xen/arch/arm/vpci.h +* Copyright (c) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#ifndef __ARCH_ARM_VPCI_H__ +#define __ARCH_ARM_VPCI_H__ + +#ifdef CONFIG_HAS_VPCI +int domain_vpci_init (struct domain *d); +#else +static inline int domain_vpci_init (struct domain *d) +{ +return 0 ; +} +#endif + +#endif /* __ARCH_ARM_VPCI_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/pci.c +7
-0@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,else iommu_enable_device (pdev); +#ifdef CONFIG_ARM + ret =vpci_add_handlers (pdev); +if ( ret ) { +printk (XENLOG_ERR "setup of vPCI for failed: %d\n" ,ret); + goto out; + } +#endif pci_enable_acs (pdev); out:xen/include/asm-arm/domain.h +0
-0
xen/include/asm-arm/domain.h enum domain_type { · line 40
Quoted 7 lines
xen/include/asm-arm/domain.h +4
-0@@ -34,6 +34,11 @@ enum domain_type {/* The hardware domain has always its memory direct mapped. */ #define is_domain_direct_mapped (d) ((d) == hardware_domain) +/* For X86 VPCI is enabled and tested for PVH DOM0 only but +* for ARM we enable support VPCI for guest domain also. +*/ +#define has_vpci (d) (true)
As mentioned we could make this configurabled based on the presence of pci=[] or something similar in device tree for dom0less guests.
Quoted 8 lines
xen/include/asm-arm/domain.h +0
-0struct vtimer {struct vcpu *v;int irq;xen/include/public/arch-arm.h +0
-0
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 427
Quoted 6 lines
xen/include/public/arch-arm.h +3
-0@@ -422,6 +422,10 @@ typedef uint64_t xen_callback_t;#define GUEST_PL011_BASE xen_mk_ullong(0x22000000)#define GUEST_PL011_SIZE xen_mk_ullong(0x00001000) +/* VPCI ECAM mappings */ +#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000) +#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
Is 256MB in size part of the ECAM standard?
Quoted 6 lines
xen/include/public/arch-arm.h +0
-0/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in. Signature · 1 line
2.17.1
Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
In reply to Rahul SinghQuoted 8 lines · Rahul Singh
libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.
We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.
Quoted 5 lines
A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).
Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
Same question
Quoted 189 lines
Signed-off-by Rahul Singh <[email protected]>--- tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + tools/xl/xl_parse.c | 7 ++ xen/include/public/arch-arm.h | 28 +++++ 4 files changed, 241 insertions(+)tools/libxl/libxl_arm.c +166
-0@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,return fdt_property (fdt,"reg" , regs,sizeof (regs)); } +static int fdt_property_vpci_bus_range (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t bus_range[num_cells]; +be32 *cells = &bus_range[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"bus-range" , bus_range,sizeof (bus_range)); +} + +static int fdt_property_vpci_interrupt_map_mask (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t interrupt_map_mask[num_cells]; +be32 *cells = &interrupt_map_mask[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map-mask" , interrupt_map_mask, +sizeof (interrupt_map_mask)); +} + +static int fdt_property_vpci_ranges (libxl__gc *gc,void *fdt, +unsigned vpci_addr_cells, +unsigned cpu_addr_cells, +unsigned vpci_size_cells, +unsigned num_regs, ...) +{ +uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)]; +be32 *cells = ®s[0 ]; +int i; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the memory bit field */ + arg =va_arg (ap, uint64_t); +set_cell (&cells,1 , arg); + +/* Set the vpci bus address */ + arg = vpci_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells,2 , arg); + +/* Set the cpu bus address where vpci address is mapped */ + arg = cpu_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, cpu_addr_cells, arg); + +/* Set the vpci size requested */ + arg = vpci_size_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, vpci_size_cells,arg); + } +va_end (ap); + +return fdt_property (fdt,"ranges" , regs,sizeof (regs)); +} + +static int fdt_property_vpci_interrupt_map (libxl__gc *gc,void *fdt, +unsigned child_unit_addr_cells, +unsigned child_interrupt_specifier_cells, +unsigned parent_unit_addr_cells, +unsigned parent_interrupt_specifier_cells, +unsigned num_regs, ...) +{ +uint32_t interrupt_map[num_regs * (child_unit_addr_cells + + child_interrupt_specifier_cells + parent_unit_addr_cells + + parent_interrupt_specifier_cells +1 )]; +be32 *cells = &interrupt_map[0 ]; +int i,j; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the child unit address*/ +for (j =0 ; j < child_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the child interrupt specifier*/ +for (j =0 ; j < child_interrupt_specifier_cells ; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the interrupt-parent*/ +set_cell (&cells,1 ,GUEST_PHANDLE_GIC ); + +/* Set the parent unit address*/ +for (j =0 ; j < parent_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the parent interrupt specifier*/ +for (j =0 ; j < parent_interrupt_specifier_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map" , interrupt_map, +sizeof (interrupt_map)); +} +static int make_root_properties (libxl__gc *gc,const libxl_version_info *vers,void *fdt)@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,return 0 ; } +static int make_vpci_node (libxl__gc *gc,void *fdt, +const struct arch_info *ainfo, +struct xc_dom_image *dom) +{ +int res; +const uint64_t vpci_ecam_base =GUEST_VPCI_ECAM_BASE ; +const uint64_t vpci_ecam_size =GUEST_VPCI_ECAM_SIZE ; +const char *name =GCSPRINTF ("pcie@%" PRIx64, vpci_ecam_base); + + res =fdt_begin_node (fdt, name); +if (res)return res; + + res =fdt_property_compat (gc, fdt,1 ,"pci-host-ecam-generic" ); +if (res)return res; + + res =fdt_property_string (fdt,"device_type" ,"pci" ); +if (res)return res; + + res =fdt_property_regs (gc, fdt,GUEST_ROOT_ADDRESS_CELLS , +GUEST_ROOT_SIZE_CELLS ,1 , vpci_ecam_base, vpci_ecam_size); +if (res)return res; + + res =fdt_property_vpci_bus_range (gc, fdt,2 ,0 ,255 ); +if (res)return res; + + res =fdt_property_cell (fdt,"linux,pci-domain" ,0 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#address-cells" ,3 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#size-cells" ,2 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#interrupt-cells" ,1 ); +if (res)return res; + + res =fdt_property_string (fdt,"status" ,"okay" ); +if (res)return res; + + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS, + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,
tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837
Quoted 10 lines
tools/libxl/libxl_arm.c +10
-0+3 , +GUEST_VPCI_ADDR_TYPE_MEM ,GUEST_VPCI_MEM_PCI_ADDR , +GUEST_VPCI_MEM_CPU_ADDR ,GUEST_VPCI_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM ,GUEST_VPCI_PREFETCH_MEM_PCI_ADDR , +GUEST_VPCI_PREFETCH_MEM_CPU_ADDR ,GUEST_VPCI_PREFETCH_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_IO ,GUEST_VPCI_IO_PCI_ADDR , + GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE); +if (res)return res; + + res =fdt_property_vpci_interrupt_map_mask (gc, fdt,4 ,0 ,0 ,0 ,7 );
it would make sense to separate out child_unit_addr_cells and child_interrupt_specifier_cells here like we do below with fdt_property_vpci_interrupt_map
tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850
Quoted 13 lines
tools/libxl/libxl_arm.c +13
-0+if (res)return res; + +/* +* Legacy interrupt is forced and assigned to the guest. +* This will be removed once we have implementation for MSI support. +* +*/ + res =fdt_property_vpci_interrupt_map (gc, fdt,3 ,1 ,0 ,3 , +4 , +0 ,0 ,0 ,1 ,0 ,136 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,2 ,0 ,137 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,3 ,0 ,138 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,4 ,0 ,139 ,DT_IRQ_TYPE_LEVEL_HIGH );
The 4 interrupt allocated for this need to be defined in xen/include/public/arch-arm.h as well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.
Quoted 11 lines
tools/libxl/libxl_arm.c +8
-0+if (res)return res; + + res =fdt_end_node (fdt); +if (res)return res; + +return 0 ; +} +static const struct arch_info *get_arch_info (libxl__gc *gc,const struct xc_dom_image *dom) {
[...]
Quoted 8 lines
xen/include/public/arch-arm.h +0
-0@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438
Quoted 10 lines
xen/include/public/arch-arm.h +10
-0+#define GUEST_PCI_ADDRESS_CELLS 3 +#define GUEST_PCI_SIZE_CELLS 2 + +/* PCI-PCIe memory space types */ +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000) + +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)
It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440
Quoted 2 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000) +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
0xC0200800 looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454
Quoted 13 lines
xen/include/public/arch-arm.h +13
-0+/* +* This is hardcoded values for the real PCI physical addresses. +* This will be removed once we read the real PCI-PCIe physical +* addresses form the config space and map to the guest memory map +* when assigning the device to guest via VPCI. +* +*/ +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000) +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000) +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000) + +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)
How did you chose these sizes? GUEST_VPCI_MEM_SIZE and/or GUEST_VPCI_PREFETCH_MEM_SIZE are supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?
If so, maybe we need to increase GUEST_VPCI_MEM_SIZE to a couple of GB and GUEST_VPCI_PREFETCH_MEM_SIZE to even more?
Quoted 5 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000) +/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniQuoted 10 lines · Stefano Stabellini
+ Jan, Andrew, Roger
Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)
Quoted 3 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
Quoted 7 lines
Quoted 6 lines
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
What is this Change-Id property?
Gerrit ;)
Quoted 61 lines
Quoted 61 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-)create mode 100644 xen/arch/arm/pci/Makefile create mode 100644
xen/arch/arm/pci/pci-access.ccreate mode 100644xen/arch/arm/pci/pci-host-common.ccreate mode 100644xen/arch/arm/pci/pci-host-generic.ccreate mode 100644xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +7
-0@@ -31,6 +31,13 @@ menu "Architecture Features" source "arch/Kconfig" +config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. + config ACPI bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64xen/arch/arm/Makefile +1
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_ARM_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.o
Quoted 8 lines
xen/arch/arm/pci/pci-access.c line 2
Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd.
I think SPDX will fit better in any new code.
Quoted 169 lines
Quoted 79 lines
xen/arch/arm/pci/pci-access.c new file +79
-0+* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len) +{ +int rc; +uint32_t val =GENMASK (0 , len *8 ); + +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val); +if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)This looks like a subset of
xen/arch/x86/x86_64/mmconfig_64.c?MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.
Quoted 72 lines
xen/arch/arm/pci/pci-access.c new file +9
-0+/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +57
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> + +/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells; +uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +}
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 65
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +}
The two functions above seem to deal with the same resources, e.g. cfg->win
and map/unmap. Would it make sense to align those, something like
s/pci_remap_cfgspace/pci_ecam_alloc and pci_ecam_alloc handles cfg->win?
Or anything which makes them look init/fini style?
Quoted 12 lines
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12
-0+ +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +int err; +struct pci_config_window *cfg; +paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; +
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 85
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) { + cfg->busn_start =0 ; + cfg->busn_end =0xff ; +printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ;
So, if bus start is, for example, 0x10 then we'll end up with bus end at (0x10 + 0xff) > 0xff
which doesn't seem to be what we want
Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c line 91
Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6
-0+ } + +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit;
Quoted 16 lines
Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12
-0+ cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size);
I am fine with "win", but can we think of something that tells us that
"win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"
while reading?
Quoted 36 lines
Quoted 36 lines
xen/arch/arm/pci/pci-host-common.c new file +36
-0+if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } + +return cfg; + +err_exit_remap: +printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void ) +{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge;
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 148
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+struct pci_config_window *cfg; +u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge ) +return -ENOMEM ; + +/* Parse and map our Configuration Space windows */
Do you expect multiple windows here as the comment says?
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c line 151
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3
-0+ cfg =gen_pci_init (dev, ops); +if ( !cfg ) +return -ENOMEM ;In case of errors the allocated bridge is not freed.
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops;
Can we have some sort of dummy ops so we don't have to check for ops != NULL every time
we read/write config? Is it really possible that we have ops set to NULL after we have
the development finished?
Quoted 19 lines
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ; + } + + bridge->segment = (u16 )segment;My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c line 165
Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?
Why? Do you expect bridge access in some specific order so ordered
list will make it faster?
Quoted 75 lines
Quoted 60 lines
xen/arch/arm/pci/pci-host-common.c new file +32
-0+return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; + +list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ; +} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).
Quoted 5 lines
Maybe
pci_config_get_addressor something like that?Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2
-0+{ +struct pci_config_window *cfg = bridge->sysdata;
I am a bit confused of the naming ;)
We already have 2 maps: win -> ECAM base and now sysdata -> cfg.
Can we please have that aligned somehow so it is easier to follow?
Quoted 1 line
xen/arch/arm/pci/pci-host-generic.c line 33
Quoted 1 line
xen/arch/arm/pci/pci-host-generic.c new file +1
-0+unsigned int devfn_shift = cfg->ops->bus_shift -8 ;
We are not checking cfg->ops for NULL, so probably we do not want bridges
with NULL ops as well?
Quoted 253 lines
Quoted 19 lines
xen/arch/arm/pci/pci-host-generic.c new file +19
-0+ +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base; + +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift); + +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; +xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);please use a switch
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) {xen/arch/arm/pci/pci-host-generic.c line 83
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ *val = ~0 ; +return -ENODEV ; + } + +if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr);please use a switch
Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5
-0+return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = {xen/arch/arm/pci/pci-host-generic.c line 99
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ .bus_shift = 20, + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write, + } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops },spurious blank line
Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31
-0+ { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +71
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 :xen/arch/arm/pci/pci.c line 81
Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10
-0+ param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } + +return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.
Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable; + +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init (); + +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +0
-0xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933
Quoted 4 lines
xen/arch/arm/setup.c +1
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init ();
pci_initshould probably be an initcallQuoted 14 lines
xen/arch/arm/setup.c +0
-0do_initcalls ();/* xen/include/asm-arm/device.h +1
-0@@ -4,6 +4,7 @@enum device_type {DEV_DT , +DEV_PCI , };
Quoted 8 lines
xen/include/asm-arm/device.h typedef struct device device_t; · line 30
Quoted 8 lines
xen/include/asm-arm/device.h +0
-3struct dev_archdata {@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT)
Didn't we have a patch for that recently or talked about?
Quoted 91 lines
Quoted 45 lines
xen/include/asm-arm/device.h +3
-0+#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum device_class {DEVICE_SERIAL ,DEVICE_IOMMU ,DEVICE_GIC , +DEVICE_PCI ,/* Use for error */ DEVICE_UNKNOWN , };xen/include/asm-arm/pci.h +25
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ +xen/include/asm-arm/pci.h line 36
Quoted 10 lines
xen/include/asm-arm/pci.h +9
-0+#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};Are you actually using struct device in struct
arch_pci_dev? struct device is already part of structdt_device_nodeand a pointer to it is stored inbridge->dt_node.Quoted 15 lines
xen/include/asm-arm/pci.h +15
-0+#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/* +* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops; +void __iomem *win;xen/include/asm-arm/pci.h line 62
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val);I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?
Can we use
pci_sbdf_tfor the sbdf parameter?
Quoted 15 lines
xen/include/asm-arm/pci.h line 73
Quoted 11 lines
xen/include/asm-arm/pci.h +11
-0+}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +};Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename
pci_ecam_opsto something else, maybe simply "pci_ops".
Yes, please, bear in mind that we are about to work with this code on
non-ECAM HW from the very beginning, so this is something that we would
like to see from the ground up
Quoted 29 lines
Quoted 29 lines
xen/include/asm-arm/pci.h +22
-1+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops; }; -#endif /* __X86_PCI_H__ */ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus); + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops); + +void pci_init (void ); + +#else /*!CONFIG_ARM_PCI*/ +struct arch_pci_dev { }; +static inline void pci_init (void ) { } +#endif /*!CONFIG_ARM_PCI*/ +#endif /* __ARM_PCI_H__ */ Signature · 1 line
2.17.1
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Stefano StabelliniQuoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
Quoted 17 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +2
-1@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h>
Quoted 9 lines
xen/arch/arm/physdev.c line 21
Quoted 9 lines
xen/arch/arm/physdev.c +5
-2int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI
In the cover letter you were saying "we are not enabling the HAS_PCI and HAS_VPCI flags for ARM".
Is this still valid?
Quoted 36 lines
Quoted 20 lines
xen/arch/arm/physdev.c +20
-0+case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
Quoted 9 lines
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctlJan, Andrew, Roger, any opinions?
I think we can also have a look at [1] by Julien. That implementation,
IMO, had some thoughts on making Arm/x86 code common where possible
[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci
Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
In reply to Stefano StabelliniQuoted 234 lines · Stefano Stabellini
Quoted 8 lines · Rahul Singh
libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.
We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.
Quoted 5 lines
A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).
Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
Same question
Quoted 189 lines
Signed-off-by Rahul Singh <[email protected]>--- tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + tools/xl/xl_parse.c | 7 ++ xen/include/public/arch-arm.h | 28 +++++ 4 files changed, 241 insertions(+)tools/libxl/libxl_arm.c +166
-0@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,return fdt_property (fdt,"reg" , regs,sizeof (regs)); } +static int fdt_property_vpci_bus_range (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t bus_range[num_cells]; +be32 *cells = &bus_range[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"bus-range" , bus_range,sizeof (bus_range)); +} + +static int fdt_property_vpci_interrupt_map_mask (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t interrupt_map_mask[num_cells]; +be32 *cells = &interrupt_map_mask[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map-mask" , interrupt_map_mask, +sizeof (interrupt_map_mask)); +} + +static int fdt_property_vpci_ranges (libxl__gc *gc,void *fdt, +unsigned vpci_addr_cells, +unsigned cpu_addr_cells, +unsigned vpci_size_cells, +unsigned num_regs, ...) +{ +uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)]; +be32 *cells = ®s[0 ]; +int i; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the memory bit field */ + arg =va_arg (ap, uint64_t); +set_cell (&cells,1 , arg); + +/* Set the vpci bus address */ + arg = vpci_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells,2 , arg); + +/* Set the cpu bus address where vpci address is mapped */ + arg = cpu_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, cpu_addr_cells, arg); + +/* Set the vpci size requested */ + arg = vpci_size_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, vpci_size_cells,arg); + } +va_end (ap); + +return fdt_property (fdt,"ranges" , regs,sizeof (regs)); +} + +static int fdt_property_vpci_interrupt_map (libxl__gc *gc,void *fdt, +unsigned child_unit_addr_cells, +unsigned child_interrupt_specifier_cells, +unsigned parent_unit_addr_cells, +unsigned parent_interrupt_specifier_cells, +unsigned num_regs, ...) +{ +uint32_t interrupt_map[num_regs * (child_unit_addr_cells + + child_interrupt_specifier_cells + parent_unit_addr_cells + + parent_interrupt_specifier_cells +1 )]; +be32 *cells = &interrupt_map[0 ]; +int i,j; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the child unit address*/ +for (j =0 ; j < child_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the child interrupt specifier*/ +for (j =0 ; j < child_interrupt_specifier_cells ; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the interrupt-parent*/ +set_cell (&cells,1 ,GUEST_PHANDLE_GIC ); + +/* Set the parent unit address*/ +for (j =0 ; j < parent_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the parent interrupt specifier*/ +for (j =0 ; j < parent_interrupt_specifier_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map" , interrupt_map, +sizeof (interrupt_map)); +} +static int make_root_properties (libxl__gc *gc,const libxl_version_info *vers,void *fdt)@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,return 0 ; } +static int make_vpci_node (libxl__gc *gc,void *fdt, +const struct arch_info *ainfo, +struct xc_dom_image *dom) +{ +int res; +const uint64_t vpci_ecam_base =GUEST_VPCI_ECAM_BASE ; +const uint64_t vpci_ecam_size =GUEST_VPCI_ECAM_SIZE ; +const char *name =GCSPRINTF ("pcie@%" PRIx64, vpci_ecam_base); + + res =fdt_begin_node (fdt, name); +if (res)return res; + + res =fdt_property_compat (gc, fdt,1 ,"pci-host-ecam-generic" ); +if (res)return res; + + res =fdt_property_string (fdt,"device_type" ,"pci" ); +if (res)return res; + + res =fdt_property_regs (gc, fdt,GUEST_ROOT_ADDRESS_CELLS , +GUEST_ROOT_SIZE_CELLS ,1 , vpci_ecam_base, vpci_ecam_size); +if (res)return res; + + res =fdt_property_vpci_bus_range (gc, fdt,2 ,0 ,255 ); +if (res)return res; + + res =fdt_property_cell (fdt,"linux,pci-domain" ,0 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#address-cells" ,3 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#size-cells" ,2 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#interrupt-cells" ,1 ); +if (res)return res; + + res =fdt_property_string (fdt,"status" ,"okay" ); +if (res)return res; + + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS, + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS,tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837
Quoted 10 lines
tools/libxl/libxl_arm.c +10
-0+3 , +GUEST_VPCI_ADDR_TYPE_MEM ,GUEST_VPCI_MEM_PCI_ADDR , +GUEST_VPCI_MEM_CPU_ADDR ,GUEST_VPCI_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM ,GUEST_VPCI_PREFETCH_MEM_PCI_ADDR , +GUEST_VPCI_PREFETCH_MEM_CPU_ADDR ,GUEST_VPCI_PREFETCH_MEM_SIZE , +GUEST_VPCI_ADDR_TYPE_IO ,GUEST_VPCI_IO_PCI_ADDR , + GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE); +if (res)return res; + + res =fdt_property_vpci_interrupt_map_mask (gc, fdt,4 ,0 ,0 ,0 ,7 );it would make sense to separate out
child_unit_addr_cellsandchild_interrupt_specifier_cellshere like we do below withfdt_property_vpci_interrupt_maptools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850
Quoted 13 lines
tools/libxl/libxl_arm.c +13
-0+if (res)return res; + +/* +* Legacy interrupt is forced and assigned to the guest. +* This will be removed once we have implementation for MSI support. +* +*/ + res =fdt_property_vpci_interrupt_map (gc, fdt,3 ,1 ,0 ,3 , +4 , +0 ,0 ,0 ,1 ,0 ,136 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,2 ,0 ,137 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,3 ,0 ,138 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,4 ,0 ,139 ,DT_IRQ_TYPE_LEVEL_HIGH );
Quoted 4 lines
The 4 interrupt allocated for this need to be defined in
xen/include/public/arch-arm.has well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.
I think this is more about shared interrupts support complexity
Quoted 71 lines
Quoted 13 lines
tools/libxl/libxl_arm.c +8
-0+if (res)return res; + + res =fdt_end_node (fdt); +if (res)return res; + +return 0 ; +} +static const struct arch_info *get_arch_info (libxl__gc *gc,const struct xc_dom_image *dom) {[...]
Quoted 8 lines
xen/include/public/arch-arm.h +0
-0@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438
Quoted 10 lines
xen/include/public/arch-arm.h +10
-0+#define GUEST_PCI_ADDRESS_CELLS 3 +#define GUEST_PCI_SIZE_CELLS 2 + +/* PCI-PCIe memory space types */ +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000) + +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440
Quoted 2 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000) +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
0xC0200800looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454
Quoted 13 lines
xen/include/public/arch-arm.h +13
-0+/* +* This is hardcoded values for the real PCI physical addresses. +* This will be removed once we read the real PCI-PCIe physical +* addresses form the config space and map to the guest memory map +* when assigning the device to guest via VPCI. +* +*/ +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000) +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000) +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000) + +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)How did you chose these sizes?
GUEST_VPCI_MEM_SIZEand/orGUEST_VPCI_PREFETCH_MEM_SIZEare supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?If so, maybe we need to increase
GUEST_VPCI_MEM_SIZEto a couple of GB andGUEST_VPCI_PREFETCH_MEM_SIZEto even more?Quoted 5 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000) +/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Oleksandr AndrushchenkoHi,
Quoted 9 lines · Oleksandr Andrushchenko
Quoted 8 lines
xen/arch/arm/pci/pci-access.c line 2
Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. I think SPDX will fit better in any new code.
While I would love to use SPDX in Xen, there was some push back in the past to use it. So the new code should use the full-blown copyright until there is an agreement to use it.
Quoted 8 lines
Quoted 4 lines
xen/arch/arm/pci/pci-access.c line 3
Quoted 2 lines
xen/arch/arm/pci/pci-access.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?Why? Do you expect bridge access in some specific order so ordered
list will make it faster?
Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.
However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Oleksandr AndrushchenkoHi,
Quoted 84 lines · Oleksandr Andrushchenko
Quoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
Quoted 17 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +2
-1@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> Quoted 9 lines
xen/arch/arm/physdev.c line 21
Quoted 9 lines
xen/arch/arm/physdev.c +5
-2int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI In the cover letter you were saying "we are not enabling the
HAS_PCIandHAS_VPCIflags for ARM".Is this still valid?
Quoted 42 lines
Quoted 20 lines
xen/arch/arm/physdev.c +20
-0+case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl
Quoted 6 lines
Quoted 3 lines
Jan, Andrew, Roger, any opinions?
I think we can also have a look at [1] by Julien. That implementation,
IMO, had some thoughts on making Arm/x86 code common where possible
There are some ideas on how I would like to see the split, although they need some cleanup. :)
In particular, I was expecting some preparatory work to get the existing PCI code non-x86 specific.
Regarding the hypercall Stefano mentionned, I think it should be possible to make it common if we abstract the NUMA node lookup part.
Cheers,
Quoted 4 lines
[1] https://xenbits.xen.org/gitweb/?p=people/julieng/xen-unstable.git;a=shortlog;h=refs/heads/dev-pci
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Rahul SinghHi Rahul,
Quoted 10 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49Signed-off-by Rahul Singh <[email protected]>
Quoted 12 lines
--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-)
As a general comment, I would suggest to split the patch in smaller chunk. This would help the review and also allow to provide more explanation on what is done.
For instance, I think it is possible to a split looking like:
- Add framework to access an hostbridge
- Add support for ECAM
- Add code to initialize the PCI subsystem
There is also some small fixes in this code that probably can move in there own patches.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniHi,
Quoted 15 lines · Stefano Stabellini
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 11 lines
xen/arch/arm/pci/pci-host-common.c new file +11
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ; + } + + bridge->segment = (u16 )segment;My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
Quoted 5 lines
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
So we definitely need a segment number in hand because this is what the admin will use to assign a PCI device to a guest.
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.
Looking at the code, I think Linux is using 'segment' as 'domain'. So they should be interchangeably. I am not sure what other OS does though.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
In reply to Stefano StabelliniHi,
Quoted 15 lines · Stefano Stabellini
tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850
Quoted 13 lines · Rahul Singh
tools/libxl/libxl_arm.c +13
-0+if (res)return res; + +/* +* Legacy interrupt is forced and assigned to the guest. +* This will be removed once we have implementation for MSI support. +* +*/ + res =fdt_property_vpci_interrupt_map (gc, fdt,3 ,1 ,0 ,3 , +4 , +0 ,0 ,0 ,1 ,0 ,136 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,2 ,0 ,137 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,3 ,0 ,138 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,4 ,0 ,139 ,DT_IRQ_TYPE_LEVEL_HIGH );
Quoted 3 lines
The 4 interrupt allocated for this need to be defined in
xen/include/public/arch-arm.has well. Also, why would we want to get rid of the legacy interrupts completely?
With legacy interrupts, there are a few cases to take into account:
1) Two PCI devices from the same hostbridge are assigned to
different cases. As SPIs (used for legacy interrupts) can only be routed to one guest, we would need to now be able to share them. This raises the question when to EOI the physical interrupts. AFAICT, Linux has some code to deal with it using a timer if it takes too long.
2) Two PCI devices from two distinct hostbridge are assigned to the
same virtual hostbridge. Legacy interrupts would need to be virtual and we would possibly need to merge multiple physical interrupts into one virtual.
3) A mix of virtual and physical PCI devices are assigned to the
same host bridges. Same as 2) legacy interrupts would need to be virtual.
Given the complexity of handling interrupts legacy and the fact they will largely be slower compare to MSIs, I would rather focus on MSIs at first.
We can then decide to implement legacy interrupts if there is a real need.
Quoted 2 lines
It would be possible to still
find device or software that rely on them.
PCI devices without MSI support is getting extremely rare. There are actually Arm HW that doesn't support legacy interrupts at all (such as Thunder-X). I can't tell for the software.
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Rahul SinghQuoted 64 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +7
-0@@ -31,6 +31,13 @@ menu "Architecture Features" source "arch/Kconfig" +config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. + config ACPI bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64xen/arch/arm/Makefile +1
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_ARM_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +0
-0
xen/arch/arm/pci/Makefile line 4
Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.o
The Kconfig option mentions the support being explicitly for ARM64, would it be better to place the code in arch/arm/arm64 then?
Quoted 19 lines
xen/arch/arm/pci/pci-access.c new file +13
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License
xen/arch/arm/pci/pci-access.c line 23
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len)
Please align with the opening parenthesis (here and everywhere in the patch series).
xen/arch/arm/pci/pci-access.c line 26
Quoted 3 lines
xen/arch/arm/pci/pci-access.c new file +3
-0+{ +int rc; +uint32_t val =GENMASK (0 , len *8 );
You can just set val = ~0. The return type of the pci_conf_readXX helpers will already truncate the value.
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);
I had a patch to add a custom modifier to out printf format in order to handle pci_sbdf_t natively:
https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
xen/arch/arm/pci/pci-access.c line 40
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val);
There's no need for the uint32_t cast, the sbdf field is already of such type.
Quoted 41 lines
xen/arch/arm/pci/pci-access.c new file +41
-0+if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \
xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)
Please use uintXX_t.
Also, It's nice to add some kind of signals for cscope and friends so they can find the autogenerated functions, ie:
#define pci_conf_read8
#undef pci_conf_read8
#define pci_conf_read16
#undef pci_conf_read16
...
It's tedious but helps future users find where the code is generated.
Quoted 46 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+ +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +30
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> +
xen/arch/arm/pci/pci-host-common.c line 40
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells;
It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use uint32_t here.
Quoted 29 lines
xen/arch/arm/pci/pci-host-common.c new file +29
-0+uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +} + +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +} + +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{
xen/arch/arm/pci/pci-host-common.c line 79
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+int err; +struct pci_config_window *cfg; +paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; + + err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) {
Braces
Quoted 37 lines
xen/arch/arm/pci/pci-host-common.c new file +37
-0+ cfg->busn_start =0 ; + cfg->busn_end =0xff ; +printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ; + } + +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit; + + cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size); +if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } +
xen/arch/arm/pci/pci-host-common.c line 126
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+return cfg; + +err_exit_remap: +printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void )
^ extra space
Quoted 15 lines
xen/arch/arm/pci/pci-host-common.c new file +15
-0+{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge; +struct pci_config_window *cfg;
xen/arch/arm/pci/pci-host-common.c line 151
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge ) +return -ENOMEM ; + +/* Parse and map our Configuration Space windows */ + cfg =gen_pci_init (dev, ops); +if ( !cfg ) +return -ENOMEM ;
You are leaking the allocation of bridge here ...
xen/arch/arm/pci/pci-host-common.c line 160
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +9
-0+ + bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ;
... and here.
Quoted 18 lines
xen/arch/arm/pci/pci-host-common.c new file +18
-0+ } + + bridge->segment = (u16 )segment; + +list_add_tail (&bridge->node, &pci_host_bridges); + +return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; +
xen/arch/arm/pci/pci-host-common.c line 188
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ;
This can be much shorter:
struct pci_host_bridge *pci_find_host_bridge(uint16_t segment, uint8_t bus)
{
struct pci_host_bridge *bridge;
list_for_each_entry( bridge, &pci_host_bridges, node )
if ( bridge->segment == segment )
return bridge;
return NULL;
}Albeit I'm confused by the fact that you pass a bus number that's completely unused.
Quoted 36 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +
xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)
You would be better passing a pci_sbdf_t directly here. Also 'where' should be renamed to offset, or reg, and be made unsigned int. AFAICT you will never pass a negative value here.
For sanity you should also assert that the offset falls between the PCI config space used by the device, in order to easily catch with wrong offsets being used.
xen/arch/arm/pci/pci-host-generic.c line 32
Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2
-0+{ +struct pci_config_window *cfg = bridge->sysdata;
const
xen/arch/arm/pci/pci-host-generic.c line 38
Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6
-0+unsigned int devfn_shift = cfg->ops->bus_shift -8 ; + +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base;
IMO adding newlines between variable definitions is not helpful, but that's my taste.
Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5
-0+ +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift);
xen/arch/arm/pci/pci-host-generic.c line 53
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where);
You can initialize at definition.
xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 9 lines
xen/arch/arm/pci/pci-host-generic.c new file +9
-0+if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);
Please use a switch, and check against specific values. The default case should be a BUG();. See pci_conf_read from x86 for an example.
Quoted 146 lines
xen/arch/arm/pci/pci-host-generic.c new file +69
-0+ +return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) { + *val = ~0 ; +return -ENODEV ; + } + +if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr); + +return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = { + .bus_shift =20 , + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write, + } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops }, + + { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +71
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 :
xen/arch/arm/pci/pci.c line 81
Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10
-0+ param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } + +return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);
You need to introduce the documentation for the parameter at docs/misc/xen-command-line.pandoc
Albeit I'm not sure I like it, why do you need to enable PCI explicitly?
Shouldn't it be discovered automatically and enabled by default?
xen/arch/arm/pci/pci.c line 89
Quoted 7 lines
xen/arch/arm/pci/pci.c new file +7
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable;
Just return here, there's not point in having a label to perform a return.
xen/arch/arm/pci/pci.c line 94
Quoted 5 lines
xen/arch/arm/pci/pci.c new file +5
-0+ +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init ();
Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
Also for ACPI you will get called by acpi_boot_init, and likely need to implement a acpi_mmcfg_init or pci_mmcfg_arch_{init,enable}. I'm not sure whether the code in acpi_mmcfg_init could be made shared between both x86 and Arm.
Quoted 49 lines
xen/arch/arm/pci/pci.c new file +18
-0+ +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +2
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init (); +do_initcalls ();/* xen/include/asm-arm/device.h +1
-2@@ -4,6 +4,7 @@enum device_type {DEV_DT , +DEV_PCI , };struct dev_archdata {@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0)
xen/include/asm-arm/device.h typedef struct device device_t; · line 37
Quoted 10 lines
xen/include/asm-arm/device.h +3
-1-#define dev_is_dt (dev) ((dev->type == DEV_DT) +#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum device_class {DEVICE_SERIAL ,DEVICE_IOMMU ,DEVICE_GIC , +DEVICE_PCI ,
It seems to be like this wants to be DEVICE_PCI_HOST_BRIDGE or some such, since this is not used to identify all PCI devices, but just bridges?
Quoted 36 lines
xen/include/asm-arm/device.h +0
-0/* Use for error */ DEVICE_UNKNOWN, };xen/include/asm-arm/pci.h +25
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ +
xen/include/asm-arm/pci.h line 36
Quoted 10 lines
xen/include/asm-arm/pci.h +9
-0+#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};
This seems to be completely unused?
Quoted 5 lines
xen/include/asm-arm/pci.h +5
-0+ +#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/*
xen/include/asm-arm/pci.h line 51
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops;
const?
Quoted 23 lines
xen/include/asm-arm/pci.h +23
-0+void __iomem *win; +}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val); +}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +}; +
xen/include/asm-arm/pci.h line 84
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops;
You seem to introduce a lot of ops structs, yet there's only one implementation the generic ECAM one, and adding such complexity should IMO be done when further implementations are added. Also given this is a fully ECAM compliant bridge you could just use most of the existing logic for x86?
I understand the discovery needs to be different, but x86 MCFG logic should already be capable of handling multiple ECAM regions.
I also agree with Julien that splitting this into separate patch would make it easier to review. For example you can start with the discovery logic, followed by the initialization and the add the accessors to the config space lastly.
Thanks, Roger.
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Stefano StabelliniQuoted 73 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
Quoted 46 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +27
-3@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI +case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
Quoted 7 lines
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctlJan, Andrew, Roger, any opinions?
Oh, physdev ops don't have a common entry point, it's all per-arch. Since Arm has no physdev ops at all, I think we should start by adding a common do_physdev_op and move PHYSDEVOP_pci_device_add into it, leaving the rest of x86 operations as arch_do_physdev_op.
Thanks, Roger.
Re: [RFC PATCH v1 3/4] xen/arm: Enable the existing x86 virtual PCI support for ARM.
In reply to Rahul SinghQuoted 11 lines · Rahul Singh
The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.
A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space,XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.
VPCI MSI support is disable for ARM as it is not tested on ARM.
I'm not seeing anything on this patch that would disable vPCI MSI support?
Quoted 70 lines
Change-Id I5501db2781f8064640403fecce53713091cd9ab4Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Makefile | 1 + xen/arch/arm/domain.c | 4 ++ xen/arch/arm/vpci.c | 102 ++++++++++++++++++++++++++++++++++ xen/arch/arm/vpci.h | 37 ++++++++++++ xen/drivers/passthrough/pci.c | 7 +++ xen/include/asm-arm/domain.h | 5 ++ xen/include/public/arch-arm.h | 4 ++ 7 files changed, 160 insertions(+) create mode 100644 xen/arch/arm/vpci.c create mode 100644 xen/arch/arm/vpci.hxen/arch/arm/Makefile +1
-0@@ -7,6 +7,7 @@ obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ obj-$(CONFIG_ARM_PCI) += pci/ +obj-$(CONFIG_HAS_VPCI) += vpci.o obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/domain.c +4
-0@@ -39,6 +39,7 @@#include <asm/vtimer.h> #include "vuart.h" +#include "vpci.h" DEFINE_PER_CPU(struct vcpu *, curr_vcpu);@@ -747,6 +748,9 @@ int arch_domain_create(struct domain *d,if (is_hardware_domain (d) && (rc =domain_vuart_init (d)) ) goto fail; +if ( (rc =domain_vpci_init (d)) !=0 ) + goto fail; +return 0 ; fail:xen/arch/arm/vpci.c new file +15
-0@@ -0,0 +1,102 @@ +/* +* xen/arch/arm/vpci.c +* Copyright (c) 2020 Arm Ltd. +* +* Based on arch/x86/hvm/io.c +* Copyright (c) 2004, Intel Corporation. +* Copyright (c) 2005, International Business Machines Corporation. +* Copyright (c) 2008, Citrix Systems, Inc. +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +*
xen/arch/arm/vpci.c line 25
Quoted 10 lines
xen/arch/arm/vpci.c new file +10
-0+* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ +#include <xen/sched.h> +#include <asm/mmio.h> + +/* Do some sanity checks. */ +static bool vpci_mmio_access_allowed (unsigned int reg,unsigned int len)
This is just a copy of vpci_access_allowed from x86, I think you should consider moving the function to common vpci code and just share it between x86 and Arm?
Quoted 40 lines
xen/arch/arm/vpci.c new file +40
-0+{ +/* Check access size. */ +if ( len !=1 && len !=2 && len !=4 && len !=8 ) +return false; + +/* Check that access is size aligned. */ +if ( (reg & (len -1 )) ) +return false; + +return true; +} + +static int vpci_mmio_read (struct vcpu *v,mmio_info_t *info, +register_t *r,void *priv) +{ +unsigned int reg; +pci_sbdf_t sbdf; +uint32_t data =0 ; +unsigned int size =1U << info->dabt.size; + + sbdf.bdf = (((info->gpa) &0x0ffff000 ) >>12 ); + reg = (((info->gpa) &0x00000ffc ) | (info->gpa &3 )); + +if ( !vpci_mmio_access_allowed (reg, size) ) +return 1 ; + + data =vpci_read (sbdf, reg, size); + +memcpy (r, &data, size); + +return 1 ; +} + +static int vpci_mmio_write (struct vcpu *v,mmio_info_t *info, +register_t r,void *priv) +{ +unsigned int reg; +pci_sbdf_t sbdf; +uint32_t data = r; +unsigned int size =1U << info->dabt.size;
xen/arch/arm/vpci.c line 75
Quoted 10 lines
xen/arch/arm/vpci.c new file +10
-0+ + sbdf.bdf = (((info->gpa) &0x0ffff000 ) >>12 ); + reg = (((info->gpa) &0x00000ffc ) | (info->gpa &3 )); + +if ( !vpci_mmio_access_allowed (reg, size) ) +return 1 ; + +vpci_write (sbdf, reg, size, data); + +return 1 ;
Both functions will only return 1 always, so can likely drop the return value completely?
xen/arch/arm/vpci.c line 83
Quoted 8 lines
xen/arch/arm/vpci.c new file +8
-0+} + +static const struct mmio_handler_ops vpci_mmio_handler = { + .read = vpci_mmio_read, + .write = vpci_mmio_write, +}; + +int domain_vpci_init (struct domain *d)
FWIW, I think you can drop the domain_ prefix, vPCI is always tied to a domain.
xen/arch/arm/vpci.c line 85
Quoted 2 lines
xen/arch/arm/vpci.c new file +2
-0+{ +if ( !has_vpci (d) ||is_hardware_domain (d) )
I wouldn't add a hardware domain exception here, and just make sure the VPCI flag is not set for the hardware domain on Arm if you don't want to use it there.
xen/arch/arm/vpci.c line 89
Quoted 4 lines
xen/arch/arm/vpci.c new file +4
-0+return 0 ; + +register_mmio_handler (d, &vpci_mmio_handler, +GUEST_VPCI_ECAM_BASE ,GUEST_VPCI_ECAM_SIZE ,NULL );
Missing spaces, and proper indentation.
xen/arch/arm/vpci.c line 91
Quoted 2 lines
xen/arch/arm/vpci.c new file +2
-0+ +return 0 ;
Doesn't seem like domain_vpci_init can fail, so you can likely skip the return value?
Quoted 59 lines
xen/arch/arm/vpci.c new file +11
-0+} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ +xen/arch/arm/vpci.h new file +37
-0@@ -0,0 +1,37 @@ +/* +* xen/arch/arm/vpci.h +* Copyright (c) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License as published by +* the Free Software Foundation; either version 2 of the License, or +* (at your option) any later version. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +*/ + +#ifndef __ARCH_ARM_VPCI_H__ +#define __ARCH_ARM_VPCI_H__ + +#ifdef CONFIG_HAS_VPCI +int domain_vpci_init (struct domain *d); +#else +static inline int domain_vpci_init (struct domain *d) +{ +return 0 ; +} +#endif + +#endif /* __ARCH_ARM_VPCI_H__ */ + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/ xen/drivers/passthrough/pci.c +0
-0
xen/drivers/passthrough/pci.c int pci_add_device(u16 seg, u8 bus, u8 devfn, · line 808
Quoted 5 lines
xen/drivers/passthrough/pci.c +2
-0@@ -804,6 +804,13 @@ int pci_add_device(u16 seg, u8 bus, u8 devfn,else iommu_enable_device (pdev); +#ifdef CONFIG_ARM + ret =vpci_add_handlers (pdev);
Don't you need to drop the __hwdom_init from that function? Or else it might be freed by the time dom0 calls pci_add_device?
Quoted 13 lines
xen/drivers/passthrough/pci.c +5
-0+if ( ret ) { +printk (XENLOG_ERR "setup of vPCI for failed: %d\n" ,ret); + goto out; + } +#endif pci_enable_acs (pdev); out:xen/include/asm-arm/domain.h +0
-0
xen/include/asm-arm/domain.h enum domain_type { · line 40
Quoted 7 lines
xen/include/asm-arm/domain.h +4
-0@@ -34,6 +34,11 @@ enum domain_type {/* The hardware domain has always its memory direct mapped. */ #define is_domain_direct_mapped (d) ((d) == hardware_domain) +/* For X86 VPCI is enabled and tested for PVH DOM0 only but +* for ARM we enable support VPCI for guest domain also. +*/ +#define has_vpci (d) (true)
Urg, that's kind of unconditional, couldn't you pass a flag for user-space in order to signal whether vPCI should be enabled? There's no point on enabling it if the domain doesn't support PCI passthrough, or if there are no PCI devices on the system.
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Roger Pau MonnéQuoted 5 lines · Roger Pau Monné
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +0
-0
Quoted 8 lines
xen/arch/arm/pci/Makefile line 4
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oThe
Kconfigoption mentions the support being explicitly for ARM64, would it be better to place the code inarch/arm/arm64then?
I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.
Quoted 11 lines
xen/arch/arm/pci/Makefile
Quoted 7 lines
xen/arch/arm/pci/Makefile new file +7
-0+ + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); + + if ( unlikely(!bridge) ) + { + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:
Quoted 7 lines
https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
[...]
Quoted 8 lines
xen/arch/arm/pci/Makefile
Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4
-0+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev, + struct pci_config_window *cfg) +{ + const __be32 *cells;It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use
uint32_there.
uint32_t would be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so __be32 is the best choice.
[...]
Quoted 11 lines
xen/arch/arm/pci/Makefile
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5
-0+ + if ( acpi_disabled ) + dt_pci_init(); + else + acpi_pci_init();Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 16 lines · Julien Grall
Quoted 5 lines · Roger Pau Monné
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +0
-0Quoted 8 lines
xen/arch/arm/pci/Makefile line 4
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oThe
Kconfigoption mentions the support being explicitly for ARM64, would it be better to place the code inarch/arm/arm64then?
Quoted 3 lines
I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.
Ack. Could the Kconfg be adjusted to not depend on ARM_64? Just stating it's only been tested on Arm64 would be enough IMO.
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/pci/Makefile
Quoted 7 lines
xen/arch/arm/pci/Makefile new file +7
-0+ + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); + + if ( unlikely(!bridge) ) + { + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:
Quoted 8 lines
Quoted 5 lines
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
I only had an Ack from Jan, so I was missing Intel and AMD Acks, which would now only be Intel since AMD has been absorbed by the x86 maintainers.
Quoted 11 lines
[...]
Quoted 8 lines
xen/arch/arm/pci/Makefile
Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4
-0+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev, + struct pci_config_window *cfg) +{ + const __be32 *cells;It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use
uint32_there.
Quoted 3 lines
uint32_twould be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so__be32is the best choice.
Oh, OK, so this is done to explicitly denote the endianness of a value on the type itself.
Quoted 8 lines
[...]
Quoted 6 lines
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5
-0+ + if ( acpi_disabled ) + dt_pci_init(); + else + acpi_pci_init();
Quoted 8 lines
Quoted 5 lines
Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.
Ack. In any case I think for ACPI PCI init will get called by acpi_mmcfg_init as part of acpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Julien Grall
Quoted 11 lines · Roger Pau Monné
xen/arch/arm/pci/pci.c line 94
Quoted 5 lines
xen/arch/arm/pci/pci.c new file +5
-0+ +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init ();Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
Quoted 7 lines
Quoted 2 lines
This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.
Ack. In any case I think for ACPI PCI init will get called by
acpi_mmcfg_initas part ofacpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?
Sorry I got confused and acpi_boot_init is a x86 specific function, wrong context. If Arm is not using that path then maybe it makes sense to init PCI here for ACPI also.
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Roger Pau MonnéQuoted 17 lines · Roger Pau Monné
Quoted 16 lines · Julien Grall
Quoted 13 lines · Roger Pau Monné
xen/arch/arm/pci/Makefile line 4
Quoted 10 lines
xen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oThe
Kconfigoption mentions the support being explicitly for ARM64, would it be better to place the code inarch/arm/arm64then?
Quoted 6 lines
Quoted 3 lines
I don't believe any of the code in this series is very arm64 specific. I guess it was just only tested on arm64. So I would rather keep that under arm/pci.
Ack. Could the Kconfg be adjusted to not depend on
ARM_64? Just stating it's only been tested on Arm64 would be enough IMO.
We already have an option to select PCI (see CONFIG_HAS_PCI). So I would prefer if we reuse it (possible renamed to CONFIG_PCI) rather than inventing one for Arm specific.
Regarding the dependency, it will depend if it is possible to make it build easily on Arm32. If not, then we will need to keep the ARM_64.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
xen/arch/arm/pci/Makefile
Quoted 8 lines
xen/arch/arm/pci/Makefile new file +7
-0+ + struct pci_host_bridge *bridge = pci_find_host_bridge(sbdf.seg, sbdf.bus); + + if ( unlikely(!bridge) ) + { + printk(XENLOG_ERR "Unable to find bridge for "PRI_pci"\n", + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
Quoted 6 lines
Quoted 2 lines
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
I only had an Ack from Jan, so I was missing Intel and AMD Acks, which would now only be Intel since AMD has been absorbed by the x86 maintainers.
Ok. So, it should be easier to get it acked now :).
Quoted 12 lines
Quoted 12 lines
[...]
Quoted 8 lines
xen/arch/arm/pci/Makefile
Quoted 4 lines
xen/arch/arm/pci/Makefile new file +4
-0+static bool __init dt_pci_parse_bus_range(struct dt_device_node *dev, + struct pci_config_window *cfg) +{ + const __be32 *cells;It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use
uint32_there.
Quoted 6 lines
Quoted 3 lines
uint32_twould be incorrect because this is a 32-bit value always in big endian. I don't think we have other typedef to show it is a 32-bit BE value, so__be32is the best choice.Oh, OK, so this is done to explicitly denote the endianness of a value on the type itself.
That's correct. On Linux, they use sparse to then check the BE and LE fields are not mixed together. We don't have that on Xen, but at least this makes more obvious what we are using.
Quoted 15 lines
Quoted 15 lines
[...]
Quoted 11 lines
xen/arch/arm/pci/Makefile
Quoted 5 lines
xen/arch/arm/pci/Makefile new file +5
-0+ + if ( acpi_disabled ) + dt_pci_init(); + else + acpi_pci_init();Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
Quoted 7 lines
Quoted 2 lines
This is the common way we do it on Arm.... I would welcome any improvement, but I don't think this should be part of this work.
Ack. In any case I think for ACPI PCI init will get called by
acpi_mmcfg_initas part ofacpi_boot_init, so I'm not sure there's much point in having something about ACPI added here, as it seems this will be DT only?
acpi_boot_init() does not exist on Arm. Looking at x86, acpi_mmcfg_init() is not event called from that function.
In general, I would prefer if each subsystem takes care of initialization itself. This makes easier to figure out the difference between ACPI and DT. FWIW, this is inline with majority of the Arm code.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 14 lines · Julien Grall
Quoted 13 lines · Roger Pau Monné
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:
Quoted 8 lines
Quoted 5 lines
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).
Jan
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Hi Jan,
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 18 lines · Roger Pau Monné
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
Quoted 6 lines
Quoted 2 lines
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).
Looking at the thread, I can see you (relunctantly) acked any components that you are the sole maintainers. Kevin gave his acked for the vtd code and I gave it mine for the common code.
I would suggest to not rehash the argument unless another maintainer agree with your position. It loosk like to me the next step is for Roger (or someone else) to resend the patch so we could collect the missing ack (I think there is only one missing from Andrew).
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 17 lines · Julien Grall
Quoted 16 lines · Stefano Stabellini
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +7
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + + if( !dt_property_read_u32(dev, "linux,pci-domain", &segment) ) + { + printk(XENLOG_ERR "\"linux,pci-domain\" property in not availablein DT\n");
xen/arch/arm/pci/pci-host-common.c new file +4
-0+return -ENODEV ; + } + + bridge->segment = (u16 )segment;My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
Quoted 8 lines
Quoted 5 lines
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
So we definitely need a segment number in hand because this is what the admin will use to assign a PCI device to a guest.
Yeah
Quoted 2 lines
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.
As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by _SEG on ACPI systems.
The link above suggests that a Linux domain corresponds to (_SEG, _BBN) where _SEG is the segment and _BBN is the "Bus Number".
I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 8 lines · Julien Grall
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
+
list_add_tail(&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?Why? Do you expect bridge access in some specific order so ordered
list will make it faster?
Quoted 7 lines
Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.
However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).
I was suggesting ordering the list because it is a rather cheap
optimization: it is easy to implement and it typically leads to decent improvements (although it varies on a case by case basis). Something more sophisticated as you mention here would be even better.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniQuoted 9 lines · Stefano Stabellini
Quoted 3 lines · Julien Grall
On 24/07/2020 00:38, Stefano Stabellini wrote:
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by
_SEGon ACPI systems.
Quoted 7 lines
The link above suggests that a Linux domain corresponds to (
_SEG,_BBN) where_SEGis the segment and_BBNis the "Bus Number".I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.
You seem to argue on the name but this doesn't resolve the underlying problem. Indeed, all our external interfaces are expecting a segment number.
If they are not equal, then I fail to see why it would be useful to have this value in Xen. In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so Dom0 and Xen can synchronize on the segment number.
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 3 lines · Julien Grall
On 24/07/2020 00:38, Stefano Stabellini wrote:
The segment number is just a value defined by the software. So as long as Linux and Xen agrees with the number, then we should be ok.As far as I understand a Linux "domain" (linux,pci-domain in device tree) is a value defined by the software. The PCI segment has a definition in the PCI spec and it is returned by
_SEGon ACPI systems.The link above suggests that a Linux domain corresponds to (
_SEG,_BBN) where_SEGis the segment and_BBNis the "Bus Number".
Quoted 7 lines
Quoted 4 lines
I just would like to be precise with the terminology: if we are talking about domains in the linux sense of the word, as it looks like we are doing, then let's call them domain instead of segments which seem to have a different definition.
You seem to argue on the name but this doesn't resolve the underlying problem. Indeed, all our external interfaces are expecting a segment number.
Yes, you are right, that is a bigger problem.
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.
I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 7 lines
Quoted 3 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
From this discussion, it also looks like there is a mismatch between the implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that really represent the segment?
Cheers,
[1] https://elixir.bootlin.com/linux/latest/source/arch/arm64/kernel/pci.c#L74
Cheers,
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 23 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 7 lines
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?
I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property. And if we look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.
I do think we need to understand the definitions and the differences. Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.
So maybe it is OK to use domains == segments, but we need to verify this in the specs and also clarify the terminology we use in a doc for our own sanity -- I am hoping that Rahul can come up with a good explanation on the topic :-)
[1] https://stackoverflow.com/questions/49050847/how-is-pci-segmentdomain-related-to-multiple-host-bridgesor-root-bridges
[2] https://wiki.osdev.org/PCI_Express
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 29 lines · Stefano Stabellini
Quoted 27 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
Quoted 6 lines
Quoted 3 lines
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)
Quoted 5 lines
And if we
look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.I do think we need to understand the definitions and the differences.
+1
Quoted 8 lines
Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.
So maybe it is OK to use domains == segments, but we need to verify this in the specs and also clarify the terminology we use in a doc for our own sanity -- I am hoping that Rahul can come up with a good explanation on the topic :-)
I am a bit confused.... You were the one arguing that domain == (segment, bus) in this thread. So may I ask why the interpretation wouldn't be valid anymore?
Cheers,
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniQuoted 37 lines · Stefano Stabellini
Quoted 23 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 7 lines
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property. And if we look at https://www.devicetree.org/open-firmware/bindings/pci/pci2_1.pdf
"PCI domain" is described and seems to match the Linux definition.
Quoted 4 lines
I do think we need to understand the definitions and the differences. Reading online [1][2] it looks like a Linux PCI domain matches a "PCI Segment Group Number" in PCI Express which is probably why Linux is making the assumption that it is making.
If I may, I'd like to put the question a little differently, in the hope for me to understand the actual issue here: On the x86 side, by way of using ACPI, Linux and Xen "naturally" agree on segment numbering (as far as normal devices go; Intel's Volume Management Device concept still needs accommodating so that it would work with Xen). This includes the multiple host bridges case then naturally. How is the Device Tree model different from ACPI?
Jan
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Stefano StabelliniSorry for the late reply.
Quoted 10 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
I think by-mistake Gerrit Change-id is added in the patch series. I will remove the Change-Id in next version of the patch.
Quoted 63 lines
Quoted 48 lines
Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/physdev.c| 42 +++++++++++++++++++++++++++++++++++++++---1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +27
-3@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI +case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
Quoted 7 lines
xen/common/sysctl.c:do_sysctl
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl
Ok sure I will check if we can create a common entry for ARM and x86 for do_physdev_op().
Quoted 1 line
Jan, Andrew, Roger, any opinions?
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 31 lines · Julien Grall
Hi Jan,
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Julien Grall
Quoted 18 lines · Roger Pau Monné
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
Quoted 6 lines
Quoted 2 lines
I forgot about this patch :/. It would be good to revive it. Which acks are you missing?
It wasn't so much missing acks, but a controversy. And that not so much about switching existing users, but whether to indeed derive this from %p (which I continue to consider inefficient).
Quoted 8 lines
Looking at the thread, I can see you (relunctantly) acked any components that you are the sole maintainers. Kevin gave his acked for the vtd code and I gave it mine for the common code.
I would suggest to not rehash the argument unless another maintainer agree with your position. It loosk like to me the next step is for Roger (or someone else) to resend the patch so we could collect the missing ack (I think there is only one missing from Andrew).
I've rebased and sent the updated patch with the collected Acks.
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?
Quoted 8 lines
Quoted 2 lines
I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)
I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?
Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniQuoted 14 lines · Stefano Stabellini
+ Jan, Andrew, Roger
Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)
Quoted 3 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
Quoted 8 lines
Quoted 6 lines
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
What is this Change-Id property?
Will remove this in next patch series.
Quoted 161 lines
Quoted 148 lines
Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-)create mode 100644 xen/arch/arm/pci/Makefile create mode 100644
xen/arch/arm/pci/pci-access.ccreate mode 100644xen/arch/arm/pci/pci-host-common.ccreate mode 100644xen/arch/arm/pci/pci-host-generic.ccreate mode 100644xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +0
-0@@ -31,6 +31,13 @@ menu "Architecture Features"source "
arch/Kconfig"xen/arch/arm/Kconfig +7
-0+config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. +config ACPI
bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends onARM_64xen/arch/arm/Makefile +0
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oxen/arch/arm/pci/pci-access.c new file +81
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len) +{ +int rc; +uint32_t val =GENMASK (0 , len *8 ); + +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val); +if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)This looks like a subset of
xen/arch/x86/x86_64/mmconfig_64.c?
Quoted 3 lines
MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.
We have just started to investigate the ACPI support for PCI within XEN on ARM. Once we have understanding how we can support the ACPI we will have a look x86 MMCFG access and will check if we can have a common interface to access the configuration space.
Quoted 104 lines
Quoted 104 lines
xen/arch/arm/pci/pci-access.c new file +9
-0+/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +86
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> + +/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells; +uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +} + +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +} + +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +int err; +struct pci_config_window *cfg; +paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; + + err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) { + cfg->busn_start =0 ; + cfg->busn_end =0xff ; +printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ; + }
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 91
Quoted 5 lines
xen/arch/arm/pci/pci-host-common.c new file +5
-0+ +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit;Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?
As of now we are supporting the PCI controllers that have "pci-host-ecam-generic” compatible node in device tree binding.As per my understanding controller that have "pci-host-ecam-generic” compatible property have only one reg property value at index 0.
Yes there are PCI controllers that support ECAM and have more than one "reg" property values.As per my understanding once we support other controller in XEN at that time we have to handle the possibility of multiple addresses and that code is specific to that controller .
Quoted 55 lines
Quoted 55 lines
xen/arch/arm/pci/pci-host-common.c new file +53
-0+ cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size); +if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } + +return cfg; + +err_exit_remap: +printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void ) +{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge; +struct pci_config_window *cfg; +u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge )
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c line 151
Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6
-0+return -ENOMEM ; + +/* Parse and map our Configuration Space windows */ + cfg =gen_pci_init (dev, ops); +if ( !cfg ) +return -ENOMEM ;In case of errors the allocated bridge is not freed.
Will fix this in next version.
Quoted 16 lines
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +11
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ; + } + + bridge->segment = (u16 )segment;My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
Quoted 6 lines
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
As per my understanding linux domain number is an alias to segment number.Device tree binding uses the property domain number and ACPI uses the property segment number. We have just started the investigation on ACPI, once we have more information about MMCFG ACPI table how it works will share the information and will update the patches and design doc accordingly.
Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c line 165
Quoted 2 lines
xen/arch/arm/pci/pci-host-common.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?
As Julien also mentioned access to the PCI config space is random there is no need to have an ordered list.
Quoted 75 lines
Quoted 59 lines
xen/arch/arm/pci/pci-host-common.c new file +32
-0+return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; + +list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ; +} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).
Quoted 1 line
Maybe
pci_config_get_addressor something like that?
OK yes will rename the function name.
Quoted 52 lines
Quoted 24 lines
xen/arch/arm/pci/pci-host-generic.c new file +22
-0+{ +struct pci_config_window *cfg = bridge->sysdata; +unsigned int devfn_shift = cfg->ops->bus_shift -8 ; + +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base; + +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift); + +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; +xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);please use a switch
Quoted 14 lines
xen/arch/arm/pci/pci-host-generic.c new file +14
-0+return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) { + *val = ~0 ; +return -ENODEV ; + } +
Quoted 8 lines
xen/arch/arm/pci/pci-host-generic.c line 83
Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6
-0+if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr);please use a switch
Ok.
Quoted 11 lines
Quoted 11 lines
xen/arch/arm/pci/pci-host-generic.c new file +9
-0+return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = { + .bus_shift =20 , + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write,
Quoted 8 lines
xen/arch/arm/pci/pci-host-generic.c line 99
Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6
-0+ } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops },spurious blank line
Ok will fix this.
Quoted 117 lines
Quoted 117 lines
xen/arch/arm/pci/pci-host-generic.c new file +31
-0+ { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +78
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 : + param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } +
Quoted 9 lines
xen/arch/arm/pci/pci.c line 81
Quoted 3 lines
xen/arch/arm/pci/pci.c new file +3
-0+return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.
Ok yes will add it to the documentation.
Quoted 37 lines
Quoted 37 lines
xen/arch/arm/pci/pci.c new file +30
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable; + +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init (); + +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +0
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,
Quoted 5 lines
xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933
Quoted 3 lines
xen/arch/arm/setup.c +1
-0setup_virt_paging (); +pci_init ();
pci_initshould probably be an initcall.
OK.
Quoted 75 lines
Quoted 75 lines
xen/arch/arm/setup.c +0
-0do_initcalls ();/* xen/include/asm-arm/device.h +0
-0@@ -4,6 +4,7 @@enum
device_type{ DEV_DT,+
DEV_PCI,
};struct
dev_archdata{xen/include/asm-arm/device.h +2
-3@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT) +#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum
device_class
{
DEVICE_SERIAL,
DEVICE_IOMMU,
DEVICE_GIC,
+DEVICE_PCI,/* Use for error */ DEVICE_UNKNOWN, };xen/include/asm-arm/pci.h +31
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ + +#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI +
Quoted 8 lines
xen/include/asm-arm/pci.h line 36
Quoted 4 lines
xen/include/asm-arm/pci.h +3
-0+/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};Are you actually using struct device in struct
arch_pci_dev? struct device is already part of structdt_device_nodeand a pointer to it is stored inbridge->dt_node.
Will be using this going forward once we have full PCI passthrough support.
Quoted 31 lines
Quoted 17 lines
xen/include/asm-arm/pci.h +15
-0+#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/* +* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops; +void __iomem *win;xen/include/asm-arm/pci.h line 62
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val);I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?
Quoted 1 line
Can we use
pci_sbdf_tfor the sbdf parameter?
Ok will fix in next version.
Quoted 18 lines
xen/include/asm-arm/pci.h line 73
Quoted 13 lines
xen/include/asm-arm/pci.h +11
-0+}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +};Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename
pci_ecam_opsto something else, maybe simply "pci_ops".
Ok will have a look and modify accordingly.
Quoted 29 lines
Quoted 29 lines
xen/include/asm-arm/pci.h +22
-1+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops; }; -#endif /* __X86_PCI_H__ */ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus); + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops); + +void pci_init (void ); + +#else /*!CONFIG_ARM_PCI*/ +struct arch_pci_dev { }; +static inline void pci_init (void ) { } +#endif /*!CONFIG_ARM_PCI*/ +#endif /* __ARM_PCI_H__ */ Signature · 1 line
2.17.1
Re: [RFC PATCH v1 4/4] arm/libxl: Emulated PCI device tree node in libxl
In reply to Stefano StabelliniQuoted 13 lines · Stefano Stabellini
Quoted 8 lines · Rahul Singh
libxl will create an emulated PCI device tree node in the device tree to enable the guest OS to discover the virtual PCI during guest boot.
We introduced the new config option [vpci="ecam"] for guests. When this config option is enabled in a guest configuration, a PCI device tree node will be created in the guest device tree.
Quoted 7 lines
Quoted 5 lines
A new area has been reserved in the arm guest physical map at which the VPCI bus is declared in the device tree (reg and ranges parameters of the node).
Change-Id: I47d39cbe8184de2226f174644df9790ecc610ccd
Same question
I will remove the change-id in the next version.
Quoted 197 lines
Quoted 197 lines
Signed-off-by Rahul Singh <[email protected]>---
tools/libxl/libxl_arm.c | 200 ++++++++++++++++++++++++++++++++++ tools/libxl/libxl_types.idl | 6 + tools/xl/xl_parse.c | 7 ++ xen/include/public/arch-arm.h | 28 +++++ 4 files changed, 241 insertions(+)tools/libxl/libxl_arm.c +172
-0@@ -268,6 +268,130 @@ static int fdt_property_regs(libxl__gc *gc, void *fdt,return fdt_property (fdt,"reg" , regs,sizeof (regs)); } +static int fdt_property_vpci_bus_range (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t bus_range[num_cells]; +be32 *cells = &bus_range[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"bus-range" , bus_range,sizeof (bus_range)); +} + +static int fdt_property_vpci_interrupt_map_mask (libxl__gc *gc,void *fdt, +unsigned num_cells, ...) +{ +uint32_t interrupt_map_mask[num_cells]; +be32 *cells = &interrupt_map_mask[0 ]; +int i; +va_list ap; +uint32_t arg; + +va_start (ap, num_cells); +for (i =0 ; i < num_cells; i++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map-mask" , interrupt_map_mask, +sizeof (interrupt_map_mask)); +} + +static int fdt_property_vpci_ranges (libxl__gc *gc,void *fdt, +unsigned vpci_addr_cells, +unsigned cpu_addr_cells, +unsigned vpci_size_cells, +unsigned num_regs, ...) +{ +uint32_t regs[num_regs*(vpci_addr_cells+cpu_addr_cells+vpci_size_cells)]; +be32 *cells = ®s[0 ]; +int i; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the memory bit field */ + arg =va_arg (ap, uint64_t); +set_cell (&cells,1 , arg); + +/* Set the vpci bus address */ + arg = vpci_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells,2 , arg); + +/* Set the cpu bus address where vpci address is mapped */ + arg = cpu_addr_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, cpu_addr_cells, arg); + +/* Set the vpci size requested */ + arg = vpci_size_cells ?va_arg (ap, uint64_t) :0 ; +set_cell (&cells, vpci_size_cells,arg); + } +va_end (ap); + +return fdt_property (fdt,"ranges" , regs,sizeof (regs)); +} + +static int fdt_property_vpci_interrupt_map (libxl__gc *gc,void *fdt, +unsigned child_unit_addr_cells, +unsigned child_interrupt_specifier_cells, +unsigned parent_unit_addr_cells, +unsigned parent_interrupt_specifier_cells, +unsigned num_regs, ...) +{ +uint32_t interrupt_map[num_regs * (child_unit_addr_cells + + child_interrupt_specifier_cells + parent_unit_addr_cells + + parent_interrupt_specifier_cells +1 )]; +be32 *cells = &interrupt_map[0 ]; +int i,j; +va_list ap; +uint64_t arg; + +va_start (ap, num_regs); +for (i =0 ; i < num_regs; i++) { +/* Set the child unit address*/ +for (j =0 ; j < child_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the child interrupt specifier*/ +for (j =0 ; j < child_interrupt_specifier_cells ; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the interrupt-parent*/ +set_cell (&cells,1 ,GUEST_PHANDLE_GIC ); + +/* Set the parent unit address*/ +for (j =0 ; j < parent_unit_addr_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + +/* Set the parent interrupt specifier*/ +for (j =0 ; j < parent_interrupt_specifier_cells; j++) { + arg =va_arg (ap, uint32_t); +set_cell (&cells,1 , arg); + } + } +va_end (ap); + +return fdt_property (fdt,"interrupt-map" , interrupt_map, +sizeof (interrupt_map)); +} +static int make_root_properties (libxl__gc *gc,const libxl_version_info *vers,void *fdt)@@ -659,6 +783,79 @@ static int make_vpl011_uart_node(libxl__gc *gc, void *fdt,return 0 ; } +static int make_vpci_node (libxl__gc *gc,void *fdt, +const struct arch_info *ainfo, +struct xc_dom_image *dom) +{ +int res; +const uint64_t vpci_ecam_base =GUEST_VPCI_ECAM_BASE ; +const uint64_t vpci_ecam_size =GUEST_VPCI_ECAM_SIZE ; +const char *name =GCSPRINTF ("pcie@%" PRIx64, vpci_ecam_base); + + res =fdt_begin_node (fdt, name); +if (res)return res; + + res =fdt_property_compat (gc, fdt,1 ,"pci-host-ecam-generic" ); +if (res)return res; + + res =fdt_property_string (fdt,"device_type" ,"pci" ); +if (res)return res; + + res =fdt_property_regs (gc, fdt,GUEST_ROOT_ADDRESS_CELLS , +GUEST_ROOT_SIZE_CELLS ,1 , vpci_ecam_base, vpci_ecam_size); +if (res)return res; + + res =fdt_property_vpci_bus_range (gc, fdt,2 ,0 ,255 ); +if (res)return res; + + res =fdt_property_cell (fdt,"linux,pci-domain" ,0 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#address-cells" ,3 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#size-cells" ,2 ); +if (res)return res; + + res =fdt_property_cell (fdt,"#interrupt-cells" ,1 ); +if (res)return res; + + res =fdt_property_string (fdt,"status" ,"okay" ); +if (res)return res; + + res = fdt_property_vpci_ranges(gc, fdt, GUEST_PCI_ADDRESS_CELLS, + GUEST_ROOT_ADDRESS_CELLS, GUEST_PCI_SIZE_CELLS, + 3, + GUEST_VPCI_ADDR_TYPE_MEM, GUEST_VPCI_MEM_PCI_ADDR, + GUEST_VPCI_MEM_CPU_ADDR, GUEST_VPCI_MEM_SIZE, + GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM, GUEST_VPCI_PREFETCH_MEM_PCI_ADDR, + GUEST_VPCI_PREFETCH_MEM_CPU_ADDR, GUEST_VPCI_PREFETCH_MEM_SIZE, + GUEST_VPCI_ADDR_TYPE_IO, GUEST_VPCI_IO_PCI_ADDR,
Quoted 8 lines
tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 837
Quoted 4 lines
tools/libxl/libxl_arm.c +4
-0+ GUEST_VPCI_IO_CPU_ADDR, GUEST_VPCI_IO_SIZE); +if (res)return res; + + res =fdt_property_vpci_interrupt_map_mask (gc, fdt,4 ,0 ,0 ,0 ,7 );it would make sense to separate out
child_unit_addr_cellsandchild_interrupt_specifier_cellshere like we do below withfdt_property_vpci_interrupt_map
Ok will fix.
Quoted 16 lines
Quoted 5 lines
tools/libxl/libxl_arm.c +3
-0+if (res)return res; + +/* tools/libxl/libxl_arm.c static int make_vpl011_uart_node(libxl__gc *gc, void *fdt, · line 850
Quoted 10 lines
tools/libxl/libxl_arm.c +10
-0+* Legacy interrupt is forced and assigned to the guest. +* This will be removed once we have implementation for MSI support. +* +*/ + res =fdt_property_vpci_interrupt_map (gc, fdt,3 ,1 ,0 ,3 , +4 , +0 ,0 ,0 ,1 ,0 ,136 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,2 ,0 ,137 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,3 ,0 ,138 ,DT_IRQ_TYPE_LEVEL_HIGH , +0 ,0 ,0 ,4 ,0 ,139 ,DT_IRQ_TYPE_LEVEL_HIGH );
Quoted 5 lines
The 4 interrupt allocated for this need to be defined in
xen/include/public/arch-arm.has well. Also, why would we want to get rid of the legacy interrupts completely? It would be possible to still find device or software that rely on them.
Ok will fix that.
Regarding legacy interrupt we have just tested on one of the board don’t know how it will work on other boards. We will mostly support MSI and will see if we have to support the legacy interrupt going forward.
Quoted 24 lines
Quoted 12 lines
tools/libxl/libxl_arm.c +8
-0+if (res)return res; + + res =fdt_end_node (fdt); +if (res)return res; + +return 0 ; +} +static const struct arch_info *get_arch_info (libxl__gc *gc,const struct xc_dom_image *dom) {[...]
Quoted 8 lines
xen/include/public/arch-arm.h +0
-0@@ -426,6 +426,34 @@ typedef uint64_t xen_callback_t;#define GUEST_VPCI_ECAM_BASE xen_mk_ullong(0x10000000)#define GUEST_VPCI_ECAM_SIZE xen_mk_ullong(0x10000000)
Quoted 12 lines
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 438
Quoted 10 lines
xen/include/public/arch-arm.h +10
-0+#define GUEST_PCI_ADDRESS_CELLS 3 +#define GUEST_PCI_SIZE_CELLS 2 + +/* PCI-PCIe memory space types */ +#define GUEST_VPCI_ADDR_TYPE_PREFETCH_MEM xen_mk_ullong(0x42000000) +#define GUEST_VPCI_ADDR_TYPE_MEM xen_mk_ullong(0x02000000) +#define GUEST_VPCI_ADDR_TYPE_IO xen_mk_ullong(0x01000000) + +/* Guest PCI-PCIe memory space where config space and BAR will be available.*/ +#define GUEST_VPCI_PREFETCH_MEM_CPU_ADDR xen_mk_ullong(0x4000000000)It looks like it could conflict with GUEST_RAM1_BASE+GUEST_RAM1_SIZE?
Ok yes will fix that and will define the address ranges for guest once we finalised the VPCI topology for the guest. We are currently investigating if we want to follow the hardware topology for the guest or we will create a different virtual topology for the guest independent of the hw topology.
Quoted 8 lines
xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 440
Quoted 4 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_MEM_CPU_ADDR xen_mk_ullong(0x04020000) +#define GUEST_VPCI_IO_CPU_ADDR xen_mk_ullong(0xC0200800)
0xC0200800looks like it could conflict with
GUEST_RAM0_BASE+GUEST_RAM0_SIZE?
Same comment above.
Quoted 19 lines
Quoted 4 lines
xen/include/public/arch-arm.h +3
-0+/* +* This is hardcoded values for the real PCI physical addresses. +* This will be removed once we read the real PCI-PCIe physical xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 454
Quoted 10 lines
xen/include/public/arch-arm.h +10
-0+* addresses form the config space and map to the guest memory map +* when assigning the device to guest via VPCI. +* +*/ +#define GUEST_VPCI_PREFETCH_MEM_PCI_ADDR xen_mk_ullong(0x4000000000) +#define GUEST_VPCI_MEM_PCI_ADDR xen_mk_ullong(0x50000000) +#define GUEST_VPCI_IO_PCI_ADDR xen_mk_ullong(0x00000000) + +#define GUEST_VPCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) +#define GUEST_VPCI_MEM_SIZE xen_mk_ullong(0x08000000)How did you chose these sizes?
GUEST_VPCI_MEM_SIZEand/orGUEST_VPCI_PREFETCH_MEM_SIZEare supposed to potentially cover all the PCI BARs, including potential future hotplug devices, right?
Quoted 2 lines
If so, maybe we need to increase
GUEST_VPCI_MEM_SIZEto a couple of GB andGUEST_VPCI_PREFETCH_MEM_SIZEto even more?
Same comments above.
Quoted 9 lines
Quoted 9 lines
xen/include/public/arch-arm.h +2
-0+#define GUEST_VPCI_IO_SIZE xen_mk_ullong(0x00800000) +/* * 16MB == 4096 pages reserved for guest to use as a region to map its * grant table in.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Oleksandr AndrushchenkoQuoted 86 lines · Oleksandr Andrushchenko
Quoted 10 lines · Stefano Stabellini
+ Jan, Andrew, Roger
Please have a look at my comment on whether we should share the MMCFG code below, feel free to ignore the rest :-)
Quoted 3 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
Quoted 7 lines
Quoted 6 lines
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id: If32f7748b7dc89dd37114dc502943222a2a36c49
What is this Change-Id property?
Gerrit ;)
Quoted 62 lines
Quoted 62 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-) create mode 100644 xen/arch/arm/pci/Makefile create mode 100644 xen/arch/arm/pci/pci-access.c create mode 100644 xen/arch/arm/pci/pci-host-common.c create mode 100644 xen/arch/arm/pci/pci-host-generic.c create mode 100644 xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +7
-0@@ -31,6 +31,13 @@ menu "Architecture Features" source "arch/Kconfig" +config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. + config ACPI bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends on ARM_64xen/arch/arm/Makefile +1
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y) obj-y += platforms/ endif obj-$(CONFIG_TEE) += tee/ +obj-$(CONFIG_ARM_PCI) += pci/ obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oxen/arch/arm/pci/pci-access.c +0
-0
Quoted 8 lines
Quoted 7 lines
xen/arch/arm/pci/pci-access.c line 2
Quoted 7 lines
new file mode 100644
index 0000000000..c53ef58336xen/arch/arm/pci/pci-access.c new file +2
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. I think SPDX will fit better in any new code.
It will be good if community helps us to decide which license is best for new files.
Quoted 180 lines
Quoted 169 lines
Quoted 79 lines
xen/arch/arm/pci/pci-access.c new file +79
-0+* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len) +{ +int rc; +uint32_t val =GENMASK (0 , len *8 ); + +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val); +if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)This looks like a subset of
xen/arch/x86/x86_64/mmconfig_64.c?MMCFG is supposed to cover ECAM-compliant host bridges too, if I am not mistaken. Is there any value in sharing the code with x86? It is OK if we don't, but I would like to understand the reasoning.
Quoted 72 lines
xen/arch/arm/pci/pci-access.c new file +9
-0+/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +57
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> + +/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells; +uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +}Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 65
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +}The two functions above seem to deal with the same resources, e.g.
cfg->win
Quoted 5 lines
and map/unmap. Would it make sense to align those, something like
s/pci_remap_cfgspace/pci_ecam_alloc and
pci_ecam_allochandlescfg->win?Or anything which makes them look init/fini style?
Ok will rename the function name.
Quoted 24 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/arm/pci/pci-host-common.c new file +12
-0+ +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +int err; +struct pci_config_window *cfg; +paddr_t addr, size; + + cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; +Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 85
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) { + cfg->busn_start =0 ; + cfg->busn_end =0xff ; +printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ;So, if bus start is, for example,
0x10then we'll end up with bus end at (0x10+0xff) >0xff
Quoted 1 line
which doesn't seem to be what we want
This is to fix if in device tree bus-ranges property, bus end corresponds to more than 256 bus.
Quoted 24 lines
Quoted 23 lines
xen/arch/arm/pci/pci-host-common.c line 91
Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +6
-0+ } + +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit;Shouldn't we handle the possibility of multiple addresses? Is it possible to have more than one range for an ECAM compliant host bridge?
Quoted 12 lines
xen/arch/arm/pci/pci-host-common.c new file +12
-0+ cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size);
Quoted 5 lines
I am fine with "win", but can we think of something that tells us that
"win" is actually ECAM base address, so one doesn't need to map "win" to "ECAM"
while reading?
Ack. Will fix.
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
xen/arch/arm/pci/pci-host-common.c new file +37
-0+if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } + +return cfg; + +err_exit_remap: +printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void ) +{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge; +struct pci_config_window *cfg;
Quoted 8 lines
Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c line 148
Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +7
-0+u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge ) +return -ENOMEM ; + +/* Parse and map our Configuration Space windows */ Do you expect multiple windows here as the comment says?
If going forward we want to support 32-bit we have to ioremap the config space for each bus individually, but as of only 64 bit is supported.
Quoted 10 lines
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c line 151
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3
-0+ cfg = gen_pci_init(dev, ops) +if ( !cfg ) +return -ENOMEM ;In case of errors the allocated bridge is not freed.
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c new file +3
-0+ bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops;
Quoted 5 lines
Can we have some sort of dummy ops so we don't have to check for ops != NULL every time
we read/write config? Is it really possible that we have ops set to NULL after we have
the development finished?
We don’t want to support host-bridges with dump ops. Proper ops has to be setup for the host-bridge to access the read/write. Once we access the config space we are sure that ops is already setup so no need to check for NULL each time.
Quoted 20 lines
Quoted 20 lines
xen/arch/arm/pci/pci-host-common.c line 163
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +8
-0+ +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ; + } + + bridge->segment = (u16 )segment;My understanding is that a Linux pci-domain doesn't correspond exactly to a PCI segment. See for instance:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03885.html
Do we need to care about the difference? If we mean pci-domain here, should we just call them as such instead of calling them "segments" in Xen (if they are not segments)?
Quoted 7 lines
Quoted 3 lines
xen/arch/arm/pci/pci-host-common.c line 165
Quoted 1 line
xen/arch/arm/pci/pci-host-common.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?Why? Do you expect bridge access in some specific order so ordered
list will make it faster?
As I mentioned in earlier no need to have ordered list as PCI config space access is random.
Quoted 79 lines
Quoted 79 lines
Quoted 61 lines
xen/arch/arm/pci/pci-host-common.c new file +32
-0+return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; + +list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ; +} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)I think the function is misnamed because reading the code below it looks like it is not just returning the base config space address but also the specific address we need to read/write (adding the device offset, "where", and everything).
Maybe
pci_config_get_addressor something like that?
Quoted 8 lines
Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c line 32
Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +2
-0+{ +struct pci_config_window *cfg = bridge->sysdata;I am a bit confused of the naming ;)
We already have 2 maps: win -> ECAM base and now sysdata -> cfg.
Can we please have that aligned somehow so it is easier to follow?
Ack.
Quoted 6 lines
Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c line 33
Quoted 2 lines
xen/arch/arm/pci/pci-host-generic.c new file +1
-0+unsigned int devfn_shift = cfg->ops->bus_shift -8 ;We are not checking
cfg->opsfor NULL, so probably we do not want bridgeswith NULL ops as well?
Yes correct we don’t want host-bridge with NULL ops. Do you see any use case to have host-bridge with NULL ops?
Quoted 257 lines
Quoted 257 lines
Quoted 20 lines
xen/arch/arm/pci/pci-host-generic.c new file +19
-0+ +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base; + +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift); + +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} + +int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; +xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);please use a switch
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) {xen/arch/arm/pci/pci-host-generic.c line 83
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ *val = ~0 ; +return -ENODEV ; + } + +if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr);please use a switch
Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c new file +5
-0+return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = {xen/arch/arm/pci/pci-host-generic.c line 99
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+ .bus_shift = 20, + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write, + } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops },spurious blank line
Quoted 108 lines
xen/arch/arm/pci/pci-host-generic.c new file +31
-0+ { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +71
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 :xen/arch/arm/pci/pci.c line 81
Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10
-0+ param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } + +return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);When adding new command line parameters please also add its documentation (docs/misc/xen-command-line.pandoc) in the same patch, unless this is meant to be just transient and we'll get removed before the final commit of the series.
Quoted 35 lines
xen/arch/arm/pci/pci.c new file +30
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable; + +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init (); + +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +0
-0xen/arch/arm/setup.c void __init start_xen(unsigned long boot_phys_offset, · line 933
Quoted 4 lines
xen/arch/arm/setup.c +1
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init ();
pci_initshould probably be an initcallQuoted 17 lines
xen/arch/arm/setup.c +0
-0do_initcalls ();/* xen/include/asm-arm/device.h +1
-0@@ -4,6 +4,7 @@enum device_type {DEV_DT , +DEV_PCI , };struct dev_archdata {@@ -25,15 +26,15 @@ typedef struct device device_t;
Quoted 6 lines
Quoted 5 lines
xen/include/asm-arm/device.h typedef struct device device_t; · line 30
Quoted 5 lines
xen/include/asm-arm/device.h +0
-3#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT)Didn't we have a patch for that recently or talked about?
Not sure need to check.
Quoted 107 lines
Quoted 91 lines
Quoted 45 lines
xen/include/asm-arm/device.h +3
-0+#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)enum device_class {DEVICE_SERIAL ,DEVICE_IOMMU ,DEVICE_GIC , +DEVICE_PCI ,/* Use for error */ DEVICE_UNKNOWN , };xen/include/asm-arm/pci.h +25
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ +xen/include/asm-arm/pci.h line 36
Quoted 10 lines
xen/include/asm-arm/pci.h +9
-0+#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};Are you actually using struct device in struct
arch_pci_dev? struct device is already part of structdt_device_nodeand a pointer to it is stored inbridge->dt_node.Quoted 15 lines
xen/include/asm-arm/pci.h +15
-0+#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/* +* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops; +void __iomem *win;xen/include/asm-arm/pci.h line 62
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val);I'd prefer if we could use explicitly-sized integers for "where" and "size" too. Also, should they be unsigned rather than signed?
Can we use
pci_sbdf_tfor the sbdf parameter?Quoted 15 lines
xen/include/asm-arm/pci.h line 73
Quoted 11 lines
xen/include/asm-arm/pci.h +11
-0+}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +};Although I realize that we are only targeting ECAM now, and the implementation is based on ECAM, the interface doesn't seem to have anything ECAM-specific in it. I would rename
pci_ecam_opsto something else, maybe simply "pci_ops".
Quoted 5 lines
Yes, please, bear in mind that we are about to work with this code on
non-ECAM HW from the very beginning, so this is something that we would
like to see from the ground up
Ok. Will take care of that in next version of the patch.
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
xen/include/asm-arm/pci.h +22
-1+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops; }; -#endif /* __X86_PCI_H__ */ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus); + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops); + +void pci_init (void ); + +#else /*!CONFIG_ARM_PCI*/ +struct arch_pci_dev { }; +static inline void pci_init (void ) { } +#endif /*!CONFIG_ARM_PCI*/ +#endif /* __ARM_PCI_H__ */ Signature · 1 line
2.17.1
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 17 lines · Julien Grall
Hi,
Quoted 9 lines · Oleksandr Andrushchenko
Quoted 8 lines
xen/arch/arm/pci/pci-access.c line 2
Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +2
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. I think SPDX will fit better in any new code.
Quoted 1 line
While I would love to use SPDX in Xen, there was some push back in the past to use it. So the new code should use the full-blown copyright until there is an agreement to use it.
Ack. We will use the copyright until we will get the confirmation from the community to use the SPDX in XEN.
Quoted 11 lines
Quoted 7 lines
Quoted 5 lines
xen/arch/arm/pci/pci-access.c line 3
Quoted 3 lines
xen/arch/arm/pci/pci-access.c new file +1
-0+list_add_tail (&bridge->node, &pci_host_bridges);It looks like
&pci_host_bridgesshould be an ordered list, ordered by segment number?Why? Do you expect bridge access in some specific order so ordered list will make it faster?
Access to the configure space will be pretty random. So I don't think ordering the list will make anything better.
However, looking up for the bridge for every config spec access is pretty slow. When I was working on the PCI passthrough, I wanted to look whether it would be possible to have a pointer to the PCI host bridge passed in argument to the helpers (rather than the segment).
Yes true every config space access has to scan the host-bridges list to find the corresponding host-bridge. I will try to find better solution so that no need to scan the host-bridge every time for read/write access.
Quoted 4 lines
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Julien GrallQuoted 27 lines · Julien Grall
Hi Rahul,
Quoted 7 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly. As of now "pci-host-ecam-generic" compatible board is supported.Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49Signed-off-by Rahul Singh <[email protected]>Quoted 12 lines
--- xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-)
Quoted 1 line
As a general comment, I would suggest to split the patch in smaller chunk. This would help the review and also allow to provide more explanation on what is done.
Ok I will split the patches in next version of the patch series.
Quoted 7 lines
For instance, I think it is possible to a split looking like:
- Add framework to access an hostbridge
- Add support for ECAM
- Add code to initialize the PCI subsystemThere is also some small fixes in this code that probably can move in there own patches.
Ack.
Quoted 5 lines
Cheers,
Signature · 1 line
Julien Grall
Re: [RFC PATCH v1 2/4] xen/arm: Discovering PCI devices and add the PCI devices in XEN.
In reply to Oleksandr AndrushchenkoQuoted 44 lines · Oleksandr Andrushchenko
Quoted 26 lines · Stefano Stabellini
Quoted 5 lines · Rahul Singh
Hardware domain is in charge of doing the PCI enumeration and will discover the PCI devices and then will communicate to XEN via hyper call
PHYSDEVOP_pci_device_addto add the PCI devices in XEN.Change-Id: Ie87e19741689503b4b62da911c8dc2ee318584ac
Same question about Change-Id
Quoted 17 lines
Signed-off-by Rahul Singh <[email protected]>--- xen/arch/arm/physdev.c | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-)xen/arch/arm/physdev.c +2
-1@@ -9,12 +9,48 @@#include <xen/errno.h> #include <xen/sched.h> #include <asm/hypercall.h> - +#include <xen/guest_access.h> +#include <xsm/xsm.h> Quoted 9 lines
xen/arch/arm/physdev.c line 21
Quoted 9 lines
xen/arch/arm/physdev.c +5
-2int do_physdev_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg) { -gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); -return -ENOSYS ; +int ret =0 ; + +switch ( cmd ) + { +#ifdef CONFIG_HAS_PCI In the cover letter you were saying "we are not enabling the
HAS_PCIandHAS_VPCIflags for ARM".
Quoted 1 line
Is this still valid?
Yes right we are not enabling it because full support of PCI passthrough is not implemented and tested.
Quoted 43 lines
Quoted 43 lines
Quoted 21 lines
xen/arch/arm/physdev.c +20
-0+case PHYSDEVOP_pci_device_add: + { +struct physdev_pci_device_add add; +struct pci_dev_info pdev_info; +nodeid_t node =NUMA_NO_NODE ; + + ret = -EFAULT ; +if (copy_from_guest (&add, arg,1 ) !=0 ) +break ; + + pdev_info.is_extfn = !!(add.flags &XEN_PCI_DEV_EXTFN ); +if ( add.flags &XEN_PCI_DEV_VIRTFN ) + { + pdev_info.is_virtfn =1 ; + pdev_info.physfn.bus = add.physfn.bus; + pdev_info.physfn.devfn = add.physfn.devfn; + } +else + pdev_info.is_virtfn =0 ; +xen/arch/arm/physdev.c line 51
Quoted 10 lines
xen/arch/arm/physdev.c +10
-0+ ret =pci_add_device (add.seg, add.bus, add.devfn, + &pdev_info, node); + + break; + } +#endif +default : +gdprintk (XENLOG_DEBUG ,"PHYSDEVOP cmd=%d: not implemented\n" , cmd); + ret = -ENOSYS ; + }I think we should make the implementation common between arm and x86 by creating xen/common/physdev.c:do_physdev_op as a shared entry point for PHYSDEVOP hypercalls implementations. See for instance:
xen/common/sysctl.c:do_sysctl
and
xen/arch/arm/sysctl.c:arch_do_sysctl
xen/arch/x86/sysctl.c:arch_do_sysctl
Quoted 6 lines
Quoted 3 lines
Jan, Andrew, Roger, any opinions?
I think we can also have a look at [1] by Julien. That implementation,
IMO, had some thoughts on making Arm/x86 code common where possible
Ok. Thanks for the pointer. We will have a look.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Roger Pau MonnéQuoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 5 lines
You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.
Thanks for the clarification!
Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?Quoted 8 lines
Quoted 2 lines
I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?
Quoted 4 lines
Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).
That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.
Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).
The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Roger Pau MonnéQuoted 61 lines · Roger Pau Monné
Quoted 56 lines · Rahul Singh
XEN during boot will read the PCI device tree node “reg” property and will map the PCI config space to the XEN memory.
XEN will read the “linux, pci-domain” property from the device tree node and configure the host bridge segment number accordingly.
As of now "pci-host-ecam-generic" compatible board is supported.
Change-Id If32f7748b7dc89dd37114dc502943222a2a36c49Signed-off-by Rahul Singh <[email protected]>---
xen/arch/arm/Kconfig | 7 + xen/arch/arm/Makefile | 1 + xen/arch/arm/pci/Makefile | 4 + xen/arch/arm/pci/pci-access.c | 101 ++++++++++++++ xen/arch/arm/pci/pci-host-common.c | 198 ++++++++++++++++++++++++++++ xen/arch/arm/pci/pci-host-generic.c | 131 ++++++++++++++++++ xen/arch/arm/pci/pci.c | 112 ++++++++++++++++ xen/arch/arm/setup.c | 2 + xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/pci.h | 97 +++++++++++++- 10 files changed, 654 insertions(+), 6 deletions(-)create mode 100644 xen/arch/arm/pci/Makefile create mode 100644
xen/arch/arm/pci/pci-access.ccreate mode 100644xen/arch/arm/pci/pci-host-common.ccreate mode 100644xen/arch/arm/pci/pci-host-generic.ccreate mode 100644xen/arch/arm/pci/pci.cxen/arch/arm/Kconfig +0
-0@@ -31,6 +31,13 @@ menu "Architecture Features"source "
arch/Kconfig"xen/arch/arm/Kconfig +7
-0+config ARM_PCI + bool "PCI Passthrough Support" + depends on ARM_64 + ---help--- + + PCI passthrough support for Xen on ARM64. +config ACPI
bool "ACPI (Advanced Configuration and Power Interface) Support" if EXPERT depends onARM_64xen/arch/arm/Makefile +0
-0@@ -6,6 +6,7 @@ ifneq ($(CONFIG_NO_PLAT),y)obj-y += platforms/
endif
obj-$(CONFIG_TEE) += tee/
+obj-$(CONFIG_ARM_PCI) += pci/
Quoted 15 lines
xen/arch/arm/pci/Makefile line 4
Quoted 12 lines
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.oxen/arch/arm/pci/Makefile new file +4
-0@@ -0,0 +1,4 @@ +obj-y += pci.o +obj-y += pci-host-generic.o +obj-y += pci-host-common.o +obj-y += pci-access.oThe
Kconfigoption mentions the support being explicitly for ARM64, would it be better to place the code inarch/arm/arm64then?
Ok.As julien also mentioned we tested on ARM64 , We have to test on ARM32 platforms.
Quoted 25 lines
Quoted 25 lines
xen/arch/arm/pci/pci-access.c new file +18
-0@@ -0,0 +1,101 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h>
Quoted 8 lines
xen/arch/arm/pci/pci-access.c line 23
Quoted 5 lines
xen/arch/arm/pci/pci-access.c new file +5
-0+#include <asm/pci.h> +#include <xen/rwlock.h> + +static uint32_t pci_config_read (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len)Please align with the opening parenthesis (here and everywhere in the patch series).
Ack.
Quoted 8 lines
xen/arch/arm/pci/pci-access.c line 26
Quoted 4 lines
xen/arch/arm/pci/pci-access.c new file +3
-0+{ +int rc; +uint32_t val =GENMASK (0 , len *8 );You can just set val = ~0. The return type of the
pci_conf_readXXhelpers will already truncate the value.
Ack.
Quoted 11 lines
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:
Quoted 7 lines
https://patchew.org/Xen/[email protected]/
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
Ok Will integrate that patch once submitted.
Quoted 11 lines
xen/arch/arm/pci/pci-access.c line 40
Quoted 8 lines
xen/arch/arm/pci/pci-access.c new file +7
-0+return val; + } + +if (unlikely (!bridge->ops->read) ) +return val; + + rc = bridge->ops->read (bridge, (uint32_t ) sbdf.sbdf, reg, len, &val);There's no need for the
uint32_tcast, the sbdf field is already of such type
Ack.
Quoted 57 lines
Quoted 41 lines
xen/arch/arm/pci/pci-access.c new file +41
-0+if ( rc ) +printk (XENLOG_ERR "Failed to read reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); + +return val; +} + +static void pci_config_write (pci_sbdf_t sbdf,unsigned int reg, +unsigned int len,uint32_t val) +{ +int rc; +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +return ; + } + +if (unlikely (!bridge->ops->write) ) +return ; + + rc = bridge->ops->write (bridge, (uint32_t ) sbdf.sbdf, reg, len, val); +if ( rc ) +printk (XENLOG_ERR "Failed to write reg %#x len %u for " PRI_pci"\n" , + reg, len, sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn); +} + +/* +* Wrappers for all PCI configuration access functions. +*/ + +#define PCI_OP_WRITE (size, type) \ + void pci_conf_write##size (pci_sbdf_t sbdf,unsigned int reg, type val) \ +{ \ + pci_config_write(sbdf, reg, size / 8, val); \ +} + +#define PCI_OP_READ (size, type) \ + type pci_conf_read##size (pci_sbdf_t sbdf, unsigned int reg) \xen/arch/arm/pci/pci-access.c line 91
Quoted 10 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+{ \ +return pci_config_read (sbdf, reg, size /8 ); \ +} + +PCI_OP_READ(8, u8) +PCI_OP_READ(16, u16) +PCI_OP_READ(32, u32) +PCI_OP_WRITE(8, u8) +PCI_OP_WRITE(16, u16) +PCI_OP_WRITE (32 , u32)Please use
uintXX_t.Also, It's nice to add some kind of signals for cscope and friends so they can find the autogenerated functions, ie:
Quoted 7 lines
#define
pci_conf_read8
#undefpci_conf_read8
#definepci_conf_read16
#undefpci_conf_read16
...It's tedious but helps future users find where the code is generated.
Ack.
Quoted 95 lines
Quoted 47 lines
xen/arch/arm/pci/pci-access.c new file +10
-0+ +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-common.c new file +30
-0@@ -0,0 +1,198 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/init.h> +#include <xen/pci.h> +#include <asm/pci.h> +#include <xen/rwlock.h> +#include <xen/vmap.h> +xen/arch/arm/pci/pci-host-common.c line 40
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+/* +* List for all the pci host bridges. +*/ + +static LIST_HEAD (pci_host_bridges); + +static bool __init dt_pci_parse_bus_range (struct dt_device_node *dev, +struct pci_config_window *cfg) +{ +const __be32 *cells;It's my impression that while based on Linux this is not a verbatim copy of a Linux file, and tries to adhere with the Xen coding style. If so please use
uint32_there.Quoted 33 lines
xen/arch/arm/pci/pci-host-common.c new file +33
-0+uint32_t len; + + cells =dt_get_property (dev,"bus-range" , &len); +/* bus-range should at least be 2 cells */ +if ( !cells || (len < (sizeof (*cells) *2 )) ) +return false; + + cfg->busn_start =dt_next_cell (1 , &cells); + cfg->busn_end =dt_next_cell (1 , &cells); + +return true; +} + +static inline void __iomem *pci_remap_cfgspace (paddr_t start,size_t len) +{ +return ioremap_nocache (start, len); +} + +static void pci_ecam_free (struct pci_config_window *cfg) +{ +if ( cfg->win ) +iounmap (cfg->win); + +xfree (cfg); +} + +static struct pci_config_window *gen_pci_init (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +int err; +struct pci_config_window *cfg; +paddr_t addr, size; +
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 79
Quoted 6 lines
xen/arch/arm/pci/pci-host-common.c new file +6
-0+ cfg = xzalloc(struct pci_config_window); +if ( !cfg ) +return NULL ; + + err =dt_pci_parse_bus_range (dev, cfg); +if ( !err ) {Braces
Ack
Quoted 41 lines
Quoted 41 lines
xen/arch/arm/pci/pci-host-common.c new file +40
-0+ cfg->busn_start =0 ; + cfg->busn_end =0xff ; +printk (XENLOG_ERR "No bus range found for pci controller\n" ); + }else { +if ( cfg->busn_end > cfg->busn_start +0xff ) + cfg->busn_end = cfg->busn_start +0xff ; + } + +/* Parse our PCI ecam register address*/ + err =dt_device_get_address (dev,0 , &addr, &size); +if ( err ) + goto err_exit; + + cfg->phys_addr = addr; + cfg->size = size; + cfg->ops = ops; + +/* +* On 64-bit systems, we do a single ioremap for the whole config space +* since we have enough virtual address range available. On 32-bit, we +* ioremap the config space for each bus individually. +* +* As of now only 64-bit is supported 32-bit is not supported. +*/ + cfg->win =pci_remap_cfgspace (cfg->phys_addr, cfg->size); +if ( !cfg->win ) + goto err_exit_remap; + +printk ("ECAM at [mem %lx-%lx] for [bus %x-%x] \n" ,cfg->phys_addr, + cfg->phys_addr + cfg->size -1 ,cfg->busn_start,cfg->busn_end); + +if ( ops->init ) { + err = ops->init (cfg); +if (err) + goto err_exit; + } + +return cfg; + +err_exit_remap:
Quoted 8 lines
xen/arch/arm/pci/pci-host-common.c line 126
Quoted 7 lines
xen/arch/arm/pci/pci-host-common.c new file +7
-0+printk (XENLOG_ERR "ECAM ioremap failed\n" ); +err_exit: +pci_ecam_free (cfg); +return NULL ; +} + +static struct pci_host_bridge *pci_alloc_host_bridge (void )^ extra space
Ack.
Quoted 82 lines
Quoted 15 lines
xen/arch/arm/pci/pci-host-common.c new file +15
-0+{ + struct pci_host_bridge *bridge = xzalloc(struct pci_host_bridge); + +if ( !bridge ) +return NULL ; + +INIT_LIST_HEAD (&bridge->node); +return bridge; +} + +int pci_host_common_probe (struct dt_device_node *dev, +struct pci_ecam_ops *ops) +{ +struct pci_host_bridge *bridge; +struct pci_config_window *cfg;xen/arch/arm/pci/pci-host-common.c line 151
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+u32 segment; + + bridge =pci_alloc_host_bridge (); +if ( !bridge ) +return -ENOMEM ; + +/* Parse and map our Configuration Space windows */ + cfg =gen_pci_init (dev, ops); +if ( !cfg ) +return -ENOMEM ;You are leaking the allocation of bridge here ...
xen/arch/arm/pci/pci-host-common.c line 160
Quoted 9 lines
xen/arch/arm/pci/pci-host-common.c new file +9
-0+ + bridge->dt_node = dev; + bridge->sysdata = cfg; + bridge->ops = &ops->pci_ops; + +if ( !dt_property_read_u32 (dev,"linux,pci-domain" , &segment) ) + { +printk (XENLOG_ERR "\"linux,pci-domain\" property in not available in DT\n" ); +return -ENODEV ;... and here.
Quoted 18 lines
xen/arch/arm/pci/pci-host-common.c new file +18
-0+ } + + bridge->segment = (u16 )segment; + +list_add_tail (&bridge->node, &pci_host_bridges); + +return 0 ; +} + +/* +* This function will lookup an hostbridge based on the segment and bus +* number. +*/ +struct pci_host_bridge *pci_find_host_bridge (uint16_t segment,uint8_t bus) +{ +struct pci_host_bridge *bridge; +bool found = false; +xen/arch/arm/pci/pci-host-common.c line 188
Quoted 10 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+list_for_each_entry ( bridge, &pci_host_bridges, node ) + { +if ( bridge->segment != segment ) +continue ; + + found = true; +break ; + } + +return (found) ? bridge :NULL ;This can be much shorter:
struct
pci_host_bridge*pci_find_host_bridge(uint16_t segment,uint8_tbus){ struct pci_host_bridge *bridge; list_for_each_entry( bridge, &pci_host_bridges, node ) if ( bridge->segment == segment ) return bridge; return NULL; }
Quoted 2 lines
Albeit I'm confused by the fact that you pass a bus number that's completely unused.
Ok Will fix.
Quoted 52 lines
Quoted 37 lines
xen/arch/arm/pci/pci-host-common.c new file +10
-0+} +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci-host-generic.c new file +20
-0@@ -0,0 +1,131 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ +xen/arch/arm/pci/pci-host-generic.c line 30
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +10
-0+#include <asm/device.h> +#include <asm/io.h> +#include <xen/pci.h> +#include <asm/pci.h> + +/* +* Function to get the config space base. +*/ +static void __iomem *pci_config_base (struct pci_host_bridge *bridge, +uint32_t sbdf,int where)You would be better passing a
pci_sbdf_tdirectly here. Also 'where' should be renamed to offset, or reg, and be made unsigned int. AFAICT you will never pass a negative value here.
Quoted 3 lines
For sanity you should also assert that the offset falls between the PCI config space used by the device, in order to easily catch with wrong offsets being used.
Ack.
Quoted 5 lines
xen/arch/arm/pci/pci-host-generic.c line 32
Quoted 3 lines
xen/arch/arm/pci/pci-host-generic.c new file +2
-0+{ +struct pci_config_window *cfg = bridge->sysdata;const
Ack.
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c line 38
Quoted 7 lines
xen/arch/arm/pci/pci-host-generic.c new file +6
-0+unsigned int devfn_shift = cfg->ops->bus_shift -8 ; + +pci_sbdf_t sbdf_t = (pci_sbdf_t ) sbdf ; + +unsigned int busn = sbdf_t.bus; +void __iomem *base;IMO adding newlines between variable definitions is not helpful, but that's my taste.
Ack.
Quoted 10 lines
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +9
-0+ +if ( busn < cfg->busn_start || busn > cfg->busn_end ) +return NULL ; + + base = cfg->win + (busn << cfg->ops->bus_shift); + +return base + (PCI_DEVFN (sbdf_t.dev, sbdf_t.fn) << devfn_shift) + where; +} +
Quoted 8 lines
xen/arch/arm/pci/pci-host-generic.c line 53
Quoted 6 lines
xen/arch/arm/pci/pci-host-generic.c new file +6
-0+int pci_ecam_config_write (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where);You can initialize at definition.
Ack.
Quoted 11 lines
xen/arch/arm/pci/pci-host-generic.c line 62
Quoted 10 lines
xen/arch/arm/pci/pci-host-generic.c new file +9
-0+if ( !addr ) +return -ENODEV ; + +if ( size ==1 ) +writeb (val, addr); +else if ( size ==2 ) +writew (val, addr); +else +writel (val, addr);
Quoted 2 lines
Please use a switch, and check against specific values. The default case should be a
BUG();. Seepci_conf_readfrom x86 for an example.
Ack.
Quoted 161 lines
Quoted 147 lines
xen/arch/arm/pci/pci-host-generic.c new file +69
-0+ +return 0 ; +} + +int pci_ecam_config_read (struct pci_host_bridge *bridge,uint32_t sbdf, +int where,int size,u32 *val) +{ +void __iomem *addr; + + addr =pci_config_base (bridge, sbdf, where); +if ( !addr ) { + *val = ~0 ; +return -ENODEV ; + } + +if ( size ==1 ) + *val =readb (addr); +else if ( size ==2 ) + *val =readw (addr); +else + *val =readl (addr); + +return 0 ; +} + +/* ECAM ops */ +struct pci_ecam_ops pci_generic_ecam_ops = { + .bus_shift =20 , + .pci_ops = { + .read = pci_ecam_config_read, + .write = pci_ecam_config_write, + } +}; + +static const struct dt_device_match gen_pci_dt_match[] = { + { .compatible ="pci-host-ecam-generic" , + .data = &pci_generic_ecam_ops }, + + { }, +}; + +static int gen_pci_dt_init (struct dt_device_node *dev,const void *data) +{ +const struct dt_device_match *of_id; +struct pci_ecam_ops *ops; + + of_id =dt_match_node (gen_pci_dt_match, dev->dev.of_node); + ops = (struct pci_ecam_ops *) of_id->data; + +printk (XENLOG_INFO "Found PCI host bridge %s compatible:%s \n" , +dt_node_full_name (dev), of_id->compatible); + +return pci_host_common_probe (dev, ops); +} + +DT_DEVICE_START (pci_gen,"PCI HOST GENERIC" ,DEVICE_PCI ) +.dt_match = gen_pci_dt_match, +.init = gen_pci_dt_init, +DT_DEVICE_END + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/pci/pci.c new file +71
-0@@ -0,0 +1,112 @@ +/* +* Copyright (C) 2020 Arm Ltd. +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/ + +#include <xen/acpi.h> +#include <xen/device_tree.h> +#include <xen/errno.h> +#include <xen/init.h> +#include <xen/pci.h> +#include <xen/param.h> + +static int __init dt_pci_init (void ) +{ +struct dt_device_node *np; +int rc; + +dt_for_each_device_node (dt_host, np) + { + rc =device_init (np,DEVICE_PCI ,NULL ); +if ( !rc ) +continue ; +/* +* Ignore the following error codes: +* - EBADF: Indicate the current is not an pci +* - ENODEV: The pci device is not present or cannot be used by +* Xen. +*/ +else if ( rc != -EBADF && rc != -ENODEV ) + { +printk (XENLOG_ERR "No driver found in XEN or driver init error.\n" ); +return rc; + } + } + +return 0 ; +} + +#ifdef CONFIG_ACPI +static void __init acpi_pci_init (void ) +{ +printk (XENLOG_ERR "ACPI pci init not supported \n" ); +return ; +} +#else +static inline void __init acpi_pci_init (void ) { } +#endif + +static bool __initdata param_pci_enable; +static int __init parse_pci_param (const char *arg) +{ +if ( !arg ) + { + param_pci_enable = false; +return 0 ; + } + +switch (parse_bool (arg,NULL ) ) + { +case 0 :xen/arch/arm/pci/pci.c line 81
Quoted 10 lines
xen/arch/arm/pci/pci.c new file +10
-0+ param_pci_enable = false; +return 0 ; +case 1 : + param_pci_enable = true; +return 0 ; + } + +return -EINVAL ; +} +custom_param ("pci" , parse_pci_param);You need to introduce the documentation for the parameter at docs/misc/xen-command-line.pandoc
Quoted 4 lines
Albeit I'm not sure I like it, why do you need to enable PCI explicitly?
Shouldn't it be discovered automatically and enabled by default?
Ack.
Quoted 11 lines
xen/arch/arm/pci/pci.c line 89
Quoted 8 lines
xen/arch/arm/pci/pci.c new file +7
-0+void __init pci_init (void ) +{ +/* +* Enable PCI when has been enabled explicitly (pci=on) +*/ +if ( !param_pci_enable) + goto disable;Just return here, there's not point in having a label to perform a return.
Ack.
Quoted 114 lines
xen/arch/arm/pci/pci.c line 94
Quoted 6 lines
xen/arch/arm/pci/pci.c new file +5
-0+ +if ( acpi_disabled ) +dt_pci_init (); +else +acpi_pci_init ();Isn't there an enum or something that tells you whether the system description is coming from ACPI or from DT?
This if .. else seems fragile.
Also for ACPI you will get called by
acpi_boot_init, and likely need to implement aacpi_mmcfg_initor pci_mmcfg_arch_{init,enable}. I'm not sure whether the code inacpi_mmcfg_initcould be made shared between both x86 and Arm.Quoted 53 lines
xen/arch/arm/pci/pci.c new file +18
-0+ +#ifdef CONFIG_HAS_PCI +pci_segments_init (); +#endif + +disable: +return ; +} + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* tab-width: 4 +* indent-tabs-mode: nil +* End: +*/ xen/arch/arm/setup.c +2
-0@@ -930,6 +930,8 @@ void __init start_xen(unsigned long boot_phys_offset,setup_virt_paging (); +pci_init (); +do_initcalls ();/* xen/include/asm-arm/device.h +0
-0@@ -4,6 +4,7 @@enum
device_type{ DEV_DT,+
DEV_PCI,
};struct
dev_archdata{xen/include/asm-arm/device.h +2
-3@@ -25,15 +26,15 @@ typedef struct device device_t;#include <xen/device_tree.h> -/* TODO: Correctly implement dev_is_pci when PCI is supported on ARM */ -#define dev_is_pci (dev) ((void)(dev), 0) -#define dev_is_dt (dev) ((dev->type == DEV_DT) +#define dev_is_pci (dev) (dev->type == DEV_PCI) +#define dev_is_dt (dev) (dev->type == DEV_DT)Quoted 6 lines
enum
device_class
{
DEVICE_SERIAL,
DEVICE_IOMMU,
DEVICE_GIC,
+DEVICE_PCI,It seems to be like this wants to be
DEVICE_PCI_HOST_BRIDGEor some such, since this is not used to identify all PCI devices, but just bridges?Quoted 33 lines
/* Use for error */ DEVICE_UNKNOWN, };xen/include/asm-arm/pci.h +22
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://www.gnu.org/licenses/>. +*/
Quoted 15 lines
xen/include/asm-arm/pci.h line 36
Quoted 13 lines
xen/include/asm-arm/pci.h +12
-0+#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ + +#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};This seems to be completely unused?
Will be using going forward.
Quoted 59 lines
Quoted 6 lines
xen/include/asm-arm/pci.h +5
-0+ +#define PRI_pci "%04x:%02x:%02x.%u" +#define pci_to_dev (pcidev) (&(pcidev)->arch.dev) + +/* xen/include/asm-arm/pci.h line 51
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+* struct to hold the mappings of a config space window. This +* is expected to be used as sysdata for PCI controllers that +* use ECAM. +*/ +struct pci_config_window { +paddr_t phys_addr; +paddr_t size; +uint8_t busn_start; +uint8_t busn_end; +struct pci_ecam_ops *ops;const?
Quoted 23 lines
xen/include/asm-arm/pci.h +23
-0+void __iomem *win; +}; + +/* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ +struct pci_host_bridge ; + +struct pci_ops { +int (*read)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 *val); +int (*write)(struct pci_host_bridge *bridge, +uint32_t sbdf,int where,int size,u32 val); +}; + +/* +* struct to hold pci ops and bus shift of the config window +* for a PCI controller. +*/ +struct pci_ecam_ops { +unsigned int bus_shift; +struct pci_ops pci_ops; +int (*init)(struct pci_config_window *); +}; +xen/include/asm-arm/pci.h line 84
Quoted 10 lines
xen/include/asm-arm/pci.h +10
-0+/* +* struct to hold pci host bridge information +* for a PCI controller. +*/ +struct pci_host_bridge { +struct dt_device_node *dt_node;/* Pointer to the associated DT node */ +struct list_head node;/* Node in list of host bridges */ +uint16_t segment;/* Segment number */ +void *sysdata;/* Pointer to the config space window*/ +const struct pci_ops *ops;You seem to introduce a lot of ops structs, yet there's only one implementation the generic ECAM one, and adding such complexity should IMO be done when further implementations are added. Also given this is a fully ECAM compliant bridge you could just use most of the existing logic for x86?
Quoted 7 lines
I understand the discovery needs to be different, but x86 MCFG logic should already be capable of handling multiple ECAM regions.
I also agree with Julien that splitting this into separate patch would make it easier to review. For example you can start with the discovery logic, followed by the initialization and the add the accessors to the config space lastly.
Ok . Will spilt the patches in the next version of the patch series.
Quoted 2 lines
Thanks, Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 18 lines · Rahul Singh
Quoted 18 lines · Roger Pau Monné
xen/arch/arm/pci/pci-access.c line 33
Quoted 7 lines · Rahul Singh
xen/arch/arm/pci/pci-access.c new file +7
-0+ +struct pci_host_bridge *bridge =pci_find_host_bridge (sbdf.seg, sbdf.bus); + +if (unlikely (!bridge) ) + { +printk (XENLOG_ERR "Unable to find bridge for " PRI_pci"\n" , + sbdf.seg, sbdf.bus, sbdf.dev, sbdf.fn);I had a patch to add a custom modifier to out printf format in order to handle
pci_sbdf_tnatively:
Quoted 7 lines
Quoted 5 lines
It missed maintainers Acks and was never committed. Since you are doing a bunch of work here, and likely adding a lot of SBDF related prints, feel free to import the modifier (%pp) and use in your code (do not attempt to switch existing users, or it's likely to get stuck again).
Ok Will integrate that patch once submitted.
I've posted an updated version to the list yesterday:
https://lore.kernel.org/xen-devel/[email protected]/
Quoted 46 lines
Quoted 30 lines
Quoted 30 lines
xen/include/asm-arm/pci.h +22
-2@@ -1,7 +1,98 @@ -#ifndef __X86_PCI_H__ -#define __X86_PCI_H__ +/* +* Copyright (C) 2020 Arm Ltd. +* +* Based on Linux drivers/pci/ecam.c +* Copyright 2016 Broadcom. +* +* Based on Linux drivers/pci/controller/pci-host-common.c +* Based on Linux drivers/pci/controller/pci-host-generic.c +* Copyright (C) 2014 ARM Limited Will Deacon <[email protected]> +* +* This program is free software; you can redistribute it and/or modify +* it under the terms of the GNU General Public License version 2 as +* published by the Free Software Foundation. +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program. If not, see <http://secure-web.cisco.com/1YIShdE6aU0WIMsRjhIYYGdRo_H_wyOVeLqk0ITHI7i_FMIXuMEmV9Y6lR76qCBB4XHss0ba_WMfQa0mLGD37bM1S3AFMBVC1WUEF0LmbURPWld2kU2zuyOamUxqnYwy4ZAv4N2EUJy7HCgDYn2s183YCQ-5YBc98vH8TO-tfBgEShkKVzAm23AgzIqikuKN_BIhcQhAbsjypr9ffeNZDsSX2jC8ClqegEPXFsCWJldrASGgWoR16rMjI-INfPNy55m9nGP5UmHIYWBmTpnHLjXtqCkruENixh20vPWIUROhcjjtzVBrR8d-Q5HnJy0hSR97WdlUSAryfxeH-8VxcPw/http%3A%2F%2Fwww.gnu.org%2Flicenses%2F>. +*/ Quoted 15 lines
xen/include/asm-arm/pci.h line 36
Quoted 13 lines
xen/include/asm-arm/pci.h +12
-0+#ifndef __ARM_PCI_H__ +#define __ARM_PCI_H__ + +#include <xen/pci.h> +#include <xen/device_tree.h> +#include <asm/device.h> + +#ifdef CONFIG_ARM_PCI + +/* Arch pci dev struct */ struct arch_pci_dev { +struct device dev; +};This seems to be completely unused?
Quoted 1 line
Will be using going forward.
Please introduce it when it's required then.
Thanks, Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
In reply to Stefano StabelliniQuoted 71 lines · Stefano Stabellini
Quoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 5 lines
You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.
Thanks for the clarification!
Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?Quoted 8 lines
Quoted 2 lines
I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?
Quoted 4 lines
Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).
That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.
Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have
CONFIG_PCI_DOMAINSenabled (including ARM64).
Quoted 5 lines
The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.
I don't think the ACPI MCFG spec contains the notion of bridges, it just describes ECAM (or MMCFG) regions, but those could be made up by concatenating different bridge ECAM regions by the firmware itself, so you could AFAICT end up with multiple bridges being aggregated into a single ECAM region, and thus using the same segment number, which seems not possible with the DT spec, where each bridge must get a different segment number?
If you could assign both a segment number and a bus start and end values to a bridge then I think it would be kind of equivalent to ACPI MCFG.
I assume we would never support a system where Xen is getting the hardware description from a DT and the hardware domain is using ACPI (or the other way around)?
If so, I don't think we care that enumeration when using DT is different than when using ACPI, as we can only guarantee consistency when both Xen and the hardware domain use the same source for the hardware description.
If when using DT each bridge has a unique segment number that's fine as long as Xen and the OS agree to not change such values.
Roger.
Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.
Quoted 78 lines · Roger Pau Monné
Quoted 71 lines · Stefano Stabellini
Quoted 18 lines · Roger Pau Monné
Quoted 13 lines · Julien Grall
Quoted 12 lines · Stefano Stabellini
Quoted 10 lines · Julien Grall
Quoted 8 lines · Stefano Stabellini
Quoted 2 lines
If they are not equal, then I fail to see why it would be useful to have this value in Xen.
I think that's because the domain is actually more convenient to use because a segment can span multiple PCI host bridges. So my understanding is that a segment alone is not sufficient to identify a host bridge. From a software implementation point of view it would be better to use domains.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
AFAICT, this would be a matter of one check vs two checks in Xen :). But... looking at Linux, they will also use domain == segment for ACPI (see [1]). So, I think, they still have to use (domain, bus) to do the lookup.
Quoted 5 lines
You have to use the (segment, bus) tuple when doing a lookup because MMCFG regions on ACPI are defined for a segment and a bus range, you can have a MMCFG region that covers segment 0 bus [0, 20) and another MMCFG region that covers segment 0 bus [20, 255], and those will use different addresses in the MMIO space.
Thanks for the clarification!
Quoted 31 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
In which case, we need to use
PHYSDEVOP_pci_mmcfg_reservedso Dom0 and Xen can synchronize on the segment number.I was hoping we could write down the assumption somewhere that for the cases we care about domain == segment, and error out if it is not the case.
Given that we have only the domain in hand, how would you enforce that?
Quoted 1 line
From this discussion, it also looks like there is a mismatch between the
implementation and the understanding on QEMU devel. So I am a bit concerned that this is not stable and may change in future Linux version.
IOW, we are know tying Xen to Linux. So could we implement
PHYSDEVOP_pci_mmcfg_reserved*or* introduce a new property that really represent the segment?Quoted 8 lines
Quoted 2 lines
I don't think we are tying Xen to Linux. Rob has already said that linux,pci-domain is basically a generic device tree property.
My concern is not so much the name of the property, but the definition of it.
AFAICT, from this thread there can be two interpretation:
- domain == segment
- domain == (segment, bus)I think domain is just an alias for segment, the difference seems to be that when using DT all bridges get a different segment (or domain) number, and thus you will always end up starting numbering at bus 0 for each bridge?
Quoted 4 lines
Ideally you would need a way to specify the segment and start/end bus numbers of each bridge, if not you cannot match what ACPI does. Albeit it might be fine as long as the OS and Xen agree on the segments and bus numbers that belong to each bridge (and thus each ECAM region).
That is what I thought and it is why I was asking to clarify the naming and/or writing a document to explain the assumptions, if any.
Then after Julien's email I followed up in the Linux codebase and clearly there is a different assumption baked in the Linux kernel for architectures that have
CONFIG_PCI_DOMAINSenabled (including ARM64).Quoted 5 lines
The assumption is that segment == domain == unique host bridge. It looks like it is coming from IEEE Std 1275-1994 but I am not certain. In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly match. So I am starting to think that domain == segment for IEEE Std 1275-1994 compliant device tree based systems.
Quoted 7 lines
I don't think the ACPI MCFG spec contains the notion of bridges, it just describes ECAM (or MMCFG) regions, but those could be made up by concatenating different bridge ECAM regions by the firmware itself, so you could AFAICT end up with multiple bridges being aggregated into a single ECAM region, and thus using the same segment number, which seems not possible with the DT spec, where each bridge must get a different segment number?
Yes, that's my understanding too
Quoted 7 lines
If you could assign both a segment number and a bus start and end values to a bridge then I think it would be kind of equivalent to ACPI MCFG.
I assume we would never support a system where Xen is getting the hardware description from a DT and the hardware domain is using ACPI (or the other way around)?
Yeah, I think it is a good assumption
Quoted 7 lines
If so, I don't think we care that enumeration when using DT is different than when using ACPI, as we can only guarantee consistency when both Xen and the hardware domain use the same source for the hardware description.
If when using DT each bridge has a unique segment number that's fine as long as Xen and the OS agree to not change such values.
I agree