Xen Q35 Bringup patches + support for PCIe Extended Capabilities for passed through devices
cover Xen Q35 Bringup patches + support for PCIe Extended Capabilities for passed through devices
This patch series introduces support of Q35 emulation for Xen HVM guests (via QEMU). This feature is present in other virtualization products and Xen can greatly benefit from this feature as well.
The main goal for implementing Q35 emulation for Xen was extending PCI/GPU passthrough capabilities. It's the main advantage of Q35 emulation
- availability of extra features for PCIe device passthrough. The most important PCIe-specific passthrough feature Q35 provides is a support for PCIe config space ECAM (aka MMCONFIG) to allow accesses to extended PCIe config space (>256), which is MMIO-based. Lots of PCIe devices and their drivers make use of PCIe Extended Capabilities, whose can be accessed only using ECAM and offsets above 0x100 in PCI config space. Supporting ECAM is a mandatory feature for PCIe passthrough. Not only this allows passthrough PCIe devices to function properly, but opens a road to extend Xen PCIe passthrough features further -- eg. providing support for AER. One of possible directions is providing support for PCIe Resizable BARs -- a feature which likely to become common for modern GPUs as video memory sizes increase.
Q35 emulation may also be useful for other purposes. In fact, the emulation of a more recent chipset partially closes a huge gap between a set of required platform features and the actual emulated platform capabilities
- lot of required functionality is actually missing in a real i440 chipset. One can look at IGD passthru support patches from Intel for example:
according to code comments, they had to create a dummy PCI-ISA bridge at BDF 0:1F.0 in order to make the old i440 system look more modern, just to make it compatible with IGD driver. Using Q35 emulation with its own emulated LPC bridge allows to avoid workarounds like this. i440 on its own is a fairly outdated system and doesn't really support lot of things, like MMIO hole above 4Gb (although it is actually emulated). Also, due to the i440 chipset's age the only fact of its usage may be used as a reliable method to detect a virtualized environment by some malicious software especially considering the fact that i440 emulation is shared among multiple virtualization products.
On top of this series I've also implemented a solution which solves existing Xen puzzle with HVM memory layout -- handling of VRAM, RMRRs and MMIO hole in general. This "puzzle" (memory layout inconsistency between libxl/libxc, hvmloader and QEMU) is a sort of fundamental problem which plagues Xen for years and among few other issues prevents Xen to become a decent GPU/PCIe passthrough platform (which it should be). This solution also allows to later resolve current PCI passthrough incompatibility issues, eg. with Populate-on-Demand. In fact, i440 support has been added as well, but it's a bit hacky as it uses NB registers which are not present in a real i440 (well, one more non-existing i440 feature won't harm anyway as there are plenty of them already). I'm planning to send RFC patches of this solution right after current patches will be reviewed and related code settle, to rebase patches on top of it. Also, a good description is required as the change is rather radical.
The good thing is that providing Q35 support for Xen at this stage neither break any existing functionality nor affect the legacy i440 emulation in any way - Q35 emulation can be enabled on demand only, using a new domain config option. Also, only existing interfaces are used, no new hypecalls were introduced, no API changes, etc. Although in the future we'll have to change some hypercall/QMP/etc interfaces to remove limitations and extend the Q35/PCIe passthru support further.
Current features and limitations:
- All basic functionality works normally - MP, networking, storage (AHCI), powering down VMs via ACPI soft off, etc
- Xen Platform Device and PV devices are supported -- PV drivers for vbd, vif, etc may be installed and used
- PCIe ECAM fully supported, with allocating space for PCIEXBAR in MMIO hole, ACPI MCFG generation, etc.
- Xen is limited to max 4 PIRQs in multiple places, while Q35 have support of 8 PIRQs / PCI router links. This was workarounded by describing only 4 usable IRQ link entries in ACPI tables and disabling PIRQE..PIRQH -- like we're on a real system which has only some of 8 available PIRQs physically connected on the chipset. Extending the number of PCI links supported is trivial, but this step will change the save/migration stream format a bit... although as it seems there was actually some place for this extension being left -- eg. field uint8_t route[4] followed by uint8_t pad0[4] in hvm_hw_pci_link structure. Anyway, there is no problem actually as we normally deal with APIC mode (or MSIs) for IRQ delivery, while PIC mode with PCI routing needed only for legacy compatibility
- PCI hotplug currently implemented via ACPI hotplug, in a way similar to i440. In future, this might be changed to native PCIe hotplug facilities (if there will be a benefit).
- For PCIe passthrough to work on Windows 7 and above, a specific workaround was implemented, which allows to use PCIe device passthrough on those guest OSes normally. In future, this should be changed to a new emulated PCI architecture for Xen -- providing support for simple PCI hierarchies, nested MMIO spaces, etc. Basically, we need at least to provide support for PCI-PCI bridges (PCIe Root Ports in our case). Currently Xen limited to bus 0 in many places, even in hypercall parameters. A detailed description of the issue can be found in the patch named "xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology check".
- VM migration was not tested as the feature primarily targets the PCIe passthrough which doesn't compatible with migration anyway.
How to use the Q35 feature:
A new domain config option was implemented: device_model_machine. It's a string which has following possible values:
- "i440" -- i440 emulation (default)
- "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI.
Note that omitting device_model_machine parameter means i440 system by default, so the default behavior doesn't change for old domain config files.
So, in order to enable Q35 emulation one need to specify the following option in the domain config file:
device_model_machine="q35"
It is recommended to install the guest OS from scratch to avoid issues due to the emulated platform change.
One extra note - if you're going to backport this series to some older QEMU version, make sure you have this patch for AHCI DMA bug applied: [1]. Otherwise you will encounter random Q35 guest hangups with "Bad RAM offset" message logged in /var/log/xen. Recent QEMU versions have this patch commited already.
Also, a commit [2] is required to be applied (for xen-pt.c) -- it is available in the upstream QEMU currently, but not present in qemu-xen.
This is my first (somewhat) large contribution to Xen, so some mistakes are to be expected. Most testing was done using previous version of patches and Xen 4.8.x.
I plan to support and extend this series further, for now I expect some comments/suggestions/testing results/bugreports.
[1]: https://lists.xen.org/archives/html/xen-devel/2017-07/msg01077.html
[2]: https://lists.gnu.org/archive/html/qemu-devel/2017-12/msg03572.html
Xen changes:
Alexey Gerasimenko (12):
libacpi: new DSDT ACPI table for Q35
Makefile: build and use new DSDT table for Q35
hvmloader: add function to query an emulated machine type (i440/Q35)
hvmloader: add ACPI enabling for Q35
hvmloader: add Q35 DSDT table loading
hvmloader: add basic Q35 support
hvmloader: allocate MMCONFIG area in the MMIO hole + minor code
refactoring
libxl: Q35 support (new option device_model_machine)
libxl: Xen Platform device support for Q35
libacpi: build ACPI MCFG table if requested
hvmloader: use libacpi to build MCFG table
docs: provide description for device_model_machine option
docs/man/xl.cfg.pod.5.in | 27 ++ tools/firmware/hvmloader/Makefile | 2 +- tools/firmware/hvmloader/config.h | 5 + tools/firmware/hvmloader/hvmloader.c | 11 +- tools/firmware/hvmloader/pci.c | 289 ++++++++++++------ tools/firmware/hvmloader/pci_regs.h | 7 + tools/firmware/hvmloader/util.c | 130 ++++++++- tools/firmware/hvmloader/util.h | 10 + tools/libacpi/Makefile | 9 +- tools/libacpi/acpi2_0.h | 21 ++ tools/libacpi/build.c | 42 +++ tools/libacpi/dsdt_q35.asl | 551 +++++++++++++++++++++++++++++++++++ tools/libacpi/libacpi.h | 4 + tools/libxl/libxl_dm.c | 20 +- tools/libxl/libxl_types.idl | 7 + tools/xl/xl_parse.c | 14 + 16 files changed, 1051 insertions(+), 98 deletions(-) create mode 100644 tools/libacpi/dsdt_q35.asl
QEMU changes:
Alexey Gerasimenko (18):
pc/xen: Xen Q35 support: provide IRQ handling for PCI devices pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35 q35/xen: Add Xen platform device support for Q35
q35: Fix incorrect values for PCIEXBAR masks
xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and
PCIe Extended Capabilities enumeration
xen/pt: avoid reading PCIe device type and cap version multiple times xen/pt: determine the legacy/PCIe mode for a passed through device xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology
check
xen/pt: add support for PCIe Extended Capabilities and larger config
space
xen/pt: handle PCIe Extended Capabilities Next register xen/pt: allow to hide PCIe Extended Capabilities
xen/pt: add Vendor-specific PCIe Extended Capability descriptor and
sizing
xen/pt: add fixed-size PCIe Extended Capabilities descriptors xen/pt: add AER PCIe Extended Capability descriptor and sizing xen/pt: add descriptors and size calculation for
RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities
xen/pt: add Resizable BAR PCIe Extended Capability descriptor and
sizing
xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and
sizing
hw/acpi/ich9.c | 24 + hw/acpi/pcihp.c | 8 +- hw/core/machine.c | 21 + hw/i386/pc_q35.c | 27 +- hw/i386/xen/xen-hvm.c | 32 +- hw/isa/lpc_ich9.c | 4 + hw/pci-host/piix.c | 2 +- hw/pci-host/q35.c | 14 +- hw/xen/xen-host-pci-device.c | 110 ++++- hw/xen/xen-host-pci-device.h | 6 +- hw/xen/xen_pt.c | 53 +- hw/xen/xen_pt.h | 19 +- hw/xen/xen_pt_config_init.c | 1109 +++++++++++++++++++++++++++++++++++++++--- include/hw/acpi/ich9.h | 2 + include/hw/acpi/pcihp.h | 2 + include/hw/boards.h | 1 + include/hw/i386/ich9.h | 1 + include/hw/i386/pc.h | 3 + include/hw/pci-host/q35.h | 4 +- include/hw/xen/xen.h | 5 +- qemu-options.hx | 1 + stubs/xen-hvm.c | 8 +- 22 files changed, 1333 insertions(+), 123 deletions(-)
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 1/12 libacpi: new DSDT ACPI table for Q35
This patch adds the DSDT table for Q35 (new tools/libacpi/dsdt_q35.asl file). There are not many differences with dsdt.asl (for i440) at the moment, namely:
- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of _OSC method to inform OSPM about PCIe features supported
As we are still using 4 PCI router links and their corresponding device/register addresses are same (offset 0x60), no need to change PCI routing descriptions.
Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
--- tools/libacpi/dsdt_q35.asl | 551 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 551 insertions(+) create mode 100644 tools/libacpi/dsdt_q35.asl
tools/libacpi/dsdt_q35.asl new file +551 -0
@@ -0,0 +1,551 @@ +/****************************************************************************** + * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; version 2.1 only. with the special + * exception on linking described in file LICENSE. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Lesser General Public License for more details. + */ + +DefinitionBlock ("DSDT.aml", "DSDT", 2, "Xen", "HVM", 0) +{ + Name (\PMBS, 0x0C00) + Name (\PMLN, 0x08) + Name (\IOB1, 0x00) + Name (\IOL1, 0x00) + Name (\APCB, 0xFEC00000) + Name (\APCL, 0x00010000) + Name (\PUID, 0x00) + + + Scope (\_SB) + { + + /* Fix HCT test for 0x400 pci memory: + * - need to report low 640 MB mem as motherboard resource + */ + Device(MEM0) + { + Name(_HID, EISAID("PNP0C02")) + Name(_CRS, ResourceTemplate() { + QWordMemory( + ResourceConsumer, PosDecode, MinFixed, + MaxFixed, Cacheable, ReadWrite, + 0x00000000, + 0x00000000, + 0x0009ffff, + 0x00000000, + 0x000a0000) + }) + } + + Device (PCI0) + { + Name (_HID, EisaId ("PNP0A03")) + Name (_UID, 0x00) + Name (_ADR, 0x00) + Name (_BBN, 0x00) + + /* _OSC, modified from ASL sample in ACPI spec */ + Name(SUPP, 0) /* PCI _OSC Support Field value */ + Name(CTRL, 0) /* PCI _OSC Control Field value */ + Method(_OSC, 4) { + /* Create DWORD-addressable fields from the Capabilities Buffer */ + CreateDWordField(Arg3, 0, CDW1) + + /* Switch by UUID. + * Only PCI Host Bridge Device capabilities UUID used for now + */ + If (LEqual(Arg0, ToUUID("33DB4D5B-1FF7-401C-9657-7441C03DD766"))) { + /* Create DWORD-addressable fields from the Capabilities Buffer */ + CreateDWordField(Arg3, 4, CDW2) + CreateDWordField(Arg3, 8, CDW3) + + /* Save Capabilities DWORD2 & 3 */ + Store(CDW2, SUPP) + Store(CDW3, CTRL) + + /* Validate Revision DWORD */ + If (LNotEqual(Arg1, One)) { + /* Unknown revision */ + /* Support and Control DWORDs will be returned anyway */ + Or(CDW1, 0x08, CDW1) + } + + /* Control field bits are: + * bit 0 PCI Express Native Hot Plug control + * bit 1 SHPC Native Hot Plug control + * bit 2 PCI Express Native Power Management Events control + * bit 3 PCI Express Advanced Error Reporting control + * bit 4 PCI Express Capability Structure control + */ + + /* Always allow native PME, AER (no dependencies) + * Never allow SHPC (no SHPC controller in this system) + * Do not allow PCIe Capability Structure control for now + * Also, ACPI hotplug is used for now instead of PCIe + * Native Hot Plug + */ + And(CTRL, 0x0C, CTRL) + + If (LNotEqual(CDW3, CTRL)) { + /* Some of Capabilities bits were masked */ + Or(CDW1, 0x10, CDW1) + } + /* Update DWORD3 in the buffer */ + Store(CTRL, CDW3) + } Else { + Or(CDW1, 4, CDW1) /* Unrecognized UUID */ + } + Return (Arg3) + } + /* end of _OSC */ + + + /* Make cirrues VGA S3 suspend/resume work in Windows XP/2003 */ + Device (VGA) + { + Name (_ADR, 0x00020000) + + Method (_S1D, 0, NotSerialized) + { + Return (0x00) + } + Method (_S2D, 0, NotSerialized) + { + Return (0x00) + } + Method (_S3D, 0, NotSerialized) + { + Return (0x00) + } + } + + Method (_CRS, 0, NotSerialized) + { + Store (ResourceTemplate () + { + /* bus number is from 0 - 255*/ + WordBusNumber( + ResourceProducer, MinFixed, MaxFixed, SubDecode, + 0x0000, + 0x0000, + 0x00FF, + 0x0000, + 0x0100) + IO (Decode16, 0x0CF8, 0x0CF8, 0x01, 0x08) + WordIO( + ResourceProducer, MinFixed, MaxFixed, PosDecode, + EntireRange, + 0x0000, + 0x0000, + 0x0CF7, + 0x0000, + 0x0CF8) + WordIO( + ResourceProducer, MinFixed, MaxFixed, PosDecode, + EntireRange, + 0x0000, + 0x0D00, + 0xFFFF, + 0x0000, + 0xF300) + + /* reserve memory for pci devices */ + DWordMemory( + ResourceProducer, PosDecode, MinFixed, MaxFixed, + WriteCombining, ReadWrite, + 0x00000000, + 0x000A0000, + 0x000BFFFF, + 0x00000000, + 0x00020000) + + DWordMemory( + ResourceProducer, PosDecode, MinFixed, MaxFixed, + NonCacheable, ReadWrite, + 0x00000000, + 0xF0000000, + 0xF4FFFFFF, + 0x00000000, + 0x05000000, + ,, _Y01) + + QWordMemory ( + ResourceProducer, PosDecode, MinFixed, MaxFixed, + NonCacheable, ReadWrite, + 0x0000000000000000, + 0x0000000FFFFFFFF0, + 0x0000000FFFFFFFFF, + 0x0000000000000000, + 0x0000000000000010, + ,, _Y02) + + }, Local1) + + CreateDWordField(Local1, \_SB.PCI0._CRS._Y01._MIN, MMIN) + CreateDWordField(Local1, \_SB.PCI0._CRS._Y01._MAX, MMAX) + CreateDWordField(Local1, \_SB.PCI0._CRS._Y01._LEN, MLEN) + + Store(\_SB.PMIN, MMIN) + Store(\_SB.PLEN, MLEN) + Add(MMIN, MLEN, MMAX) + Subtract(MMAX, One, MMAX) + + /* + * WinXP / Win2K3 blue-screen for operations on 64-bit values. + * Therefore we need to split the 64-bit calculations needed + * here, but different iasl versions evaluate name references + * to integers differently: + * Year (approximate) 2006 2008 2012 + * \_SB.PCI0._CRS._Y02 zero valid valid + * \_SB.PCI0._CRS._Y02._MIN valid valid huge + */ + If(LEqual(Zero, \_SB.PCI0._CRS._Y02)) { + Subtract(\_SB.PCI0._CRS._Y02._MIN, 14, Local0) + } Else { + Store(\_SB.PCI0._CRS._Y02, Local0) + } + CreateDWordField(Local1, Add(Local0, 14), MINL) + CreateDWordField(Local1, Add(Local0, 18), MINH) + CreateDWordField(Local1, Add(Local0, 22), MAXL) + CreateDWordField(Local1, Add(Local0, 26), MAXH) + CreateDWordField(Local1, Add(Local0, 38), LENL) + CreateDWordField(Local1, Add(Local0, 42), LENH) + + Store(\_SB.LMIN, MINL) + Store(\_SB.HMIN, MINH) + Store(\_SB.LLEN, LENL) + Store(\_SB.HLEN, LENH) + Add(MINL, LENL, MAXL) + Add(MINH, LENH, MAXH) + If(LLess(MAXL, MINL)) { + Add(MAXH, One, MAXH) + } + If(LOr(MINH, LENL)) { + If(LEqual(MAXL, 0)) { + Subtract(MAXH, One, MAXH) + } + Subtract(MAXL, One, MAXL) + } + + Return (Local1) + } + + Device(HPET) { + Name(_HID, EISAID("PNP0103")) + Name(_UID, 0) + Method (_STA, 0, NotSerialized) { + If(LEqual(\_SB.HPET, 0)) { + Return(0x00) + } Else { + Return(0x0F) + } + } + Name(_CRS, ResourceTemplate() { + DWordMemory( + ResourceConsumer, PosDecode, MinFixed, MaxFixed, + NonCacheable, ReadWrite, + 0x00000000, + 0xFED00000, + 0xFED003FF, + 0x00000000, + 0x00000400 /* 1K memory: FED00000 - FED003FF */ + ) + }) + } + + + /**************************************************************** + * LPC ISA bridge + ****************************************************************/ + + Device (ISA) + { + Name (_ADR, 0x001f0000) /* device 31, fn 0 */ + + /* PCI Interrupt Routing Register 1 - PIRQA..PIRQD */ + OperationRegion(PIRQ, PCI_Config, 0x60, 0x4) + Scope(\) { + Field (\_SB.PCI0.ISA.PIRQ, ByteAcc, NoLock, Preserve) { + PIRA, 8, + PIRB, 8, + PIRC, 8, + PIRD, 8 + } + } + /* + PCI Interrupt Routing Register 2 (PIRQE..PIRQH) cannot be + used because of existing Xen IRQ limitations (4 PCI links + only) + */ + + /* LPC_I/O: I/O Decode Ranges Register */ + OperationRegion(LPCD, PCI_Config, 0x80, 0x2) + Field(LPCD, AnyAcc, NoLock, Preserve) { + COMA, 3, + , 1, + COMB, 3, + + Offset(0x01), + LPTD, 2, + , 2, + FDCD, 2 + } + + /* LPC_EN: LPC I/F Enables Register */ + OperationRegion(LPCE, PCI_Config, 0x82, 0x2) + Field(LPCE, AnyAcc, NoLock, Preserve) { + CAEN, 1, + CBEN, 1, + LPEN, 1, + FDEN, 1 + } + + Device (SYSR) + { + Name (_HID, EisaId ("PNP0C02")) + Name (_UID, 0x01) + Name (CRS, ResourceTemplate () + { + /* TODO: list hidden resources */ + IO (Decode16, 0x0010, 0x0010, 0x00, 0x10) + IO (Decode16, 0x0022, 0x0022, 0x00, 0x0C) + IO (Decode16, 0x0030, 0x0030, 0x00, 0x10) + IO (Decode16, 0x0044, 0x0044, 0x00, 0x1C) + IO (Decode16, 0x0062, 0x0062, 0x00, 0x02) + IO (Decode16, 0x0065, 0x0065, 0x00, 0x0B) + IO (Decode16, 0x0072, 0x0072, 0x00, 0x0E) + IO (Decode16, 0x0080, 0x0080, 0x00, 0x01) + IO (Decode16, 0x0084, 0x0084, 0x00, 0x03) + IO (Decode16, 0x0088, 0x0088, 0x00, 0x01) + IO (Decode16, 0x008C, 0x008C, 0x00, 0x03) + IO (Decode16, 0x0090, 0x0090, 0x00, 0x10) + IO (Decode16, 0x00A2, 0x00A2, 0x00, 0x1C) + IO (Decode16, 0x00E0, 0x00E0, 0x00, 0x10) + IO (Decode16, 0x08A0, 0x08A0, 0x00, 0x04) + IO (Decode16, 0x0CC0, 0x0CC0, 0x00, 0x10) + IO (Decode16, 0x04D0, 0x04D0, 0x00, 0x02) + }) + Method (_CRS, 0, NotSerialized) + { + Return (CRS) + } + } + + Device (PIC) + { + Name (_HID, EisaId ("PNP0000")) + Name (_CRS, ResourceTemplate () + { + IO (Decode16, 0x0020, 0x0020, 0x01, 0x02) + IO (Decode16, 0x00A0, 0x00A0, 0x01, 0x02) + IRQNoFlags () {2} + }) + } + + Device (DMA0) + { + Name (_HID, EisaId ("PNP0200")) + Name (_CRS, ResourceTemplate () + { + DMA (Compatibility, BusMaster, Transfer8) {4} + IO (Decode16, 0x0000, 0x0000, 0x00, 0x10) + IO (Decode16, 0x0081, 0x0081, 0x00, 0x03) + IO (Decode16, 0x0087, 0x0087, 0x00, 0x01) + IO (Decode16, 0x0089, 0x0089, 0x00, 0x03) + IO (Decode16, 0x008F, 0x008F, 0x00, 0x01) + IO (Decode16, 0x00C0, 0x00C0, 0x00, 0x20) + IO (Decode16, 0x0480, 0x0480, 0x00, 0x10) + }) + } + + Device (TMR) + { + Name (_HID, EisaId ("PNP0100")) + Name (_CRS, ResourceTemplate () + { + IO (Decode16, 0x0040, 0x0040, 0x00, 0x04) + IRQNoFlags () {0} + }) + } + + Device (RTC) + { + Name (_HID, EisaId ("PNP0B00")) + Name (_CRS, ResourceTemplate () + { + IO (Decode16, 0x0070, 0x0070, 0x00, 0x02) + IRQNoFlags () {8} + }) + } + + Device (SPKR) + { + Name (_HID, EisaId ("PNP0800")) + Name (_CRS, ResourceTemplate () + { + IO (Decode16, 0x0061, 0x0061, 0x00, 0x01) + }) + } + + Device (PS2M) + { + Name (_HID, EisaId ("PNP0F13")) + Name (_CID, 0x130FD041) + Method (_STA, 0, NotSerialized) + { + Return (0x0F) + } + + Name (_CRS, ResourceTemplate () + { + IRQNoFlags () {12} + }) + } + + Device (PS2K) + { + Name (_HID, EisaId ("PNP0303")) + Name (_CID, 0x0B03D041) + Method (_STA, 0, NotSerialized) + { + Return (0x0F) + } + + Name (_CRS, ResourceTemplate () + { + IO (Decode16, 0x0060, 0x0060, 0x00, 0x01) + IO (Decode16, 0x0064, 0x0064, 0x00, 0x01) + IRQNoFlags () {1} + }) + } + + Device(FDC0) + { + Name(_HID, EisaId("PNP0700")) + Method(_STA, 0, NotSerialized) + { + Store(FDEN, Local0) + If (LEqual(Local0, 0)) { + Return (0x00) + } Else { + Return (0x0F) + } + } + + Name(_CRS, ResourceTemplate() + { + IO(Decode16, 0x03F2, 0x03F2, 0x00, 0x04) + IO(Decode16, 0x03F7, 0x03F7, 0x00, 0x01) + IRQNoFlags() { 6 } + DMA(Compatibility, NotBusMaster, Transfer8) { 2 } + }) + } + + Device (UAR1) + { + Name (_HID, EisaId ("PNP0501")) + Name (_UID, 0x01) + Method (_STA, 0, NotSerialized) + { + If(LEqual(\_SB.UAR1, 0)) { + Return(0x00) + } Else { + Return(0x0F) + } + } + + Name (_CRS, ResourceTemplate() + { + IO (Decode16, 0x03F8, 0x03F8, 8, 8) + IRQNoFlags () {4} + }) + } + + Device (UAR2) + { + Name (_HID, EisaId ("PNP0501")) + Name (_UID, 0x02) + Method (_STA, 0, NotSerialized) + { + If(LEqual(\_SB.UAR2, 0)) { + Return(0x00) + } Else { + Return(0x0F) + } + } + + Name (_CRS, ResourceTemplate() + { + IO (Decode16, 0x02F8, 0x02F8, 8, 8) + IRQNoFlags () {3} + }) + } + + Device (LTP1) + { + Name (_HID, EisaId ("PNP0400")) + Name (_UID, 0x02) + Method (_STA, 0, NotSerialized) + { + If(LEqual(\_SB.LTP1, 0)) { + Return(0x00) + } Else { + Return(0x0F) + } + } + + Name (_CRS, ResourceTemplate() + { + IO (Decode16, 0x0378, 0x0378, 0x08, 0x08) + IRQNoFlags () {7} + }) + } + + Device(VGID) { + Name(_HID, EisaId ("XEN0000")) + Name(_UID, 0x00) + Name(_CID, "VM_Gen_Counter") + Name(_DDN, "VM_Gen_Counter") + Method(_STA, 0, NotSerialized) + { + If(LEqual(\_SB.VGIA, 0x00000000)) { + Return(0x00) + } Else { + Return(0x0F) + } + } + Name(PKG, Package () + { + 0x00000000, + 0x00000000 + }) + Method(ADDR, 0, NotSerialized) + { + Store(\_SB.VGIA, Index(PKG, 0)) + Return(PKG) + } + } + } + } + } + /* _S3 and _S4 are in separate SSDTs */ + Name (\_S5, Package (0x04) { + 0x00, /* PM1a_CNT.SLP_TYP */ + 0x00, /* PM1b_CNT.SLP_TYP */ + 0x00, /* reserved */ + 0x00 /* reserved */ + }) + Name(PICD, 0) + Method(_PIC, 1) { + Store(Arg0, PICD) + } +}
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 2/12 Makefile: build and use new DSDT table for Q35
In reply to Alexey GerasimenkoProvide building for newly added dsdt_q35.asl file, in a way similar to dsdt.asl.
Note that '15cpu' ACPI tables are only applicable to qemu-traditional (which have no support for Q35), so we need to use 'anycpu' version only.
--- tools/firmware/hvmloader/Makefile | 2 +- tools/libacpi/Makefile | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-)
tools/firmware/hvmloader/Makefile +1 -1
@@ -75,7 +75,7 @@ rombios.o: roms.inc smbios.o: CFLAGS += -D__SMBIOS_DATE__="\"$(SMBIOS_REL_DATE)\"" ACPI_PATH = ../../libacpi -DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c +DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c dsdt_q35_anycpu_qemu_xen.c ACPI_OBJS = $(patsubst %.c,%.o,$(DSDT_FILES)) build.o static_tables.o $(ACPI_OBJS): CFLAGS += -I. -DLIBACPI_STDUTILS=\"$(CURDIR)/util.h\" CFLAGS += -I$(ACPI_PATH)
tools/libacpi/Makefile +8 -1
@@ -21,7 +21,7 @@ endif MK_DSDT = $(ACPI_BUILD_DIR)/mk_dsdt -C_SRC-$(CONFIG_X86) = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c dsdt_pvh.c +C_SRC-$(CONFIG_X86) = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c dsdt_q35_anycpu_qemu_xen.c dsdt_pvh.c C_SRC-$(CONFIG_ARM_64) = dsdt_anycpu_arm.c DSDT_FILES ?= $(C_SRC-y) C_SRC = $(addprefix $(ACPI_BUILD_DIR)/, $(DSDT_FILES))
@@ -56,6 +56,13 @@ $(ACPI_BUILD_DIR)/dsdt_anycpu_qemu_xen.asl: dsdt.asl dsdt_acpi_info.asl $(MK_DSD $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) mv -f $@.$(TMP_SUFFIX) $@ +$(ACPI_BUILD_DIR)/dsdt_q35_anycpu_qemu_xen.asl: dsdt_q35.asl dsdt_acpi_info.asl $(MK_DSDT) + # Remove last bracket + awk 'NR > 1 {print s} {s=$$0}' $< > $@.$(TMP_SUFFIX) + cat dsdt_acpi_info.asl >> $@.$(TMP_SUFFIX) + $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) + mv -f $@.$(TMP_SUFFIX) $@ + # NB. awk invocation is a portable alternative to 'head -n -1' $(ACPI_BUILD_DIR)/dsdt_%cpu.asl: dsdt.asl dsdt_acpi_info.asl $(MK_DSDT) # Remove last bracket
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 3/12 hvmloader: add function to query an emulated machine type (i440/Q35)
In reply to Alexey GerasimenkoThis adds a new function get_pc_machine_type() which allows to determine the emulated chipset type. Supported return values:
- MACHINE_TYPE_I440
- MACHINE_TYPE_Q35
- MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by calling BUG() in hvmloader.
--- tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47 +++++++++++++++++++++++++++++++++++++ tools/firmware/hvmloader/util.h | 8 +++++++ 3 files changed, 60 insertions(+)
tools/firmware/hvmloader/pci_regs.h +5 -0
@@ -107,6 +107,11 @@#define PCI_INTEL_OPREGION 0xfc/* 4 bits */ +#define PCI_VENDOR_ID_INTEL 0x8086 +#define PCI_DEVICE_ID_INTEL_82441 0x1237 +#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 + +#endif /* __HVMLOADER_PCI_REGS_H__ */ /*
tools/firmware/hvmloader/util.c +47 -0
59 lines not on this page · raw message
tools/firmware/hvmloader/util.h +8 -0
@@ -100,6 +100,14 @@ void pci_write(uint32_t devfn, uint32_t reg, uint32_t len, uint32_t val);#define pci_writew (devfn, reg, val) pci_write(devfn, reg, 2, (uint16_t)(val))#define pci_writel (devfn, reg, val) pci_write(devfn, reg, 4, (uint32_t)(val)) +/* Emulated machine types */ +#define MACHINE_TYPE_UNDEFINED 0 +#define MACHINE_TYPE_I440 1 +#define MACHINE_TYPE_Q35 2 +#define MACHINE_TYPE_UNKNOWN (-1) + +int get_pc_machine_type (void ); +/* Get a pointer to the shared-info page */ struct shared_info *get_shared_info (void ) __attribute__ ((const));
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 4/12 hvmloader: add ACPI enabling for Q35
In reply to Alexey GerasimenkoIn order to turn on ACPI for OS, we need to write a chipset-specific value to SMI_CMD register (sort of imitation of the APM->ACPI switch on real systems). Modify acpi_enable_sci() function to support both i440 and Q35 emulation.
--- tools/firmware/hvmloader/hvmloader.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
tools/firmware/hvmloader/hvmloader.c +9 -2
@@ -257,9 +257,16 @@ static const struct bios_config *detect_bios(void)static void acpi_enable_sci (void ) {uint8_t pm1a_cnt_val; +uint8_t acpi_enable_val; -#define PIIX4_SMI_CMD_IOPORT 0xb2 +#define SMI_CMD_IOPORT 0xb2#define PIIX4_ACPI_ENABLE 0xf1 +#define ICH9_ACPI_ENABLE 0x02 + +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + acpi_enable_val =ICH9_ACPI_ENABLE ; +else + acpi_enable_val =PIIX4_ACPI_ENABLE ;/* * PIIX4 emulation in QEMU has SCI_EN=0 by default. We have no legacy
@@ -267,7 +274,7 @@ static void acpi_enable_sci(void)*/ pm1a_cnt_val =inb (ACPI_PM1A_CNT_BLK_ADDRESS_V1 );if ( !(pm1a_cnt_val &ACPI_PM1C_SCI_EN ) ) -outb (PIIX4_SMI_CMD_IOPORT ,PIIX4_ACPI_ENABLE ); +outb (SMI_CMD_IOPORT , acpi_enable_val); pm1a_cnt_val =inb (ACPI_PM1A_CNT_BLK_ADDRESS_V1 );BUG_ON (!(pm1a_cnt_val &ACPI_PM1C_SCI_EN ));
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 5/12 hvmloader: add Q35 DSDT table loading
In reply to Alexey GerasimenkoAllows to select Q35 DSDT table in hvmloader_acpi_build_tables(). Function get_pc_machine_type() is used to select a proper table (i440/q35).
As we are bound to the qemu-xen device model for Q35, no need to initialize config->dsdt_15cpu/config->dsdt_15cpu_len fields.
--- tools/firmware/hvmloader/util.c | 13 +++++++++++-- tools/firmware/hvmloader/util.h | 2 ++ 2 files changed, 13 insertions(+), 2 deletions(-)
tools/firmware/hvmloader/util.c +11 -2
@@ -955,8 +955,17 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, }else if ( !strncmp (s,"qemu_xen" ,9 ) ) { - config->dsdt_anycpu = dsdt_anycpu_qemu_xen; - config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + { + config->dsdt_anycpu = dsdt_q35_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_q35_anycpu_qemu_xen_len; + } +else + { + config->dsdt_anycpu = dsdt_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; + } + config->dsdt_15cpu =NULL ; config->dsdt_15cpu_len =0 ; }
tools/firmware/hvmloader/util.h +2 -0
@@ -288,7 +288,9 @@ bool check_overlap(uint64_t start, uint64_t size, uint64_t reserved_start, uint64_t reserved_size);extern const unsigned char dsdt_anycpu_qemu_xen[], dsdt_anycpu[], dsdt_15cpu[]; +extern const unsigned char dsdt_q35_anycpu_qemu_xen[];extern const int dsdt_anycpu_qemu_xen_len, dsdt_anycpu_len, dsdt_15cpu_len; +extern const int dsdt_q35_anycpu_qemu_xen_len;struct acpi_config ; void hvmloader_acpi_build_tables(struct acpi_config *config,
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 6/12 hvmloader: add basic Q35 support
In reply to Alexey GerasimenkoThis patch does following:
1. Move PCI-device specific initialization out of pci_setup function to the newly created class_specific_pci_device_setup function to simplify code.
2. PCI-device specific initialization extended with LPC controller initialization
3. Initialize PIRQA...{PIRQD, PIRQH} routing accordingly to the emulated south bridge (either located on PCI_ISA_DEVFN or PCI_ICH9_LPC_DEVFN).
--- tools/firmware/hvmloader/config.h | 1 + tools/firmware/hvmloader/pci.c | 162 ++++++++++++++++++++++++-------------- 2 files changed, 104 insertions(+), 59 deletions(-)
tools/firmware/hvmloader/config.h +1 -0
@@ -52,6 +52,7 @@ extern uint8_t ioapic_version;#define PCI_ISA_DEVFN 0x08/* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U/* ISA IRQs 5,10,11 are PCI connected */ +#define PCI_ICH9_LPC_DEVFN 0xf8/* dev 31, fn 0 */ /* MMIO hole: Hardcoded defaults, which can be dynamically expanded. */ #define PCI_MEM_END 0xfc000000
tools/firmware/hvmloader/pci.c +103 -59
209 lines not on this page · raw message
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 7/12 hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GerasimenkoMuch like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via PCI_MAX_MCFG_BUSES in hvmloader/config.h. When multiple PCI buses support for Xen will be implemented, PCI_MAX_MCFG_BUSES may be changed to calculation of the number of buses according to results of the PCI devices enumeration.
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
One important new field is addr_mask, which tells which bits of the base address can (should) be written. Different address types (ROM, MMIO BAR, PCIEXBAR) will have different addr_mask values.
For every assignable BAR range we store its size, PCI device BDF (devfn actually) to which it belongs, BAR type (mem/io/mem64) and corresponding register offset in device PCI conf space. This way we can insert MMCONFIG entry into bars array in the same manner like for any other BARs. In this case, the devfn field will point to MCH PCI device and bar_reg will contain PCIEXBAR register offset. It will be assigned a slot in MMIO hole later in a very same way like for plain PCI BARs, with respect to its size alignment.
Also, to reduce code complexity, all long mem/mem64 BAR flags checks are replaced by simple bars[i] field probing, eg.:
- if ( (bar_reg == PCI_ROM_ADDRESS) ||
- ((bar_data & PCI_BASE_ADDRESS_SPACE) ==
- PCI_BASE_ADDRESS_SPACE_MEMORY) )
+ if ( bars[i].is_mem )
--- tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127 ++++++++++++++++++++++++++++-------- tools/firmware/hvmloader/pci_regs.h | 2 + 3 files changed, 106 insertions(+), 27 deletions(-)
tools/firmware/hvmloader/config.h +4 -0
@@ -53,10 +53,14 @@ extern uint8_t ioapic_version;#define PCI_ISA_DEVFN 0x08/* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U/* ISA IRQs 5,10,11 are PCI connected */ #define PCI_ICH9_LPC_DEVFN 0xf8/* dev 31, fn 0 */ +#define PCI_MCH_DEVFN 0/* bus 0, dev 0, func 0 */ /* MMIO hole: Hardcoded defaults, which can be dynamically expanded. */ #define PCI_MEM_END 0xfc000000 +/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64 +#define ACPI_TIS_HDR_ADDRESS 0xFED40F00ULextern unsigned long pci_mem_start, pci_mem_end;
tools/firmware/hvmloader/pci.c +100 -27
224 lines not on this page · raw message
tools/firmware/hvmloader/pci_regs.h +2 -0
@@ -111,6 +111,8 @@#define PCI_DEVICE_ID_INTEL_82441 0x1237#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 +#define PCI_MCH_PCIEXBAR 0x60 +#endif /* __HVMLOADER_PCI_REGS_H__ */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 8/12 libxl: Q35 support (new option device_model_machine)
In reply to Alexey GerasimenkoProvide a new domain config option to select the emulated machine type, device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.
Note that omitting device_model_machine parameter means i440 system by default, so the default behavior doesn't change for existing domain config files.
Setting device_model_machine to "q35" sends '-machine q35,accel=xen' argument to QEMU. Unlike i440, there no separate machine type to enable/disable Xen platform device, it is controlled via a machine property only. See 'libxl: Xen Platform device support for Q35' patch for a detailed description.
--- tools/libxl/libxl_dm.c | 16 ++++++++++------ tools/libxl/libxl_types.idl | 7 +++++++ tools/xl/xl_parse.c | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-)
tools/libxl/libxl_dm.c +10 -6
@@ -1443,13 +1443,17 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,flexarray_append (dm_args, b_info->extra_pv[i]);break ;case LIBXL_DOMAIN_TYPE_HVM : -if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { -/* Switching here to the machine "pc" which does not add -* the xen-platform device instead of the default "xenfv" machine. -*/ - machinearg =libxl__strdup (gc,"pc,accel=xen" ); +if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); }else { - machinearg =libxl__strdup (gc,"xenfv" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { +/* Switching here to the machine "pc" which does not add +* the xen-platform device instead of the default "xenfv" machine. +*/ + machinearg =libxl__strdup (gc,"pc,accel=xen" ); + }else { + machinearg =libxl__strdup (gc,"xenfv" ); + } }if (b_info->u.hvm.mmio_hole_memkb) { uint64_t max_ram_below_4g = (1ULL << 32) -
tools/libxl/libxl_types.idl +7 -0
@@ -101,6 +101,12 @@ libxl_device_model_version = Enumeration("device_model_version", [ (2, "QEMU_XEN"), # Upstream based qemu-xen device model ]) +libxl_device_model_machine = Enumeration("device_model_machine", [ + (0, "UNKNOWN"), + (1, "I440"), + (2, "Q35"), + ]) + libxl_console_type = Enumeration("console_type", [ (0, "UNKNOWN"), (1, "SERIAL"),
@@ -491,6 +497,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("device_model_ssid_label", string), # device_model_user is not ready for use yet ("device_model_user", string), + ("device_model_machine", libxl_device_model_machine), # extra parameters pass directly to qemu, NULL terminated ("extra", libxl_string_list),
tools/xl/xl_parse.c +14 -0
@@ -2110,6 +2110,20 @@ skip_usbdev:xlu_cfg_replace_string (config,"device_model_user" , &b_info->device_model_user,0 ); +if (!xlu_cfg_get_string (config,"device_model_machine" , &buf,0 )) { +if (!strcmp (buf,"i440" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_I440 ; + }else if (!strcmp (buf,"q35" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_Q35 ; + }else { +fprintf (stderr, +"Unknown device_model_machine \"%s\" specified\n" , buf); +exit (1 ); + } + }else { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_UNKNOWN ; + } +#define parse_extra_args (type) \ e = xlu_cfg_get_list_as_string_list(config, "device_model_args"#type, \ &b_info->extra##type, 0); \
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 9/12 libxl: Xen Platform device support for Q35
In reply to Alexey GerasimenkoCurrent Xen/QEMU method to control Xen Platform device is a bit odd -- changing 'xen_platform_device' option value actually modifies QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use the new way to control Xen Platform device for QEMU -- xen-platform-dev property. To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation uses the old method (xenfv/pc machine) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
--- tools/libxl/libxl_dm.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
tools/libxl/libxl_dm.c +5 -1
@@ -1444,7 +1444,11 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,break ;case LIBXL_DOMAIN_TYPE_HVM :if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { - machinearg =libxl__sprintf (gc,"q35,accel=xen" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); + }else { + machinearg =libxl__sprintf (gc,"q35,accel=xen,xen-platform-dev=on" ); + } }else {if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) {/* Switching here to the machine "pc" which does not add
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 10/12 libacpi: build ACPI MCFG table if requested
In reply to Alexey GerasimenkoThis adds construct_mcfg() function to libacpi which allows to build MCFG table for a given mmconfig_addr/mmconfig_len pair if the ACPI_HAS_MCFG flag was specified in acpi_config struct.
The maximum bus number is calculated from mmconfig_len using MCFG_SIZE_TO_NUM_BUSES macro (1MByte of MMIO space per bus).
--- tools/libacpi/acpi2_0.h | 21 +++++++++++++++++++++ tools/libacpi/build.c | 42 ++++++++++++++++++++++++++++++++++++++++++ tools/libacpi/libacpi.h | 4 ++++ 3 files changed, 67 insertions(+)
tools/libacpi/acpi2_0.h +21 -0
@@ -422,6 +422,25 @@ struct acpi_20_slit { };/* +* PCI Express Memory Mapped Configuration Description Table +*/ +struct mcfg_range_entry { +uint64_t base_address; +uint16_t pci_segment; +uint8_t start_pci_bus_num; +uint8_t end_pci_bus_num; +uint32_t reserved; +}; + +struct acpi_mcfg { +struct acpi_header header; +uint8_t reserved[8 ]; +struct mcfg_range_entry entries[1 ]; +}; + +#define MCFG_SIZE_TO_NUM_BUSES (size) ((size) >> 20) + +/* * Table Signatures. */ #define ACPI_2_0_RSDP_SIGNATURE ASCII64('R','S','D',' ','P','T','R',' ')
@@ -435,6 +454,7 @@ struct acpi_20_slit {#define ACPI_2_0_WAET_SIGNATURE ASCII32('W','A','E','T')#define ACPI_2_0_SRAT_SIGNATURE ASCII32('S','R','A','T')#define ACPI_2_0_SLIT_SIGNATURE ASCII32('S','L','I','T') +#define ACPI_MCFG_SIGNATURE ASCII32('M','C','F','G')/* * Table revision numbers.
@@ -449,6 +469,7 @@ struct acpi_20_slit {#define ACPI_1_0_FADT_REVISION 0x01#define ACPI_2_0_SRAT_REVISION 0x01#define ACPI_2_0_SLIT_REVISION 0x01 +#define ACPI_1_0_MCFG_REVISION 0x01#pragma pack ()
tools/libacpi/build.c +42 -0
60 lines not on this page · raw message
tools/libacpi/libacpi.h +4 -0
@@ -36,6 +36,7 @@#define ACPI_HAS_8042 (1<<13)#define ACPI_HAS_CMOS_RTC (1<<14)#define ACPI_HAS_SSDT_LAPTOP_SLATE (1<<15) +#define ACPI_HAS_MCFG (1<<16)struct xen_vmemrange ;struct acpi_numa {
@@ -96,6 +97,9 @@ struct acpi_config {uint32_t ioapic_base_address;uint16_t pci_isa_irq_mask;uint8_t ioapic_id; + +uint64_t mmconfig_addr; +uint32_t mmconfig_len; };int acpi_build_tables (struct acpi_ctxt *ctxt,struct acpi_config *config);
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 11/12 hvmloader: use libacpi to build MCFG table
In reply to Alexey GerasimenkoThis patch extends hvmloader_acpi_build_tables() with code which detects if MMCONFIG is available -- i.e. initialized and enabled (+we're running on Q35), obtains its base address and size and asks libacpi to build MCFG table for it via setting the flag ACPI_HAS_MCFG in a manner similar to other optional ACPI tables building.
--- tools/firmware/hvmloader/util.c | 70 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
tools/firmware/hvmloader/util.c +70 -0
@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; } +#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1 + +static uint64_t mmconfig_get_base (void ) +{ +uint64_t base; +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + + base = reg | (uint64_t )pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ) <<32 ; + +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ; +case 3 : +BUG ();/* a reserved value encountered */ + } + +return base; +} + +static uint32_t mmconfig_get_size (void ) +{ +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 :return MB (256 ); +case 1 :return MB (128 ); +case 2 :return MB (64 ); +case 3 : +BUG ();/* a reserved value encountered */ + } + +return 0 ; +} + +static uint32_t mmconfig_is_enabled (void ) +{ +return pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ) &PCIEXBAREN ; +} + +static int is_mmconfig_used (void ) +{ +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + { +if (mmconfig_is_enabled () &&mmconfig_get_base ()) +return 1 ; + } + +return 0 ; +} +static void validate_hvm_info (struct hvm_info_table *t) {uint8_t *ptr = (uint8_t *)t;
@@ -993,6 +1056,13 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, config->pci_hi_len = pci_hi_mem_end - pci_hi_mem_start; } +if (is_mmconfig_used () ) + { + config->table_flags |=ACPI_HAS_MCFG ; + config->mmconfig_addr =mmconfig_get_base (); + config->mmconfig_len =mmconfig_get_size (); + } + s =xenstore_read ("platform/generation-id" ,"0:0" );if ( s ) {
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 12/12 docs: provide description for device_model_machine option
In reply to Alexey GerasimenkoThis patch adds description for 'device_model_machine' option which allows to control which chipset will be emulated by device model.
--- docs/man/xl.cfg.pod.5.in | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
docs/man/xl.cfg.pod.5.in +27 -0
@@ -2484,6 +2484,33 @@ you have existing guests then, depending on the nature of the guest Operating System, you may wish to force them to use the device model which they were installed with. +=item B<device_model_machine="STRING"> + +Selects which chipset the device model should emulate for this +guest. + +Valid options are: + +=over 4 + +=item B<"i440"> + +Use i440 emulation (a default setting) + +=item B<"q35"> + +Use Q35/ICH9 emulation. This enables additional features for +PCIe device passthrough + +=back + +Note that omitting device_model_machine parameter means i440 system +by default, so the default behavior doesn't change for old domain +config files. + +It is recommended to install the guest OS from scratch to avoid issues +due to the emulated platform change. + =item B<device_model_override="PATH"> Override the path to the binary to be used as the device-model. The
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 13/30 pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
In reply to Alexey GerasimenkoThe primary difference in PCI device IRQ management between Xen HVM and QEMU is that Xen PCI IRQs are "device-centric" while QEMU PCI IRQs are "chipset-centric". Namely, Xen uses PCI device BDF and INTx as coordinates to assert IRQ while QEMU finds out to which chipset PIRQ the IRQ is routed through the hierarchy of PCI buses and manages IRQ assertion on chipset side (as PIRQ inputs).
Two callback functions are used for this purpose: .map_irq and .set_irq (named after corresponding structure fields). Corresponding Xen-specific callback functions are piix3_set_irq() and pci_slot_get_pirq(). In Xen case these functions do not operate on pirq pin numbers. Instead, they use a specific value to pass BDF/INTx information between .map_irq and .set_irq -- PCI device devfn and INTx pin number are combined into pseudo-PIRQ in pci_slot_get_pirq, which piix3_set_irq later decodes back into devfn and INTx number for passing to *set_pci_intx_level() call.
For Xen on Q35 this scheme is still applicable, with the exception that function names are non-descriptive now and need to be renamed to show their common i440/Q35 nature. Proposed new names are:
xen_pci_slot_get_pirq --> xen_cmn_pci_slot_get_pirqxen_piix3_set_irq --> xen_cmn_set_irq
Another IRQ-related difference between i440 and Q35 is the number of PIRQ inputs and PIRQ routers (PCI IRQ links in terms of ACPI) available. i440 has 4 PCI interrupt links, while Q35 has 8 (PIRQA...PIRQH). Currently Xen have support for only 4 PCI links, so we describe only 4 of 8 PCI links in ACPI tables. Also, hvmloader disables PIRQ routing for PIRQE..PIRQH by writing 80h into corresponding PIRQ[n]_ROUT registers.
All this PCI interrupt routing stuff is largely an ancient legacy from PIC era. It's hardly worth to extend number of PCI links supported as we normally deal with APIC mode and/or MSI interrupts.
The only useful thing to do with PIRQE..PIRQH routing currently is to check if guest actually attempts to use it for some reason (despite ACPI PCI routing information provided). In this case, a warning is logged.
--- hw/i386/pc_q35.c | 13 ++++++++++--- hw/i386/xen/xen-hvm.c | 32 +++++++++++++++++++++++++++++--- hw/isa/lpc_ich9.c | 4 ++++ hw/pci-host/piix.c | 2 +- include/hw/i386/ich9.h | 1 + include/hw/xen/xen.h | 5 +++-- stubs/xen-hvm.c | 8 ++++++-- 7 files changed, 54 insertions(+), 11 deletions(-)
hw/i386/pc_q35.c +10 -3
@@ -203,9 +203,16 @@ static void pc_q35_init(MachineState *machine)for (i =0 ; i <GSI_NUM_PINS ; i++) {qdev_connect_gpio_out_named (lpc_dev,ICH9_GPIO_GSI , i, pcms->gsi[i]); } -pci_bus_irqs (host_bus, ich9_lpc_set_irq, ich9_lpc_map_irq, ich9_lpc, -ICH9_LPC_NB_PIRQS ); -pci_bus_set_route_irq_fn (host_bus, ich9_route_intx_pin_to_irq); + +if (xen_enabled ()) { +pci_bus_irqs (host_bus, xen_cmn_set_irq, xen_cmn_pci_slot_get_pirq, + ich9_lpc,ICH9_XEN_NUM_IRQ_SOURCES ); + }else { +pci_bus_irqs (host_bus, ich9_lpc_set_irq, ich9_lpc_map_irq, ich9_lpc, +ICH9_LPC_NB_PIRQS ); +pci_bus_set_route_irq_fn (host_bus, ich9_route_intx_pin_to_irq); + } + isa_bus = ich9_lpc->isa_bus;if (kvm_pic_in_kernel ()) {
hw/i386/xen/xen-hvm.c +29 -3
@@ -13,6 +13,7 @@#include "cpu.h" #include "hw/pci/pci.h" #include "hw/i386/pc.h" +#include "hw/i386/ich9.h" #include "hw/i386/apic-msidef.h" #include "hw/xen/xen_common.h" #include "hw/xen/xen_backend.h"
@@ -115,14 +116,14 @@ typedef struct XenIOState {Notifier wakeup; } XenIOState; -/* Xen specific function for piix pci */ +/* Xen-specific functions for pci dev IRQ handling */ -int xen_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num) +int xen_cmn_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num) {return irq_num + ((pci_dev->devfn >>3 ) <<2 ); } -void xen_piix3_set_irq (void *opaque,int irq_num,int level) +void xen_cmn_set_irq (void *opaque,int irq_num,int level) {xen_set_pci_intx_level (xen_domid,0 ,0 , irq_num >>2 , irq_num &3 , level);
@@ -145,6 +146,31 @@ void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len) } } +void xen_ich9_pci_write_config_client (uint32_t address,uint32_t val,int len) +{ +static bool pirqe_f_warned = false; + +if (ranges_overlap (address, len,ICH9_LPC_PIRQA_ROUT ,4 )) { +/* handle PIRQA..PIRQD routing */ +xen_piix_pci_write_config_client (address, val, len); + }else if (ranges_overlap (address, len,ICH9_LPC_PIRQE_ROUT ,4 )) { +while (len--) { +if (range_covers_byte (ICH9_LPC_PIRQE_ROUT ,4 , address) && + (val &0x80 ) ==0 ) { +/* print warning only once */ +if (!pirqe_f_warned) { + pirqe_f_warned = true; +fprintf (stderr,"WARNING: guest domain attempted to use PIRQ%c " +"routing which is not supported for Xen/Q35 currently\n" , + (char )(address -ICH9_LPC_PIRQE_ROUT +'E' )); +break ; + } + } + address++, val >>=8 ; + } + } +} +int xen_is_pirq_msi (uint32_t msi_data) {/* If vector is 0, the msi is remapped into a pirq, passed as
hw/isa/lpc_ich9.c +4 -0
@@ -49,6 +49,7 @@#include "qom/cpu.h" #include "hw/nvram/fw_cfg.h" #include "qemu/cutils.h" +#include "hw/xen/xen.h" /*****************************************************************************/ /* ICH9 LPC PCI to ISA bridge */
@@ -514,6 +515,9 @@ static void ich9_lpc_config_write(PCIDevice *d,ICH9LPCState *lpc =ICH9_LPC_DEVICE (d);uint32_t rcba_old =pci_get_long (d->config +ICH9_LPC_RCBA ); +if (xen_enabled ()){ +xen_ich9_pci_write_config_client (addr, val, len); + }pci_default_write_config (d, addr, val, len);if (ranges_overlap (addr, len,ICH9_LPC_PMBASE ,4 ) ||ranges_overlap (addr, len,ICH9_LPC_ACPI_CTRL ,1 )) {
hw/pci-host/piix.c +1 -1
@@ -415,7 +415,7 @@ PCIBus *i440fx_init(const char *host_type, const char *pci_type,PCIDevice *pci_dev =pci_create_simple_multifunction (b,-1 , true,"PIIX3-xen" ); piix3 =PIIX3_PCI_DEVICE (pci_dev); -pci_bus_irqs (b, xen_piix3_set_irq, xen_pci_slot_get_pirq, +pci_bus_irqs (b, xen_cmn_set_irq, xen_cmn_pci_slot_get_pirq, piix3,XEN_PIIX_NUM_PIRQS ); }else { PCIDevice *pci_dev = pci_create_simple_multifunction(b,
include/hw/i386/ich9.h +1 -0
@@ -143,6 +143,7 @@ Object *ich9_lpc_find(void);#define ICH9_A2_LPC_REVISION 0x2#define ICH9_LPC_NB_PIRQS 8/* PCI A-H */ +#define ICH9_XEN_NUM_IRQ_SOURCES 128#define ICH9_LPC_PMBASE 0x40#define ICH9_LPC_PMBASE_BASE_ADDRESS_MASK Q35_MASK(32, 15, 7)
include/hw/xen/xen.h +3 -2
@@ -30,9 +30,10 @@ static inline bool xen_enabled(void)return xen_allowed; } -int xen_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num); -void xen_piix3_set_irq (void *opaque,int irq_num,int level); +int xen_cmn_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num); +void xen_cmn_set_irq (void *opaque,int irq_num,int level);void xen_piix_pci_write_config_client (uint32_t address,uint32_t val,int len); +void xen_ich9_pci_write_config_client (uint32_t address,uint32_t val,int len);void xen_hvm_inject_msi (uint64_t addr,uint32_t data);int xen_is_pirq_msi (uint32_t msi_data);
stubs/xen-hvm.c +6 -2
@@ -14,12 +14,12 @@#include "exec/memory.h" #include "qapi/qapi-commands-misc.h" -int xen_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num) +int xen_cmn_pci_slot_get_pirq (PCIDevice *pci_dev,int irq_num) {return -1 ; } -void xen_piix3_set_irq (void *opaque,int irq_num,int level) +void xen_cmn_set_irq (void *opaque,int irq_num,int level) { }
@@ -27,6 +27,10 @@ void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len) { } +void xen_ich9_pci_write_config_client (uint32_t address,uint32_t val,int len) +{ +} +void xen_hvm_inject_msi (uint64_t addr,uint32_t data) { }
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 14/30 pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug
In reply to Alexey GerasimenkoOn Q35 we still need to assign BSEL property to bus(es) for PCI device add/hotplug to work.
Extend acpi_set_pci_info() function to support Q35 as well. Previously it was limited to find_i440fx() call, this patch adds new (trivial) function find_q35() which returns root PCIBus object on Q35, in a way similar to what find_i440fx does.
--- hw/acpi/pcihp.c | 6 +++++- hw/pci-host/q35.c | 8 ++++++++ include/hw/i386/pc.h | 3 +++ 3 files changed, 16 insertions(+), 1 deletion(-)
hw/acpi/pcihp.c +5 -1
@@ -105,7 +105,11 @@ static void acpi_set_pci_info(void) } bsel_is_set = true; - bus =find_i440fx ();/* TODO: Q35 support */ + bus =find_i440fx (); +if (!bus) { + bus =find_q35 (); + } +if (bus) {/* Scan all PCI buses. Set property to enable acpi based hotplug. */ pci_for_each_bus_depth_first (bus, acpi_set_bsel,NULL , &bsel_alloc);
hw/pci-host/q35.c +8 -0
@@ -258,6 +258,14 @@ static void q35_host_initfn(Object *obj) IO_APIC_DEFAULT_ADDRESS - 1); } +PCIBus *find_q35 (void ) +{ +PCIHostState *s =OBJECT_CHECK (PCIHostState, +object_resolve_path ("/machine/q35" ,NULL ), +TYPE_PCI_HOST_BRIDGE ); +return s ? s->bus :NULL ; +} +static const TypeInfo q35_host_info = { .name =TYPE_Q35_HOST_DEVICE , .parent =TYPE_PCIE_HOST_BRIDGE ,
include/hw/i386/pc.h +3 -0
@@ -302,6 +302,9 @@ PCIBus *find_i440fx(void);extern PCIDevice *piix4_dev;int piix4_init (PCIBus *bus,ISABus **isa_bus,int devfn); +/* q35.c */ +PCIBus *find_q35 (void ); +/* pc_sysfw.c */ void pc_system_firmware_init (MemoryRegion *rom_memory,bool isapc_ram_fw);
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 15/30 q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35
In reply to Alexey GerasimenkoThis patch allows to use ACPI PCI hotplug functionality for Xen on Q35. All added code depends on xen_enabled(), so no functionality change for non-Xen usage.
We need to call the acpi_set_pci_info function from ich9_pm_init as well, so it was made globally visible again (as it was before).
--- hw/acpi/ich9.c | 24 ++++++++++++++++++++++++ hw/acpi/pcihp.c | 2 +- include/hw/acpi/ich9.h | 2 ++ include/hw/acpi/pcihp.h | 2 ++ 4 files changed, 29 insertions(+), 1 deletion(-)
hw/acpi/ich9.c +24 -0
@@ -37,6 +37,7 @@#include "hw/i386/ich9.h" #include "hw/mem/pc-dimm.h" +#include "hw/xen/xen.h" //#define DEBUG
@@ -258,6 +259,10 @@ static void pm_reset(void *opaque) pm->smi_en_wmask = ~0 ;acpi_update_sci (&pm->acpi_regs, pm->irq); + +if (xen_enabled ()) { +acpi_pcihp_reset (&pm->acpi_pci_hotplug); + } }static void pm_powerdown_req (Notifier *n,void *opaque)
@@ -300,6 +305,17 @@ void ich9_pm_init(PCIDevice *lpc_pci, ICH9LPCPMRegs *pm, pm->powerdown_notifier.notify = pm_powerdown_req;qemu_register_powerdown_notifier (&pm->powerdown_notifier); +if (xen_enabled ()) { +PCIBus *bus =pci_get_bus (lpc_pci); + +qbus_set_hotplug_handler (BUS (bus),DEVICE (lpc_pci), &error_abort); + +acpi_pcihp_init (OBJECT (lpc_pci), &pm->acpi_pci_hotplug, bus, +pci_address_space_io (lpc_pci), false); + +acpi_set_pci_info (); + } +legacy_acpi_cpu_hotplug_init (pci_address_space_io (lpc_pci),OBJECT (lpc_pci), &pm->gpe_cpu,ICH9_CPU_HOTPLUG_IO_BASE );
@@ -496,6 +512,10 @@ void ich9_pm_device_plug_cb(HotplugHandler *hotplug_dev, DeviceState *dev,acpi_memory_plug_cb (hotplug_dev, &lpc->pm.acpi_memory_hotplug, dev, errp); } + }else if (xen_enabled () && +object_dynamic_cast (OBJECT (dev),TYPE_PCI_DEVICE )) { +acpi_pcihp_device_plug_cb (hotplug_dev, &lpc->pm.acpi_pci_hotplug, + dev, errp); }else if (object_dynamic_cast (OBJECT (dev),TYPE_CPU )) {if (lpc->pm.cpu_hotplug_legacy) {legacy_acpi_cpu_plug_cb (hotplug_dev, &lpc->pm.gpe_cpu, dev, errp);
@@ -522,6 +542,10 @@ void ich9_pm_device_unplug_request_cb(HotplugHandler *hotplug_dev, !lpc->pm.cpu_hotplug_legacy) {acpi_cpu_unplug_request_cb (hotplug_dev, &lpc->pm.cpuhp_state, dev, errp); + } else if (xen_enabled() && +object_dynamic_cast (OBJECT (dev ),TYPE_PCI_DEVICE )) { +acpi_pcihp_device_unplug_cb (hotplug_dev, &lpc->pm.acpi_pci_hotplug, + dev, errp); } else {error_setg (errp,"acpi: device unplug request for not supported device" " type: %s" ,object_get_typename (OBJECT (dev)));
hw/acpi/pcihp.c +1 -1
@@ -94,7 +94,7 @@ static void *acpi_set_bsel(PCIBus *bus, void *opaque)return bsel_alloc; } -static void acpi_set_pci_info (void ) +void acpi_set_pci_info (void ) {static bool bsel_is_set;PCIBus *bus;
include/hw/acpi/ich9.h +2 -0
@@ -26,6 +26,7 @@#include "hw/acpi/cpu.h" #include "hw/acpi/memory_hotplug.h" #include "hw/acpi/acpi_dev_interface.h" +#include "hw/acpi/pcihp.h" #include "hw/acpi/tco.h" typedef struct ICH9LPCPMRegs {
@@ -52,6 +53,7 @@ typedef struct ICH9LPCPMRegs {bool cpu_hotplug_legacy;AcpiCpuHotplug gpe_cpu;CPUHotplugState cpuhp_state; +AcpiPciHpState acpi_pci_hotplug;MemHotplugState acpi_memory_hotplug;
include/hw/acpi/pcihp.h +2 -0
@@ -64,6 +64,8 @@ void acpi_pcihp_device_unplug_cb(HotplugHandler *hotplug_dev, AcpiPciHpState *s,/* Called on reset */ void acpi_pcihp_reset (AcpiPciHpState *s); +void acpi_set_pci_info (void ); +extern const VMStateDescription vmstate_acpi_pcihp_pci_status;#define VMSTATE_PCI_HOTPLUG (pcihp, state, test_pcihp) \
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 16/30 q35/xen: Add Xen platform device support for Q35
In reply to Alexey GerasimenkoCurrent Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
--- hw/core/machine.c | 21 +++++++++++++++++++++ hw/i386/pc_q35.c | 14 ++++++++++++++ include/hw/boards.h | 1 + qemu-options.hx | 1 + 4 files changed, 37 insertions(+)
hw/core/machine.c +21 -0
@@ -290,6 +290,20 @@ static void machine_set_igd_gfx_passthru(Object *obj, bool value, Error **errp) ms->igd_gfx_passthru = value; } +static bool machine_get_xen_platform_dev (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->xen_platform_dev; +} + +static void machine_set_xen_platform_dev (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->xen_platform_dev = value; +} +static char *machine_get_firmware (Object *obj,Error **errp) {MachineState *ms =MACHINE (obj);
@@ -595,6 +609,13 @@ static void machine_class_init(ObjectClass *oc, void *data)object_class_property_set_description (oc,"igd-passthru" ,"Set on/off to enable/disable igd passthrou" , &error_abort); +object_class_property_add_bool (oc,"xen-platform-dev" , + machine_get_xen_platform_dev, + machine_set_xen_platform_dev, &error_abort); +object_class_property_set_description (oc,"xen-platform-dev" , +"Set on/off to enable/disable Xen Platform device" , + &error_abort); +object_class_property_add_str (oc,"firmware" , machine_get_firmware, machine_set_firmware, &error_abort);
hw/i386/pc_q35.c +14 -0
@@ -56,6 +56,18 @@/* ICH9 AHCI has 6 ports */ #define MAX_SATA_PORTS 6 +static void q35_xen_hvm_init (MachineState *machine) +{ +PCMachineState *pcms =PC_MACHINE (machine); + +if (xen_enabled ()) { +/* check if Xen Platform device is enabled */ +if (machine->xen_platform_dev) { +pci_create_simple (pcms->bus,-1 ,"xen-platform" ); + } + } +} +/* PC hardware initialisation */ static void pc_q35_init (MachineState *machine) {
@@ -207,6 +219,8 @@ static void pc_q35_init(MachineState *machine)if (xen_enabled ()) {pci_bus_irqs (host_bus, xen_cmn_set_irq, xen_cmn_pci_slot_get_pirq, ich9_lpc,ICH9_XEN_NUM_IRQ_SOURCES ); + +q35_xen_hvm_init (machine); }else {pci_bus_irqs (host_bus, ich9_lpc_set_irq, ich9_lpc_map_irq, ich9_lpc,ICH9_LPC_NB_PIRQS );
include/hw/boards.h +1 -0
@@ -238,6 +238,7 @@ struct MachineState {bool usb;bool usb_disabled;bool igd_gfx_passthru; +bool xen_platform_dev;char *firmware;bool iommu;bool suppress_vmdesc;
qemu-options.hx +1 -0
@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory in a core dump (default=on)\n" " mem-merge=on|off controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n" + " xen-platform-dev=on|off controls Xen Platform device (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n" " dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n" " suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
Signature · 1 line
2.11.0
RFC PATCH 17/30 q35: Fix incorrect values for PCIEXBAR masks
In reply to Alexey GerasimenkoThere are two small issues in PCIEXBAR address mask handling:
- wrong bit positions for address mask bits (see PCIEXBAR description in Q35 datasheet)
- incorrect usage of 64ADR_MASK
Due to this, attempting to write a valid PCIEXBAR address may cause it to shift to another address, causing memory layout corruption where emulated MMIO regions may overlap real (passed through) MMIO ranges. Fix this by providing correct values.
--- hw/pci-host/q35.c | 6 +++--- include/hw/pci-host/q35.h | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
hw/pci-host/q35.c +3 -3
@@ -322,12 +322,12 @@ static void mch_update_pciexbar(MCHPCIState *mch)break ;case MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_128M : length =128 *1024 *1024 ; - addr_mask |=MCH_HOST_BRIDGE_PCIEXBAR_128ADMSK | -MCH_HOST_BRIDGE_PCIEXBAR_64ADMSK ; + addr_mask |=MCH_HOST_BRIDGE_PCIEXBAR_128ADMSK ;break ;case MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_64M : length =64 *1024 *1024 ; - addr_mask |=MCH_HOST_BRIDGE_PCIEXBAR_64ADMSK ; + addr_mask |=MCH_HOST_BRIDGE_PCIEXBAR_64ADMSK | +MCH_HOST_BRIDGE_PCIEXBAR_128ADMSK ;break ;case MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_RVD :default :
include/hw/pci-host/q35.h +2 -2
@@ -103,8 +103,8 @@ typedef struct Q35PCIHost {#define MCH_HOST_BRIDGE_PCIEXBAR_DEFAULT 0xb0000000#define MCH_HOST_BRIDGE_PCIEXBAR_MAX (0x10000000)/* 256M */ #define MCH_HOST_BRIDGE_PCIEXBAR_ADMSK Q35_MASK(64, 35, 28) -#define MCH_HOST_BRIDGE_PCIEXBAR_128ADMSK ((uint64_t)(1 << 26)) -#define MCH_HOST_BRIDGE_PCIEXBAR_64ADMSK ((uint64_t)(1 << 25)) +#define MCH_HOST_BRIDGE_PCIEXBAR_128ADMSK ((uint64_t)(1 << 27)) +#define MCH_HOST_BRIDGE_PCIEXBAR_64ADMSK ((uint64_t)(1 << 26))#define MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_MASK ((uint64_t)(0x3 << 1))#define MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_256M ((uint64_t)(0x0 << 1))#define MCH_HOST_BRIDGE_PCIEXBAR_LENGTH_128M ((uint64_t)(0x1 << 1))
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 18/30 xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and PCIe Extended Capabilities enumeration
In reply to Alexey GerasimenkoThis patch introduces 2 new functions,
- xen_host_pci_find_next_ext_cap (actually a reworked xen_host_pci_find_ext_cap_offset function which is unused)
- xen_host_pci_find_next_cap
These functions allow to search for PCI/PCIe capabilities in a uniform way. Both functions allow to search either a specific capability or any encountered next (by specifying CAP_ID_ANY as a capability ID) -- this may be useful when we merely need to traverse the capability list one-by-one. In both functions the 'pos' argument allows to continue searching from last position (0 means to start from beginning).
In order not to probe PCIe Extended Capabilities existence every time, xen_host_pci_find_next_ext_cap makes use of the new 'has_pcie_ext_caps' field in XenHostPCIDevice structure which is filled only once (in xen_host_pci_device_get).
--- hw/xen/xen-host-pci-device.c | 95 +++++++++++++++++++++++++++++++++++++------- hw/xen/xen-host-pci-device.h | 5 ++- 2 files changed, 85 insertions(+), 15 deletions(-)
hw/xen/xen-host-pci-device.c +81 -14
@@ -14,6 +14,7 @@#define XEN_HOST_PCI_MAX_EXT_CAP \ ((PCIE_CONFIG_SPACE_SIZE - PCI_CONFIG_SPACE_SIZE) / (PCI_CAP_SIZEOF + 4)) +#define XEN_HOST_PCI_CAP_MAX 48#ifdef XEN_HOST_PCI_DEVICE_DEBUG # define XEN_HOST_PCI_LOG(f, a...) fprintf(stderr, "%s: " f, __func__, ##a)
@@ -199,6 +200,19 @@ static bool xen_host_pci_dev_is_virtfn(XenHostPCIDevice *d)return !stat (path, &buf); } +static bool xen_host_pci_dev_has_pcie_ext_caps (XenHostPCIDevice *d) +{ +uint32_t header; + +if (xen_host_pci_get_long (d,PCI_CONFIG_SPACE_SIZE , &header)) +return false; + +if (header ==0 || header == ~0U ) +return false; + +return true; +} +static void xen_host_pci_config_open (XenHostPCIDevice *d,Error **errp) {char path[PATH_MAX ];
@@ -297,37 +311,89 @@ int xen_host_pci_set_block(XenHostPCIDevice *d, int pos, uint8_t *buf, int len)return xen_host_pci_config_write (d, pos, buf, len); } -int xen_host_pci_find_ext_cap_offset (XenHostPCIDevice *d,uint32_t cap) +int xen_host_pci_find_next_ext_cap (XenHostPCIDevice *d,int pos,uint32_t cap) {uint32_t header =0 ;int max_cap =XEN_HOST_PCI_MAX_EXT_CAP ; -int pos =PCI_CONFIG_SPACE_SIZE ; + +if (!d->has_pcie_ext_caps) +return 0 ; + +if (!pos) { + pos =PCI_CONFIG_SPACE_SIZE ; + }else { +if (xen_host_pci_get_long (d, pos, &header)) +return 0 ; + + pos =PCI_EXT_CAP_NEXT (header); + }do { -if (xen_host_pci_get_long (d, pos, &header)) { +if (!pos || pos <PCI_CONFIG_SPACE_SIZE ) +break ; + +if (xen_host_pci_get_long (d, pos, &header))break ; - }/* * If we have no capabilities, this is indicated by cap ID, * cap version and next pointer all being 0. +* Also check for all F's returned (which means PCIe ext conf space +* is unreadable for some reason) */ -if (header ==0 ) { +if (header ==0 || header == ~0U )break ; - } -if (PCI_EXT_CAP_ID (header) == cap) { +if (cap ==CAP_ID_ANY ) +return pos; +else if (PCI_EXT_CAP_ID (header) == cap)return pos; - } pos =PCI_EXT_CAP_NEXT (header); -if (pos <PCI_CONFIG_SPACE_SIZE ) { + }while (--max_cap); + +return 0 ; +} + +int xen_host_pci_find_next_cap (XenHostPCIDevice *d,int pos,uint32_t cap) +{ +uint8_t id; +unsigned max_cap =XEN_HOST_PCI_CAP_MAX ; +uint8_t status =0 ; +uint8_t curpos; + +if (xen_host_pci_get_byte (d,PCI_STATUS , &status)) +return 0 ; + +if ((status &PCI_STATUS_CAP_LIST ) ==0 ) +return 0 ; + +if (pos <PCI_CAPABILITY_LIST ) { + curpos =PCI_CAPABILITY_LIST ; + }else { + curpos = (uint8_t ) pos; + } + +while (max_cap--) { +if (xen_host_pci_get_byte (d, curpos, &curpos)) +break ; +if (!curpos)break ; - } - max_cap--; - }while (max_cap >0 ); +if (cap ==CAP_ID_ANY ) +return curpos; -return -1 ; +if (xen_host_pci_get_byte (d, curpos +PCI_CAP_LIST_ID , &id)) +break ; + +if (id ==0xff ) +break ; +else if (id == cap) +return curpos; + + curpos +=PCI_CAP_LIST_NEXT ; + } + +return 0 ; } void xen_host_pci_device_get(XenHostPCIDevice *d, uint16_t domain,
@@ -377,7 +443,8 @@ void xen_host_pci_device_get(XenHostPCIDevice *d, uint16_t domain, } d->class_code = v; - d->is_virtfn =xen_host_pci_dev_is_virtfn (d); + d->is_virtfn =xen_host_pci_dev_is_virtfn (d); + d->has_pcie_ext_caps =xen_host_pci_dev_has_pcie_ext_caps (d);return ;
hw/xen/xen-host-pci-device.h +4 -1
@@ -32,6 +32,7 @@ typedef struct XenHostPCIDevice {XenHostPCIIORegion rom;bool is_virtfn; +bool has_pcie_ext_caps;int config_fd; } XenHostPCIDevice;
@@ -53,6 +54,8 @@ int xen_host_pci_set_long(XenHostPCIDevice *d, int pos, uint32_t data);int xen_host_pci_set_block (XenHostPCIDevice *d,int pos,uint8_t *buf,int len); -int xen_host_pci_find_ext_cap_offset (XenHostPCIDevice *s,uint32_t cap); +#define CAP_ID_ANY (~0U) +int xen_host_pci_find_next_cap (XenHostPCIDevice *s,int pos,uint32_t cap); +int xen_host_pci_find_next_ext_cap (XenHostPCIDevice *d,int pos,uint32_t cap);#endif /* XEN_HOST_PCI_DEVICE_H */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 19/30 xen/pt: avoid reading PCIe device type and cap version multiple times
In reply to Alexey Gerasimenkoxen_pt_config_init.c reads Device/Port Type and Capability version fields in many places. Two functions are used for this purpose:get_capability_version and get_device_type. These functions perform PCI conf space reading every time they're called. Another bad thing is that these functions know nothing about where PCI Expess Capability is located, so its offset must be provided explicitly in function arguments. Their typical usage is like this:
uint8_t cap_ver = get_capability_version(s, real_offset - reg->offset);
uint8_t dev_type = get_device_type(s, real_offset - reg->offset);To avoid this, the PCI Express Capability register now being read only once and stored in XenHostPCIDevice structure (pcie_flags field). The capabiliy offset parameter is no longer needed, simplifying functions usage. Also, get_device_type and get_capability_version were renamed to more descriptive get_pcie_device_type and get_pcie_capability_version.
--- hw/xen/xen-host-pci-device.c | 15 +++++++++++++++ hw/xen/xen-host-pci-device.h | 1 + hw/xen/xen_pt_config_init.c | 34 ++++++++++++++-------------------- 3 files changed, 30 insertions(+), 20 deletions(-)
hw/xen/xen-host-pci-device.c +15 -0
@@ -402,6 +402,7 @@ void xen_host_pci_device_get(XenHostPCIDevice *d, uint16_t domain, {unsigned int v;Error *err =NULL ; +int pcie_cap_pos; d->config_fd =-1 ; d->domain = domain;
@@ -446,6 +447,20 @@ void xen_host_pci_device_get(XenHostPCIDevice *d, uint16_t domain, d->is_virtfn =xen_host_pci_dev_is_virtfn (d); d->has_pcie_ext_caps =xen_host_pci_dev_has_pcie_ext_caps (d); +/* read and store PCIe Capabilities field for later use */ + pcie_cap_pos =xen_host_pci_find_next_cap (d,0 ,PCI_CAP_ID_EXP ); + +if (pcie_cap_pos) { +if (xen_host_pci_get_word (d, pcie_cap_pos +PCI_EXP_FLAGS , + &d->pcie_flags)) { +error_setg (&err,"Unable to read from PCI Express capability " +"structure at 0x%x" , pcie_cap_pos); + goto error; + } + }else { + d->pcie_flags =0xFFFF ; + } +return ; error:
hw/xen/xen-host-pci-device.h +1 -0
@@ -27,6 +27,7 @@ typedef struct XenHostPCIDevice {uint16_t device_id;uint32_t class_code;int irq; +uint16_t pcie_flags;XenHostPCIIORegion io_regions[PCI_NUM_REGIONS -1 ];XenHostPCIIORegion rom;
hw/xen/xen_pt_config_init.c +14 -20
@@ -828,24 +828,18 @@ static XenPTRegInfo xen_pt_emu_reg_vendor[] = {* PCI Express Capability */ -static inline uint8_t get_capability_version (XenPCIPassthroughState *s, -uint32_t offset) +static inline uint8_t get_pcie_capability_version (XenPCIPassthroughState *s) { -uint8_t flag; -if (xen_host_pci_get_byte (&s->real_device, offset +PCI_EXP_FLAGS , &flag)) { -return 0 ; - } -return flag &PCI_EXP_FLAGS_VERS ; +assert (s->real_device.pcie_flags !=0xFFFF ); + +return (uint8_t ) (s->real_device.pcie_flags &PCI_EXP_FLAGS_VERS ); } -static inline uint8_t get_device_type (XenPCIPassthroughState *s, -uint32_t offset) +static inline uint8_t get_pcie_device_type (XenPCIPassthroughState *s) { -uint8_t flag; -if (xen_host_pci_get_byte (&s->real_device, offset +PCI_EXP_FLAGS , &flag)) { -return 0 ; - } -return (flag &PCI_EXP_FLAGS_TYPE ) >>4 ; +assert (s->real_device.pcie_flags !=0xFFFF ); + +return (uint8_t ) ((s->real_device.pcie_flags &PCI_EXP_FLAGS_TYPE ) >>4 ); }/* initialize Link Control register */
@@ -853,8 +847,8 @@ static int xen_pt_linkctrl_reg_init(XenPCIPassthroughState *s, XenPTRegInfo *reg, uint32_t real_offset, uint32_t *data) { - uint8_t cap_ver = get_capability_version(s, real_offset - reg->offset); -uint8_t dev_type =get_device_type (s, real_offset - reg->offset); + uint8_t cap_ver = get_pcie_capability_version(s); +uint8_t dev_type =get_pcie_device_type (s);/* no need to initialize in case of Root Complex Integrated Endpoint * with cap_ver 1.x
@@ -871,7 +865,7 @@ static int xen_pt_devctrl2_reg_init(XenPCIPassthroughState *s, XenPTRegInfo *reg, uint32_t real_offset, uint32_t *data) { - uint8_t cap_ver = get_capability_version(s, real_offset - reg->offset); + uint8_t cap_ver = get_pcie_capability_version(s);/* no need to initialize in case of cap_ver 1.x */ if (cap_ver ==1 ) {
@@ -886,7 +880,7 @@ static int xen_pt_linkctrl2_reg_init(XenPCIPassthroughState *s, XenPTRegInfo *reg, uint32_t real_offset, uint32_t *data) { - uint8_t cap_ver = get_capability_version(s, real_offset - reg->offset); + uint8_t cap_ver = get_pcie_capability_version(s);uint32_t reg_field =0 ;/* no need to initialize in case of cap_ver 1.x */
@@ -1586,8 +1580,8 @@ static int xen_pt_pcie_size_init(XenPCIPassthroughState *s, uint32_t base_offset, uint8_t *size) { PCIDevice *d = &s->dev; -uint8_t version =get_capability_version (s, base_offset); -uint8_t type =get_device_type (s, base_offset); +uint8_t version =get_pcie_capability_version (s); +uint8_t type =get_pcie_device_type (s);uint8_t pcie_size =0 ;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 20/30 xen/pt: determine the legacy/PCIe mode for a passed through device
In reply to Alexey GerasimenkoEven if we have some real PCIe device being passed through to a guest, there are situations when we cannot use its PCIe features, primarily allowing to access extended (>256) config space.
Basically, we can allow reading PCIe extended config space only if both the device and emulated system are PCIe-capable. So it's a combination of checks:
- PCI Express capability presence
- pci_is_express(device)
- pci_bus_is_express(device bus)
The AND-product of these checks is stored to pcie_enabled_dev flag in XenPCIPassthroughState for later use in functions like xen_pt_pci_config_access_check.
This way we get consistent behavior when the same PCIe device being passed through to either i440 domain or Q35 one.
--- hw/xen/xen_pt.c | 28 ++++++++++++++++++++++++++-- hw/xen/xen_pt.h | 1 + 2 files changed, 27 insertions(+), 2 deletions(-)
hw/xen/xen_pt.c +26 -2
@@ -687,6 +687,21 @@ static const MemoryListener xen_pt_io_listener = { .priority = 10, }; +static inline bool xen_pt_dev_is_pcie_mode (PCIDevice *d) +{ +XenPCIPassthroughState *s =XEN_PT_DEVICE (d); +PCIBus *bus =pci_get_bus (d); + +if (bus !=NULL ) { +if (pci_is_express (d) &&pci_bus_is_express (bus) && +xen_host_pci_find_next_cap (&s->real_device,0 ,PCI_CAP_ID_EXP )) { +return true; + } + } + +return false; +} +static void xen_igd_passthrough_isa_bridge_create (XenPCIPassthroughState *s,XenHostPCIDevice *dev)
@@ -794,8 +809,17 @@ static void xen_pt_realize(PCIDevice *d, Error **errp) s->real_device.dev, s->real_device.func); } -/* Initialize virtualized PCI configuration (Extended 256 Bytes) */ -memset (d->config,0 ,PCI_CONFIG_SPACE_SIZE ); + s->pcie_enabled_dev =xen_pt_dev_is_pcie_mode (d); +if (s->pcie_enabled_dev) { +XEN_PT_LOG (d,"Host device %04x:%02x:%02x.%d passed thru " +"in PCIe mode\n" , s->real_device.domain, + s->real_device.bus, s->real_device.dev, + s->real_device.func); + } + +/* Initialize virtualized PCI configuration space (256/4K bytes) */ +memset (d->config,0 ,pci_is_express (d) ?PCIE_CONFIG_SPACE_SIZE + :PCI_CONFIG_SPACE_SIZE ); s->memory_listener = xen_pt_memory_listener; s->io_listener = xen_pt_io_listener;
hw/xen/xen_pt.h +1 -0
@@ -212,6 +212,7 @@ struct XenPCIPassthroughState {PCIHostDeviceAddress hostaddr;bool is_virtfn; +bool pcie_enabled_dev;bool permissive;bool permissive_warned;XenHostPCIDevice real_device;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 21/30 xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology check
In reply to Alexey GerasimenkoCompared to legacy i440 system, there are certain difficulties while passing through PCIe devices to guest OSes like Windows 7 and above on platforms with native support of PCIe bus (in our case Q35). This problem is not applicable to older OSes like Windows XP -- PCIe passthrough on such OSes can be used normally as these OSes have no support for PCIe-specific features and treat all PCIe devices as legacy PCI ones.
The problem manifests itself as "Code 10" error for a passed thru PCIe device in Windows Device Manager (along with exclamation mark on it). The device with such error do not function no matter the fact that Windows successfully booted while actually using this device, ex. as a primary VGA card with VBE features, LFB, etc. working properly during boot time. It doesn't matter which PCI class the device have -- the problem is common to GPUs, NIC cards, USB controllers, etc. In the same time, all these devices can be passed thru successfully using i440 emulation on same Windows 7+ OSes.
The actual root cause of the problem lies in the fact that Windows kernel (PnP manager particularly) while processing StartDevice IRP refuses to continue to start the device and control flow actually doesn't even reach the IRP handler in the device driver at all. The real reason for this typically does not appear at the time PnP manager tries to start the device, but happens much earlier -- during the Windows boot stage, while enumerating devices on a PCI/PCIe bus in the Windows pci.sys driver. There is a set of checks for every discovered device on the PCIe bus. Failing some of them leads to marking the discovered PCIe device as 'invalid' by setting the flag. Later on, StartDevice attempt will fail due to this flag, finally resulting in Code 10 error.
The actual check in pci.sys which results in the PCIe device being marked as 'invalid' in our case is a validation of upstream PCIe bus hierarchy to which passed through device belongs. Basically, pci.sys checks if the PCIe device has parent devices, such as PCIe Root Port or upstream PCIe switch. In our case the PCIe device has no parents and resides on bus 0 without eg. corresponding Root Port.
Therefore, in order to resolve this problem in a architecturally correct way, we need to introduce to Xen some support of at least trivial non-flat PCI bus hierarchy. In very simplest case - just one virtual Root Port, on secondary bus of which all physical functions of the real passed thru device will reside, eg. GPU and its HDAudio function.
This solution is not hard to implement technically, but there are multiple affecting limitations present in Xen (many related to each other) currently:
- in many places the code is limited to use bus 0 only. This applicable to both hypervisor and supplemental modules like hvmloader. This limitation is enforced on API level -- many functions and interfaces allow to specify only devfn argument while bus 0 being implied.
- lot of code assumes Type0 PCI config space layout only, while we need to handle Type1 PCI devices as well
- currently there no way to assign to a guest domain even a simplest linked hierarchy of passed thru PCI devices. In some cases we might need to passthrough a real PCIe Switch/Root Port with his downstream child devices.
- in a similar way Xen/hvmloader lacks the concept of IO/MMIO space nesting. Both code which does MMIO hole sizing and code which allocates BARs to MMIO hole have no idea of MMIO ranges nesting and their relations. In case of virtual Root Port we have basically an emulated PCI-PCI bridge with some parts of its MMIO range used for real MMIO ranges of passed through device(s).
So, adding to Xen multiple PCI buses support will require a bit of effort and discussions regarding the actual design of the feature. Nevertheless, this task is crucial for PCI/GPU passthrough features of Xen to work properly.
To summarize, we need to implement following things in the future:
1) Get rid of PCI bus 0 limitation everywhere. This could've been a simplest of subtasks but in reality this will require to change interfaces as well - AFAIR even adding a PCI device via QMP only allows to specify a device slot while we need to have some way to place the device on an arbitrary bus.
2) Fully or partially emulated PCI-PCI bridge which will provide a secondary bus for PCIe device placement - there might be a possibility to reuse some existing emulation QEMU provides. This also includes Type1 devices support.
The task will become more complicated if there arise necessity, for example, to control the PCIe link for a passed through PCIe device. As PT device reset is mandatory in most cases, there might be a chance to encounter a situation when we need to retrain the PCIe link to restore PCIe link speed after the reset. In this case there will be a need to selectively translate accesses to certain registers of emulated PCIe Switch/Root Port to the corresponding physical upstream PCIe Switch/RootPort. This will require some interaction with Dom0, hopefully extending xen-pciback will be enough.
3) The concept of I/O and MMIO ranges nesting, for tasks like sizing MMIO hole or PCI BAR allocation. This one should be pretty simple.
The actual implementation still is a matter to discuss of course.
In the meantime there can be used a very simple workaround which allows to bypass pci.sys limitation for PCIe topology check - there exist one good exception to "must have upstream PCIe parent" rule of pci.sys. It's chipset-integrated devices. How pci.sys can tell if it deals with a chipset built-in device? It checks one of PCI Express Capability fields in the device PCI conf space. For chipset built-in devices this field will state "root complex integrated device" while in our case for a normal passed thru PCIe device there will be a "PCIe endpoint" type. So that's what the workaround does - it intercepts reading of this particular field for passed through devices and returns the "root complex integrated device" value for PCIe endpoints. This makes pci.sys happy and allows Windows 7 and above to use PT device on PCIe-capable system normally. So far no negative side effects were encountered while using this approach, so it's a good temporary solution until multiple PCI bus support will be added to Xen.
--- hw/xen/xen_pt_config_init.c | 60 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+)
hw/xen/xen_pt_config_init.c +60 -0
@@ -902,6 +902,55 @@ static int xen_pt_linkctrl2_reg_init(XenPCIPassthroughState *s, *data = reg_field;return 0 ; } +/* initialize PCI Express Capabilities register */ +static int xen_pt_pcie_capabilities_reg_init (XenPCIPassthroughState *s, +XenPTRegInfo *reg, +uint32_t real_offset, +uint32_t *data) +{ +uint8_t dev_type =get_pcie_device_type (s); +uint16_t reg_field; + +if (xen_host_pci_get_word (&s->real_device, + real_offset - reg->offset +PCI_EXP_FLAGS , + ®_field)) { +XEN_PT_ERR (&s->dev,"Error reading PCIe Capabilities reg\n" ); + *data =0 ; +return 0 ; + } + +/* +* Q35 workaround for Win7+ pci.sys PCIe topology check. +* As our PT device currently located on a bus 0, fake the +* device/port type field to the "Root Complex integrated device" +* value to bypass the check +*/ +switch (dev_type) { +case PCI_EXP_TYPE_ENDPOINT : +case PCI_EXP_TYPE_LEG_END : +XEN_PT_LOG (&s->dev,"Original PCIe Capabilities reg is 0x%04X\n" , + reg_field); + reg_field &= ~PCI_EXP_FLAGS_TYPE ; + reg_field |= ((PCI_EXP_TYPE_RC_END /*9*/ <<4 ) &PCI_EXP_FLAGS_TYPE ); +XEN_PT_LOG (&s->dev,"Q35 PCIe topology check workaround: " +"faking Capabilities reg to 0x%04X\n" , reg_field); +break ; + +case PCI_EXP_TYPE_ROOT_PORT : +case PCI_EXP_TYPE_UPSTREAM : +case PCI_EXP_TYPE_DOWNSTREAM : +case PCI_EXP_TYPE_PCI_BRIDGE : +case PCI_EXP_TYPE_PCIE_BRIDGE : +case PCI_EXP_TYPE_RC_END : +case PCI_EXP_TYPE_RC_EC : +default : +/* do nothing, return as is */ +break ; + } + + *data = reg_field; +return 0 ; +}/* PCI Express Capability Structure reg static information table */ static XenPTRegInfo xen_pt_emu_reg_pcie[] = {
@@ -916,6 +965,17 @@ static XenPTRegInfo xen_pt_emu_reg_pcie[] = { .u.b.read = xen_pt_byte_reg_read, .u.b.write = xen_pt_byte_reg_write, }, +/* PCI Express Capabilities Register */ + { + .offset = PCI_EXP_FLAGS, + .size = 2, + .init_val = 0x0000, + .ro_mask = 0xFFFF, + .emu_mask = 0xFFFF, + .init = xen_pt_pcie_capabilities_reg_init, + .u.w.read = xen_pt_word_reg_read, + .u.w.write = xen_pt_word_reg_write, + }, /* Device Capabilities reg */ { .offset = PCI_EXP_DEVCAP,
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 22/30 xen/pt: add support for PCIe Extended Capabilities and larger config space
In reply to Alexey GerasimenkoThis patch provides basic facilities for PCIe Extended Capabilities and support for controlled (via s->pcie_enabled_dev flag) access to PCIe config space (>256).
PCIe Extended Capabilities make use of 16-bit capability ID. Also, a capability size might exceed 8-bit width. So as the very first step we need to increase type size for grp_id, grp_size, etc -- they were limited to 8-bit.
The only troublesome issue with PCIe Extended Capability IDs is that their value range is actually same as for basic PCI capabilities. Eg. capability ID 3 means VPD Capability for PCI and at the same time Device Serial Number Capability for PCIe Extended caps. This adds a bit of inconvenience.
In order to distinguish between two sets of same capability IDs, the patch introduces a set of macros to mark a capability ID as PCIe Extended one (or check if it is basic/extended + get a raw ID value):
- PCIE_EXT_CAP_ID(cap_id)
- IS_PCIE_EXT_CAP_ID(grp_id)
- GET_PCIE_EXT_CAP_ID(grp_id)
Here is how it's used:
/* Intel IGD Opregion group */
{
.grp_id = XEN_PCI_INTEL_OPREGION, /* no change */
.grp_type = XEN_PT_GRP_TYPE_EMU,
.grp_size = 0x4,
.size_init = xen_pt_reg_grp_size_init,
.emu_regs = xen_pt_emu_reg_igd_opregion,
},
/* Vendor-specific Extended Capability reg group */
{
.grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_VNDR),
.grp_type = XEN_PT_GRP_TYPE_EMU,
.grp_size = 0xFF,
.size_init = xen_pt_ext_cap_vendor_size_init,
.emu_regs = xen_pt_ext_cap_emu_reg_vendor,
},
By using the PCIE_EXT_CAP_ID() macro it is possible to reuse existing header files with already defined PCIe Extended Capability ID values.
find_cap_offset() receive capabily ID and checks if it's an Extended one by using IS_PCIE_EXT_CAP_ID(cap) macro, passing the real capabiliy ID value to either xen_host_pci_find_next_ext_cap
or xen_host_pci_find_next_cap.
--- hw/xen/xen_pt.c | 14 +++++- hw/xen/xen_pt.h | 13 +++-- hw/xen/xen_pt_config_init.c | 113 +++++++++++++++++++++----------------------- 3 files changed, 74 insertions(+), 66 deletions(-)
hw/xen/xen_pt.c +12 -2
@@ -82,10 +82,20 @@ void xen_pt_log(const PCIDevice *d, const char *f, ...)/* Config Space */ -static int xen_pt_pci_config_access_check (PCIDevice *d,uint32_t addr,int len) +static int xen_pt_pci_config_access_check (PCIDevice *d, +uint32_t addr,int len) { +XenPCIPassthroughState *s =XEN_PT_DEVICE (d); +/* check offset range */ -if (addr >0xFF ) { +if (s->pcie_enabled_dev) { +if (addr >=PCIE_CONFIG_SPACE_SIZE ) { +XEN_PT_ERR (d,"Failed to access register with offset " +"exceeding 0xFFF. (addr: 0x%02x, len: %d)\n" , + addr, len); +return -1 ; + } + }else if (addr >=PCI_CONFIG_SPACE_SIZE ) {XEN_PT_ERR (d,"Failed to access register with offset exceeding 0xFF. " "(addr: 0x%02x, len: %d)\n" , addr, len);return -1 ;
hw/xen/xen_pt.h +9 -4
@@ -31,6 +31,11 @@ void xen_pt_log(const PCIDevice *d, const char *f, ...) GCC_FMT_ATTR(2, 3);/* Helper */ #define XEN_PFN (x) ((x) >> XC_PAGE_SHIFT) +/* Macro's for PCIe Extended Capabilities */ +#define PCIE_EXT_CAP_ID (cap_id) ((cap_id) | (1U << 16)) +#define IS_PCIE_EXT_CAP_ID (grp_id) ((grp_id) & (1U << 16)) +#define GET_PCIE_EXT_CAP_ID (grp_id) ((grp_id) & 0xFFFF) +typedef const struct XenPTRegInfo XenPTRegInfo ;typedef struct XenPTReg XenPTReg ;
@@ -152,13 +157,13 @@ typedef const struct XenPTRegGroupInfo XenPTRegGroupInfo;/* emul reg group size initialize method */ typedef int (*xen_pt_reg_size_init_fn ) (XenPCIPassthroughState *,XenPTRegGroupInfo *, -uint32_t base_offset,uint8_t *size); +uint32_t base_offset,uint32_t *size);/* emulated register group information */ struct XenPTRegGroupInfo { -uint8_t grp_id; +uint32_t grp_id;XenPTRegisterGroupType grp_type; -uint8_t grp_size; +uint32_t grp_size;xen_pt_reg_size_init_fn size_init;XenPTRegInfo *emu_regs; };
@@ -168,7 +173,7 @@ typedef struct XenPTRegGroup {QLIST_ENTRY (XenPTRegGroup) entries;XenPTRegGroupInfo *reg_grp;uint32_t base_offset; -uint8_t size; +uint32_t size; QLIST_HEAD(, XenPTReg) reg_tbl_list; } XenPTRegGroup;
hw/xen/xen_pt_config_init.c +53 -60
161 lines not on this page · raw message
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 23/30 xen/pt: handle PCIe Extended Capabilities Next register
In reply to Alexey GerasimenkoThe patch adds new xen_pt_ext_cap_ptr_reg_init function which is used to initialize the emulated next pcie extended capability pointer.
Primary mission of this function is to have a method to selectively hide some extended capabilities from the capability linked list, skipping them by altering the Next capability pointer value.
--- hw/xen/xen_pt_config_init.c | 73 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 2 deletions(-)
hw/xen/xen_pt_config_init.c +71 -2
@@ -23,11 +23,14 @@#define XEN_PT_INVALID_REG 0xFFFFFFFF/* invalid register value */ -/* prototype */ +/* prototypes */ static int xen_pt_ptr_reg_init (XenPCIPassthroughState *s,XenPTRegInfo *reg,uint32_t real_offset,uint32_t *data); - +static int xen_pt_ext_cap_ptr_reg_init (XenPCIPassthroughState *s, +XenPTRegInfo *reg, +uint32_t real_offset, +uint32_t *data);/* helper */
@@ -1932,6 +1935,72 @@ out:return 0 ; } +#define PCIE_EXT_CAP_NEXT_SHIFT 4 +#define PCIE_EXT_CAP_VER_MASK 0xF + +static int xen_pt_ext_cap_ptr_reg_init (XenPCIPassthroughState *s, +XenPTRegInfo *reg, +uint32_t real_offset, +uint32_t *data) +{ +int i, rc; +XenHostPCIDevice *d = &s->real_device; +uint16_t reg_field; +uint16_t cur_offset, version, cap_id; +uint32_t header; + +if (real_offset <PCI_CONFIG_SPACE_SIZE ) { +XEN_PT_ERR (&s->dev,"Incorrect PCIe extended capability offset" +"encountered: 0x%04x\n" , real_offset); +return -EINVAL ; + } + + rc =xen_host_pci_get_word (d, real_offset, ®_field); +if (rc) +return rc; + +/* preserve version field */ + version = reg_field &PCIE_EXT_CAP_VER_MASK ; + cur_offset = reg_field >>PCIE_EXT_CAP_NEXT_SHIFT ; + +while (cur_offset && cur_offset !=0xFFF ) { + rc =xen_host_pci_get_long (d, cur_offset, &header); +if (rc) { +XEN_PT_ERR (&s->dev,"Failed to read PCIe extended capability " +"@0x%x (rc:%d)\n" , cur_offset, rc); +return rc; + } + + cap_id =PCI_EXT_CAP_ID (header); + +for (i =0 ; xen_pt_emu_reg_grps[i].grp_size !=0 ; i++) { +uint32_t cur_grp_id = xen_pt_emu_reg_grps[i].grp_id; + +if (!IS_PCIE_EXT_CAP_ID (cur_grp_id)) +continue ; + +if (xen_pt_hide_dev_cap (d, cur_grp_id)) +continue ; + +if (GET_PCIE_EXT_CAP_ID (cur_grp_id) == cap_id) { +if (xen_pt_emu_reg_grps[i].grp_type ==XEN_PT_GRP_TYPE_EMU ) + goto out; + +/* skip TYPE_HARDWIRED capability, move the ptr to next one */ +break ; + } + } + +/* next capability */ + cur_offset =PCI_EXT_CAP_NEXT (header); + } + +out: + *data = (cur_offset <<PCIE_EXT_CAP_NEXT_SHIFT ) | version; +return 0 ; +} + +/************* * Main
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 24/30 xen/pt: allow to hide PCIe Extended Capabilities
In reply to Alexey GerasimenkoWe need to hide some unwanted PCI/PCIe capabilities for passed through devices.
Normally we do this by marking the capability register group as XEN_PT_GRP_TYPE_HARDWIRED which exclude this capability from the capability list and returns zeroes on attempts to read capability body. Skipping the capability in the linked list of capabilities can be done by changing Next Capability register to skip one or many unwanted capabilities.
One difference between PCI and PCIe Extended capabilities is that we don't have the list head field anymore. PCIe Extended capabilities always start at offset 0x100 if they're present. Unfortunately, there are typically only few PCIe extended capabilities present which means there is a chance that some capability we want to hide will reside at offset 0x100 in PCIe config space.
The simplest way to hide such capabilities from guest OS or drivers is faking their capability ID value.
This patch adds the Capability ID register handler which checks
- if the capability to which this register belong starts at offset 0x100 in PCIe config space
- if this capability is marked as XEN_PT_GRP_TYPE_HARDWIRED
If it is the case, then a fake Capability ID value is returned.
--- hw/xen/xen_pt.c | 11 +++++++- hw/xen/xen_pt.h | 5 ++++ hw/xen/xen_pt_config_init.c | 62 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 76 insertions(+), 2 deletions(-)
hw/xen/xen_pt.c +10 -1
@@ -154,7 +154,16 @@ static uint32_t xen_pt_pci_read_config(PCIDevice *d, uint32_t addr, int len) reg_grp_entry =xen_pt_find_reg_grp (s, addr);if (reg_grp_entry) {/* check 0-Hardwired register group */ -if (reg_grp_entry->reg_grp->grp_type ==XEN_PT_GRP_TYPE_HARDWIRED ) { +if (reg_grp_entry->reg_grp->grp_type ==XEN_PT_GRP_TYPE_HARDWIRED && +/* +* For PCIe Extended Capabilities we need to emulate +* CapabilityID and NextCapability/Version registers for a +* hardwired reg group located at the offset 0x100 in PCIe +* config space. This allows us to hide the first extended +* capability as well. +*/ + !(reg_grp_entry->base_offset ==PCI_CONFIG_SPACE_SIZE && +ranges_overlap (addr, len,0x100 ,4 ))) {/* no need to emulate, just return 0 */ val =0 ; goto exit;
hw/xen/xen_pt.h +5 -0
@@ -78,6 +78,11 @@ typedef int (*xen_pt_conf_byte_read)#define XEN_PCI_INTEL_OPREGION 0xfc +#define XEN_PCIE_CAP_ID 0 +#define XEN_PCIE_CAP_LIST_NEXT 2 + +#define XEN_PCIE_FAKE_CAP_ID_BASE 0xFE00 + typedef enum { XEN_PT_GRP_TYPE_HARDWIRED = 0, /* 0 Hardwired reg group */ XEN_PT_GRP_TYPE_EMU, /* emul reg group */
hw/xen/xen_pt_config_init.c +61 -1
80 lines not on this page · raw message
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 25/30 xen/pt: add Vendor-specific PCIe Extended Capability descriptor and sizing
In reply to Alexey GerasimenkoThe patch provides Vendor-specific PCIe Extended Capability description structure and corresponding sizing function. In this particular case the size of the Vendor capability is available in the VSEC Length field.
--- hw/xen/xen_pt_config_init.c | 77 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 75 insertions(+), 2 deletions(-)
hw/xen/xen_pt_config_init.c +75 -2
@@ -129,6 +129,18 @@ static uint32_t get_throughable_mask(const XenPCIPassthroughState *s,return throughable_mask & valid_mask; } +static void log_pcie_extended_cap (XenPCIPassthroughState *s, +const char *cap_name, +uint32_t base_offset,uint32_t size) +{ +if (size) { +XEN_PT_LOG (&s->dev,"Found PCIe Extended Capability: %s at 0x%04x, " +"size 0x%x bytes\n" , cap_name, + (uint16_t ) base_offset, size); + } +} + +/**************** * general register functions */
@@ -1684,6 +1696,44 @@ static int xen_pt_ext_cap_capid_reg_init(XenPCIPassthroughState *s, } +/* Vendor-specific Ext Capability Structure reg static information table */ +static XenPTRegInfo xen_pt_ext_cap_emu_reg_vendor[] = { + { + .offset =XEN_PCIE_CAP_ID , + .size =2 , + .init_val =0x0000 , + .ro_mask =0xFFFF , + .emu_mask =0xFFFF , + .init = xen_pt_ext_cap_capid_reg_init, + .u.w.read = xen_pt_word_reg_read, + .u.w.write = xen_pt_word_reg_write, + }, + { + .offset =XEN_PCIE_CAP_LIST_NEXT , + .size =2 , + .init_val =0x0000 , + .ro_mask =0xFFFF , + .emu_mask =0xFFFF , + .init = xen_pt_ext_cap_ptr_reg_init, + .u.w.read = xen_pt_word_reg_read, + .u.w.write = xen_pt_word_reg_write, + }, + { + .offset =PCI_VNDR_HEADER , + .size =4 , + .init_val =0x00000000 , + .ro_mask =0xFFFFFFFF , + .emu_mask =0x00000000 , + .init = xen_pt_common_reg_init, + .u.dw.read = xen_pt_long_reg_read, + .u.dw.write = xen_pt_long_reg_write, + }, + { + .size =0 , + }, +}; + +/**************************** * Capabilities */
@@ -1708,6 +1758,23 @@ static int xen_pt_vendor_size_init(XenPCIPassthroughState *s, *size = sz;return ret; } + +static int xen_pt_ext_cap_vendor_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t vsec_hdr =0 ; +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_VNDR_HEADER , + &vsec_hdr); + + *size =PCI_VNDR_HEADER_LEN (vsec_hdr); + +log_pcie_extended_cap (s,"Vendor-specific" , base_offset, *size); + +return ret; +}/* get PCI Express Capability Structure register group size */ static int xen_pt_pcie_size_init(XenPCIPassthroughState *s, const XenPTRegGroupInfo *grp_reg,
@@ -1934,6 +2001,14 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .size_init = xen_pt_reg_grp_size_init, .emu_regs = xen_pt_emu_reg_igd_opregion, }, +/* Vendor-specific Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_VNDR), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_vendor_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_vendor, + }, { .grp_size = 0, },
@@ -2054,8 +2129,6 @@ out:return 0 ; } - -/************* * Main */
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 26/30 xen/pt: add fixed-size PCIe Extended Capabilities descriptors
In reply to Alexey GerasimenkoThis adds description structures for all fixed-size PCIe Extended Capabilities.
For every capability register group, only 2 registers are emulated
currently: Capability ID (16 bit) and Next Capability Offset/Version (16 bit). Both needed to implement selective capability hiding. All other registers are passed through at the moment (unless they belong to a "hardwired" capability which is hidden)
--- hw/xen/xen_pt_config_init.c | 183 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+)
hw/xen/xen_pt_config_init.c +183 -0
@@ -1734,6 +1734,37 @@ static XenPTRegInfo xen_pt_ext_cap_emu_reg_vendor[] = { }; +/* Common reg static information table for all passthru-type +* PCIe Extended Capabilities. Only Extended Cap ID and +* Next pointer are handled (to support capability hiding). +*/ +static XenPTRegInfo xen_pt_ext_cap_emu_reg_dummy[] = { + { + .offset =XEN_PCIE_CAP_ID , + .size =2 , + .init_val =0x0000 , + .ro_mask =0xFFFF , + .emu_mask =0xFFFF , + .init = xen_pt_ext_cap_capid_reg_init, + .u.w.read = xen_pt_word_reg_read, + .u.w.write = xen_pt_word_reg_write, + }, + { + .offset =XEN_PCIE_CAP_LIST_NEXT , + .size =2 , + .init_val =0x0000 , + .ro_mask =0xFFFF , + .emu_mask =0xFFFF , + .init = xen_pt_ext_cap_ptr_reg_init, + .u.w.read = xen_pt_word_reg_read, + .u.w.write = xen_pt_word_reg_write, + }, + { + .size =0 , + }, +}; + +/**************************** * Capabilities */
@@ -2009,6 +2040,158 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .size_init = xen_pt_ext_cap_vendor_size_init, .emu_regs = xen_pt_ext_cap_emu_reg_vendor, }, + /* Device Serial Number Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_DSN), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_DSN_SIZEOF, /*0x0C*/ + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Power Budgeting Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_PWR), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_PWR_SIZEOF, /*0x10*/ + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Root Complex Internal Link Control Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_RCILC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x0C, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Root Complex Event Collector Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_RCEC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x08, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Root Complex Register Block Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_RCRB), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x14, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Configuration Access Correlation Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_CAC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x08, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Alternate Routing ID Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_ARI), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_ARI_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Address Translation Services Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_ATS), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_ATS_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Single Root I/O Virtualization Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_SRIOV), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_SRIOV_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Page Request Interface Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_PRI), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_PRI_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Latency Tolerance Reporting Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_LTR), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_LTR_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Secondary PCIe Capability Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_SECPCI), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x10, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Process Address Space ID Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_PASID), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = PCI_EXT_CAP_PASID_SIZEOF, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* L1 PM Substates Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_L1SS), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x10, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Precision Time Measurement Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_PTM), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x0C, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* M-PCIe Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(0x20), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x1C, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* LN Requester (LNR) Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(0x1C), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x08, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Function Readiness Status (FRS) Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(0x21), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x10, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Readiness Time Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(0x22), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0x0C, + .size_init = xen_pt_reg_grp_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, { .grp_size = 0, },
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 27/30 xen/pt: add AER PCIe Extended Capability descriptor and sizing
In reply to Alexey GerasimenkoThe patch provides Advanced Error Reporting PCIe Extended Capability description structure and corresponding capability sizing function.
--- hw/xen/xen_pt_config_init.c | 72 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+)
hw/xen/xen_pt_config_init.c +72 -0
@@ -1924,6 +1924,70 @@ static int xen_pt_msix_size_init(XenPCIPassthroughState *s,return 0 ; } +/* get Advanced Error Reporting Extended Capability register group size */ +#define PCI_ERR_CAP_TLP_PREFIX_LOG (1U << 11) +#define PCI_DEVCAP2_END_END_TLP_PREFIX (1U << 21) +static int xen_pt_ext_cap_aer_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint8_t dev_type =get_pcie_device_type (s); +uint32_t aer_caps =0 ; +uint32_t sz =0 ; +int pcie_cap_pos; +uint32_t devcaps2; +int ret =0 ; + + pcie_cap_pos =xen_host_pci_find_next_cap (&s->real_device,0 , +PCI_CAP_ID_EXP ); +if (!pcie_cap_pos) { +XEN_PT_ERR (&s->dev, +"Cannot find a required PCI Express Capability\n" ); +return -1 ; + } + +if (get_pcie_capability_version (s) >1 ) { + ret =xen_host_pci_get_long (&s->real_device, + pcie_cap_pos +PCI_EXP_DEVCAP2 , + &devcaps2); +if (ret) { +XEN_PT_ERR (&s->dev,"Error while reading Device " +"Capabilities 2 Register \n" ); +return -1 ; + } + } + +if (devcaps2 &PCI_DEVCAP2_END_END_TLP_PREFIX ) { + ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_ERR_CAP , + &aer_caps); +if (ret) { +XEN_PT_ERR (&s->dev, +"Error while reading AER Extended Capability\n" ); +return -1 ; + } + +if (aer_caps &PCI_ERR_CAP_TLP_PREFIX_LOG ) { + sz =0x48 ; + } + } + +if (!sz) { +if (dev_type ==PCI_EXP_TYPE_ROOT_PORT || + dev_type ==PCI_EXP_TYPE_RC_EC ) { + sz =0x38 ; + }else { + sz =0x2C ; + } + } + + *size = sz; + +log_pcie_extended_cap (s,"AER" , base_offset, *size); +return ret; +} +static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = {/* Header Type0 reg group */
@@ -2192,6 +2256,14 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .size_init = xen_pt_reg_grp_size_init, .emu_regs = xen_pt_ext_cap_emu_reg_dummy, }, +/* Advanced Error Reporting Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_ERR), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_aer_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, { .grp_size = 0, },
Signature · 1 line
2.11.0
RFC PATCH 28/30 xen/pt: add descriptors and size calculation for RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities
In reply to Alexey GerasimenkoAdd few more PCIe Extended Capabilities entries to the xen_pt_emu_reg_grps[] array along with their corresponding *_size_init() functions.
All these capabilities have non-fixed size but their size calculation is very simple, hence adding them in a single batch.
For every capability register group, only 2 registers are emulated
currently: Capability ID (16 bit) and Next Capability Offset/Version (16 bit). Both needed to implement the selective capability hiding. All other registers are passed through at the moment (unless they belong to a capability marked as "hardwired" which is hidden)
--- hw/xen/xen_pt_config_init.c | 224 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 224 insertions(+)
hw/xen/xen_pt_config_init.c +224 -0
@@ -1988,6 +1988,174 @@ static int xen_pt_ext_cap_aer_size_init(XenPCIPassthroughState *s,return ret; } +/* get Root Complex Link Declaration Extended Capability register group size */ +#define RCLD_GET_NUM_ENTRIES (x) (((x) >> 8) & 0xFF) +static int xen_pt_ext_cap_rcld_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t elem_self_descr =0 ; + +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +4 , + &elem_self_descr); + + *size =0x10 +RCLD_GET_NUM_ENTRIES (elem_self_descr) *0x10 ; + +log_pcie_extended_cap (s,"Root Complex Link Declaration" , + base_offset, *size); +return ret; +} + +/* get Access Control Services Extended Capability register group size */ +#define ACS_VECTOR_SIZE_BITS (x) ((((x) >> 8) & 0xFF) ?: 256) +static int xen_pt_ext_cap_acs_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint16_t acs_caps =0 ; + +int ret =xen_host_pci_get_word (&s->real_device, + base_offset +PCI_ACS_CAP , + &acs_caps); + +if (acs_caps &PCI_ACS_EC ) { +uint32_t vector_sz =ACS_VECTOR_SIZE_BITS (acs_caps); + + *size =PCI_ACS_EGRESS_CTL_V + ((vector_sz +7 ) & ~7 ) /8 ; + }else { + *size =PCI_ACS_EGRESS_CTL_V ; + } + +log_pcie_extended_cap (s,"ACS" , base_offset, *size); +return ret; +} + +/* get Multicast Extended Capability register group size */ +static int xen_pt_ext_cap_multicast_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint8_t dev_type =get_pcie_device_type (s); + +switch (dev_type) { +case PCI_EXP_TYPE_ENDPOINT : +case PCI_EXP_TYPE_LEG_END : +case PCI_EXP_TYPE_RC_END : +case PCI_EXP_TYPE_RC_EC : +default : + *size =PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF ; +break ; + +case PCI_EXP_TYPE_ROOT_PORT : +case PCI_EXP_TYPE_UPSTREAM : +case PCI_EXP_TYPE_DOWNSTREAM : + *size =0x30 ; +break ; + } + +log_pcie_extended_cap (s,"Multicast" , base_offset, *size); +return 0 ; +} + +/* get Dynamic Power Allocation Extended Capability register group size */ +static int xen_pt_ext_cap_dpa_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t dpa_caps =0 ; +uint32_t num_entries; + +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_DPA_CAP , + &dpa_caps); + + num_entries = (dpa_caps &PCI_DPA_CAP_SUBSTATE_MASK ) +1 ; + + *size =PCI_DPA_BASE_SIZEOF + num_entries/*byte-size registers*/ ; + +log_pcie_extended_cap (s,"Dynamic Power Allocation" , base_offset, *size); +return ret; +} + +/* get TPH Requester Extended Capability register group size */ +static int xen_pt_ext_cap_tph_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t tph_caps =0 ; +uint32_t num_entries; + +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_TPH_CAP , + &tph_caps); + +switch (tph_caps &PCI_TPH_CAP_LOC_MASK ) { +case PCI_TPH_LOC_CAP : + num_entries = (tph_caps &PCI_TPH_CAP_ST_MASK ) >>PCI_TPH_CAP_ST_SHIFT ; + num_entries++; +break ; + +case PCI_TPH_LOC_NONE : +case PCI_TPH_LOC_MSIX : +default : +/* not in the capability */ + num_entries =0 ; + } + + *size =PCI_TPH_BASE_SIZEOF + num_entries *2 ; + +log_pcie_extended_cap (s,"TPH Requester" , base_offset, *size); +return ret; +} + +/* get Downstream Port Containment Extended Capability register group size */ +static int xen_pt_ext_cap_dpc_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint16_t dpc_caps =0 ; + +int ret =xen_host_pci_get_word (&s->real_device, + base_offset +PCI_EXP_DPC_CAP , + &dpc_caps); + +if (dpc_caps &PCI_EXP_DPC_CAP_RP_EXT ) { + *size =0x20 + ((dpc_caps &PCI_EXP_DPC_RP_PIO_LOG_SIZE ) >>8 ) *4 ; + }else { + *size =0xC ; + } + +log_pcie_extended_cap (s,"Downstream Port Containment" , + base_offset, *size); +return ret; +} + +/* get Protocol Multiplexing Extended Capability register group size */ +#define PMUX_GET_NUM_ENTRIES (x) ((x) & 0x3F) +static int xen_pt_ext_cap_pmux_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t pmux_caps =0 ; + +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +4 , + &pmux_caps); + + *size =0x10 +PMUX_GET_NUM_ENTRIES (pmux_caps) *4 ; + +log_pcie_extended_cap (s,"PMUX" , base_offset, *size); +return ret; +} +static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = {/* Header Type0 reg group */
@@ -2264,6 +2432,62 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .size_init = xen_pt_ext_cap_aer_size_init, .emu_regs = xen_pt_ext_cap_emu_reg_dummy, }, +/* Root Complex Link Declaration Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_RCLD), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_rcld_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Access Control Services Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_ACS), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_acs_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Multicast Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_MCAST), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_multicast_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Dynamic Power Allocation Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_DPA), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_dpa_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* TPH Requester Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_TPH), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_tph_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Protocol Multiplexing Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_PMUX), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_pmux_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Downstream Port Containment Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_DPC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_dpc_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, { .grp_size = 0, },
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 29/30 xen/pt: add Resizable BAR PCIe Extended Capability descriptor and sizing
In reply to Alexey GerasimenkoUnlike other PCIe Extended Capabilities, we currently cannot allow attempts to use Resizable BAR Capability. Without specifically handling BAR resizing we're likely end up with corrupted MMIO hole layout if guest OS will attempt to use this feature. Actually, recent Windows versions started to understand and use the Resizable BAR Capability (see [1]).
For now, we need to hide the Resizable BAR Capability from guest OS until BAR resizing emulation support will be implemented in Xen. This support is a pretty much mandatory todo-feature as the effect of writing to Resizable BAR control registers can be considered similar to reprogramming normal BAR registers -- i.e. this needs to be handled explicitly, resulting in corresponding MMIO BAR range(s) remapping. Until then, we mark the Resizable BAR Capability as XEN_PT_GRP_TYPE_HARDWIRED.
[1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/display/resizable-bar-support
--- hw/xen/xen_pt_config_init.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
hw/xen/xen_pt_config_init.c +27 -0
@@ -2156,6 +2156,26 @@ static int xen_pt_ext_cap_pmux_size_init(XenPCIPassthroughState *s,return ret; } +/* get Resizable BAR Extended Capability register group size */ +static int xen_pt_ext_cap_rebar_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t rebar_ctl =0 ; +uint32_t num_entries; + +int ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_REBAR_CTRL , + &rebar_ctl); + num_entries = + (rebar_ctl &PCI_REBAR_CTRL_NBAR_MASK ) >>PCI_REBAR_CTRL_NBAR_SHIFT ; + + *size = num_entries*8 +4 ; + +log_pcie_extended_cap (s,"Resizable BAR" , base_offset, *size); +return ret; +}static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = {/* Header Type0 reg group */
@@ -2488,6 +2508,13 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .size_init = xen_pt_ext_cap_dpc_size_init, .emu_regs = xen_pt_ext_cap_emu_reg_dummy, }, +/* Resizable BAR Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_REBAR), + .grp_type = XEN_PT_GRP_TYPE_HARDWIRED, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_rebar_size_init, + }, { .grp_size = 0, },
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
RFC PATCH 30/30 xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and sizing
In reply to Alexey GerasimenkoVirtual Channel/MFVC capabilities are relatively useless for emulation (passing through accesses to them should be enough in most cases) yet they have hardest format of all PCIe Extended Capabilities, mostly because VC capability format allows the sparse config space layout with gaps between the parts which make up the VC capability.
We have the main capability body followed by variable number of entries where each entry may additionally reference the arbitration table outside main capability body. There are no constrains on these arbitration table offsets -- in theory, they may reside outside the VC capability range anywhere in PCIe extended config space. Also, every arbitration table size is not fixed - it depends on current VC/Port Arbitration Select field value.
To simplify things, this patch assume that changing VC/Port Arbitration Select value (i.e. resizing arbitration tables) do not cause arbitration table offsets to change. Normally the device must place arbitration tables considering their maximum size, not current one. Maximum arbitration table size depends on VC/Port Arbitration Capability bitmask -- this is what actually used to calculate the arbitration table size.
--- hw/xen/xen_pt_config_init.c | 192 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 192 insertions(+)
hw/xen/xen_pt_config_init.c +192 -0
@@ -2177,6 +2177,174 @@ static int xen_pt_ext_cap_rebar_size_init(XenPCIPassthroughState *s,return ret; } +/* get VC/VC9/MFVC Extended Capability register group size */ +static uint32_t get_arb_table_len_max (XenPCIPassthroughState *s, +uint32_t max_bit_supported, +uint32_t arb_cap) +{ +int n_bit; +uint32_t table_max_size =0 ; + +if (!arb_cap) { +return 0 ; + } + +for (n_bit =7 ; n_bit >=0 && !(arb_cap & (1 << n_bit)); n_bit--); + +if (n_bit > max_bit_supported) { +XEN_PT_ERR (&s->dev,"Warning: encountered unknown VC arbitration " +"capability supported: 0x%02x\n" , (uint8_t ) arb_cap); + } + +switch (n_bit) { +case 0 :break ; +case 1 :return 32 ; +case 2 :return 64 ; +case 3 :/*128 too*/ +case 4 :return 128 ; +default : + table_max_size =8 << n_bit; + } + +return table_max_size; +} + +#define GET_ARB_TABLE_OFFSET (x) (((x) >> 24) * 0x10) +#define GET_VC_ARB_CAPABILITY (x) ((x) & 0xFF) +#define ARB_TABLE_ENTRY_SIZE_BITS (x) (1 << (((x) & PCI_VC_CAP1_ARB_SIZE)\ + >> 10)) +static int xen_pt_ext_cap_vchan_size_init (XenPCIPassthroughState *s, +const XenPTRegGroupInfo *grp_reg, +uint32_t base_offset, +uint32_t *size) +{ +uint32_t header; +uint32_t vc_cap_max_size =PCIE_CONFIG_SPACE_SIZE - base_offset; +uint32_t next_ptr; +uint32_t arb_table_start_max =0 , arb_table_end_max =0 ; +uint32_t port_vc_cap1, port_vc_cap2, vc_rsrc_cap; +uint32_t ext_vc_count =0 ; +uint32_t arb_table_entry_size;/* in bits */ +const char *cap_name; +int ret; +int i; + + ret =xen_host_pci_get_long (&s->real_device, base_offset, &header); +if (ret) { + goto err_read; + } + + next_ptr =PCI_EXT_CAP_NEXT (header); + +switch (PCI_EXT_CAP_ID (header)) { +case PCI_EXT_CAP_ID_VC : +case PCI_EXT_CAP_ID_VC9 : + cap_name ="Virtual Channel" ; +break ; +case PCI_EXT_CAP_ID_MFVC : + cap_name ="Multi-Function VC" ; +break ; +default : +XEN_PT_ERR (&s->dev,"Unknown VC Extended Capability ID " +"encountered: 0x%04x\n" ,PCI_EXT_CAP_ID (header)); +return -1 ; + } + +if (next_ptr && next_ptr > base_offset) { + vc_cap_max_size = next_ptr - base_offset; + } + + ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_VC_PORT_CAP1 , + &port_vc_cap1); +if (ret) { + goto err_read; + } + + ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_VC_PORT_CAP2 , + &port_vc_cap2); +if (ret) { + goto err_read; + } + + ext_vc_count = port_vc_cap1 &PCI_VC_CAP1_EVCC ; + + arb_table_start_max =GET_ARB_TABLE_OFFSET (port_vc_cap2); + +/* check arbitration table offset for validity */ +if (arb_table_start_max >= vc_cap_max_size) { +XEN_PT_ERR (&s->dev,"Warning: VC arbitration table offset points " +"outside the expected range: %#04x\n" , + (uint16_t ) arb_table_start_max); +/* skip this arbitration table */ + arb_table_start_max =0 ; + } + +if (arb_table_start_max) { +uint32_t vc_arb_cap =GET_VC_ARB_CAPABILITY (port_vc_cap2); +uint32_t num_phases =get_arb_table_len_max (s,3 , vc_arb_cap); +uint32_t arb_tbl_sz =QEMU_ALIGN_UP (num_phases *4 ,32 ) /8 ; + + arb_table_end_max = base_offset + arb_table_start_max + arb_tbl_sz; + } + +/* get Function/Port Arbitration Table Entry size */ + arb_table_entry_size =ARB_TABLE_ENTRY_SIZE_BITS (port_vc_cap1); + +/* process all VC Resource entries */ +for (i =0 ; i < ext_vc_count; i++) { +uint32_t arb_table_offset; + +/* read VC Resource Capability */ + ret =xen_host_pci_get_long (&s->real_device, + base_offset +PCI_VC_RES_CAP + i *PCI_CAP_VC_PER_VC_SIZEOF , + &vc_rsrc_cap); +if (ret) { + goto err_read; + } + + arb_table_offset =GET_ARB_TABLE_OFFSET (vc_rsrc_cap); + +if (arb_table_offset > arb_table_start_max) { +/* check arbitration table offset for validity */ +if (arb_table_offset >= vc_cap_max_size) { +XEN_PT_ERR (&s->dev,"Warning: Port/Function arbitration table " +"offset points outside the expected range: %#04x\n" , + (uint16_t ) arb_table_offset); +/* skip this arbitration table */ + arb_table_offset =0 ; + }else { + arb_table_start_max = arb_table_offset; + } + +if (arb_table_offset) { +uint32_t vc_arb_cap =GET_VC_ARB_CAPABILITY (vc_rsrc_cap); +uint32_t num_phases =get_arb_table_len_max (s,5 , vc_arb_cap); +uint32_t arb_tbl_sz = +QEMU_ALIGN_UP (num_phases * arb_table_entry_size,32 ) /8 ; + + arb_table_end_max = base_offset + arb_table_offset + arb_tbl_sz; + } + } + } + +if (arb_table_end_max) { + *size = arb_table_end_max - base_offset; + }else { + *size =PCI_CAP_VC_BASE_SIZEOF + + ext_vc_count *PCI_CAP_VC_PER_VC_SIZEOF ; + } + +log_pcie_extended_cap (s, cap_name, base_offset, *size); +return 0 ; + +err_read: +XEN_PT_ERR (&s->dev,"Error while reading VC Extended Capability\n" ); +return ret; +} + +static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = {/* Header Type0 reg group */ {
@@ -2515,6 +2683,30 @@ static const XenPTRegGroupInfo xen_pt_emu_reg_grps[] = { .grp_size = 0xFF, .size_init = xen_pt_ext_cap_rebar_size_init, }, +/* Virtual Channel Extended Capability reg group (2) */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_VC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_vchan_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Virtual Channel Extended Capability reg group (9) */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_VC9), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_vchan_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, + /* Multi-Function Virtual Channel Extended Capability reg group */ + { + .grp_id = PCIE_EXT_CAP_ID(PCI_EXT_CAP_ID_MFVC), + .grp_type = XEN_PT_GRP_TYPE_EMU, + .grp_size = 0xFF, + .size_init = xen_pt_ext_cap_vchan_size_init, + .emu_regs = xen_pt_ext_cap_emu_reg_dummy, + }, { .grp_size = 0, },
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Alexey GerasimenkoQuoted 27 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 551 insertions(+) create mode 100644 tools/libacpi/dsdt_q35.asltools/libacpi/dsdt_q35.asl new file +0
-0
tools/libacpi/dsdt_q35.asl line 9
Quoted 9 lines
tools/libacpi/dsdt_q35.asl new file +9
-0@@ -0,0 +1,551 @@ +/****************************************************************************** + * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU Lesser General Public License as published + * by the Free Software Foundation; version 2.1 only. with the special + * exception on linking described in file LICENSE.
I don't see the 'LICENSE' file in Xen's directory?
Also, your email does not seem to be coming from Intel, so I have to ask, where did this file originally come from?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Alexey GerasimenkoQuoted 16 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
qemu-options.hx DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ · line 44
Quoted 12 lines
qemu-options.hx +1
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory in a core dump (default=on)\n" " mem-merge=on|off controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n" + " xen-platform-dev=on|off controls Xen Platform device (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n" " dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n" " suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
Signature · 1 line
Eduardo
Re: [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Alexey GerasimenkoQuoted 16 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
qemu-options.hx DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ · line 44
Quoted 12 lines
qemu-options.hx +1
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory in a core dump (default=on)\n" " mem-merge=on|off controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n" + " xen-platform-dev=on|off controls Xen Platform device (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n" " dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n" " suppress-vmdesc=on|off disables self-describing migration (default=off)\n"
What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
Signature · 6 lines
Eduardo _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Konrad Rzeszutek WilkQuoted 44 lines · On Mon, 12 Mar 2018 15:38:03 -0400 Konrad Rzeszutek Wilk…
Quoted 34 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551+++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 551 insertions(+) create mode 100644
tools/libacpi/dsdt_q35.asltools/libacpi/dsdt_q35.asl new file +6
-0@@ -0,0 +1,551 @@ +/****************************************************************************** + * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/orQuoted 6 lines
modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.I don't see the 'LICENSE' file in Xen's directory?
Quoted 2 lines
Also, your email does not seem to be coming from Intel, so I have to ask, where did this file originally come from?
It's basically Xen's dsdt.asl with some modifications related to Q35. Currently only few modifications needed, but in the future dsdt.asl and dsdt_q35.asl will diverge more from each other -- that's the reason why a separate file was forked instead applying these changes to dsdt.asl directly, for example, as #ifdef-parts.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
Quoted 47 lines · Alexey G
Quoted 44 lines · On Mon, 12 Mar 2018 15:38:03 -0400 Konrad Rzeszutek Wilk…
Quoted 34 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551+++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 551 insertions(+) create mode 100644
tools/libacpi/dsdt_q35.asltools/libacpi/dsdt_q35.asl new file +6
-0@@ -0,0 +1,551 @@ +/****************************************************************************** + * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/orQuoted 6 lines
modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.I don't see the 'LICENSE' file in Xen's directory?
Quoted 8 lines
Quoted 2 lines
Also, your email does not seem to be coming from Intel, so I have to ask, where did this file originally come from?
It's basically Xen's dsdt.asl with some modifications related to Q35. Currently only few modifications needed, but in the future dsdt.asl and dsdt_q35.asl will diverge more from each other -- that's the reason why a separate file was forked instead applying these changes to dsdt.asl directly, for example, as #ifdef-parts.
OK, as such you should make a seperate patch that adds this file (and be completly unmodified) and make sure you CC Intel folks (Kevin, et all) so they can Ack it.
Thank you.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Eduardo HabkostQuoted 31 lines · On Mon, 12 Mar 2018 16:44:06 -0300 Eduardo Habkost <[email protected]> wrote:
Quoted 17 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
Quoted 12 lines
qemu-options.hx +0
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory ina core dump (default=on)\n" " mem-merge=on|off
controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n"
+ " xen-platform-dev=on|off controls Xen Platform
device (default=off)\n" " aes-key-wrap=on|off
Quoted 8 lines
Quoted 4 lines
controls support for AES key wrapping (default=on)\n"
" dea-key-wrap=on|off controls support for DEA key
wrapping (default=on)\n" " suppress-vmdesc=on|off
disables self-describing migration (default=off)\n"What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
I guess the initial assumption was that changing the xen_platform_device value in Xen's options may cause some additional changes in platform configuration besides adding (or not) the Xen Platform device, hence a completely different machine type was chosen (xenfv).
At the moment pc,accel=xen/xenfv selection mostly governs only the Xen Platform device presence. Also setting max_cpus to HVM_MAX_VCPUS depends on it, but this doesn't applicable to a 'pc,accel=xen' machine for some reason.
If applying HVM_MAX_VCPUS to max_cpus is really necessary I think it's better to set it unconditionally for all 'accel=xen' HVM machine types inside xen_enabled() block. Right now it's missing for pc,accel=xen and q35,accel=xen.
I'll check if supplying the Xen platform device via the '-device' option will be ok for all usage cases.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Konrad Rzeszutek WilkQuoted 58 lines · On Mon, 12 Mar 2018 16:32:27 -0400 Konrad Rzeszutek Wilk…
Quoted 48 lines · Alexey G
Quoted 45 lines · On Mon, 12 Mar 2018 15:38:03 -0400 Konrad Rzeszutek Wilk…
Quoted 34 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551+++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 551 insertions(+) create mode 100644
tools/libacpi/dsdt_q35.asl(quoted diff) +0
-0b/tools/libacpi/dsdt_q35.asl new file mode 100644
index 0000000000..cd02946a07tools/libacpi/dsdt_q35.asl new file +6
-0@@ -0,0 +1,551 @@ +/****************************************************************************** + * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/orQuoted 6 lines
modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.I don't see the 'LICENSE' file in Xen's directory?
Quoted 8 lines
Quoted 2 lines
Also, your email does not seem to be coming from Intel, so I have to ask, where did this file originally come from?
It's basically Xen's dsdt.asl with some modifications related to Q35. Currently only few modifications needed, but in the future dsdt.asl and dsdt_q35.asl will diverge more from each other -- that's the reason why a separate file was forked instead applying these changes to dsdt.asl directly, for example, as #ifdef-parts.
Quoted 3 lines
OK, as such you should make a seperate patch that adds this file (and be completly unmodified) and make sure you CC Intel folks (Kevin, et all) so they can Ack it.
Kevin -- I assume you mean Kevin Tian <[email protected]>? Cc'ing him.
Please let me know other persons from Intel who are also responsible, the MAINTAINERS file doesn't tell much about Intel people regarding /libacpi, unfortunately.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Alexey GQuoted 54 lines · Alexey G
Quoted 31 lines · On Mon, 12 Mar 2018 16:44:06 -0300 Eduardo Habkost <[email protected]> wrote:
Quoted 17 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
Quoted 12 lines
qemu-options.hx +0
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory ina core dump (default=on)\n" " mem-merge=on|off
controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n"
+ " xen-platform-dev=on|off controls Xen Platform
device (default=off)\n" " aes-key-wrap=on|offQuoted 8 lines
Quoted 4 lines
controls support for AES key wrapping (default=on)\n"
" dea-key-wrap=on|off controls support for DEA key
wrapping (default=on)\n" " suppress-vmdesc=on|off
disables self-describing migration (default=off)\n"What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
I guess the initial assumption was that changing the
xen_platform_devicevalue in Xen's options may cause some additional changes in platform configuration besides adding (or not) the Xen Platform device, hence a completely different machine type was chosen (xenfv).At the moment pc,accel=xen/xenfv selection mostly governs only the Xen Platform device presence. Also setting
max_cpustoHVM_MAX_VCPUSdepends on it, but this doesn't applicable to a 'pc,accel=xen' machine for some reason.
Quoted 4 lines
If applying
HVM_MAX_VCPUStomax_cpusis really necessary I think it's better to set it unconditionally for all 'accel=xen' HVM machine types insidexen_enabled()block. Right now it's missing for pc,accel=xen and q35,accel=xen.
If you are talking about MachineClass::max_cpus, note that it is returned by query-machines, so it's supposed to be a static value. Changing it a runtime would mean the query-machines value is incorrect.
Is HVM_MAX_CPUS higher or lower than 255? If it's higher, does it mean the current value on pc and q35 isn't accurate?
Quoted 3 lines
I'll check if supplying the Xen platform device via the '-device' option will be ok for all usage cases.
Is HVM_MAX_CPUS something that needs to be enabled because of accel=xen or because or the xen-platform device?
If it's just because of accel=xen, we could introduce a AccelClass::max_cpus() method (we also have KVM-imposed CPU count limits, currently implemented inside kvm_init()).
Signature · 1 line
Eduardo
Re: [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Alexey GQuoted 54 lines · Alexey G
Quoted 31 lines · On Mon, 12 Mar 2018 16:44:06 -0300 Eduardo Habkost <[email protected]> wrote:
Quoted 17 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
Quoted 12 lines
qemu-options.hx +0
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory ina core dump (default=on)\n" " mem-merge=on|off
controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n"
+ " xen-platform-dev=on|off controls Xen Platform
device (default=off)\n" " aes-key-wrap=on|offQuoted 8 lines
Quoted 4 lines
controls support for AES key wrapping (default=on)\n"
" dea-key-wrap=on|off controls support for DEA key
wrapping (default=on)\n" " suppress-vmdesc=on|off
disables self-describing migration (default=off)\n"What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
I guess the initial assumption was that changing the
xen_platform_devicevalue in Xen's options may cause some additional changes in platform configuration besides adding (or not) the Xen Platform device, hence a completely different machine type was chosen (xenfv).At the moment pc,accel=xen/xenfv selection mostly governs only the Xen Platform device presence. Also setting
max_cpustoHVM_MAX_VCPUSdepends on it, but this doesn't applicable to a 'pc,accel=xen' machine for some reason.
Quoted 4 lines
If applying
HVM_MAX_VCPUStomax_cpusis really necessary I think it's better to set it unconditionally for all 'accel=xen' HVM machine types insidexen_enabled()block. Right now it's missing for pc,accel=xen and q35,accel=xen.
If you are talking about MachineClass::max_cpus, note that it is returned by query-machines, so it's supposed to be a static value. Changing it a runtime would mean the query-machines value is incorrect.
Is HVM_MAX_CPUS higher or lower than 255? If it's higher, does it mean the current value on pc and q35 isn't accurate?
Quoted 3 lines
I'll check if supplying the Xen platform device via the '-device' option will be ok for all usage cases.
Is HVM_MAX_CPUS something that needs to be enabled because of accel=xen or because or the xen-platform device?
If it's just because of accel=xen, we could introduce a AccelClass::max_cpus() method (we also have KVM-imposed CPU count limits, currently implemented inside kvm_init()).
Signature · 6 lines
Eduardo _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Alexey GQuoted 71 lines
From: Alexey G [mailto:[email protected]]
Sent: Tuesday, March 13, 2018 5:20 AMQuoted 19 lines · On Mon, 12 Mar 2018 16:32:27 -0400 Konrad Rzeszutek Wilk…
Quoted 18 lines · Alexey G
Quoted 15 lines · On Mon, 12 Mar 2018 15:38:03 -0400 Konrad Rzeszutek Wilk…
Quoted 13 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551+++++++++++++++++++++++++++++++++++++++++++++ 1 file changed,
551
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
tools/libacpi/dsdt_q35.asl
Quoted 9 lines
insertions(+) create mode 100644
tools/libacpi/dsdt_q35.asl(quoted diff) +0
-0b/tools/libacpi/dsdt_q35.asl new file mode 100644
index 0000000000..cd02946a07tools/libacpi/dsdt_q35.asl new file +0
-0@@ -0,0 +1,551 @@+/************************************************************
******************Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
Quoted 11 lines
tools/libacpi/dsdt_q35.asl new file +5
-0+ * DSDT for Xen with Qemu device model (for Q35 machine) + * + * Copyright (c) 2004, Intel Corporation. + * + * This program is free software; you can redistribute it and/ormodify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.I don't see the 'LICENSE' file in Xen's directory?
Quoted 8 lines
Quoted 2 lines
Also, your email does not seem to be coming from Intel, so I have to ask, where did this file originally come from?
It's basically Xen's dsdt.asl with some modifications related to Q35. Currently only few modifications needed, but in the future dsdt.asl and dsdt_q35.asl will diverge more from each other -- that's the reason why a separate file was forked instead applying these changes to dsdt.asl directly, for example, as #ifdef-parts.
Quoted 3 lines
OK, as such you should make a seperate patch that adds this file (and be completly unmodified) and make sure you CC Intel folks (Kevin, et all) so they can Ack it.
Quoted 5 lines
Kevin -- I assume you mean Kevin Tian <[email protected]>? Cc'ing him.
Please let me know other persons from Intel who are also responsible, theMAINTAINERSfile doesn't tell much about Intel people regarding /libacpi, unfortunately.
I'm not the maintainer of libacpi (should be Jan?). But CC my colleague (Chao Peng) here who did some study of Q35 support before and can help review.
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
The subject line says to expect 30 patches, but you've only sent 18 to the list here. I eventually figured out that the first 12 patches were in Xen code and so not sent to qemu-devel.
For future if you have changes that affect multiple completely separate projects, send them as separate series. ie just send PATCH 00/18 to QEMU devel so it doesn't look like a bunch of patches have gone missing.
Quoted 7 lines · Alexey Gerasimenko
How to use the Q35 feature:
A new domain config option was implemented:
device_model_machine. It's a string which has following possible values:
- "i440" -- i440 emulation (default)
- "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI.
Presumably this is mapping to the QEMU -machine arg, so it feels desirable to keep the same naming scheme. ie allow any of the versioned machine names that QEMU uses. eg any of "pc-q35-2.x" versioned types, or 'q35' as an alias for latest, and use "pc-i440fx-2.x" versioned types of 'pc' as an alias for latest, rather than 'i440' which is needlessly divering from the QEMU machine type.
Regards,
Daniel
Signature · 8 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Alexey GerasimenkoQuoted 10 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
The change you made to q35 is pretty tiny, so I imagine the equiv change to pc machine is equally small. IOW, I think you should just convert them both straight away rather than providing an inconsistent configuration approach for q35 vs pc.
Quoted 110 lines
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>--- hw/core/machine.c | 21 +++++++++++++++++++++ hw/i386/pc_q35.c | 14 ++++++++++++++ include/hw/boards.h | 1 + qemu-options.hx | 1 + 4 files changed, 37 insertions(+)hw/core/machine.c +21
-0@@ -290,6 +290,20 @@ static void machine_set_igd_gfx_passthru(Object *obj, bool value, Error **errp) ms->igd_gfx_passthru = value; } +static bool machine_get_xen_platform_dev (Object *obj,Error **errp) +{ +MachineState *ms =MACHINE (obj); + +return ms->xen_platform_dev; +} + +static void machine_set_xen_platform_dev (Object *obj,bool value,Error **errp) +{ +MachineState *ms =MACHINE (obj); + + ms->xen_platform_dev = value; +} +static char *machine_get_firmware (Object *obj,Error **errp) {MachineState *ms =MACHINE (obj);@@ -595,6 +609,13 @@ static void machine_class_init(ObjectClass *oc, void *data)object_class_property_set_description (oc,"igd-passthru" ,"Set on/off to enable/disable igd passthrou" , &error_abort); +object_class_property_add_bool (oc,"xen-platform-dev" , + machine_get_xen_platform_dev, + machine_set_xen_platform_dev, &error_abort); +object_class_property_set_description (oc,"xen-platform-dev" , +"Set on/off to enable/disable Xen Platform device" , + &error_abort); +object_class_property_add_str (oc,"firmware" , machine_get_firmware, machine_set_firmware, &error_abort);hw/i386/pc_q35.c +14
-0@@ -56,6 +56,18 @@/* ICH9 AHCI has 6 ports */ #define MAX_SATA_PORTS 6 +static void q35_xen_hvm_init (MachineState *machine) +{ +PCMachineState *pcms =PC_MACHINE (machine); + +if (xen_enabled ()) { +/* check if Xen Platform device is enabled */ +if (machine->xen_platform_dev) { +pci_create_simple (pcms->bus,-1 ,"xen-platform" ); + } + } +} +/* PC hardware initialisation */ static void pc_q35_init (MachineState *machine) {@@ -207,6 +219,8 @@ static void pc_q35_init(MachineState *machine)if (xen_enabled ()) {pci_bus_irqs (host_bus, xen_cmn_set_irq, xen_cmn_pci_slot_get_pirq, ich9_lpc,ICH9_XEN_NUM_IRQ_SOURCES ); + +q35_xen_hvm_init (machine); }else {pci_bus_irqs (host_bus, ich9_lpc_set_irq, ich9_lpc_map_irq, ich9_lpc,ICH9_LPC_NB_PIRQS );include/hw/boards.h +1
-0@@ -238,6 +238,7 @@ struct MachineState {bool usb;bool usb_disabled;bool igd_gfx_passthru; +bool xen_platform_dev;char *firmware;bool iommu;bool suppress_vmdesc;qemu-options.hx +1
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memory in a core dump (default=on)\n" " mem-merge=on|off controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n" + " xen-platform-dev=on|off controls Xen Platform device (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrapping (default=on)\n" " dea-key-wrap=on|off controls support for DEA key wrapping (default=on)\n" " suppress-vmdesc=on|off disables self-describing migration (default=off)\n"Signature · 1 line
2.11.0
Regards,
Daniel
Signature · 8 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 8 lines · On Tue, 13 Mar 2018 09:21:54 +0000 Daniel P. Berrangé <[email protected]>…
The subject line says to expect 30 patches, but you've only sent 18 to the list here. I eventually figured out that the first 12 patches were in Xen code and so not sent to qemu-devel.
For future if you have changes that affect multiple completely separate projects, send them as separate series. ie just send PATCH 00/18 to QEMU devel so it doesn't look like a bunch of patches have gone missing.
OK, we'll do for next versions.
Quoted 6 lines
Quoted 5 lines
A new domain config option was implemented:
device_model_machine. It's a string which has following possible values:
- "i440" -- i440 emulation (default)
- "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI.
Quoted 6 lines
Presumably this is mapping to the QEMU -machine arg, so it feels desirable to keep the same naming scheme. ie allow any of the versioned machine names that QEMU uses. eg any of "pc-q35-2.x" versioned types, or 'q35' as an alias for latest, and use "pc-i440fx-2.x" versioned types of 'pc' as an alias for latest, rather than 'i440' which is needlessly divering from the QEMU machine type.
Yes, it is translated into the '-machine' argument.
A direct mapping between the Xen device_model_machine option and QEMU '-machine' argument won't be accepted by Xen maintainers I guess.
The main problem with this approach is a requirement to have a match between Xen/libxl and QEMU versions. If, for example, device_model_machine tells something like "pc-q35-2.11" and later we downgrade QEMU to some older version we'll likely have a problem without changing anything in the domain config. So I guess the "use the latest available" approach for machine selection (pc, q35, etc) is the only possible option. Perhaps having the way to specify the exact QEMU machine name and version in a separate domain config parameter (for advanced use) might be feasible.
Also, parameter names do not speak for themselves I'm afraid. This way we'll have, for example, device_model_machine="pc" vs device_model_machine="q35"... a bit unclear I think. This may be obvious for a QEMU user, but many Xen users didn't get used to QEMU machines and there might be some wondering why "q35" is not "pc" and why "pc" is an i440 system precisely.
Another obstacle here is xen_platform_device option which indirectly selects QEMU machine type for i440 at the moment (pc/xenfv), but this may be addressed by controlling the Xen platform device independently via a separate machine property or '-device xen-platform' like Eduardo Habkost suggested.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 32 lines · Alexey G
Quoted 8 lines · On Tue, 13 Mar 2018 09:21:54 +0000 Daniel P. Berrangé <[email protected]>…
The subject line says to expect 30 patches, but you've only sent 18 to the list here. I eventually figured out that the first 12 patches were in Xen code and so not sent to qemu-devel.
For future if you have changes that affect multiple completely separate projects, send them as separate series. ie just send PATCH 00/18 to QEMU devel so it doesn't look like a bunch of patches have gone missing.
OK, we'll do for next versions.
Quoted 6 lines
Quoted 5 lines
A new domain config option was implemented:
device_model_machine. It's a string which has following possible values:
- "i440" -- i440 emulation (default)
- "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI.Quoted 6 lines
Presumably this is mapping to the QEMU -machine arg, so it feels desirable to keep the same naming scheme. ie allow any of the versioned machine names that QEMU uses. eg any of "pc-q35-2.x" versioned types, or 'q35' as an alias for latest, and use "pc-i440fx-2.x" versioned types of 'pc' as an alias for latest, rather than 'i440' which is needlessly divering from the QEMU machine type.
Yes, it is translated into the '-machine' argument.
A direct mapping between the Xen
device_model_machineoption and QEMU '-machine' argument won't be accepted by Xen maintainers I guess.
Quoted 9 lines
The main problem with this approach is a requirement to have a match between Xen/libxl and QEMU versions. If, for example,
device_model_machinetells something like "pc-q35-2.11" and later we downgrade QEMU to some older version we'll likely have a problem without changing anything in the domain config. So I guess the "use the latest available" approach for machine selection (pc, q35, etc) is the only possible option. Perhaps having the way to specify the exact QEMU machine name and version in a separate domain config parameter (for advanced use) might be feasible.
At least with plain QEMU or KVM, using the versioned machine type names is important as that is what guarantees you a stable guest machine ABI, independant of QEMU version. If your deployment has a mixture of QEMU versions on different hosts, then you very much want to pick a versioned machine type to ensure compatibility for live migration. With libvirt we accept the short "pc" or "q35" names on input, but expand them to the fully versioned name when saving the config file, so no matter which QEMU version is used each time the guest is launched, the ABI is always the same.
Quoted 8 lines
Also, parameter names do not speak for themselves I'm afraid. This way we'll have, for example, device_model_machine="pc" vs device_model_machine="q35"... a bit unclear I think. This may be obvious for a QEMU user, but many Xen users didn't get used to QEMU machines and there might be some wondering why "q35" is not "pc" and why "pc" is an i440 system precisely.
Quoted 5 lines
Another obstacle here is
xen_platform_deviceoption which indirectly selects QEMU machine type for i440 at the moment (pc/xenfv), but this may be addressed by controlling the Xen platform device independently via a separate machine property or '-device xen-platform' like Eduardo Habkost suggested.
Regards,
Daniel
Signature · 8 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 32 lines · Alexey G
Quoted 8 lines · On Tue, 13 Mar 2018 09:21:54 +0000 Daniel P. Berrangé <[email protected]>…
The subject line says to expect 30 patches, but you've only sent 18 to the list here. I eventually figured out that the first 12 patches were in Xen code and so not sent to qemu-devel.
For future if you have changes that affect multiple completely separate projects, send them as separate series. ie just send PATCH 00/18 to QEMU devel so it doesn't look like a bunch of patches have gone missing.
OK, we'll do for next versions.
Quoted 6 lines
Quoted 5 lines
A new domain config option was implemented:
device_model_machine. It's a string which has following possible values:
- "i440" -- i440 emulation (default)
- "q35" -- emulate a Q35 machine. By default, the storage interface is AHCI.Quoted 6 lines
Presumably this is mapping to the QEMU -machine arg, so it feels desirable to keep the same naming scheme. ie allow any of the versioned machine names that QEMU uses. eg any of "pc-q35-2.x" versioned types, or 'q35' as an alias for latest, and use "pc-i440fx-2.x" versioned types of 'pc' as an alias for latest, rather than 'i440' which is needlessly divering from the QEMU machine type.
Yes, it is translated into the '-machine' argument.
A direct mapping between the Xen
device_model_machineoption and QEMU '-machine' argument won't be accepted by Xen maintainers I guess.
Quoted 9 lines
The main problem with this approach is a requirement to have a match between Xen/libxl and QEMU versions. If, for example,
device_model_machinetells something like "pc-q35-2.11" and later we downgrade QEMU to some older version we'll likely have a problem without changing anything in the domain config. So I guess the "use the latest available" approach for machine selection (pc, q35, etc) is the only possible option. Perhaps having the way to specify the exact QEMU machine name and version in a separate domain config parameter (for advanced use) might be feasible.
At least with plain QEMU or KVM, using the versioned machine type names is important as that is what guarantees you a stable guest machine ABI, independant of QEMU version. If your deployment has a mixture of QEMU versions on different hosts, then you very much want to pick a versioned machine type to ensure compatibility for live migration. With libvirt we accept the short "pc" or "q35" names on input, but expand them to the fully versioned name when saving the config file, so no matter which QEMU version is used each time the guest is launched, the ABI is always the same.
Quoted 8 lines
Also, parameter names do not speak for themselves I'm afraid. This way we'll have, for example, device_model_machine="pc" vs device_model_machine="q35"... a bit unclear I think. This may be obvious for a QEMU user, but many Xen users didn't get used to QEMU machines and there might be some wondering why "q35" is not "pc" and why "pc" is an i440 system precisely.
Quoted 5 lines
Another obstacle here is
xen_platform_deviceoption which indirectly selects QEMU machine type for i440 at the moment (pc/xenfv), but this may be addressed by controlling the Xen platform device independently via a separate machine property or '-device xen-platform' like Eduardo Habkost suggested.
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Alexey GerasimenkoCc Anthony
IIRC there are changes needed on QEMU side? Do we need to wait until that lands?
Wei.
Quoted 103 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.Note that omitting
device_model_machineparameter means i440 system by default, so the default behavior doesn't change for existing domain config files.Setting
device_model_machineto "q35" sends '-machine q35,accel=xen' argument to QEMU. Unlike i440, there no separate machine type to enable/disable Xen platform device, it is controlled via a machine property only. See 'libxl: Xen Platform device support for Q35' patch for a detailed description.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libxl/libxl_dm.c | 16 ++++++++++------ tools/libxl/libxl_types.idl | 7 +++++++ tools/xl/xl_parse.c | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-)tools/libxl/libxl_dm.c +10
-6@@ -1443,13 +1443,17 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,flexarray_append (dm_args, b_info->extra_pv[i]);break ;case LIBXL_DOMAIN_TYPE_HVM : -if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { -/* Switching here to the machine "pc" which does not add -* the xen-platform device instead of the default "xenfv" machine. -*/ - machinearg =libxl__strdup (gc,"pc,accel=xen" ); +if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); }else { - machinearg =libxl__strdup (gc,"xenfv" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { +/* Switching here to the machine "pc" which does not add +* the xen-platform device instead of the default "xenfv" machine. +*/ + machinearg =libxl__strdup (gc,"pc,accel=xen" ); + }else { + machinearg =libxl__strdup (gc,"xenfv" ); + } }if (b_info->u.hvm.mmio_hole_memkb) { uint64_t max_ram_below_4g = (1ULL << 32) -tools/libxl/libxl_types.idl +7
-0@@ -101,6 +101,12 @@ libxl_device_model_version = Enumeration("device_model_version", [ (2, "QEMU_XEN"), # Upstream based qemu-xen device model ]) +libxl_device_model_machine = Enumeration("device_model_machine", [ + (0, "UNKNOWN"), + (1, "I440"), + (2, "Q35"), + ]) + libxl_console_type = Enumeration("console_type", [ (0, "UNKNOWN"), (1, "SERIAL"),@@ -491,6 +497,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("device_model_ssid_label", string), # device_model_user is not ready for use yet ("device_model_user", string), + ("device_model_machine", libxl_device_model_machine), # extra parameters pass directly to qemu, NULL terminated ("extra", libxl_string_list),tools/xl/xl_parse.c +14
-0@@ -2110,6 +2110,20 @@ skip_usbdev:xlu_cfg_replace_string (config,"device_model_user" , &b_info->device_model_user,0 ); +if (!xlu_cfg_get_string (config,"device_model_machine" , &buf,0 )) { +if (!strcmp (buf,"i440" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_I440 ; + }else if (!strcmp (buf,"q35" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_Q35 ; + }else { +fprintf (stderr, +"Unknown device_model_machine \"%s\" specified\n" , buf); +exit (1 ); + } + }else { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_UNKNOWN ; + } +#define parse_extra_args (type) \ e = xlu_cfg_get_list_as_string_list(config, "device_model_args"#type, \ &b_info->extra##type, 0); \Signature · 1 line
2.11.0
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 03/12] hvmloader: add function to query an emulated machine type (i440/Q35)
In reply to Alexey GerasimenkoQuoted 20 lines · Alexey Gerasimenko
This adds a new function
get_pc_machine_type()which allows to determine the emulated chipset type. Supported return values:-
MACHINE_TYPE_I440
-MACHINE_TYPE_Q35
-MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by callingBUG()in hvmloader.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47 +++++++++++++++++++++++++++++++++++++ tools/firmware/hvmloader/util.h | 8 +++++++ 3 files changed, 60 insertions(+)tools/firmware/hvmloader/pci_regs.h +0
-0
tools/firmware/hvmloader/pci_regs.h line 114
Quoted 8 lines
tools/firmware/hvmloader/pci_regs.h +5
-0@@ -107,6 +107,11 @@#define PCI_INTEL_OPREGION 0xfc/* 4 bits */ +#define PCI_VENDOR_ID_INTEL 0x8086 +#define PCI_DEVICE_ID_INTEL_82441 0x1237 +#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 + +
Too many blank lines.
Quoted 27 lines
tools/firmware/hvmloader/pci_regs.h +0
-0#endif /* __HVMLOADER_PCI_REGS_H__ */ /* tools/firmware/hvmloader/util.c +9
-0@@ -22,6 +22,7 @@#include "hypercall.h" #include "ctype.h" #include "vnuma.h" +#include "pci_regs.h" #include <acpi2_0.h> #include <libacpi.h> #include <stdint.h> @@ -735,6 +736,52 @@ void __bug(char *file, int line)crash (); } + +static int machine_type =MACHINE_TYPE_UNDEFINED ; + +int get_pc_machine_type (void ) +{ +uint16_t vendor_id; +uint16_t device_id; +
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 756
Quoted 10 lines
tools/firmware/hvmloader/util.c +10
-0+if (machine_type !=MACHINE_TYPE_UNDEFINED ) +return machine_type; + + machine_type =MACHINE_TYPE_UNKNOWN ; + + vendor_id =pci_readw (0 ,PCI_VENDOR_ID ); + device_id =pci_readw (0 ,PCI_DEVICE_ID ); + +/* only Intel platforms are emulated currently */ +if (vendor_id ==PCI_VENDOR_ID_INTEL )
Coding style.
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 758
Quoted 2 lines
tools/firmware/hvmloader/util.c +2
-0+ { + switch (device_id)
Ditto.
And this patch should be folded into its user, unless the patch that uses it is very big on its own.
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 04/12] hvmloader: add ACPI enabling for Q35
In reply to Alexey GerasimenkoQuoted 16 lines · Alexey Gerasimenko
In order to turn on ACPI for OS, we need to write a chipset-specific value to
SMI_CMDregister (sort of imitation of theAPM->ACPIswitch on real systems). Modifyacpi_enable_sci()function to support both i440 and Q35 emulation.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/hvmloader.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)tools/firmware/hvmloader/hvmloader.c +0
-0@@ -257,9 +257,16 @@ static const struct bios_config *detect_bios(void)static void acpi_enable_sci (void )
tools/firmware/hvmloader/hvmloader.c static const struct bios_config *detect_bios(void) · line 266
Quoted 10 lines
tools/firmware/hvmloader/hvmloader.c +5
-1{uint8_t pm1a_cnt_val; +uint8_t acpi_enable_val; -#define PIIX4_SMI_CMD_IOPORT 0xb2 +#define SMI_CMD_IOPORT 0xb2#define PIIX4_ACPI_ENABLE 0xf1 +#define ICH9_ACPI_ENABLE 0x02 + +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 )
Coding style.
And the previous patch can be folded into this one.
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Wei LiuQuoted 3 lines · Wei Liu
Cc Anthony
IIRC there are changes needed on QEMU side?
Yes, there are actually QEMU patches in the patch series. I'm CCed on them.
Quoted 1 line
Do we need to wait until that lands?
That's depends. I don't have an answer.
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 03/12] hvmloader: add function to query an emulated machine type (i440/Q35)
In reply to Wei LiuQuoted 25 lines · On Tue, 13 Mar 2018 17:26:04 +0000 Wei Liu <[email protected]> wrote:
Quoted 24 lines · Alexey Gerasimenko
This adds a new function
get_pc_machine_type()which allows to determine the emulated chipset type. Supported return values:-
MACHINE_TYPE_I440
-MACHINE_TYPE_Q35
-MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by callingBUG()in hvmloader.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47+++++++++++++++++++++++++++++++++++++
tools/firmware/hvmloader/util.h| 8 +++++++ 3 files changed, 60
insertions(+)(quoted diff) +0
-0
b/tools/firmware/hvmloader/pci_regs.hindex7bf2d873ab..ba498b840e100644 ---a/tools/firmware/hvmloader/pci_regs.h
+++b/tools/firmware/hvmloader/pci_regs.htools/firmware/hvmloader/pci_regs.h +0
-0@@ -107,6 +107,11 @@#define PCI_INTEL_OPREGION 0xfc/* 4 bits */
Quoted 7 lines
tools/firmware/hvmloader/pci_regs.h line 114
Quoted 5 lines
tools/firmware/hvmloader/pci_regs.h +5
-0+#define PCI_VENDOR_ID_INTEL 0x8086 +#define PCI_DEVICE_ID_INTEL_82441 0x1237 +#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 + +Too many blank lines.
Will fix.
Quoted 10 lines
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 740
Quoted 6 lines
tools/firmware/hvmloader/util.c +2
-0@@ -735,6 +736,52 @@ void __bug(char *file, int line)crash (); } +/* only Intel platforms are emulated currently */ +if (vendor_id ==PCI_VENDOR_ID_INTEL )Coding style.
Ditto.
Will fix.
Quoted 2 lines
And this patch should be folded into its user, unless the patch that uses it is very big on its own.
Hmm, looks like I overfollowed the recommendation about making atomic patches for easier review. There are multiple users of these function, it was made in a separate patch just because of this. In the next version I'll merge it with some of the patches which use this function then.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 03/12] hvmloader: add function to query an emulated machine type (i440/Q35)
Quoted 51 lines · Alexey G
Quoted 25 lines · On Tue, 13 Mar 2018 17:26:04 +0000 Wei Liu <[email protected]> wrote:
Quoted 24 lines · Alexey Gerasimenko
This adds a new function
get_pc_machine_type()which allows to determine the emulated chipset type. Supported return values:-
MACHINE_TYPE_I440
-MACHINE_TYPE_Q35
-MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by callingBUG()in hvmloader.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47+++++++++++++++++++++++++++++++++++++
tools/firmware/hvmloader/util.h| 8 +++++++ 3 files changed, 60
insertions(+)(quoted diff) +0
-0
b/tools/firmware/hvmloader/pci_regs.hindex7bf2d873ab..ba498b840e100644 ---a/tools/firmware/hvmloader/pci_regs.h
+++b/tools/firmware/hvmloader/pci_regs.htools/firmware/hvmloader/pci_regs.h +0
-0@@ -107,6 +107,11 @@#define PCI_INTEL_OPREGION 0xfc/* 4 bits */ Quoted 7 lines
tools/firmware/hvmloader/pci_regs.h line 114
Quoted 5 lines
tools/firmware/hvmloader/pci_regs.h +5
-0+#define PCI_VENDOR_ID_INTEL 0x8086 +#define PCI_DEVICE_ID_INTEL_82441 0x1237 +#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 + +Too many blank lines.
Will fix.
Quoted 10 lines
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 740
Quoted 6 lines
tools/firmware/hvmloader/util.c +2
-0@@ -735,6 +736,52 @@ void __bug(char *file, int line)crash (); } +/* only Intel platforms are emulated currently */ +if (vendor_id ==PCI_VENDOR_ID_INTEL )Coding style.
Ditto.
Will fix.
Quoted 8 lines
Quoted 2 lines
And this patch should be folded into its user, unless the patch that uses it is very big on its own.
Hmm, looks like I overfollowed the recommendation about making atomic patches for easier review. There are multiple users of these function, it was made in a separate patch just because of this. In the next version I'll merge it with some of the patches which use this function then.
It really depends. It will take some back-and-forth to find the right balance. I can't say I'm very consistent on this either.
If you think leaving it in a separate patch is better, I won't object.
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 16/30] q35/xen: Add Xen platform device support for Q35
In reply to Eduardo HabkostQuoted 62 lines · On Mon, 12 Mar 2018 18:44:02 -0300 Eduardo Habkost <[email protected]> wrote:
Quoted 56 lines · Alexey G
Quoted 33 lines · On Mon, 12 Mar 2018 16:44:06 -0300 Eduardo Habkost <[email protected]> wrote:
Quoted 17 lines · Alexey Gerasimenko
Current Xen/QEMU method to control Xen Platform device on i440 is a bit odd -- enabling/disabling Xen platform device actually modifies the QEMU emulated machine type, namely xenfv <--> pc.
In order to avoid multiplying machine types, use a new way to control Xen Platform device for QEMU -- "xen-platform-dev" machine property (bool). To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation still uses the old method (i.e. xenfv/pc machine selection) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
This way we can use a single machine type (q35) and change just xen-platform-dev value to on/off to control Xen platform device.
Signed-off-by Alexey Gerasimenko <[email protected]>---
[...]
Quoted 13 lines
qemu-options.hx +0
-0@@ -38,6 +38,7 @@ DEF("machine", HAS_ARG, QEMU_OPTION_machine, \ " dump-guest-core=on|off include guest memoryin a core dump (default=on)\n" " mem-merge=on|off
controls memory merge support (default: on)\n" " igd-passthru=on|off controls IGD GFX passthrough support (default=off)\n"
+ " xen-platform-dev=on|off controls Xen
Platform device (default=off)\n" " aes-key-wrap=on|off controls support for AES key wrappingQuoted 8 lines
Quoted 4 lines
(default=on)\n" " dea-key-wrap=on|off controls
support for DEA key wrapping (default=on)\n" " suppress-vmdesc=on|off disables self-describing migration (default=off)\n"What are the obstacles preventing "-device xen-platform" from working? It would be better than adding a new boolean option to -machine.
I guess the initial assumption was that changing the
xen_platform_devicevalue in Xen's options may cause some additional changes in platform configuration besides adding (or not) the Xen Platform device, hence a completely different machine type was chosen (xenfv).At the moment pc,accel=xen/xenfv selection mostly governs only the Xen Platform device presence. Also setting
max_cpustoHVM_MAX_VCPUSdepends on it, but this doesn't applicable to a 'pc,accel=xen' machine for some reason.Quoted 4 lines
If applying
HVM_MAX_VCPUStomax_cpusis really necessary I think it's better to set it unconditionally for all 'accel=xen' HVM machine types insidexen_enabled()block. Right now it's missing for pc,accel=xen and q35,accel=xen.
Quoted 7 lines
If you are talking about
MachineClass::max_cpus, note that it is returned by query-machines, so it's supposed to be a static value. Changing it a runtime would mean the query-machines value is incorrect.Is
HVM_MAX_CPUShigher or lower than 255? If it's higher, does it mean the current value on pc and q35 isn't accurate?
HVM_MAX_VCPUS is 128 currently, but there is an ongoing work from Intel to support more vcpus and >8bit APIC IDs, so this number will likely change soon.
According to the code, using HVM_MAX_VCPUS in QEMU is a bit excessive as the maximum number of vcpus is controlled on Xen side anyway. Currently HVM_MAX_VCPUS is used in a one-time check for the maxcpus value (which itself comes from libxl).
I think for future compatibility it's better to set mc->max_cpus to HVM_MAX_VCPUS for all accel=xen HVM-supported machine types, not just xenfv.
The '-device' approach you suggested seems more preferable than a machine bool property, I'll try switching to it.
Quoted 6 lines
Is
HVM_MAX_CPUSsomething that needs to be enabled because of accel=xen or because or the xen-platform device?If it's just because of accel=xen, we could introduce a
AccelClass::max_cpus()method (we also have KVM-imposed CPU count limits, currently implemented insidekvm_init()).
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 13/30] pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
In reply to Alexey GerasimenkoQuoted 2 lines · Alexey Gerasimenko
xen_pci_slot_get_pirq-->xen_cmn_pci_slot_get_pirqxen_piix3_set_irq-->xen_cmn_set_irq
Don't abbrvt names, xen_hvm_ is a better prefix.
hw/i386/xen/xen-hvm.c void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len) · line 165
Quoted 4 lines
hw/i386/xen/xen-hvm.c +3
-0+fprintf (stderr,"WARNING: guest domain attempted to use PIRQ%c " +"routing which is not supported for Xen/Q35 currently\n" , + (char )(address -ICH9_LPC_PIRQE_ROUT +'E' ));
Use error_report instead.
Paolo
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Qemu-devel] [RFC PATCH 13/30] pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
In reply to Alexey GerasimenkoQuoted 2 lines · Alexey Gerasimenko
xen_pci_slot_get_pirq-->xen_cmn_pci_slot_get_pirqxen_piix3_set_irq-->xen_cmn_set_irq
Don't abbrvt names, xen_hvm_ is a better prefix.
hw/i386/xen/xen-hvm.c void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len) · line 165
Quoted 4 lines
hw/i386/xen/xen-hvm.c +3
-0+fprintf (stderr,"WARNING: guest domain attempted to use PIRQ%c " +"routing which is not supported for Xen/Q35 currently\n" , + (char )(address -ICH9_LPC_PIRQE_ROUT +'E' ));
Use error_report instead.
Paolo
Re: [Qemu-devel] [RFC PATCH 13/30] pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
Quoted 5 lines · On Wed, 14 Mar 2018 11:48:46 +0100 Paolo Bonzini <[email protected]> wrote:
Quoted 2 lines · Alexey Gerasimenko
xen_pci_slot_get_pirq-->xen_cmn_pci_slot_get_pirqxen_piix3_set_irq-->xen_cmn_set_irqDon't abbrvt names,
xen_hvm_is a better prefix.
Agree, will rename xen_cmn_* to xen_hvm_*
Quoted 8 lines
Quoted 6 lines
+ fprintf(stderr, "WARNING: guest domain
attempted to use PIRQ%c "
+ "routing which is not supported for
Xen/Q35 currently\n",
+ (char)(address -ICH9_LPC_PIRQE_ROUT+
'E'));Use
error_reportinstead.
OK, will change to error_report().
There are multiple fprintf(stderr,...)'s still left in the file though, an additional cleanup patch to replace all such instances to error_report() calls might be needed later.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Alexey GerasimenkoQuoted 11 lines · On Tue, 13 Mar 2018 04:33:56 +1000 Alexey Gerasimenko <[email protected]> wrote:
This patch extends
hvmloader_acpi_build_tables()with code which detects if MMCONFIG is available -- i.e. initialized and enabled (+we're running on Q35), obtains its base address and size and asks libacpi to build MCFG table for it via setting the flagACPI_HAS_MCFGin a manner similar to other optional ACPI tables building.Signed-off-by Alexey Gerasimenko <[email protected]>---
tools/firmware/hvmloader/util.c| 70 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
Looks like I missed the patch for reserving MMCONFIG area in E820 map, it is required for Linux guests (otherwise MMCONFIG info will be rejected by linux kernel). Windows guests allow to use MMCONFIG without a corresponding E820 entry.
Following lines need to be added to /hvmloader/e820.c:
+ /* mark MMCONFIG area */
+ if ( is_mmconfig_used() )
+ {
+ e820[nr].addr = mmconfig_get_base();
+ e820[nr].size = mmconfig_get_size();
+ e820[nr].type = E820_RESERVED;
+ nr++;+ }
The corresponding patch-file is attached, will include it in v2 patches.
From c16186e0c1ad388362f61136b8da2e02e76d1840 Mon Sep 17 00:00:00 2001
From: Alexey Gerasimenko <[email protected]>
Date: Thu, 15 Mar 2018 03:06:39 +1000
Subject: [PATCH] hvmloader: mark MMCONFIG in E820 map
--- tools/firmware/hvmloader/e820.c | 9 +++++++++ tools/firmware/hvmloader/util.c | 6 +++--- tools/firmware/hvmloader/util.h | 5 +++++ 3 files changed, 17 insertions(+), 3 deletions(-)
tools/firmware/hvmloader/e820.c +9 -0
@@ -233,6 +233,15 @@ int build_e820_table(struct e820entry *e820, nr++; } +/* mark MMCONFIG area */ +if (is_mmconfig_used () ) + { + e820[nr].addr =mmconfig_get_base (); + e820[nr].size =mmconfig_get_size (); + e820[nr].type =E820_RESERVED ; + nr++; + } +/* Low RAM goes here. Reserve space for special pages. */ BUG_ON (low_mem_end <MB (2 ));
tools/firmware/hvmloader/util.c +3 -3
@@ -788,7 +788,7 @@ int get_pc_machine_type(void)#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3)#define PCIEXBAREN 1 -static uint64_t mmconfig_get_base (void ) +uint64_t mmconfig_get_base (void ) {uint64_t base;uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR );
@@ -813,7 +813,7 @@ static uint64_t mmconfig_get_base(void)return base; } -static uint32_t mmconfig_get_size (void ) +uint32_t mmconfig_get_size (void ) {uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR );
@@ -834,7 +834,7 @@ static uint32_t mmconfig_is_enabled(void)return pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ) &PCIEXBAREN ; } -static int is_mmconfig_used (void ) +int is_mmconfig_used (void ) {if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) {
tools/firmware/hvmloader/util.h +5 -0
@@ -296,6 +296,11 @@ struct acpi_config;void hvmloader_acpi_build_tables (struct acpi_config *config,unsigned int physical); +/* MMCONFIG-related */ +uint64_t mmconfig_get_base (void ); +uint32_t mmconfig_get_size (void ); +int is_mmconfig_used (void ); +#endif /* __HVMLOADER_UTIL_H__ */ /*
Signature · 1 line
2.11.0
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
A gentle RFC-ping.
Any thoughts on this? Regarding the feature as a whole. So far there were responses mostly targeting individual patches, while I'd like to hear about chosen approaches in general, whether the overall direction is correct (or not), etc. It's just RFC after all, not v11. :)
I can split it into two series if that would be preferable, one for general Q35 bring up and basic access to PCIe extended config space via ECAM (this is what the feature was used for initially) and the second part is providing support for PCIe Extended Capabilities emulation infrastructure (hw/xen/xen-pt*.c in QEMU).
Quoted 4 lines · On Tue, 13 Mar 2018 04:33:45 +1000 Alexey Gerasimenko <[email protected]> wrote:
This patch series introduces support of Q35 emulation for Xen HVM guests (via QEMU). This feature is present in other virtualization products and Xen can greatly benefit from this feature as well.
Quoted 14 lines
The main goal for implementing Q35 emulation for Xen was extending PCI/GPU passthrough capabilities. It's the main advantage of Q35 emulation
- availability of extra features for PCIe device passthrough. The most important PCIe-specific passthrough feature Q35 provides is a support for PCIe config space ECAM (aka MMCONFIG) to allow accesses to extended PCIe config space (>256), which is MMIO-based. Lots of PCIe devices and their drivers make use of PCIe Extended Capabilities, whose can be accessed only using ECAM and offsets above0x100in PCI config space. Supporting ECAM is a mandatory feature for PCIe passthrough. Not only this allows passthrough PCIe devices to function properly, but opens a road to extend Xen PCIe passthrough features further -- eg. providing support for AER. One of possible directions is providing support for PCIe Resizable BARs -- a feature which likely
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Alexey, thanks for the ping. I think this is a good feature to have and I would like to check it in when it is ready. I spoke with Anthony and agreed that he will be reviewing it. Please be patient but we'll get there :-)
Quoted 7 lines · Alexey G
A gentle RFC-ping.
Any thoughts on this? Regarding the feature as a whole. So far there were responses mostly targeting individual patches, while I'd like to hear about chosen approaches in general, whether the overall direction is correct (or not), etc. It's just RFC after all, not v11. :)
Quoted 28 lines
I can split it into two series if that would be preferable, one for general Q35 bring up and basic access to PCIe extended config space via ECAM (this is what the feature was used for initially) and the second part is providing support for PCIe Extended Capabilities emulation infrastructure (
hw/xen/xen-pt*.cin QEMU).Quoted 18 lines · On Tue, 13 Mar 2018 04:33:45 +1000 Alexey Gerasimenko <[email protected]> wrote:
This patch series introduces support of Q35 emulation for Xen HVM guests (via QEMU). This feature is present in other virtualization products and Xen can greatly benefit from this feature as well.
The main goal for implementing Q35 emulation for Xen was extending PCI/GPU passthrough capabilities. It's the main advantage of Q35 emulation
- availability of extra features for PCIe device passthrough. The most important PCIe-specific passthrough feature Q35 provides is a support for PCIe config space ECAM (aka MMCONFIG) to allow accesses to extended PCIe config space (>256), which is MMIO-based. Lots of PCIe devices and their drivers make use of PCIe Extended Capabilities, whose can be accessed only using ECAM and offsets above0x100in PCI config space. Supporting ECAM is a mandatory feature for PCIe passthrough. Not only this allows passthrough PCIe devices to function properly, but opens a road to extend Xen PCIe passthrough features further -- eg. providing support for AER. One of possible directions is providing support for PCIe Resizable BARs -- a feature which likely
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 6 lines · Alexey G
A gentle RFC-ping.
Any thoughts on this? Regarding the feature as a whole. So far there were responses mostly targeting individual patches, while I'd like to hear about chosen approaches in general, whether the overall direction is correct (or not), etc. It's just RFC after all, not v11. :)
I plan to look at the series, but in general you should wait at least 7 days (one week) before pinging.
Quoted 5 lines
I can split it into two series if that would be preferable, one for general Q35 bring up and basic access to PCIe extended config space via ECAM (this is what the feature was used for initially) and the second part is providing support for PCIe Extended Capabilities emulation infrastructure (
hw/xen/xen-pt*.cin QEMU).
I would wait a bit before doing more work, as said I plan to look at the series, and other probably are too, it's just that you gave us too little time ;).
Keep in mind we are approaching feature freeze, and there are some more mature series that will likely get more review attention than yours ATM, in order to try to get them in before the freeze.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Alexey GerasimenkoQuoted 12 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.
Quoted 6 lines
Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551 +++++++++++++++++++++++++++++++++++++++++++++
So this is basically a modified dupe of the current dsdt.asl? AFAICT there are a bunch of common bits, which ideally we want to have defined in a single place.
Can't you factor out the common parts of the dsdt.asl into smaller parts an include them for both dsdt.asl and dsdt_q35.asl?
I would first have a patch that extract the common parts of the dsdt into file(s), and then a second patch which creates a dsdt_q35.asl based on those common bits plus the specific q35 code.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 02/12] Makefile: build and use new DSDT table for Q35
In reply to Alexey GerasimenkoQuoted 5 lines · Alexey Gerasimenko
Provide building for newly added dsdt_q35.asl file, in a way similar to dsdt.asl.
Note that '15cpu' ACPI tables are only applicable to qemu-traditional (which have no support for Q35), so we need to use 'anycpu' version only.
You should do this in the same patch that adds dsdt_q35.asl, at the end without this the previous patch just adds dead code.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 03/12] hvmloader: add function to query an emulated machine type (i440/Q35)
In reply to Alexey GerasimenkoQuoted 7 lines · Alexey Gerasimenko
This adds a new function
get_pc_machine_type()which allows to determine the emulated chipset type. Supported return values:-
MACHINE_TYPE_I440
-MACHINE_TYPE_Q35
-MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by callingBUG()in hvmloader.
This is not correct, the return values are strictly MACHINE_TYPE_I440 or MACHINE_TYPE_Q35. Everything else ends up in a BUG().
Also makes me wonder whether this should instead be init_machine_type, and users should just read machine_type directly.
Quoted 37 lines
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47 +++++++++++++++++++++++++++++++++++++ tools/firmware/hvmloader/util.h | 8 +++++++ 3 files changed, 60 insertions(+)tools/firmware/hvmloader/pci_regs.h +5
-0@@ -107,6 +107,11 @@#define PCI_INTEL_OPREGION 0xfc/* 4 bits */ +#define PCI_VENDOR_ID_INTEL 0x8086 +#define PCI_DEVICE_ID_INTEL_82441 0x1237 +#define PCI_DEVICE_ID_INTEL_Q35_MCH 0x29c0 + +#endif /* __HVMLOADER_PCI_REGS_H__ */ /* tools/firmware/hvmloader/util.c +1
-0@@ -22,6 +22,7 @@#include "hypercall.h" #include "ctype.h" #include "vnuma.h" +#include "pci_regs.h" #include <acpi2_0.h> #include <libacpi.h> #include <stdint.h>
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 740
Quoted 5 lines
tools/firmware/hvmloader/util.c +2
-0@@ -735,6 +736,52 @@ void __bug(char *file, int line)crash (); } + +static int machine_type =MACHINE_TYPE_UNDEFINED ;
There's no need to init this, _UNDEFINED is 0 which is the default value.
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 742
Quoted 2 lines
tools/firmware/hvmloader/util.c +2
-0+ +int get_pc_machine_type (void )
You introduce a function that's not used anywhere, and the commit log doesn't mention why this is needed at all. In general I prefer functions to be introduced with at least a caller, or else it needs to be described in the commit message why this is not the case.
Quoted 4 lines
tools/firmware/hvmloader/util.c +4
-0+{ +uint16_t vendor_id; +uint16_t device_id; +
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 756
Quoted 10 lines
tools/firmware/hvmloader/util.c +10
-0+if (machine_type !=MACHINE_TYPE_UNDEFINED ) +return machine_type; + + machine_type =MACHINE_TYPE_UNKNOWN ; + + vendor_id =pci_readw (0 ,PCI_VENDOR_ID ); + device_id =pci_readw (0 ,PCI_DEVICE_ID ); + +/* only Intel platforms are emulated currently */ +if (vendor_id ==PCI_VENDOR_ID_INTEL )
Should this maybe be a BUG_ON(vendor_id != PCI_VENDOR_ID_INTEL) then? Note that in this case you end up with a BUG later anyway.
Quoted 13 lines
tools/firmware/hvmloader/util.c +13
-0+ { +switch (device_id) + { +case PCI_DEVICE_ID_INTEL_82441 : + machine_type =MACHINE_TYPE_I440 ; +printf ("Detected i440 chipset\n" ); +break ; + +case PCI_DEVICE_ID_INTEL_Q35_MCH : + machine_type =MACHINE_TYPE_Q35 ; +printf ("Detected Q35 chipset\n" ); +break ; +
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 779
Quoted 10 lines
tools/firmware/hvmloader/util.c +10
-0+default : +break ; + } + } + +if (machine_type ==MACHINE_TYPE_UNKNOWN ) + { +printf ("Unknown emulated chipset encountered, VID=%04Xh, DID=%04Xh\n" , + vendor_id, device_id); +BUG ();
Why not place this in the default switch label? That would allow you to get rid of the MACHINE_TYPE_UNKNOWN define also.
Quoted 13 lines
tools/firmware/hvmloader/util.c +5
-0+ } + +return machine_type; +} +static void validate_hvm_info (struct hvm_info_table *t) {uint8_t *ptr = (uint8_t *)t;tools/firmware/hvmloader/util.h +0
-0
tools/firmware/hvmloader/util.h void pci_write(uint32_t devfn, uint32_t reg, uint32_t len, uint32_t val); · line 107
Quoted 8 lines
tools/firmware/hvmloader/util.h +5
-0@@ -100,6 +100,14 @@ void pci_write(uint32_t devfn, uint32_t reg, uint32_t len, uint32_t val);#define pci_writew (devfn, reg, val) pci_write(devfn, reg, 2, (uint16_t)(val))#define pci_writel (devfn, reg, val) pci_write(devfn, reg, 4, (uint32_t)(val)) +/* Emulated machine types */ +#define MACHINE_TYPE_UNDEFINED 0 +#define MACHINE_TYPE_I440 1 +#define MACHINE_TYPE_Q35 2 +#define MACHINE_TYPE_UNKNOWN (-1)
An enum seems better suited for this.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 04/12] hvmloader: add ACPI enabling for Q35
In reply to Alexey GerasimenkoQuoted 19 lines · Alexey Gerasimenko
In order to turn on ACPI for OS, we need to write a chipset-specific value to
SMI_CMDregister (sort of imitation of theAPM->ACPIswitch on real systems). Modifyacpi_enable_sci()function to support both i440 and Q35 emulation.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/hvmloader.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)tools/firmware/hvmloader/hvmloader.c +1
-0@@ -257,9 +257,16 @@ static const struct bios_config *detect_bios(void)static void acpi_enable_sci (void ) {uint8_t pm1a_cnt_val; +uint8_t acpi_enable_val;
tools/firmware/hvmloader/hvmloader.c static const struct bios_config *detect_bios(void) · line 269
Quoted 10 lines
tools/firmware/hvmloader/hvmloader.c +7
-1-#define PIIX4_SMI_CMD_IOPORT 0xb2 +#define SMI_CMD_IOPORT 0xb2#define PIIX4_ACPI_ENABLE 0xf1 +#define ICH9_ACPI_ENABLE 0x02 + +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + acpi_enable_val =ICH9_ACPI_ENABLE ; +else + acpi_enable_val =PIIX4_ACPI_ENABLE ;
Coding style, but I would rather:
switch ( get_pc_machine_type() )
{
case MACHINE_TYPE_Q35:
...
case MACHINE_TYPE_I440:
...
default:BUG();
}
I think storing the machine type in a global variable is better than calling get_pc_machine_type each time.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 02/12] Makefile: build and use new DSDT table for Q35
In reply to Alexey Gerasimenkotools/firmware/hvmloader/Makefile rombios.o: roms.inc · line 78
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 12.03.18 at 19:33, <[email protected]> wrote:
tools/firmware/hvmloader/Makefile +1
-1@@ -75,7 +75,7 @@ rombios.o: roms.inc smbios.o: CFLAGS += -D__SMBIOS_DATE__="\"$(SMBIOS_REL_DATE)\"" ACPI_PATH = ../../libacpi -DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c +DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c dsdt_q35_anycpu_qemu_xen.c
Unless you intend to add a second flavor, please omit the "anycpu" part from the name of the new instance.
tools/firmware/hvmloader/Makefile $(ACPI_BUILD_DIR)/dsdt_anycpu_qemu_xen.asl: dsdt.asl dsdt_acpi_info.asl $(MK_DSD · line 64
Quoted 10 lines
tools/firmware/hvmloader/Makefile +6
-0@@ -56,6 +56,13 @@ $(ACPI_BUILD_DIR)/dsdt_anycpu_qemu_xen.asl: dsdt.asl dsdt_acpi_info.asl $(MK_DSD $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) mv -f $@.$(TMP_SUFFIX) $@ +$(ACPI_BUILD_DIR)/dsdt_q35_anycpu_qemu_xen.asl: dsdt_q35.asl dsdt_acpi_info.asl $(MK_DSDT) + # Remove last bracket + awk 'NR > 1 {print s} {s=$$0}' $< > $@.$(TMP_SUFFIX) + cat dsdt_acpi_info.asl >> $@.$(TMP_SUFFIX) + $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) + mv -f $@.$(TMP_SUFFIX) $@
The commands look to be exactly the same as those for dsdt_anycpu_qemu_xen.asl - please let's not duplicate such things, but instead use a pattern rule.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 01/12] libacpi: new DSDT ACPI table for Q35
In reply to Roger Pau MonnéQuoted 26 lines · On Mon, 19 Mar 2018 12:43:05 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 13 lines · Alexey Gerasimenko
This patch adds the DSDT table for Q35 (new
tools/libacpi/dsdt_q35.aslfile). There are not many differences with dsdt.asl (for i440) at the moment, namely:- BDF location of LPC Controller
- Minor changes related to FDC detection
- Addition of_OSCmethod to inform OSPM about PCIe features supportedAs we are still using 4 PCI router links and their corresponding device/register addresses are same (offset
0x60), no need to change PCI routing descriptions.Quoted 7 lines
Also, ACPI hotplug is still used to control passed through device hot (un)plug (as it was for i440).
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/dsdt_q35.asl | 551+++++++++++++++++++++++++++++++++++++++++++++
So this is basically a modified dupe of the current dsdt.asl? AFAICT there are a bunch of common bits, which ideally we want to have defined in a single place.
Quoted 6 lines
Can't you factor out the common parts of the dsdt.asl into smaller parts an include them for both dsdt.asl and dsdt_q35.asl?
I would first have a patch that extract the common parts of the dsdt into file(s), and then a second patch which creates a dsdt_q35.asl based on those common bits plus the specific q35 code.
Yes, it's a good thing that many registers have same addresses on i440 and Q35. Some encountered common things were unexpected though -- AFAIR _S5 SLP_TYP value do not correspond to the ICH9 datasheet, a different value used instead to trigger ACPI Soft-Off emulation.
Regarding dsdt.asl/dsdt_q35.asl -- OK, I'll split these files into common/specific parts.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 02/12] Makefile: build and use new DSDT table for Q35
In reply to Jan BeulichQuoted 6 lines · On Mon, 19 Mar 2018 07:07:34 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 12.03.18 at 19:33, <[email protected]> wrote:
tools/firmware/hvmloader/Makefile +0
-0@@ -75,7 +75,7 @@ rombios.o: roms.inc smbios.o: CFLAGS += -D__SMBIOS_DATE__="\"$(SMBIOS_REL_DATE)\""
Quoted 7 lines
Quoted 4 lines
tools/firmware/hvmloader/Makefile +1
-1ACPI_PATH = ../../libacpi -DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c +DSDT_FILES = dsdt_anycpu.c dsdt_15cpu.c dsdt_anycpu_qemu_xen.c
dsdt_q35_anycpu_qemu_xen.cUnless you intend to add a second flavor, please omit the "anycpu" part from the name of the new instance.
Just following same "anycpu/15cpu" naming scheme, there will be no need for dsdt_q35_15cpu.c, so I guess its ok to drop anycpu/15cpu part of the name, will rename it.
Quoted 4 lines
Quoted 4 lines
tools/firmware/hvmloader/Makefile +0
-1@@ -56,6 +56,13 @@ $(ACPI_BUILD_DIR)/dsdt_anycpu_qemu_xen.asl: dsdt.asl dsdt_acpi_info.asl $(MK_DSD $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) mv -f $@.$(TMP_SUFFIX) $@
Quoted 12 lines
tools/firmware/hvmloader/Makefile $(ACPI_BUILD_DIR)/dsdt_anycpu_qemu_xen.asl: · line 65
Quoted 8 lines
tools/firmware/hvmloader/Makefile +6
-0+$(ACPI_BUILD_DIR)/dsdt_q35_anycpu_qemu_xen.asl: dsdt_q35.asl dsdt_acpi_info.asl $(MK_DSDT) + # Remove last bracket + awk 'NR > 1 {print s} {s=$$0}' $< > $@.$(TMP_SUFFIX) + cat dsdt_acpi_info.asl >> $@.$(TMP_SUFFIX) + $(MK_DSDT) --debug=$(debug) --dm-version qemu-xen >> $@.$(TMP_SUFFIX) + mv -f $@.$(TMP_SUFFIX) $@The commands look to be exactly the same as those for dsdt_anycpu_qemu_xen.asl - please let's not duplicate such things, but instead use a pattern rule.
Agree, reusing the rule will be better.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 02/12] Makefile: build and use new DSDT table for Q35
In reply to Roger Pau MonnéQuoted 4 lines · On Mon, 19 Mar 2018 12:46:05 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 3 lines · Alexey Gerasimenko
Provide building for newly added dsdt_q35.asl file, in a way similar to dsdt.asl.
Quoted 8 lines
Quoted 3 lines
Note that '15cpu' ACPI tables are only applicable to qemu-traditional (which have no support for Q35), so we need to use 'anycpu' version only.
You should do this in the same patch that adds dsdt_q35.asl, at the end without this the previous patch just adds dead code.
Thanks, Roger.
Agree, I've abused recommendation to granulate patches for easier review. :) Will merge it with the previous patch.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 05/12] hvmloader: add Q35 DSDT table loading
In reply to Alexey GerasimenkoQuoted 17 lines · Alexey Gerasimenko
Allows to select Q35 DSDT table in
hvmloader_acpi_build_tables(). Functionget_pc_machine_type()is used to select a proper table (i440/q35).As we are bound to the qemu-xen device model for Q35, no need to initialize config->dsdt_15cpu/config->dsdt_15cpu_len fields.
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/util.c | 13 +++++++++++-- tools/firmware/hvmloader/util.h | 2 ++ 2 files changed, 13 insertions(+), 2 deletions(-)tools/firmware/hvmloader/util.c +0
-0
tools/firmware/hvmloader/util.c void hvmloader_acpi_build_tables(struct acpi_config *config, · line 958
Quoted 6 lines
tools/firmware/hvmloader/util.c +1
-2@@ -955,8 +955,17 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, }else if ( !strncmp (s,"qemu_xen" ,9 ) ) { - config->dsdt_anycpu = dsdt_anycpu_qemu_xen; - config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 )
Coding style (missing spaces between parentheses), and I would prefer a switch here.
IMO you should add a BUG_ON(Q35) in the qemu_xen_traditional condition above this one..
Quoted 18 lines
tools/firmware/hvmloader/util.c +10
-0+ { + config->dsdt_anycpu = dsdt_q35_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_q35_anycpu_qemu_xen_len; + } + else + { + config->dsdt_anycpu = dsdt_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; + } + config->dsdt_15cpu =NULL ; config->dsdt_15cpu_len =0 ; }tools/firmware/hvmloader/util.h +0
-0
tools/firmware/hvmloader/util.h bool check_overlap(uint64_t start, uint64_t size, · line 293
Quoted 6 lines
tools/firmware/hvmloader/util.h +2
-0@@ -288,7 +288,9 @@ bool check_overlap(uint64_t start, uint64_t size, uint64_t reserved_start, uint64_t reserved_size);extern const unsigned char dsdt_anycpu_qemu_xen[], dsdt_anycpu[], dsdt_15cpu[]; +extern const unsigned char dsdt_q35_anycpu_qemu_xen[];extern const int dsdt_anycpu_qemu_xen_len, dsdt_anycpu_len, dsdt_15cpu_len; +extern const int dsdt_q35_anycpu_qemu_xen_len;
Since you are adding this, maybe unsigned int? (or size_t?)
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 09/12] libxl: Xen Platform device support for Q35
In reply to Alexey GerasimenkoQuoted 36 lines · On Tue, 13 Mar 2018 04:33:54 +1000 Alexey Gerasimenko <[email protected]> wrote:
Current Xen/QEMU method to control Xen Platform device is a bit odd -- changing '
xen_platform_device' option value actually modifies QEMU emulated machine type, namely xenfv <--> pc.In order to avoid multiplying machine types, use the new way to control Xen Platform device for QEMU -- xen-platform-dev property. To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation uses the old method (xenfv/pc machine) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
Signed-off-by Alexey Gerasimenko <[email protected]>---
tools/libxl/libxl_dm.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)tools/libxl/libxl_dm.c +5
-1@@ -1444,7 +1444,11 @@ static intlibxl__build_device_model_args_new (libxl__gc *gc, break;case LIBXL_DOMAIN_TYPE_HVM :if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { - machinearg =libxl__sprintf (gc,"q35,accel=xen" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); + }else { + machinearg =libxl__sprintf (gc,"q35,accel=xen,xen-platform-dev=on" ); + }} else { if (!libxl_defbool_val(b_info->u.hvm.xen_platform_pci)) { /* Switching here to the machine "pc" which does not
Quoted 1 line
add
Regarding this one -- QEMU maintainers suggested that supplying '-device xen-platform' directly should be a better approach than a machine property, so this patch is kinda obsolete.
Right now "xenfv" machine usage for qemu-xen seems to be limited to controlling the Xen platform device and applying the HVM_MAX_VCPUS value to maxcpus + minor changes related to IGD passthrough. Both should be applicable for a "pc,accel=xen" machine as well I think, which in fact currently lacks the HVM_MAX_VCPUS check for some reason.
Adding a distinct method to control Xen platform device for the q35 machine suggests to propagate the same approach to i440 machine types, but... it depends on who else can use xenfv for qemu-xen (not to be confused with xenfv usage on qemu-traditional).
Is there any other toolstacks/code which use xenfv machine solely to turn on/off Xen platform device?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 06/12] hvmloader: add basic Q35 support
In reply to Alexey GerasimenkoQuoted 35 lines · Alexey Gerasimenko
This patch does following:
1. Move PCI-device specific initialization out of
pci_setupfunction to the newly createdclass_specific_pci_device_setupfunction to simplify code.2. PCI-device specific initialization extended with LPC controller initialization
3. Initialize PIRQA...{PIRQD, PIRQH} routing accordingly to the emulated south bridge (either located on
PCI_ISA_DEVFNorPCI_ICH9_LPC_DEVFN).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/config.h | 1 + tools/firmware/hvmloader/pci.c | 162 ++++++++++++++++++++++++-------------- 2 files changed, 104 insertions(+), 59 deletions(-)tools/firmware/hvmloader/config.h +1
-0@@ -52,6 +52,7 @@ extern uint8_t ioapic_version;#define PCI_ISA_DEVFN 0x08/* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U/* ISA IRQs 5,10,11 are PCI connected */ +#define PCI_ICH9_LPC_DEVFN 0xf8/* dev 31, fn 0 */ /* MMIO hole: Hardcoded defaults, which can be dynamically expanded. */ #define PCI_MEM_END 0xfc000000tools/firmware/hvmloader/pci.c +0
-0
tools/firmware/hvmloader/pci.c unsigned long pci_mem_end = PCI_MEM_END; · line 38
Quoted 4 lines
tools/firmware/hvmloader/pci.c +1
-0@@ -35,6 +35,7 @@ unsigned long pci_mem_end = PCI_MEM_END;uint64_t pci_hi_mem_start =0 , pci_hi_mem_end =0 ;enum virtual_vga virtual_vga = VGA_none; +uint32_t vga_devfn =256 ;
uint8_t should be enough to store a devfn. Also this should be static maybe?
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 82
Quoted 10 lines
tools/firmware/hvmloader/pci.c +3
-0unsigned long igd_opregion_pgbase =0 ;/* Check if the specified range conflicts with any reserved device memory. */ @@ -76,14 +77,93 @@ static int find_next_rmrr(uint32_t base)return next_rmrr; } +#define SCI_EN_IOPORT (ACPI_PM1A_EVT_BLK_ADDRESS_V1 + 0x30) +#define GBL_SMI_EN (1 << 0) +#define APMC_EN (1 << 5)
Alignment.
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 92
Quoted 10 lines
tools/firmware/hvmloader/pci.c +10
-0+ +static void class_specific_pci_device_setup (uint16_t vendor_id, +uint16_t device_id, +uint8_t bus,uint8_t devfn) +{ +uint16_t class; + + class =pci_readw (devfn,PCI_CLASS_DEVICE ); + + switch ( class )
switch ( pci_readw(devfn, PCI_CLASS_DEVICE) ) ?
I don't see class being used elsewhere.
Also why is vendor_id/device_id provided by the caller but not class? It seems kind of pointless.
Why not fetch vendor/device from the function itself and move the (vendor_id == 0xffff) && (device_id == 0xffff) check inside the function?
Also in this case I think it would be better to have a non-functional patch that introduces class_specific_pci_device_setup and a second patch that adds support for ICH9.
Having code movement and new code in the same patch makes it harder to very what you are actually moving vs introducing.
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 94
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2
-0+ { +case 0x0300 :
All this values need to be defines documented somewhere.
Quoted 64 lines
tools/firmware/hvmloader/pci.c +64
-0+/* If emulated VGA is found, preserve it as primary VGA. */ +if ( (vendor_id ==0x1234 ) && (device_id ==0x1111 ) ) + { + vga_devfn = devfn; + virtual_vga = VGA_std; + } +else if ( (vendor_id ==0x1013 ) && (device_id ==0xb8 ) ) + { + vga_devfn = devfn; + virtual_vga = VGA_cirrus; + } +else if ( virtual_vga == VGA_none ) + { + vga_devfn = devfn; + virtual_vga = VGA_pt; +if ( vendor_id ==0x8086 ) + { + igd_opregion_pgbase =mem_hole_alloc (IGD_OPREGION_PAGES ); +/* +* Write the the OpRegion offset to give the opregion +* address to the device model. The device model will trap +* and map the OpRegion at the give address. +*/ +pci_writel (vga_devfn,PCI_INTEL_OPREGION , + igd_opregion_pgbase <<PAGE_SHIFT ); + } + } +break ; + +case 0x0680 : +/* PIIX4 ACPI PM. Special device with special PCI config space. */ +ASSERT ((vendor_id ==0x8086 ) && (device_id ==0x7113 )); +pci_writew (devfn,0x20 ,0x0000 );/* No smb bus IO enable */ +pci_writew (devfn,0xd2 ,0x0000 );/* No smb bus IO enable */ +pci_writew (devfn,0x22 ,0x0000 ); +pci_writew (devfn,0x3c ,0x0009 );/* Hardcoded IRQ9 */ +pci_writew (devfn,0x3d ,0x0001 ); +pci_writel (devfn,0x40 ,ACPI_PM1A_EVT_BLK_ADDRESS_V1 |1 ); +pci_writeb (devfn,0x80 ,0x01 );/* enable PM io space */ +break ; + +case 0x0601 : +/* LPC bridge */ +if (vendor_id ==0x8086 && device_id ==0x2918 ) + { +pci_writeb (devfn,0x3c ,0x09 );/* Hardcoded IRQ9 */ +pci_writeb (devfn,0x3d ,0x01 ); +pci_writel (devfn,0x40 ,ACPI_PM1A_EVT_BLK_ADDRESS_V1 |1 ); +pci_writeb (devfn,0x44 ,0x80 );/* enable PM io space */ +outl (SCI_EN_IOPORT ,inl (SCI_EN_IOPORT ) |GBL_SMI_EN |APMC_EN ); + } +break ; + +case 0x0101 : +if ( vendor_id ==0x8086 ) + { +/* Intel ICHs since PIIX3: enable IDE legacy mode. */ +pci_writew (devfn,0x40 ,0x8000 );/* enable IDE0 */ +pci_writew (devfn,0x42 ,0x8000 );/* enable IDE1 */ + } +break ; + } +} +
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 166
Quoted 10 lines
tools/firmware/hvmloader/pci.c +2
-2void pci_setup (void ) {uint8_t is_64bar, using_64bar, bar64_relocate =0 ;uint32_t devfn, bar_reg, cmd, bar_data, bar_data_upper;uint64_t base, bar_sz, bar_sz_upper, mmio_total =0 ; -uint32_t vga_devfn =256 ; -uint16_t class, vendor_id, device_id; +uint16_t vendor_id, device_id;unsigned int bar, pin, link, isa_irq; +int is_running_on_q35 =0 ;
bool is_running_on_q35 = (get_pc_machine_type() == MACHINE_TYPE_Q35);
Quoted 9 lines
tools/firmware/hvmloader/pci.c +2
-0/* Resources assignable to PCI devices via BARs. */ struct resource {@@ -130,13 +210,28 @@ void pci_setup(void)if ( s ) mmio_hole_size =strtoll (s,NULL ,0 ); +/* check if we are on Q35 and set the flag if it is the case */ + is_running_on_q35 =get_pc_machine_type () ==MACHINE_TYPE_Q35 ;
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 223
Quoted 10 lines
tools/firmware/hvmloader/pci.c +3
-1+/* Program PCI-ISA bridge with appropriate link routes. */ isa_irq =0 ;for ( link =0 ; link <4 ; link++ ) {do { isa_irq = (isa_irq +1 ) &15 ; }while ( !(PCI_ISA_IRQ_MASK & (1U << isa_irq)) ); -pci_writeb (PCI_ISA_DEVFN ,0x60 + link, isa_irq); + +if (is_running_on_q35)
Coding style.
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 228
Quoted 5 lines
tools/firmware/hvmloader/pci.c +5
-0+ { +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x60 + link, isa_irq); + +/* PIRQE..PIRQH are unused */ +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x68 + link,0x80 );
According to the spec 0x80 is the default value for this registers, do you really need to write it?
Is maybe QEMU not correctly setting the default value?
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 232
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4
-0+ } +else + { +pci_writeb (PCI_ISA_DEVFN ,0x60 + link, isa_irq);
Is all this magic described somewhere that you can reference?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GerasimenkoQuoted 12 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration.
Quoted 4 lines
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see IOREQ_TYPE_PCI_CONFIG). With this change you will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.
Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Adding Paul who knows more about all this.
Quoted 15 lines
One important new field is
addr_mask, which tells which bits of the base address can (should) be written. Different address types (ROM, MMIO BAR, PCIEXBAR) will have differentaddr_maskvalues.For every assignable BAR range we store its size, PCI device BDF (devfn actually) to which it belongs, BAR type (mem/io/mem64) and corresponding register offset in device PCI conf space. This way we can insert MMCONFIG entry into bars array in the same manner like for any other BARs. In this case, the devfn field will point to MCH PCI device and
bar_regwill contain PCIEXBAR register offset. It will be assigned a slot in MMIO hole later in a very same way like for plain PCI BARs, with respect to its size alignment.Also, to reduce code complexity, all long mem/mem64 BAR flags checks are replaced by simple bars[i] field probing, eg.:
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 622
Quoted 4 lines
tools/firmware/hvmloader/pci.c +1
-3-if ( (bar_reg ==PCI_ROM_ADDRESS ) || - ((bar_data &PCI_BASE_ADDRESS_SPACE ) == -PCI_BASE_ADDRESS_SPACE_MEMORY ) ) +if ( bars[i].is_mem )
This should be a separate change IMO.
Quoted 13 lines
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127 ++++++++++++++++++++++++++++-------- tools/firmware/hvmloader/pci_regs.h | 2 + 3 files changed, 106 insertions(+), 27 deletions(-)tools/firmware/hvmloader/config.h +0
-0
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 10 lines
tools/firmware/hvmloader/config.h +3
-0@@ -53,10 +53,14 @@ extern uint8_t ioapic_version;#define PCI_ISA_DEVFN 0x08/* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U/* ISA IRQs 5,10,11 are PCI connected */ #define PCI_ICH9_LPC_DEVFN 0xf8/* dev 31, fn 0 */ +#define PCI_MCH_DEVFN 0/* bus 0, dev 0, func 0 */ /* MMIO hole: Hardcoded defaults, which can be dynamically expanded. */ #define PCI_MEM_END 0xfc000000 +/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64
What the reasoning for this value? Do we know which devices need ECAM areas?
Quoted 23 lines
tools/firmware/hvmloader/config.h +1
-0+#define ACPI_TIS_HDR_ADDRESS 0xFED40F00ULextern unsigned long pci_mem_start, pci_mem_end;tools/firmware/hvmloader/pci.c +2
-1@@ -158,9 +158,10 @@ static void class_specific_pci_device_setup(uint16_t vendor_id,void pci_setup (void ) { -uint8_t is_64bar, using_64bar, bar64_relocate =0 ; +uint8_t is_64bar, using_64bar, bar64_relocate =0 , is_mem;uint32_t devfn, bar_reg, cmd, bar_data, bar_data_upper;uint64_t base, bar_sz, bar_sz_upper, mmio_total =0 ; +uint64_t addr_mask;uint16_t vendor_id, device_id;unsigned int bar, pin, link, isa_irq;int is_running_on_q35 =0 ;@@ -172,10 +173,14 @@ void pci_setup(void)/* Create a list of device BARs in descending order of size. */
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 183
Quoted 10 lines
tools/firmware/hvmloader/pci.c +5
-1struct bars { -uint32_t is_64bar;uint32_t devfn;uint32_t bar_reg;uint64_t bar_sz; +uint64_t addr_mask;/* which bits of the base address can be written */ +uint32_t bar_data;/* initial value - BAR flags here */ +uint8_t is_64bar; +uint8_t is_mem; +uint8_t padding[2 ];
Why are you manually adding a padding here? Also why not make this fields bool?
Quoted 41 lines
tools/firmware/hvmloader/pci.c +14
-4} *bars = (struct bars *)scratch_start;unsigned int i, nr_bars =0 ;uint64_t mmio_hole_size =0 ;@@ -259,13 +264,21 @@ void pci_setup(void) bar_reg =PCI_ROM_ADDRESS ; bar_data =pci_readl (devfn, bar_reg); + + is_mem = !!(((bar_data &PCI_BASE_ADDRESS_SPACE ) == +PCI_BASE_ADDRESS_SPACE_MEMORY ) || + (bar_reg ==PCI_ROM_ADDRESS )); +if ( bar_reg !=PCI_ROM_ADDRESS ) { - is_64bar = !!((bar_data & (PCI_BASE_ADDRESS_SPACE | -PCI_BASE_ADDRESS_MEM_TYPE_MASK )) == - (PCI_BASE_ADDRESS_SPACE_MEMORY | + is_64bar = !!(is_mem && + ((bar_data &PCI_BASE_ADDRESS_MEM_TYPE_MASK ) ==PCI_BASE_ADDRESS_MEM_TYPE_64 )); +pci_writel (devfn, bar_reg, ~0 ); + + addr_mask = is_mem ?PCI_BASE_ADDRESS_MEM_MASK + :PCI_BASE_ADDRESS_IO_MASK ; }else {@@ -273,28 +286,35 @@ void pci_setup(void)pci_writel (devfn, bar_reg, (bar_data |PCI_ROM_ADDRESS_MASK ) & ~PCI_ROM_ADDRESS_ENABLE ); + + addr_mask =PCI_ROM_ADDRESS_MASK ; } + bar_sz =pci_readl (devfn, bar_reg);pci_writel (devfn, bar_reg, bar_data);if ( bar_reg !=PCI_ROM_ADDRESS ) - bar_sz &= (((bar_data & PCI_BASE_ADDRESS_SPACE) ==
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 302
Quoted 10 lines
tools/firmware/hvmloader/pci.c +4
-4- PCI_BASE_ADDRESS_SPACE_MEMORY) ? -PCI_BASE_ADDRESS_MEM_MASK : - (PCI_BASE_ADDRESS_IO_MASK & 0xffff)); + bar_sz &= is_mem ?PCI_BASE_ADDRESS_MEM_MASK : + (PCI_BASE_ADDRESS_IO_MASK &0xffff ); else bar_sz &= PCI_ROM_ADDRESS_MASK; -if (is_64bar) { + +if (is_64bar)
Coding style (spaces between parentheses).
Quoted 44 lines
tools/firmware/hvmloader/pci.c +13
-7+ { bar_data_upper =pci_readl (devfn, bar_reg +4 );pci_writel (devfn, bar_reg +4 , ~0 ); bar_sz_upper =pci_readl (devfn, bar_reg +4 );pci_writel (devfn, bar_reg +4 , bar_data_upper); bar_sz = (bar_sz_upper <<32 ) | bar_sz; } + bar_sz &= ~(bar_sz -1 );if ( bar_sz ==0 )continue ; +/* leave only memtype/enable bits etc */ + bar_data &= ~addr_mask; +for ( i =0 ; i < nr_bars; i++ )if ( bars[i].bar_sz < bar_sz )break ;@@ -302,14 +322,15 @@ void pci_setup(void)if ( i != nr_bars )memmove (&bars[i+1 ], &bars[i], (nr_bars-i) *sizeof (*bars)); - bars[i].is_64bar = is_64bar; - bars[i].devfn = devfn; - bars[i].bar_reg = bar_reg; - bars[i].bar_sz = bar_sz; + bars[i].is_64bar = is_64bar; + bars[i].is_mem = is_mem; + bars[i].devfn = devfn; + bars[i].bar_reg = bar_reg; + bars[i].bar_sz = bar_sz; + bars[i].addr_mask = addr_mask; + bars[i].bar_data = bar_data; -if ( ((bar_data &PCI_BASE_ADDRESS_SPACE ) == -PCI_BASE_ADDRESS_SPACE_MEMORY ) || - (bar_reg ==PCI_ROM_ADDRESS ) ) +if ( is_mem ) mmio_total += bar_sz; nr_bars++;@@ -339,6 +360,63 @@ void pci_setup(void)pci_writew (devfn,PCI_COMMAND , cmd); }
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 371
Quoted 10 lines
tools/firmware/hvmloader/pci.c +9
-0+/* +* Calculate MMCONFIG area size and squeeze it into the bars array +* for assigning a slot in the MMIO hole +*/ +if (is_running_on_q35) + { +/* disable PCIEXBAR decoding for now */ +pci_writel (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ,0 ); +pci_writel (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ,0 );
I'm afraid I will need some context here, where is the description for the config space of dev 0 fn 0? I don't seem to be able to find it in the ich9 spec.
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 376
Quoted 5 lines
tools/firmware/hvmloader/pci.c +5
-0+ +#define PCIEXBAR_64_BUSES (2 << 1) +#define PCIEXBAR_128_BUSES (1 << 1) +#define PCIEXBAR_256_BUSES (0 << 1) +#define PCIEXBAR_ENABLE (1 << 0)
Why those strange definitions? (0 << 1)? (2 << 1) instead of (1 << 2)?
Quoted 12 lines
tools/firmware/hvmloader/pci.c +12
-0+ +switch (PCI_MAX_MCFG_BUSES ) + { +case 64 : + bar_data =PCIEXBAR_64_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (64 ); +break ; + +case 128 : + bar_data =PCIEXBAR_128_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (128 ); +break ;
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 398
Quoted 10 lines
tools/firmware/hvmloader/pci.c +10
-0+ +case 256 : + bar_data =PCIEXBAR_256_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (256 ); +break ; + +default : +/* unsupported number of buses specified */ +BUG (); + }
I don't see how PCI_MAX_MCFG_BUSES should be used. Is the user supposed to know what value to use at compile time? What about distro packagers?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 03/12] hvmloader: add function to query an emulated machine type (i440/Q35)
In reply to Roger Pau MonnéQuoted 9 lines · On Mon, 19 Mar 2018 12:56:51 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 7 lines · Alexey Gerasimenko
This adds a new function
get_pc_machine_type()which allows to determine the emulated chipset type. Supported return values:-
MACHINE_TYPE_I440
-MACHINE_TYPE_Q35
-MACHINE_TYPE_UNKNOWN, results in the error message being printed followed by callingBUG()in hvmloader.
Quoted 5 lines
This is not correct, the return values are strictly
MACHINE_TYPE_I440orMACHINE_TYPE_Q35. Everything else ends up in aBUG().Also makes me wonder whether this should instead be
init_machine_type, and users should just readmachine_typedirectly.
Completely agree here, get_-style function should normally return a value, not to perform extra checks and call BUG().
Renaming the function to init_machine_type() and replacing get_pc_machine_type() usage to reading the machine_type (extern) variable should be more clear (or, perhaps, one-line function to return its value).
This way we can assume the machine type was successfully validated, hence no need for additional checks for MACHINE_TYPE_UNKNOWN value (and the MACHINE_TYPE_UNKNOWN itself).
Quoted 11 lines
Quoted 10 lines
tools/firmware/hvmloader/pci_regs.h | 5 ++++ tools/firmware/hvmloader/util.c | 47+++++++++++++++++++++++++++++++++++++
tools/firmware/hvmloader/util.h| 8 +++++++ 3 files changed, 60
insertions(+)(quoted diff) +0
-0
b/tools/firmware/hvmloader/pci_regs.hindex7bf2d873ab..ba498b840e100644 ---a/tools/firmware/hvmloader/pci_regs.h
+++b/tools/firmware/hvmloader/pci_regs.h
Quoted 4 lines
Quoted 1 line
+static int
machine_type=MACHINE_TYPE_UNDEFINED;There's no need to init this,
_UNDEFINEDis 0 which is the default value.
Using the explicit initialization with the named constant here merely improves readability. Comparing the enum-style variable later with MACHINE_TYPE_UNDEFINED seems better than comparing it with 0. It's zero difference for a compiler, but makes difference for a human. :)
Besides, it will be converted to enum type anyway, so some named entry for the 'unassigned' value will be appropriate I think.
Quoted 6 lines
Quoted 1 line
+int
get_pc_machine_type(void)You introduce a function that's not used anywhere, and the commit log doesn't mention why this is needed at all. In general I prefer functions to be introduced with at least a caller, or else it needs to be described in the commit message why this is not the case.
There are multiple users, will merge the function with some of its callers (Wei suggested the same).
Quoted 9 lines
Quoted 9 lines
tools/firmware/hvmloader/util.c +9
-0+{ +uint16_t vendor_id; +uint16_t device_id; + +if (machine_type !=MACHINE_TYPE_UNDEFINED ) +return machine_type; + + machine_type =MACHINE_TYPE_UNKNOWN ; +
Quoted 8 lines
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 756
Quoted 5 lines
tools/firmware/hvmloader/util.c +5
-0+ vendor_id =pci_readw (0 ,PCI_VENDOR_ID ); + device_id =pci_readw (0 ,PCI_DEVICE_ID ); + +/* only Intel platforms are emulated currently */ +if (vendor_id ==PCI_VENDOR_ID_INTEL )Should this maybe be a
BUG_ON(vendor_id != PCI_VENDOR_ID_INTEL)then? Note that in this case you end up with a BUG later anyway.
Yes, this is intentional. Non-Intel vendor => unknown machine.
Quoted 19 lines
Quoted 19 lines
tools/firmware/hvmloader/util.c +19
-0+ { +switch (device_id) + { +case PCI_DEVICE_ID_INTEL_82441 : + machine_type =MACHINE_TYPE_I440 ; +printf ("Detected i440 chipset\n" ); +break ; + +case PCI_DEVICE_ID_INTEL_Q35_MCH : + machine_type =MACHINE_TYPE_Q35 ; +printf ("Detected Q35 chipset\n" ); +break ; + +default : +break ; + } + } + +if (machine_type ==MACHINE_TYPE_UNKNOWN )
Quoted 8 lines
tools/firmware/hvmloader/util.c void __bug(char *file, int line) · line 779
Quoted 5 lines
(quoted diff) +2
-0+ { + printf("Unknown emulated chipset encountered, VID=%04Xh,DID=%04Xh\n",
tools/firmware/hvmloader/util.c +2
-0+ vendor_id, device_id); +BUG ();Why not place this in the default switch label? That would allow you to get rid of the
MACHINE_TYPE_UNKNOWNdefine also.
This check outside the switch covers cases (Vendor is not Intel) OR (Vendor is Intel but the host bridge is unknown).
I guess it can be moved into the switch, but it means there will be two copies of printf(VID:DID)/BUG() block -- one for Vendor ID check, second is for Device ID processing. Placing this check outside allows to reuse it for both cases.
Quoted 15 lines
Quoted 15 lines
tools/firmware/hvmloader/util.c +5
-0+ } + +return machine_type; +} +static void validate_hvm_info (struct hvm_info_table *t) {uint8_t *ptr = (uint8_t *)t;(quoted diff) +0
-0
b/tools/firmware/hvmloader/util.hindex7bca6418d2..7c77bedb00100644tools/firmware/hvmloader/util.h +0
-0@@ -100,6 +100,14 @@ void pci_write(uint32_t devfn, uint32_t reg, uint32_t len, uint32_t val); #define pci_writew(devfn, reg, val) pci_write(devfn, reg, 2, (uint16_t)(val)) #define pci_writel(devfn,
Quoted 8 lines
tools/firmware/hvmloader/util.h void pci_write(uint32_t devfn, uint32_t reg, uint32_t len, uint32_t val); · line 107
Quoted 6 lines
tools/firmware/hvmloader/util.h +5
-0reg, val) pci_write(devfn, reg, 4, (uint32_t)(val)) +/* Emulated machine types */ +#define MACHINE_TYPE_UNDEFINED 0 +#define MACHINE_TYPE_I440 1 +#define MACHINE_TYPE_Q35 2 +#define MACHINE_TYPE_UNKNOWN (-1)An enum seems better suited for this.
Agree, + MACHINE_TYPE_UNKNOWN will be dropped in the next version.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Alexey GerasimenkoQuoted 4 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.
I would rather name this machine_chipset or device_model_chipset.
Quoted 8 lines
Note that omitting
device_model_machineparameter means i440 system by default, so the default behavior doesn't change for existing domain config files.Setting
device_model_machineto "q35" sends '-machine q35,accel=xen' argument to QEMU. Unlike i440, there no separate machine type to enable/disable Xen platform device, it is controlled via a machine
But I assume the xen_platform_pci option still works as expected?
Quoted 43 lines
property only. See 'libxl: Xen Platform device support for Q35' patch for a detailed description.
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libxl/libxl_dm.c | 16 ++++++++++------ tools/libxl/libxl_types.idl | 7 +++++++ tools/xl/xl_parse.c | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-)tools/libxl/libxl_dm.c +10
-6@@ -1443,13 +1443,17 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,flexarray_append (dm_args, b_info->extra_pv[i]);break ;case LIBXL_DOMAIN_TYPE_HVM : -if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { -/* Switching here to the machine "pc" which does not add -* the xen-platform device instead of the default "xenfv" machine. -*/ - machinearg =libxl__strdup (gc,"pc,accel=xen" ); +if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); }else { - machinearg =libxl__strdup (gc,"xenfv" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { +/* Switching here to the machine "pc" which does not add +* the xen-platform device instead of the default "xenfv" machine. +*/ + machinearg =libxl__strdup (gc,"pc,accel=xen" ); + }else { + machinearg =libxl__strdup (gc,"xenfv" ); + } }if (b_info->u.hvm.mmio_hole_memkb) { uint64_t max_ram_below_4g = (1ULL << 32) -tools/libxl/libxl_types.idl +0
-0
tools/libxl/libxl_types.idl libxl_device_model_version = Enumeration("device_model_version", [ · line 105
Quoted 5 lines
tools/libxl/libxl_types.idl +2
-0@@ -101,6 +101,12 @@ libxl_device_model_version = Enumeration("device_model_version", [ (2, "QEMU_XEN"), # Upstream based qemu-xen device model ]) +libxl_device_model_machine = Enumeration("device_model_machine", [ + (0, "UNKNOWN"),
Shouldn't this be named DEFAULT?
Quoted 25 lines
tools/libxl/libxl_types.idl +5
-0+ (1, "I440"), + (2, "Q35"), + ]) + libxl_console_type = Enumeration("console_type", [ (0, "UNKNOWN"), (1, "SERIAL"),@@ -491,6 +497,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("device_model_ssid_label", string), # device_model_user is not ready for use yet ("device_model_user", string), + ("device_model_machine", libxl_device_model_machine), # extra parameters pass directly to qemu, NULL terminated ("extra", libxl_string_list),tools/xl/xl_parse.c +2
-0@@ -2110,6 +2110,20 @@ skip_usbdev:xlu_cfg_replace_string (config,"device_model_user" , &b_info->device_model_user,0 ); +if (!xlu_cfg_get_string (config,"device_model_machine" , &buf,0 )) { +if (!strcmp (buf,"i440" )) {
tools/xl/xl_parse.c skip_usbdev: · line 2124
Quoted 10 lines
tools/xl/xl_parse.c +10
-0+ b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_I440 ; + }else if (!strcmp (buf,"q35" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_Q35 ; + }else { +fprintf (stderr, +"Unknown device_model_machine \"%s\" specified\n" , buf); +exit (1 ); + } + }else { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_UNKNOWN ;
That seems to be it's usage. I'm not sure you should explicitly set it in the default case (DEFAULT == 0 already).
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 10/12] libacpi: build ACPI MCFG table if requested
In reply to Alexey GerasimenkoQuoted 27 lines · Alexey Gerasimenko
This adds
construct_mcfg()function to libacpi which allows to build MCFG table for a given mmconfig_addr/mmconfig_len pair if theACPI_HAS_MCFGflag was specified inacpi_configstruct.The maximum bus number is calculated from
mmconfig_lenusingMCFG_SIZE_TO_NUM_BUSESmacro (1MByte of MMIO space per bus).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/acpi2_0.h | 21 +++++++++++++++++++++ tools/libacpi/build.c | 42 ++++++++++++++++++++++++++++++++++++++++++ tools/libacpi/libacpi.h | 4 ++++ 3 files changed, 67 insertions(+)tools/libacpi/acpi2_0.h +5
-0@@ -422,6 +422,25 @@ struct acpi_20_slit { };/* +* PCI Express Memory Mapped Configuration Description Table +*/ +struct mcfg_range_entry { +uint64_t base_address; +uint16_t pci_segment;
tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 439
Quoted 10 lines
tools/libacpi/acpi2_0.h +10
-0+uint8_t start_pci_bus_num; +uint8_t end_pci_bus_num; +uint32_t reserved; +}; + +struct acpi_mcfg { +struct acpi_header header; +uint8_t reserved[8 ]; +struct mcfg_range_entry entries[1 ]; +};
I would define this as:
struct acpi_10_mcfg {
struct acpi_header header;
uint8_t reserved[8];
struct acpi_10_mcfg_entry {
uint64_t base_address;
uint16_t pci_segment;
uint8_t start_pci_bus;
uint8_t end_pci_bus;
uint32_t reserved;
} entries[1];
};tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 441
Quoted 2 lines
tools/libacpi/acpi2_0.h +2
-0+ +#define MCFG_SIZE_TO_NUM_BUSES (size) ((size) >> 20)
I'm not sure the following macro belongs here. This is not directly related to ACPI.
Quoted 27 lines
tools/libacpi/acpi2_0.h +4
-0+ +/* * Table Signatures. */ #define ACPI_2_0_RSDP_SIGNATURE ASCII64('R','S','D',' ','P','T','R',' ')@@ -435,6 +454,7 @@ struct acpi_20_slit {#define ACPI_2_0_WAET_SIGNATURE ASCII32('W','A','E','T')#define ACPI_2_0_SRAT_SIGNATURE ASCII32('S','R','A','T')#define ACPI_2_0_SLIT_SIGNATURE ASCII32('S','L','I','T') +#define ACPI_MCFG_SIGNATURE ASCII32('M','C','F','G')/* * Table revision numbers. @@ -449,6 +469,7 @@ struct acpi_20_slit {#define ACPI_1_0_FADT_REVISION 0x01#define ACPI_2_0_SRAT_REVISION 0x01#define ACPI_2_0_SLIT_REVISION 0x01 +#define ACPI_1_0_MCFG_REVISION 0x01#pragma pack ()tools/libacpi/build.c +0
-0@@ -303,6 +303,37 @@ static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt,return slit;
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 313
Quoted 10 lines
tools/libacpi/build.c +8
-0} +static struct acpi_mcfg *construct_mcfg (struct acpi_ctxt *ctxt, +const struct acpi_config *config) +{ +struct acpi_mcfg *mcfg; + +/* Warning: this code expects that we have only one PCI segment */ + mcfg = ctxt->mem_ops.alloc (ctxt,sizeof (*mcfg),16 ); +if (!mcfg)
Coding style.
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 324
Quoted 11 lines
tools/libacpi/build.c +11
-0+return NULL ; + +memset (mcfg,0 ,sizeof (*mcfg)); + mcfg->header.signature =ACPI_MCFG_SIGNATURE ; + mcfg->header.revision =ACPI_1_0_MCFG_REVISION ; +fixed_strcpy (mcfg->header.oem_id,ACPI_OEM_ID ); +fixed_strcpy (mcfg->header.oem_table_id,ACPI_OEM_TABLE_ID ); + mcfg->header.oem_revision =ACPI_OEM_REVISION ; + mcfg->header.creator_id =ACPI_CREATOR_ID ; + mcfg->header.creator_revision =ACPI_CREATOR_REVISION ; + mcfg->header.length =sizeof (*mcfg);
As said before, if you want to align things, please do it for the whole block.
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 330
Quoted 6 lines
tools/libacpi/build.c +6
-0+ + mcfg->entries[0 ].base_address = config->mmconfig_addr; + mcfg->entries[0 ].pci_segment =0 ; + mcfg->entries[0 ].start_pci_bus_num =0 ; + mcfg->entries[0 ].end_pci_bus_num = +MCFG_SIZE_TO_NUM_BUSES (config->mmconfig_len) -1 ;
Why not pass the start_bus and end_bus values in acpi_config at least?
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 334
Quoted 4 lines
tools/libacpi/build.c +4
-0+ +set_checksum (mcfg, offsetof(struct acpi_header , checksum),sizeof (*mcfg)); + +return mcfg;;
Double ;;
Quoted 14 lines
tools/libacpi/build.c +3
-0+} + static int construct_passthrough_tables(struct acpi_ctxt *ctxt, unsigned long *table_ptrs, int nr_tables,@@ -350,6 +381,7 @@ static int construct_secondary_tables(struct acpi_ctxt *ctxt,struct acpi_20_hpet *hpet;struct acpi_20_waet *waet;struct acpi_20_tcpa *tcpa; +struct acpi_mcfg *mcfg;unsigned char *ssdt;static const uint16_t tis_signature[] = {0x0001 ,0x0001 ,0x0001 };void *lasa;
tools/libacpi/build.c static int construct_secondary_tables(struct acpi_ctxt *ctxt, · line 456
Quoted 8 lines
tools/libacpi/build.c +5
-0@@ -417,6 +449,16 @@ static int construct_secondary_tables(struct acpi_ctxt *ctxt,printf ("CONV disabled\n" ); } +/* MCFG */ +if ( config->table_flags &ACPI_HAS_MCFG ) + { + mcfg =construct_mcfg (ctxt, config); +if (!mcfg)
Coding style.
Quoted 21 lines
tools/libacpi/build.c +5
-0+return -1 ; + + table_ptrs[nr_tables++] = ctxt->mem_ops.v2p (ctxt, mcfg); + } +/* TPM TCPA and SSDT. */ if ( (config->table_flags & ACPI_HAS_TCPA) && (config->tis_hdr[0] == tis_signature[0]) &&tools/libacpi/libacpi.h +1
-0@@ -36,6 +36,7 @@#define ACPI_HAS_8042 (1<<13)#define ACPI_HAS_CMOS_RTC (1<<14)#define ACPI_HAS_SSDT_LAPTOP_SLATE (1<<15) +#define ACPI_HAS_MCFG (1<<16)struct xen_vmemrange ;struct acpi_numa {
tools/libacpi/libacpi.h struct acpi_config { · line 102
Quoted 6 lines
tools/libacpi/libacpi.h +3
-0@@ -96,6 +97,9 @@ struct acpi_config {uint32_t ioapic_base_address;uint16_t pci_isa_irq_mask;uint8_t ioapic_id; + +uint64_t mmconfig_addr; +uint32_t mmconfig_len;
This interface is quite limited because it only allows us to create a single MCFG entry, but since this is not a public interface I guess it doesn't matter that much, it can always be expanded when required.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Alexey GerasimenkoQuoted 16 lines · Alexey Gerasimenko
This patch extends
hvmloader_acpi_build_tables()with code which detects if MMCONFIG is available -- i.e. initialized and enabled (+we're running on Q35), obtains its base address and size and asks libacpi to build MCFG table for it via setting the flagACPI_HAS_MCFGin a manner similar to other optional ACPI tables building.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/util.c | 70 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)tools/firmware/hvmloader/util.c +0
-0
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 789
Quoted 8 lines
tools/firmware/hvmloader/util.c +5
-0@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; } +#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1
PCIEXBAR_ENABLE maybe?
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 796
Quoted 7 lines
tools/firmware/hvmloader/util.c +7
-0+ +static uint64_t mmconfig_get_base (void ) +{ +uint64_t base; +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + + base = reg | (uint64_t )pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ) <<32 ;
Please add parentheses in the above expression.
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 808
Quoted 12 lines
tools/firmware/hvmloader/util.c +12
-0+ +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;
Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
They are very related, and in fact I'm not sure why we need to write this info to the device in patch 7 and then fetch it from the device here. Isn't there an easier way to pass this information? At the end this is all in hvmloader.
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 809
Quoted 1 line
tools/firmware/hvmloader/util.c +1
-0+case 3 :
default:
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 816
Quoted 7 lines
tools/firmware/hvmloader/util.c +7
-0+BUG ();/* a reserved value encountered */ + } + +return base; +} + +static uint32_t mmconfig_get_size (void )
unsigned int or size_t?
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 826
Quoted 10 lines
tools/firmware/hvmloader/util.c +10
-0+{ +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 :return MB (256 ); +case 1 :return MB (128 ); +case 2 :return MB (64 ); +case 3 : +BUG ();/* a reserved value encountered */
Same comments as above about the labels and the case 3 label.
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 837
Quoted 11 lines
tools/firmware/hvmloader/util.c +11
-0+ } + +return 0 ; +} + +static uint32_t mmconfig_is_enabled (void ) +{ +return pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ) &PCIEXBAREN ; +} + +static int is_mmconfig_used (void )
bool
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 841
Quoted 4 lines
tools/firmware/hvmloader/util.c +4
-0+{ +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + { +if (mmconfig_is_enabled () &&mmconfig_get_base ())
Coding style.
Also you can join the conditions:
if ( get_pc_machine_type() == MACHINE_TYPE_Q35 && mmconfig_is_enabled() &&
mmconfig_get_base() )
return true;Looking at this, is it actually a valid state to have mmconfig_is_enabled() == true and mmconfig_get_base() == 0?
Quoted 31 lines
tools/firmware/hvmloader/util.c +13
-0+return 1 ; + } + +return 0 ; +} +static void validate_hvm_info (struct hvm_info_table *t) {uint8_t *ptr = (uint8_t *)t;@@ -993,6 +1056,13 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, config->pci_hi_len = pci_hi_mem_end - pci_hi_mem_start; } +if (is_mmconfig_used () ) + { + config->table_flags |=ACPI_HAS_MCFG ; + config->mmconfig_addr =mmconfig_get_base (); + config->mmconfig_len =mmconfig_get_size (); + } + s =xenstore_read ("platform/generation-id" ,"0:0" );if ( s ) {Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 24 lines · On Mon, 19 Mar 2018 15:58:02 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 12 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration.Quoted 4 lines
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see
IOREQ_TYPE_PCI_CONFIG). With this change you will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.
Quoted 4 lines
Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Adding Paul who knows more about all this.
In which use cases multiple PCI-device emulators are used for a single HVM domain? Is it a proprietary setup?
I assume it is somehow related to this code in xen-hvm.c:
/* Fake a write to port 0xCF8 so that
* the config space access will target the
* correct device model.
*/
val = (1u << 31) | ((req->addr & 0x0f00) <...>
do_outp(0xcf8, 4, val);if yes, similar thing can be made for IOREQ_TYPE_COPY accesses to the emulated MMCONFIG if needed.
In HVM+QEMU case we are not limited to merely passed through devices, most of the observable PCI config space devices belong to one particular QEMU instance. This dictates the overall emulated MMCONFIG layout for a domain which should be in sync to what QEMU emulates via CF8h/CFCh accesses... and between multiple device model instances (if there are any, still not sure what multiple PCI-device emulators you mentioned really are).
Basically, we have an emulated MMCONFIG area of 64/128/256MB size in the MMIO hole of the guest HVM domain. (BTW, this area itself can be considered a feature of the chipset the device model emulates.) It can be relocated to some other place in MMIO hole, this means that QEMU will trap accesses to the specific to the emulated chipset PCIEXBAR register and will issue same MMIO unmap/map calls as for any normal emulated MMIO range.
On the other hand, it won't be easy to provide emulated MMCONFIG translation into IOREQ_TYPE_PCI_CONFIG from Xen side. Xen should know current emulated MMCONFIG area position and size in order to translate (or not) accesses to it into corresponding BDF/reg pair (+whether that area is enabled for decoding or not). This will likely require to introduce new hypercall(s).
The question is if there will be any difference or benefit at all.
It's basically the same emulated MMIO range after all, but in one case we trap accesses to it in Xen and translate them into IOREQ_TYPE_PCI_CONFIG requests.
We have to provide some infrastructure to let Xen know where the device model/guest expects to use the MMCONFIG area (and its size). The device model will need to use this infrastructure, informing Xen of any changes. Also, due to MMCONFIG nature there might be some pitfalls like a necessity to send multiple IOREQ_TYPE_PCI_CONFIG ioreqs caused by a single memory read/write operation.
In another case, we still have an emulated MMIO range, but Xen will send plain IOREQ_TYPE_COPY requests to QEMU which it handles itself. In such case, all code to work with MMCONFIG accesses is available for reuse right away (mmcfg -> pci_* translation in QEMU), no new functionality required neither in Xen or QEMU.
Quoted 13 lines
Quoted 13 lines
One important new field is
addr_mask, which tells which bits of the base address can (should) be written. Different address types (ROM, MMIO BAR, PCIEXBAR) will have differentaddr_maskvalues.For every assignable BAR range we store its size, PCI device BDF (devfn actually) to which it belongs, BAR type (mem/io/mem64) and corresponding register offset in device PCI conf space. This way we can insert MMCONFIG entry into bars array in the same manner like for any other BARs. In this case, the devfn field will point to MCH PCI device and
bar_regwill contain PCIEXBAR register offset. It will be assigned a slot in MMIO hole later in a very same way like for plain PCI BARs, with respect to its size alignment.
Quoted 8 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 622
Quoted 6 lines
Also, to reduce code complexity, all long mem/mem64 BAR flags checks are replaced by simple bars[i] field probing, eg.:
tools/firmware/hvmloader/pci.c +1
-3-if ( (bar_reg ==PCI_ROM_ADDRESS ) || - ((bar_data &PCI_BASE_ADDRESS_SPACE ) == -PCI_BASE_ADDRESS_SPACE_MEMORY ) ) +if ( bars[i].is_mem )This should be a separate change IMO.
OK, no problem.
Quoted 16 lines
Quoted 16 lines
tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127++++++++++++++++++++++++++++--------
tools/firmware/hvmloader/pci_regs.h| 2 + 3 files changed, 106
insertions(+), 27 deletions(-)(quoted diff) +0
-0
b/tools/firmware/hvmloader/config.hindex6fde6b7b60..5443ecd804100644 ---a/tools/firmware/hvmloader/config.h
+++b/tools/firmware/hvmloader/config.h(quoted diff) +0
-0@@ -53,10 +53,14 @@ extern uint8_t ioapic_version; #define PCI_ISA_DEVFN 0x08 /* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U /* ISA IRQs 5,10,11 are PCIconnected */ #define
PCI_ICH9_LPC_DEVFN0xf8/* dev 31, fn 0 */
+#definePCI_MCH_DEVFN0 /* bus 0, dev 0, func 0 */
Quoted 8 lines
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 5 lines
/* MMIO hole: Hardcoded defaults, which can be dynamically
expanded. */ #definePCI_MEM_END0xfc000000tools/firmware/hvmloader/config.h +2
-0+/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64What the reasoning for this value? Do we know which devices need ECAM areas?
Yes, Xen is limited to bus 0 emulation currently, the description states "When multiple PCI buses support for Xen will be implemented, PCI_MAX_MCFG_BUSES may be changed to calculation of the number of buses according to results of the PCI devices enumeration".
I think it might be better to replace 'switch (PCI_MAX_MCFG_BUSES)' with the real code right away, i.e. change it to
'switch (max_bus_num, aligned up to 64/128/256 boundary)', where max_bus_num should be set in PCI device enumeration code in pci_setup(). As we are limited to bus 0 currently, we'll just set it to 0 for now, before/after the PCI device enumeration loop (which should became multi-bus capable eventually).
Quoted 21 lines
Quoted 21 lines
#define
ACPI_TIS_HDR_ADDRESS0xFED40F00ULextern unsigned long
pci_mem_start,pci_mem_end;(quoted diff) +0
-0
b/tools/firmware/hvmloader/pci.cindex033bd20992..6de124bbd5100644tools/firmware/hvmloader/pci.c +2
-1@@ -158,9 +158,10 @@ static void class_specific_pci_device_setup(uint16_t vendor_id, void pci_setup(void) { - uint8_t is_64bar, using_64bar, bar64_relocate = 0; + uint8_t is_64bar, using_64bar, bar64_relocate = 0, is_mem;uint32_t devfn, bar_reg, cmd, bar_data, bar_data_upper;uint64_t base, bar_sz, bar_sz_upper, mmio_total =0 ; +uint64_t addr_mask;uint16_t vendor_id, device_id;unsigned int bar, pin, link, isa_irq;int is_running_on_q35 =0 ;@@ -172,10 +173,14 @@ void pci_setup(void)
Quoted 15 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 183
Quoted 12 lines
tools/firmware/hvmloader/pci.c +1
-1/* Create a list of device BARs in descending order of size. */ struct bars { -uint32_t is_64bar;uint32_t devfn;uint32_t bar_reg;uint64_t bar_sz; +uint64_t addr_mask;/* which bits of the base address can be written */
tools/firmware/hvmloader/pci.c +4
-0+uint32_t bar_data;/* initial value - BAR flags here */ +uint8_t is_64bar; +uint8_t is_mem; +uint8_t padding[2 ];Why are you manually adding a padding here? Also why not make this fields bool?
Just following existing code style, hvmloader/pci.c for some reason prefers to specify uint8_t for bool vars. OK, will change it to bools.
Quoted 39 lines
Quoted 39 lines
} *bars = (struct bars *)scratch_start; unsigned int i, nr_bars = 0; uint64_t mmio_hole_size = 0;tools/firmware/hvmloader/pci.c +14
-3@@ -259,13 +264,21 @@ void pci_setup(void) bar_reg =PCI_ROM_ADDRESS ; bar_data =pci_readl (devfn, bar_reg); + + is_mem = !!(((bar_data &PCI_BASE_ADDRESS_SPACE ) == +PCI_BASE_ADDRESS_SPACE_MEMORY ) || + (bar_reg ==PCI_ROM_ADDRESS )); +if ( bar_reg !=PCI_ROM_ADDRESS ) { - is_64bar = !!((bar_data & (PCI_BASE_ADDRESS_SPACE | -PCI_BASE_ADDRESS_MEM_TYPE_MASK )) == - (PCI_BASE_ADDRESS_SPACE_MEMORY | + is_64bar = !!(is_mem && + ((bar_data &PCI_BASE_ADDRESS_MEM_TYPE_MASK ) ==PCI_BASE_ADDRESS_MEM_TYPE_64 )); +pci_writel (devfn, bar_reg, ~0 ); + + addr_mask = is_mem ?PCI_BASE_ADDRESS_MEM_MASK + :PCI_BASE_ADDRESS_IO_MASK ; }else {@@ -273,28 +286,35 @@ void pci_setup(void)pci_writel (devfn, bar_reg, (bar_data |PCI_ROM_ADDRESS_MASK ) & ~PCI_ROM_ADDRESS_ENABLE ); + + addr_mask =PCI_ROM_ADDRESS_MASK ; } + bar_sz =pci_readl (devfn, bar_reg);pci_writel (devfn, bar_reg, bar_data);
Quoted 14 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 302
Quoted 12 lines
tools/firmware/hvmloader/pci.c +2
-4if ( bar_reg !=PCI_ROM_ADDRESS ) - bar_sz &= (((bar_data &PCI_BASE_ADDRESS_SPACE ) == -PCI_BASE_ADDRESS_SPACE_MEMORY ) ? -PCI_BASE_ADDRESS_MEM_MASK : - (PCI_BASE_ADDRESS_IO_MASK &0xffff )); + bar_sz &= is_mem ? PCI_BASE_ADDRESS_MEM_MASK : + (PCI_BASE_ADDRESS_IO_MASK &
0xffff); elsebar_sz &= PCI_ROM_ADDRESS_MASK;tools/firmware/hvmloader/pci.c +2
-1-if (is_64bar) { + +if (is_64bar)Coding style (spaces between parentheses).
OK, will add.
Quoted 45 lines
Quoted 45 lines
tools/firmware/hvmloader/pci.c +13
-7+ { bar_data_upper =pci_readl (devfn, bar_reg +4 );pci_writel (devfn, bar_reg +4 , ~0 ); bar_sz_upper =pci_readl (devfn, bar_reg +4 );pci_writel (devfn, bar_reg +4 , bar_data_upper); bar_sz = (bar_sz_upper <<32 ) | bar_sz; } + bar_sz &= ~(bar_sz -1 );if ( bar_sz ==0 )continue ; +/* leave only memtype/enable bits etc */ + bar_data &= ~addr_mask; +for ( i =0 ; i < nr_bars; i++ )if ( bars[i].bar_sz < bar_sz )break ;@@ -302,14 +322,15 @@ void pci_setup(void)if ( i != nr_bars )memmove (&bars[i+1 ], &bars[i], (nr_bars-i) *sizeof (*bars)); - bars[i].is_64bar = is_64bar; - bars[i].devfn = devfn; - bars[i].bar_reg = bar_reg; - bars[i].bar_sz = bar_sz; + bars[i].is_64bar = is_64bar; + bars[i].is_mem = is_mem; + bars[i].devfn = devfn; + bars[i].bar_reg = bar_reg; + bars[i].bar_sz = bar_sz; + bars[i].addr_mask = addr_mask; + bars[i].bar_data = bar_data; -if ( ((bar_data &PCI_BASE_ADDRESS_SPACE ) == -PCI_BASE_ADDRESS_SPACE_MEMORY ) || - (bar_reg ==PCI_ROM_ADDRESS ) ) +if ( is_mem ) mmio_total += bar_sz; nr_bars++;@@ -339,6 +360,63 @@ void pci_setup(void)pci_writew (devfn,PCI_COMMAND , cmd); }
Quoted 14 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 371
Quoted 10 lines
tools/firmware/hvmloader/pci.c +2
-0+/* +* Calculate MMCONFIG area size and squeeze it into the bars array
tools/firmware/hvmloader/pci.c +7
-0+* for assigning a slot in the MMIO hole +*/ +if (is_running_on_q35) + { +/* disable PCIEXBAR decoding for now */ +pci_writel (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ,0 ); +pci_writel (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ,0 );I'm afraid I will need some context here, where is the description for the config space of dev 0 fn 0? I don't seem to be able to find it in the ich9 spec.
ICH9 is a south bridge, you need to check the NB/MCH datasheet, namely "Intel® 3 Series Express Chipset Family".
Quoted 7 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 376
Quoted 5 lines
tools/firmware/hvmloader/pci.c +5
-0+ +#define PCIEXBAR_64_BUSES (2 << 1) +#define PCIEXBAR_128_BUSES (1 << 1) +#define PCIEXBAR_256_BUSES (0 << 1) +#define PCIEXBAR_ENABLE (1 << 0)Why those strange definitions? (0 << 1)? (2 << 1) instead of (1 << 2)?
These are bitfields. It's just to show their bitfield nature, bits[2..1] and bit0. I'll change them to something more readable (like shifts with _BITPOS-defines) in non-RFC patches.
Quoted 18 lines
Quoted 18 lines
tools/firmware/hvmloader/pci.c +18
-0+ +switch (PCI_MAX_MCFG_BUSES ) + { +case 64 : + bar_data =PCIEXBAR_64_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (64 ); +break ; + +case 128 : + bar_data =PCIEXBAR_128_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (128 ); +break ; + +case 256 : + bar_data =PCIEXBAR_256_BUSES |PCIEXBAR_ENABLE ; + bar_sz =MB (256 ); +break ; +
Quoted 8 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 398
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4
-0+default : +/* unsupported number of buses specified */ +BUG (); + }I don't see how
PCI_MAX_MCFG_BUSESshould be used. Is the user supposed to know what value to use at compile time? What about distro packagers?
Answered above mostly.
We're limited to bus 0 currently. However, it is possible to change MMCONFIG size manually for now (eg. to 256MB which allows to cover the whole 0-FF bus range).
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Roger Pau MonnéQuoted 22 lines · On Mon, 19 Mar 2018 17:49:09 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 21 lines · Alexey Gerasimenko
This patch extends
hvmloader_acpi_build_tables()with code which detects if MMCONFIG is available -- i.e. initialized and enabled (+we're running on Q35), obtains its base address and size and asks libacpi to build MCFG table for it via setting the flagACPI_HAS_MCFGin a manner similar to other optional ACPI tables building.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/util.c | 70+++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
(quoted diff) +0
-0
b/tools/firmware/hvmloader/util.cindexd8db9e3c8e..c6fc81d52a100644tools/firmware/hvmloader/util.c +0
-0@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; }
Quoted 7 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 789
Quoted 5 lines
tools/firmware/hvmloader/util.c +5
-0+#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1
PCIEXBAR_ENABLEmaybe?
PCIEXBAREN is just an official name of this bit from the Intel datasheet. :) OK, will rename it to PCIEXBAR_ENABLE.
Quoted 10 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void)
Quoted 8 lines
tools/firmware/hvmloader/util.c +7
-0+ +static uint64_t mmconfig_get_base (void ) +{ +uint64_t base; +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + + base = reg | (uint64_t )pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ) <<32 ;Please add parentheses in the above expression.
Agree, parentheses will make the op priority clearer.
Quoted 13 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 808
Quoted 12 lines
tools/firmware/hvmloader/util.c +12
-0+ +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;
Quoted 3 lines
Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
I think all these #defines should find a way to pci_regs.h, it seems like an appropriate place for them.
Regarding the order of hvmloader patches -- will verify this for the next version.
Quoted 4 lines
They are very related, and in fact I'm not sure why we need to write this info to the device in patch 7 and then fetch it from the device here. Isn't there an easier way to pass this information? At the end this is all in hvmloader.
Well, the hvmloader_acpi_build_tables() function mostly does device probing (using I/O instruction) and xenstore reads to collect system information in order to discover which ACPI_HAS_* flags it should pass to acpi_build_tables(), but using global variables to pass this kind of information for MMCONFIG will be OK too I think.
Quoted 3 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 810
Quoted 1 line
tools/firmware/hvmloader/util.c +1
-0+case 3 :default:
There is '& 3' for the switch argument, but ok I guess, it's clearer with 'default'.
Quoted 9 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 816
Quoted 7 lines
tools/firmware/hvmloader/util.c +7
-0+BUG ();/* a reserved value encountered */ + } + +return base; +} + +static uint32_t mmconfig_get_size (void )unsigned int or
size_t?
Using types which are common to the existing code.
size_t have almost zero use in hvmloader.
unsigned int instead of uint32_t... well, the uint32_t still used more often as a type name anyway, but I have no objections to either choice.
Quoted 12 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 826
Quoted 10 lines
tools/firmware/hvmloader/util.c +10
-0+{ +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 :return MB (256 ); +case 1 :return MB (128 ); +case 2 :return MB (64 ); +case 3 : +BUG ();/* a reserved value encountered */ Same comments as above about the labels and the case 3 label.
Quoted 13 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 837
Quoted 11 lines
tools/firmware/hvmloader/util.c +11
-0+ } + +return 0 ; +} + +static uint32_t mmconfig_is_enabled (void ) +{ +return pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ) &PCIEXBAREN ; +} + +static int is_mmconfig_used (void )bool
OK
Quoted 9 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 841
Quoted 4 lines
tools/firmware/hvmloader/util.c +4
-0+{ +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + { +if (mmconfig_is_enabled () &&mmconfig_get_base ())Coding style.
Also you can join the conditions:
Quoted 7 lines
if (
get_pc_machine_type()==MACHINE_TYPE_Q35&&mmconfig_is_enabled()&&mmconfig_get_base() ) return true;Looking at this, is it actually a valid state to have
mmconfig_is_enabled()== true andmmconfig_get_base()== 0?
Yes, in theory we can have either PCIEXBAREN=0 and a valid PCIEXBAR base, or vice versa.
Of course normally we should not encounter a situation where base=0 and PCIEXBAREN=1, just covering here possible cases which the register format allows.
Regarding check merging -- ok, sure. Short-circuit evaluation should guaranty that these registers are not touched on a different machine.
Quoted 13 lines
Quoted 13 lines
tools/firmware/hvmloader/util.c +6
-0+return 1 ; + } + +return 0 ; +} +static void validate_hvm_info (struct hvm_info_table *t) {uint8_t *ptr = (uint8_t *)t;tools/firmware/hvmloader/util.c +0
-0@@ -993,6 +1056,13 @@ void hvmloader_acpi_build_tables(struct
acpi_config*config,config->pci_hi_len=pci_hi_mem_end-pci_hi_mem_start; }
Quoted 7 lines
tools/firmware/hvmloader/util.c void hvmloader_acpi_build_tables(struct acpi_config *config, · line 1065
Quoted 7 lines
tools/firmware/hvmloader/util.c +7
-0+if (is_mmconfig_used () ) + { + config->table_flags |=ACPI_HAS_MCFG ; + config->mmconfig_addr =mmconfig_get_base (); + config->mmconfig_len =mmconfig_get_size (); + } +
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 10/12] libacpi: build ACPI MCFG table if requested
In reply to Roger Pau MonnéQuoted 41 lines · On Mon, 19 Mar 2018 17:33:34 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 27 lines · Alexey Gerasimenko
This adds
construct_mcfg()function to libacpi which allows to build MCFG table for a given mmconfig_addr/mmconfig_len pair if theACPI_HAS_MCFGflag was specified inacpi_configstruct.The maximum bus number is calculated from
mmconfig_lenusingMCFG_SIZE_TO_NUM_BUSESmacro (1MByte of MMIO space per bus).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/acpi2_0.h | 21 +++++++++++++++++++++ tools/libacpi/build.c | 42++++++++++++++++++++++++++++++++++++++++++
tools/libacpi/libacpi.h| 4 ++++ 3 files changed, 67 insertions(+)tools/libacpi/acpi2_0.h +5
-0@@ -422,6 +422,25 @@ struct acpi_20_slit { };/* +* PCI Express Memory Mapped Configuration Description Table +*/ +struct mcfg_range_entry { +uint64_t base_address; +uint16_t pci_segment;tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 439
Quoted 10 lines
tools/libacpi/acpi2_0.h +10
-0+uint8_t start_pci_bus_num; +uint8_t end_pci_bus_num; +uint32_t reserved; +}; + +struct acpi_mcfg { +struct acpi_header header; +uint8_t reserved[8 ]; +struct mcfg_range_entry entries[1 ]; +};I would define this as:
Quoted 11 lines
struct acpi_10_mcfg {
struct acpi_header header;
uint8_t reserved[8];
struct acpi_10_mcfg_entry {
uint64_t base_address;
uint16_t pci_segment;
uint8_t start_pci_bus;
uint8_t end_pci_bus;
uint32_t reserved;
} entries[1];
};Hmm, a choice of preference, but OK, will move it inside.
Quoted 5 lines
tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 441
Quoted 2 lines
tools/libacpi/acpi2_0.h +2
-0+ +#define MCFG_SIZE_TO_NUM_BUSES (size) ((size) >> 20)I'm not sure the following macro belongs here. This is not directly related to ACPI.
Yeah, pci_regs.h might be better I think.
Quoted 29 lines
Quoted 29 lines
tools/libacpi/acpi2_0.h +2
-0+ +/* * Table Signatures. */ #define ACPI_2_0_RSDP_SIGNATURE ASCII64('R','S','D','','P','T','R',' ') @@ -435,6 +454,7 @@ struct
acpi_20_slit{
#defineACPI_2_0_WAET_SIGNATUREASCII32('W','A','E','T')#defineACPI_2_0_SRAT_SIGNATUREASCII32('S','R','A','T')#defineACPI_2_0_SLIT_SIGNATUREASCII32('S','L','I','T')tools/libacpi/acpi2_0.h +2
-0+#define ACPI_MCFG_SIGNATURE ASCII32('M','C','F','G')/* * Table revision numbers. @@ -449,6 +469,7 @@ struct acpi_20_slit {#define ACPI_1_0_FADT_REVISION 0x01#define ACPI_2_0_SRAT_REVISION 0x01#define ACPI_2_0_SLIT_REVISION 0x01 +#define ACPI_1_0_MCFG_REVISION 0x01#pragma pack ()tools/libacpi/build.c +0
-0@@ -303,6 +303,37 @@ static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, return slit; }
Quoted 11 lines
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 313
Quoted 9 lines
tools/libacpi/build.c +7
-0+static struct acpi_mcfg *construct_mcfg (struct acpi_ctxt *ctxt, +const struct acpi_config *config) +{ +struct acpi_mcfg *mcfg; + +/* Warning: this code expects that we have only one PCI segment */ + mcfg = ctxt->mem_ops.alloc (ctxt,sizeof (*mcfg),16 ); +if (!mcfg)Coding style.
OK
Quoted 12 lines
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 324
Quoted 11 lines
tools/libacpi/build.c +11
-0+return NULL ; + +memset (mcfg,0 ,sizeof (*mcfg)); + mcfg->header.signature =ACPI_MCFG_SIGNATURE ; + mcfg->header.revision =ACPI_1_0_MCFG_REVISION ; +fixed_strcpy (mcfg->header.oem_id,ACPI_OEM_ID ); +fixed_strcpy (mcfg->header.oem_table_id,ACPI_OEM_TABLE_ID ); + mcfg->header.oem_revision =ACPI_OEM_REVISION ; + mcfg->header.creator_id =ACPI_CREATOR_ID ; + mcfg->header.creator_revision =ACPI_CREATOR_REVISION ; + mcfg->header.length =sizeof (*mcfg);
Quoted 2 lines
As said before, if you want to align things, please do it for the whole block.
Agree, will reorder lines.
Quoted 8 lines
tools/libacpi/build.c static struct acpi_20_slit *construct_slit(struct acpi_ctxt *ctxt, · line 330
Quoted 6 lines
tools/libacpi/build.c +6
-0+ + mcfg->entries[0 ].base_address = config->mmconfig_addr; + mcfg->entries[0 ].pci_segment =0 ; + mcfg->entries[0 ].start_pci_bus_num =0 ; + mcfg->entries[0 ].end_pci_bus_num = +MCFG_SIZE_TO_NUM_BUSES (config->mmconfig_len) -1 ;Why not pass the
start_busandend_busvalues inacpi_configat least?
start_pci_bus_num will be always 0.
It will be kinda ugly to pass config->mmconfig_addr along with config->end_pci_bus_num, baseaddr+size combo looks nicer I think.
Quoted 6 lines
Quoted 4 lines
tools/libacpi/build.c +2
-0+ + set_checksum(mcfg, offsetof(struct acpi_header, checksum),
sizeof(*mcfg)); +
+ return mcfg;;Double ;;
Oops, missed this one.
Quoted 17 lines
Quoted 17 lines
tools/libacpi/build.c +3
-0+} + static int construct_passthrough_tables(struct acpi_ctxt *ctxt, unsigned long *table_ptrs, int nr_tables,@@ -350,6 +381,7 @@ static int construct_secondary_tables(struct acpi_ctxt *ctxt, struct acpi_20_hpet *hpet;struct acpi_20_waet *waet;struct acpi_20_tcpa *tcpa; +struct acpi_mcfg *mcfg;unsigned char *ssdt;static const uint16_t tis_signature[] = {0x0001 ,0x0001 ,0x0001 };void *lasa;@@ -417,6 +449,16 @@ static int construct_secondary_tables(struct acpi_ctxt *ctxt,printf ("CONV disabled\n" ); }
Quoted 7 lines
tools/libacpi/build.c static int construct_secondary_tables(struct acpi_ctxt *ctxt, · line 456
Quoted 5 lines
tools/libacpi/build.c +5
-0+/* MCFG */ +if ( config->table_flags &ACPI_HAS_MCFG ) + { + mcfg =construct_mcfg (ctxt, config); +if (!mcfg)Coding style.
Will fix.
Quoted 18 lines
Quoted 18 lines
tools/libacpi/build.c +5
-0+return -1 ; + + table_ptrs[nr_tables++] = ctxt->mem_ops.v2p (ctxt, mcfg); + } +/* TPM TCPA and SSDT. */ if ( (config->table_flags & ACPI_HAS_TCPA) && (config->tis_hdr[0] == tis_signature[0]) &&tools/libacpi/libacpi.h +1
-0@@ -36,6 +36,7 @@#define ACPI_HAS_8042 (1<<13)#define ACPI_HAS_CMOS_RTC (1<<14)#define ACPI_HAS_SSDT_LAPTOP_SLATE (1<<15) +#define ACPI_HAS_MCFG (1<<16)
Quoted 13 lines
tools/libacpi/libacpi.h struct acpi_config { · line 102
Quoted 9 lines
tools/libacpi/libacpi.h +3
-0struct xen_vmemrange ;struct acpi_numa {@@ -96,6 +97,9 @@ struct acpi_config {uint32_t ioapic_base_address;uint16_t pci_isa_irq_mask;uint8_t ioapic_id; + +uint64_t mmconfig_addr; +uint32_t mmconfig_len;This interface is quite limited because it only allows us to create a single MCFG entry, but since this is not a public interface I guess it doesn't matter that much, it can always be expanded when required.
We will be limited to a single MMCONFIG area for a long time I'm afraid, it will good to move away from the bus 0 limitation first.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Roger Pau MonnéQuoted 8 lines · On Mon, 19 Mar 2018 17:01:18 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 5 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.I would rather name this
machine_chipsetordevice_model_chipset.
device_model_ prefix is a must I think -- multiple device model related options have names starting with device_model_.
device_model_chipset... well, maybe, but we're actually specifying a QEMU machine here. In QEMU mailing list there was even a suggestion to allow to pass a machine version number here, like "pc-q35-2.10". I think some opinions are needed here.
Quoted 11 lines
Quoted 9 lines
Note that omitting
device_model_machineparameter means i440 system by default, so the default behavior doesn't change for existing domain config files.Setting
device_model_machineto "q35" sends '-machine q35,accel=xen' argument to QEMU. Unlike i440, there no separate machine type to enable/disable Xen platform device, it is controlled via a machineBut I assume the
xen_platform_pcioption still works as expected?
Yes, xen_platform_pci should work as before.
Quoted 51 lines
Quoted 51 lines
property only. See 'libxl: Xen Platform device support for Q35' patch for a detailed description.
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libxl/libxl_dm.c | 16 ++++++++++------ tools/libxl/libxl_types.idl | 7 +++++++ tools/xl/xl_parse.c | 14 ++++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-)tools/libxl/libxl_dm.c +0
-3@@ -1443,13 +1443,17 @@ static intlibxl__build_device_model_args_new (libxl__gc *gc, flexarray_append(dm_args, b_info->extra_pv[i]); break;case LIBXL_DOMAIN_TYPE_HVM : -if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { -/* Switching here to the machine "pc" which does not add -* the xen-platform device instead of the default "xenfv" machine.
tools/libxl/libxl_dm.c +1
-2-*/ - machinearg =libxl__strdup (gc,"pc,accel=xen" ); + if (b_info->device_model_machine ==
LIBXL_DEVICE_MODEL_MACHINE_Q35) {tools/libxl/libxl_dm.c +2
-1+ machinearg =libxl__sprintf (gc,"q35,accel=xen" ); }else { - machinearg =libxl__strdup (gc,"xenfv" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)){
+ /* Switching here to the machine "pc" which does
not add
+ * the xen-platform device instead of the default
"xenfv" machine.tools/libxl/libxl_dm.c +5
-0+*/ + machinearg =libxl__strdup (gc,"pc,accel=xen" ); + }else { + machinearg =libxl__strdup (gc,"xenfv" ); + } }if (b_info->u.hvm.mmio_hole_memkb) { uint64_t max_ram_below_4g = (1ULL << 32) -(quoted diff) +0
-0b/tools/libxl/libxl_types.idl index
35038120ca..f3ef3cbdde100644tools/libxl/libxl_types.idl +0
-0@@ -101,6 +101,12 @@ libxl_device_model_version =Enumeration("device_model_version", [ (2, "
QEMU_XEN"), #
Upstream based qemu-xen device model ])
Quoted 4 lines
tools/libxl/libxl_types.idl libxl_device_model_version = Enumeration("device_model_version", [ · line 105
Quoted 2 lines
tools/libxl/libxl_types.idl +2
-0+libxl_device_model_machine = Enumeration("device_model_machine", [ + (0, "UNKNOWN"),Shouldn't this be named DEFAULT?
"Unknown" here should be read as "unspecified", but I guess DEFAULT will be clearer anyway.
Quoted 35 lines
Quoted 35 lines
tools/libxl/libxl_types.idl +5
-0+ (1, "I440"), + (2, "Q35"), + ]) + libxl_console_type = Enumeration("console_type", [ (0, "UNKNOWN"), (1, "SERIAL"),@@ -491,6 +497,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ ("device_model_ssid_label", string), # device_model_user is not ready for use yet ("device_model_user", string), + ("device_model_machine", libxl_device_model_machine), # extra parameters pass directly to qemu, NULL terminated ("extra", libxl_string_list),tools/xl/xl_parse.c +8
-0@@ -2110,6 +2110,20 @@ skip_usbdev:xlu_cfg_replace_string (config,"device_model_user" , &b_info->device_model_user,0 ); +if (!xlu_cfg_get_string (config,"device_model_machine" , &buf,0 )) { +if (!strcmp (buf,"i440" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_I440 ; + }else if (!strcmp (buf,"q35" )) { + b_info->device_model_machine =LIBXL_DEVICE_MODEL_MACHINE_Q35 ; + }else { + fprintf(stderr, + "Unknown device_model_machine \"%s\"specified\n", buf);
Quoted 8 lines
Quoted 5 lines
tools/xl/xl_parse.c +4
-0+exit (1 ); + } + }else { + b_info->device_model_machine =
LIBXL_DEVICE_MODEL_MACHINE_UNKNOWN;That seems to be it's usage. I'm not sure you should explicitly set it in the default case (DEFAULT == 0 already).
Will check this, although setting the variable value explicitly is good for code readability I think.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 06/12] hvmloader: add basic Q35 support
In reply to Roger Pau MonnéQuoted 40 lines · On Mon, 19 Mar 2018 15:30:14 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 39 lines · Alexey Gerasimenko
This patch does following:
1. Move PCI-device specific initialization out of
pci_setupfunction to the newly createdclass_specific_pci_device_setupfunction to simplify code.2. PCI-device specific initialization extended with LPC controller initialization
3. Initialize PIRQA...{PIRQD, PIRQH} routing accordingly to the emulated south bridge (either located on
PCI_ISA_DEVFNorPCI_ICH9_LPC_DEVFN).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/config.h | 1 + tools/firmware/hvmloader/pci.c | 162++++++++++++++++++++++++-------------- 2 files changed, 104 insertions(+), 59 deletions(-)
(quoted diff) +0
-0
b/tools/firmware/hvmloader/config.hindex6e00413f2e..6fde6b7b60100644 ---a/tools/firmware/hvmloader/config.h
+++b/tools/firmware/hvmloader/config.htools/firmware/hvmloader/config.h +0
-0@@ -52,6 +52,7 @@ extern uint8_t ioapic_version;#define PCI_ISA_DEVFN 0x08/* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U/* ISA IRQs 5,10,11 are PCI connected */ +#define
PCI_ICH9_LPC_DEVFN0xf8/* dev 31, fn 0 *//* MMIO hole: Hardcoded defaults, which can be dynamically
expanded. */ #definePCI_MEM_END0xfc000000(quoted diff) +0
-0
b/tools/firmware/hvmloader/pci.cindex0b708bf578..033bd20992100644tools/firmware/hvmloader/pci.c +0
-0@@ -35,6 +35,7 @@ unsigned long pci_mem_end = PCI_MEM_END;uint64_t pci_hi_mem_start =0 , pci_hi_mem_end =0 ;
Quoted 5 lines
tools/firmware/hvmloader/pci.c unsigned long pci_mem_end = PCI_MEM_END; · line 38
Quoted 2 lines
tools/firmware/hvmloader/pci.c +1
-0enum virtual_vga virtual_vga = VGA_none; +uint32_t vga_devfn =256 ;
uint8_tshould be enough to store a devfn. Also this should be static maybe?
Yep, forgot 'static'. Changing uint32_t to uint8_t here will require to change the
' if ( vga_devfn != 256 )' condition as well -- it's a bit out of the
patch scope, probably a separate tiny patch would be better.
Quoted 7 lines
Quoted 7 lines
tools/firmware/hvmloader/pci.c +0
-0unsigned long igd_opregion_pgbase =0 ;/* Check if the specified range conflicts with any reserved device memory. */ @@ -76,14 +77,93 @@ static int
find_next_rmrr(uint32_t base)returnnext_rmrr;
}
Quoted 5 lines
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 82
Quoted 3 lines
tools/firmware/hvmloader/pci.c +3
-0+#define SCI_EN_IOPORT (ACPI_PM1A_EVT_BLK_ADDRESS_V1 + 0x30) +#define GBL_SMI_EN (1 << 0) +#define APMC_EN (1 << 5)Alignment.
Will correct.
Quoted 13 lines
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 92
Quoted 10 lines
tools/firmware/hvmloader/pci.c +4
-0+ +static void class_specific_pci_device_setup (uint16_t vendor_id, +uint16_t device_id, + uint8_t bus, uint8_tdevfn) +{
tools/firmware/hvmloader/pci.c +5
-0+uint16_t class; + + class =pci_readw (devfn,PCI_CLASS_DEVICE ); + + switch ( class )switch (
pci_readw(devfn, PCI_CLASS_DEVICE)) ?
Quoted 4 lines
I don't see class being used elsewhere.
Also why is vendor_id/device_id provided by the caller but not class? It seems kind of pointless.
'class' is not used by pci_setup(), thus moved toclass_specific_pci_device_setup().
pci_readw(devfn, PCI_CLASS_DEVICE) inside the switch condition to drop the variable -- sure, agree.
Passing vendor_id/device_id pair via function args allows to avoid reading the vendor_id/device_id from PCI conf twice -- a bit less garbage in the polluted PCI setup debuglog. It's not a big problem really, so this can be changed to passing only BDF to class_specific_pci_device_setup().
Quoted 3 lines
Why not fetch vendor/device from the function itself and move the (
vendor_id==0xffff) && (device_id==0xffff) check inside the function?
Hmm, this is a part of the PCI bus enumeration, not PCI device setup.
Quoted 6 lines
Also in this case I think it would be better to have a non-functional patch that introduces
class_specific_pci_device_setupand a second patch that adds support for ICH9.Having code movement and new code in the same patch makes it harder to very what you are actually moving vs introducing.
Agree, will split this actions to separate patches for the next version.
Quoted 4 lines
tools/firmware/hvmloader/pci.c
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2
-0+ { +case 0x0300 :All this values need to be defines documented somewhere.
Agree... although it was not me who introduced all these hardcoded PCI class values. :) I'll change these numbers into newly added pci_regs.h #defines in the non-functional patch.
Quoted 74 lines
Quoted 74 lines
tools/firmware/hvmloader/pci.c +18
-0+/* If emulated VGA is found, preserve it as primary VGA. */ +if ( (vendor_id ==0x1234 ) && (device_id ==0x1111 ) ) + { + vga_devfn = devfn; + virtual_vga = VGA_std; + } +else if ( (vendor_id ==0x1013 ) && (device_id ==0xb8 ) ) + { + vga_devfn = devfn; + virtual_vga = VGA_cirrus; + } +else if ( virtual_vga == VGA_none ) + { + vga_devfn = devfn; + virtual_vga = VGA_pt; +if ( vendor_id ==0x8086 ) + { + igd_opregion_pgbase =
mem_hole_alloc(IGD_OPREGION_PAGES);tools/firmware/hvmloader/pci.c +2
-0+/* +* Write the the OpRegion offset to give the opregion
+ * address to the device model. The device model
will traptools/firmware/hvmloader/pci.c +10
-0+* and map the OpRegion at the give address. +*/ +pci_writel (vga_devfn,PCI_INTEL_OPREGION , + igd_opregion_pgbase <<PAGE_SHIFT ); + } + } +break ; + +case 0x0680 : +/* PIIX4 ACPI PM. Special device with special PCI config space. */
tools/firmware/hvmloader/pci.c +17
-0+ASSERT ((vendor_id ==0x8086 ) && (device_id ==0x7113 )); +pci_writew (devfn,0x20 ,0x0000 );/* No smb bus IO enable */ +pci_writew (devfn,0xd2 ,0x0000 );/* No smb bus IO enable */ +pci_writew (devfn,0x22 ,0x0000 ); +pci_writew (devfn,0x3c ,0x0009 );/* Hardcoded IRQ9 */ +pci_writew (devfn,0x3d ,0x0001 ); +pci_writel (devfn,0x40 ,ACPI_PM1A_EVT_BLK_ADDRESS_V1 |1 ); +pci_writeb (devfn,0x80 ,0x01 );/* enable PM io space */ +break ; + +case 0x0601 : +/* LPC bridge */ +if (vendor_id ==0x8086 && device_id ==0x2918 ) + { +pci_writeb (devfn,0x3c ,0x09 );/* Hardcoded IRQ9 */ +pci_writeb (devfn,0x3d ,0x01 ); + pci_writel(devfn, 0x40, ACPI_PM1A_EVT_BLK_ADDRESS_V1 |1);
tools/firmware/hvmloader/pci.c +2
-0+pci_writeb (devfn,0x44 ,0x80 );/* enable PM io space */ + outl(SCI_EN_IOPORT, inl(SCI_EN_IOPORT) | GBL_SMI_EN |
APMC_EN);tools/firmware/hvmloader/pci.c +14
-0+ } +break ; + +case 0x0101 : +if ( vendor_id ==0x8086 ) + { +/* Intel ICHs since PIIX3: enable IDE legacy mode. */ +pci_writew (devfn,0x40 ,0x8000 );/* enable IDE0 */ +pci_writew (devfn,0x42 ,0x8000 );/* enable IDE1 */ + } +break ; + } +} +void pci_setup (void ) {uint8_t is_64bar, using_64bar, bar64_relocate =0 ;uint32_t devfn, bar_reg, cmd, bar_data, bar_data_upper;
Quoted 8 lines
tools/firmware/hvmloader/pci.c static int find_next_rmrr(uint32_t base) · line 166
Quoted 6 lines
uint64_t base, bar_sz, bar_sz_upper, mmio_total = 0;tools/firmware/hvmloader/pci.c +2
-2-uint32_t vga_devfn =256 ; -uint16_t class, vendor_id, device_id; +uint16_t vendor_id, device_id;unsigned int bar, pin, link, isa_irq; +int is_running_on_q35 =0 ;bool
is_running_on_q35= (get_pc_machine_type()==MACHINE_TYPE_Q35);
OK
Quoted 7 lines
Quoted 7 lines
/* Resources assignable to PCI devices via BARs. */ struct resource {tools/firmware/hvmloader/pci.c +0
-0@@ -130,13 +210,28 @@ void pci_setup(void)if ( s ) mmio_hole_size =strtoll (s,NULL ,0 );
Quoted 14 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 223
Quoted 12 lines
tools/firmware/hvmloader/pci.c +5
-1+/* check if we are on Q35 and set the flag if it is the case */ + is_running_on_q35 =get_pc_machine_type () ==MACHINE_TYPE_Q35 ; +/* Program PCI-ISA bridge with appropriate link routes. */ isa_irq =0 ;for ( link =0 ; link <4 ; link++ ) {do { isa_irq = (isa_irq +1 ) &15 ; }while ( !(PCI_ISA_IRQ_MASK & (1U << isa_irq)) ); -pci_writeb (PCI_ISA_DEVFN ,0x60 + link, isa_irq); + +if (is_running_on_q35)Coding style.
OK
Quoted 10 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 228
Quoted 5 lines
tools/firmware/hvmloader/pci.c +5
-0+ { +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x60 + link, isa_irq); + +/* PIRQE..PIRQH are unused */ +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x68 + link,0x80 );According to the spec
0x80is the default value for this registers, do you really need to write it?Is maybe QEMU not correctly setting the default value?
Won't agree here. We're initializing PIRQ[n] routing in this fragment, it's better not to rely on any values but simply initialize all PIRQ[n]_ROUT registers, this makes it explicit.
Even if it is unnecessary due to defaults it's more obvious to set these registers to our own values than to force a reader to either look up their emulation in QEMU code or read the ICH9 pdf to confirm assumptions.
Quoted 6 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 232
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4
-0+ } +else + { +pci_writeb (PCI_ISA_DEVFN ,0x60 + link, isa_irq);Is all this magic described somewhere that you can reference?
It's setting up PCI interrupt routing for PIC mode. All this PIRQ[n]_ROUT stuff basically needed for legacy compatibility only, normally we deal with APIC mode (+MSIs).
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 04/12] hvmloader: add ACPI enabling for Q35
In reply to Roger Pau MonnéQuoted 33 lines · On Mon, 19 Mar 2018 13:01:58 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 19 lines · Alexey Gerasimenko
In order to turn on ACPI for OS, we need to write a chipset-specific value to
SMI_CMDregister (sort of imitation of theAPM->ACPIswitch on real systems). Modifyacpi_enable_sci()function to support both i440 and Q35 emulation.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/hvmloader.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/tools/firmware/hvmloader/hvmloader.cindexf603f68ded..070698440e100644 ---a/tools/firmware/hvmloader/hvmloader.c
+++b/tools/firmware/hvmloader/hvmloader.ctools/firmware/hvmloader/hvmloader.c +1
-0@@ -257,9 +257,16 @@ static const struct bios_config *detect_bios(void) static void acpi_enable_sci(void) {uint8_t pm1a_cnt_val; +uint8_t acpi_enable_val;tools/firmware/hvmloader/hvmloader.c static const struct bios_config · line 269
Quoted 10 lines
tools/firmware/hvmloader/hvmloader.c +7
-1-#define PIIX4_SMI_CMD_IOPORT 0xb2 +#define SMI_CMD_IOPORT 0xb2#define PIIX4_ACPI_ENABLE 0xf1 +#define ICH9_ACPI_ENABLE 0x02 + +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + acpi_enable_val =ICH9_ACPI_ENABLE ; +else + acpi_enable_val =PIIX4_ACPI_ENABLE ;Coding style, but I would rather:
Quoted 9 lines
switch (
get_pc_machine_type())
{
caseMACHINE_TYPE_Q35:
...
caseMACHINE_TYPE_I440:
...
default:BUG();
}
Agree, a better code maintainability.
Quoted 2 lines
I think storing the machine type in a global variable is better than calling
get_pc_machine_typeeach time.
OK, will switch to it.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 05/12] hvmloader: add Q35 DSDT table loading
In reply to Roger Pau MonnéQuoted 14 lines · On Mon, 19 Mar 2018 14:45:29 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 13 lines · Alexey Gerasimenko
Allows to select Q35 DSDT table in
hvmloader_acpi_build_tables(). Functionget_pc_machine_type()is used to select a proper table (i440/q35).As we are bound to the qemu-xen device model for Q35, no need to initialize config->dsdt_15cpu/config->dsdt_15cpu_len fields.
Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/util.c | 13 +++++++++++-- tools/firmware/hvmloader/util.h | 2 ++ 2 files changed, 13 insertions(+), 2 deletions(-)
Quoted 14 lines
tools/firmware/hvmloader/util.c void hvmloader_acpi_build_tables(struct · line 958
Quoted 11 lines
(quoted diff) +0
-0
b/tools/firmware/hvmloader/util.cindex5739a87628..d8db9e3c8e100644tools/firmware/hvmloader/util.c +1
-2@@ -955,8 +955,17 @@ void hvmloader_acpi_build_tables(struct acpi_config *config, }else if ( !strncmp (s,"qemu_xen" ,9 ) ) { - config->dsdt_anycpu = dsdt_anycpu_qemu_xen; - config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 )Coding style (missing spaces between parentheses), and I would prefer a switch here.
OK, will change to a switch.
Quoted 2 lines
IMO you should add a
BUG_ON(Q35)in theqemu_xen_traditionalcondition above this one..
AFAIR qemu-traditional knows nothing about Q35 emulation, so we won't ever encounter a Q35 chipset while using qemu-traditional.
Quoted 20 lines
Quoted 20 lines
tools/firmware/hvmloader/util.c +10
-0+ { + config->dsdt_anycpu = dsdt_q35_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_q35_anycpu_qemu_xen_len; + } + else + { + config->dsdt_anycpu = dsdt_anycpu_qemu_xen; + config->dsdt_anycpu_len = dsdt_anycpu_qemu_xen_len; + } + config->dsdt_15cpu =NULL ; config->dsdt_15cpu_len =0 ; }(quoted diff) +0
-0
b/tools/firmware/hvmloader/util.hindex7c77bedb00..fd2d885c96100644tools/firmware/hvmloader/util.h +0
-0@@ -288,7 +288,9 @@ bool check_overlap(uint64_t start, uint64_t size, uint64_t reserved_start, uint64_t reserved_size);
Quoted 6 lines
tools/firmware/hvmloader/util.h bool check_overlap(uint64_t start, uint64_t size, · line 293
Quoted 4 lines
tools/firmware/hvmloader/util.h +0
-0extern const unsigned char dsdt_anycpu_qemu_xen[], dsdt_anycpu[], dsdt_15cpu[]; +extern const unsigned char dsdt_q35_anycpu_qemu_xen[];extern const int dsdt_anycpu_qemu_xen_len, dsdt_anycpu_len, dsdt_15cpu_len; +extern const int dsdt_q35_anycpu_qemu_xen_len;Since you are adding this, maybe unsigned int? (or
size_t?)
No problem, ok.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 27 lines · Alexey G
Quoted 24 lines · On Mon, 19 Mar 2018 15:58:02 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 12 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration.Quoted 4 lines
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see
IOREQ_TYPE_PCI_CONFIG). With this change you will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.
Quoted 7 lines
Quoted 4 lines
Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Adding Paul who knows more about all this.
In which use cases multiple PCI-device emulators are used for a single HVM domain? Is it a proprietary setup?
Likely. I think XenGT might be using it. It's a feature of the IOREQ implementation in Xen.
Traditional PCI config space accesses are not IO port space accesses. The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for example xen_map_pcidev in the QEMU code.
By treating MCFG accesses as MMIO you are bypassing the IOREQ PCI layer, and thus a IOREQ server could register a PCI device and only receive PCI configuration accesses from the IO port space, while MCFG accesses would be forwarded somewhere else.
I think you need to make the IOREQ code aware of the MCFG area and XEN_DMOP_IO_RANGE_PCI needs to forward both IO space and MCFG accesses to the right IOREQ server.
Quoted 9 lines
I assume it is somehow related to this code in
xen-hvm.c:/* Fake a write to port 0xCF8 so that * the config space access will target the * correct device model. */ val = (1u << 31) | ((req->addr & 0x0f00) <...> do_outp(0xcf8, 4, val);if yes, similar thing can be made for
IOREQ_TYPE_COPYaccesses to the emulated MMCONFIG if needed.
I have to admit I don't know that much about QEMU, and I have no idea what the chunk above is supposed to accomplish.
Quoted 8 lines
In HVM+QEMU case we are not limited to merely passed through devices, most of the observable PCI config space devices belong to one particular QEMU instance. This dictates the overall emulated MMCONFIG layout for a domain which should be in sync to what QEMU emulates via CF8h/CFCh accesses... and between multiple device model instances (if there are any, still not sure what multiple PCI-device emulators you mentioned really are).
In newer versions of Xen (>4.5 IIRC, Paul knows more), QEMU doesn't directly trap accesses to the 0xcf8/0xcfc IO ports, it's Xen instead the one that detects and decodes such accesses, and then forwards them to the IOREQ server that has been registered to handle them.
You cannot simply forward all MCFG accesses to QEMU as MMIO accesses, Xen needs to decode them and they need to be handled as IOREQ_TYPE_PCI_CONFIG requests, not IOREQ_TYPE_COPY IMO.
Quoted 9 lines
Basically, we have an emulated MMCONFIG area of 64/128/256MB size in the MMIO hole of the guest HVM domain. (BTW, this area itself can be considered a feature of the chipset the device model emulates.) It can be relocated to some other place in MMIO hole, this means that QEMU will trap accesses to the specific to the emulated chipset PCIEXBAR register and will issue same MMIO unmap/map calls as for any normal emulated MMIO range.
Quoted 6 lines
On the other hand, it won't be easy to provide emulated MMCONFIG translation into
IOREQ_TYPE_PCI_CONFIGfrom Xen side. Xen should know current emulated MMCONFIG area position and size in order to translate (or not) accesses to it into corresponding BDF/reg pair (+whether that area is enabled for decoding or not). This will likely require to introduce new hypercall(s).
Yes, you will have to introduce new hypercalls to tell Xen the position/size of the MCFG hole. Likely you want to tell it the start address, the pci segment, start bus and end bus. I know pci segment and start bus is always going to be 0 ATM, but it would be nice to have a complete interface.
By your comment above I think you want an interface that allows you to remove/add those MCFG areas at runtime.
Quoted 1 line
The question is if there will be any difference or benefit at all.
IMO it's not about benefits or differences, it's about correctness. Xen currently detects accesses to the PCI configuration space from IO ports and for consistency it should also detect accesses to this space by any other means.
Quoted 9 lines
It's basically the same emulated MMIO range after all, but in one case we trap accesses to it in Xen and translate them into
IOREQ_TYPE_PCI_CONFIGrequests.
We have to provide some infrastructure to let Xen know where the device model/guest expects to use the MMCONFIG area (and its size). The device model will need to use this infrastructure, informing Xen of any changes. Also, due to MMCONFIG nature there might be some pitfalls like a necessity to send multipleIOREQ_TYPE_PCI_CONFIGioreqs caused by a single memory read/write operation.
This seems all fine. Why do you expect MCFG access to create multiple IOREQ_TYPE_PCI_CONFIG but not multiple IOREQ_TYPE_COPY?
Quoted 5 lines
In another case, we still have an emulated MMIO range, but Xen will send plain
IOREQ_TYPE_COPYrequests to QEMU which it handles itself. In such case, all code to work with MMCONFIG accesses is available for reuse right away (mmcfg -> pci_* translation in QEMU), no new functionality required neither in Xen or QEMU.
As I tried to argument above, I think this is not correct, but I would also like that Paul expresses his opinion as the IOREQ maintainer.
Quoted 30 lines
Quoted 16 lines
Quoted 16 lines
tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127++++++++++++++++++++++++++++--------
tools/firmware/hvmloader/pci_regs.h| 2 + 3 files changed, 106
insertions(+), 27 deletions(-)(quoted diff) +0
-0
b/tools/firmware/hvmloader/config.hindex6fde6b7b60..5443ecd804100644 ---a/tools/firmware/hvmloader/config.h
+++b/tools/firmware/hvmloader/config.h(quoted diff) +0
-0@@ -53,10 +53,14 @@ extern uint8_t ioapic_version; #define PCI_ISA_DEVFN 0x08 /* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U /* ISA IRQs 5,10,11 are PCIconnected */ #define
PCI_ICH9_LPC_DEVFN0xf8/* dev 31, fn 0 */
+#definePCI_MCH_DEVFN0 /* bus 0, dev 0, func 0 */Quoted 8 lines
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 5 lines
/* MMIO hole: Hardcoded defaults, which can be dynamically
expanded. */ #definePCI_MEM_END0xfc000000tools/firmware/hvmloader/config.h +2
-0+/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64What the reasoning for this value? Do we know which devices need ECAM areas?
Yes, Xen is limited to bus 0 emulation currently, the description states "When multiple PCI buses support for Xen will be implemented,
PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration".
Quoted 8 lines
I think it might be better to replace 'switch (
PCI_MAX_MCFG_BUSES)' with the real code right away, i.e. change it to'switch (
max_bus_num, aligned up to 64/128/256 boundary)', wheremax_bus_numshould be set in PCI device enumeration code inpci_setup(). As we are limited to bus 0 currently, we'll just set it to 0 for now, before/after the PCI device enumeration loop (which should became multi-bus capable eventually).
I guess this is all pretty much hardcoded to bus 0 in several places, so I'm not sure it's worth to add PCI_MAX_MCFG_BUSES. IMO if something like this should be added it should be PCI_MAX_BUSES, and several places should be changed to make use of it. Or ideally we should find a way to detect this at runtime, without needed any hardcoded defines.
I think it would be good if you can add a note comment describing the different MCFG sizes supported by the Q35 chipset (64/128/256).
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Alexey GQuoted 33 lines · Alexey G
Quoted 22 lines · On Mon, 19 Mar 2018 17:49:09 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 21 lines · Alexey Gerasimenko
This patch extends
hvmloader_acpi_build_tables()with code which detects if MMCONFIG is available -- i.e. initialized and enabled (+we're running on Q35), obtains its base address and size and asks libacpi to build MCFG table for it via setting the flagACPI_HAS_MCFGin a manner similar to other optional ACPI tables building.Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/firmware/hvmloader/util.c | 70+++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
(quoted diff) +0
-0
b/tools/firmware/hvmloader/util.cindexd8db9e3c8e..c6fc81d52a100644tools/firmware/hvmloader/util.c +0
-0@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; }Quoted 7 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 789
Quoted 5 lines
tools/firmware/hvmloader/util.c +5
-0+#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1
PCIEXBAR_ENABLEmaybe?
Quoted 2 lines
PCIEXBAREN is just an official name of this bit from the Intel datasheet. :) OK, will rename it to
PCIEXBAR_ENABLE.
Oh, if that's the name on the spec then leave it as-is. It's always best to be able to search directly on the spec.
Quoted 26 lines
Quoted 10 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void)
Quoted 8 lines
tools/firmware/hvmloader/util.c +7
-0+ +static uint64_t mmconfig_get_base (void ) +{ +uint64_t base; +uint32_t reg =pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR ); + + base = reg | (uint64_t )pci_readl (PCI_MCH_DEVFN ,PCI_MCH_PCIEXBAR +4 ) <<32 ;Please add parentheses in the above expression.
Agree, parentheses will make the op priority clearer.
Quoted 13 lines
Quoted 12 lines
tools/firmware/hvmloader/util.c +12
-0+ +switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;
Quoted 6 lines
Quoted 3 lines
Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
I think all these #defines should find a way to
pci_regs.h, it seems like an appropriate place for them.
Hm, pci_regs.h seems to contain the generic PCI registers. Those should maybe live in a q35.h header, since it's very device specific AFAICT.
Quoted 8 lines
Regarding the order of hvmloader patches -- will verify this for the next version.
Quoted 4 lines
They are very related, and in fact I'm not sure why we need to write this info to the device in patch 7 and then fetch it from the device here. Isn't there an easier way to pass this information? At the end this is all in hvmloader.
Quoted 5 lines
Well, the
hvmloader_acpi_build_tables()function mostly does device probing (using I/O instruction) and xenstore reads to collect system information in order to discover which ACPI_HAS_* flags it should pass toacpi_build_tables(), but using global variables to pass this kind of information for MMCONFIG will be OK too I think.
It was just a suggestion, it seems kind of cumbersome to write something to a register and then fetch it afterwards, when it's all done in the same binary.
Quoted 17 lines
Quoted 3 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 810
Quoted 1 line
tools/firmware/hvmloader/util.c +1
-0+case 3 :default:
There is '& 3' for the switch argument, but ok I guess, it's clearer with 'default'.
Quoted 9 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 816
Quoted 7 lines
tools/firmware/hvmloader/util.c +7
-0+BUG ();/* a reserved value encountered */ + } + +return base; +} + +static uint32_t mmconfig_get_size (void )unsigned int or
size_t?
Quoted 3 lines
Using types which are common to the existing code.
size_thave almost zero use in hvmloader.
If it's available I would rather use it.
Quoted 14 lines
Quoted 14 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 841
Quoted 4 lines
tools/firmware/hvmloader/util.c +4
-0+{ +if (get_pc_machine_type () ==MACHINE_TYPE_Q35 ) + { +if (mmconfig_is_enabled () &&mmconfig_get_base ())Coding style.
Also you can join the conditions:
if (
get_pc_machine_type()==MACHINE_TYPE_Q35&&mmconfig_is_enabled()&&mmconfig_get_base() ) return true;
Quoted 8 lines
Quoted 2 lines
Looking at this, is it actually a valid state to have
mmconfig_is_enabled()== true andmmconfig_get_base()== 0?Yes, in theory we can have either PCIEXBAREN=0 and a valid PCIEXBAR base, or vice versa.
Of course normally we should not encounter a situation where base=0 and PCIEXBAREN=1, just covering here possible cases which the register format allows.
But those registers are set by hvmloader, and I don't think hvmloader will ever set PCIEXBAREN == 1 and PCIEXBAR base == 0?
Quoted 3 lines
Regarding check merging -- ok, sure. Short-circuit evaluation should guaranty that these registers are not touched on a different machine.
Yes, if you first check for the chipset type.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 10/12] libacpi: build ACPI MCFG table if requested
In reply to Alexey GQuoted 56 lines · Alexey G
Quoted 41 lines · On Mon, 19 Mar 2018 17:33:34 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 27 lines · Alexey Gerasimenko
This adds
construct_mcfg()function to libacpi which allows to build MCFG table for a given mmconfig_addr/mmconfig_len pair if theACPI_HAS_MCFGflag was specified inacpi_configstruct.The maximum bus number is calculated from
mmconfig_lenusingMCFG_SIZE_TO_NUM_BUSESmacro (1MByte of MMIO space per bus).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/acpi2_0.h | 21 +++++++++++++++++++++ tools/libacpi/build.c | 42++++++++++++++++++++++++++++++++++++++++++
tools/libacpi/libacpi.h| 4 ++++ 3 files changed, 67 insertions(+)tools/libacpi/acpi2_0.h +5
-0@@ -422,6 +422,25 @@ struct acpi_20_slit { };/* +* PCI Express Memory Mapped Configuration Description Table +*/ +struct mcfg_range_entry { +uint64_t base_address; +uint16_t pci_segment;tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 439
Quoted 10 lines
tools/libacpi/acpi2_0.h +10
-0+uint8_t start_pci_bus_num; +uint8_t end_pci_bus_num; +uint32_t reserved; +}; + +struct acpi_mcfg { +struct acpi_header header; +uint8_t reserved[8 ]; +struct mcfg_range_entry entries[1 ]; +};I would define this as:
Quoted 11 lines
struct acpi_10_mcfg { struct acpi_header header; uint8_t reserved[8]; struct acpi_10_mcfg_entry { uint64_t base_address; uint16_t pci_segment; uint8_t start_pci_bus; uint8_t end_pci_bus; uint32_t reserved; } entries[1]; };
Quoted 1 line
Hmm, a choice of preference, but OK, will move it inside.
Note the name change also (acpi_10_mcfg). Also I think you can drop the acpi_10_mcfg_entry name and just use an anonymous struct.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
tools/libacpi/acpi2_0.h +6
-0+ + mcfg->entries[0 ].base_address = config->mmconfig_addr; + mcfg->entries[0 ].pci_segment =0 ; + mcfg->entries[0 ].start_pci_bus_num =0 ; + mcfg->entries[0 ].end_pci_bus_num = +MCFG_SIZE_TO_NUM_BUSES (config->mmconfig_len) -1 ;
Quoted 6 lines
Quoted 1 line
Why not pass the
start_busandend_busvalues inacpi_configat least?
start_pci_bus_numwill be always 0.It will be kinda ugly to pass
config->mmconfig_addralong withconfig->end_pci_bus_num, baseaddr+size combo looks nicer I think.
I'm not going to insist, but ACPI doesn't really care about the size, it just needs to know the start and end. Seems pointless to write a value here that later libacpi needs to convert to the value it actually needs. Also start/end buses are uint8_t, size is uint32_t.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Alexey GQuoted 12 lines · Alexey G
Quoted 8 lines · On Mon, 19 Mar 2018 17:01:18 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 5 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.I would rather name this
machine_chipsetordevice_model_chipset.
Quoted 7 lines
device_model_prefix is a must I think -- multiple device model related options have names starting withdevice_model_.
device_model_chipset... well, maybe, but we're actually specifying a QEMU machine here. In QEMU mailing list there was even a suggestion to allow to pass a machine version number here, like "pc-q35-2.10". I think some opinions are needed here.
I'm not sure what a 'machine' is in QEMU speak, but in my mind I would consider PC a machine (vs ARM for example).
I think 'chipset' is clearer, but again others should express their opinion.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 06/12] hvmloader: add basic Q35 support
In reply to Alexey GQuoted 4 lines · Alexey G
Quoted 1 line · On Mon, 19 Mar 2018 15:30:14 +0000 Roger Pau Monné <[email protected]> wrote:
On Tue, Mar 13, 2018 at 04:33:51AM +1000, Alexey Gerasimenko wrote:
Quoted 8 lines
Quoted 4 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ { + case 0x0300:All this values need to be defines documented somewhere.
Agree... although it was not me who introduced all these hardcoded PCI class values. :) I'll change these numbers into newly added
pci_regs.h#defines in the non-functional patch.
Right. I've realized that later. If you place this code moment in a separate patch without any other modifications I won't complain about the lack of defines (although it would be nice to have them :)).
Quoted 11 lines
Quoted 10 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 228
Quoted 5 lines
tools/firmware/hvmloader/pci.c +5
-0+ { +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x60 + link, isa_irq); + +/* PIRQE..PIRQH are unused */ +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x68 + link,0x80 );According to the spec
0x80is the default value for this registers, do you really need to write it?Is maybe QEMU not correctly setting the default value?
Quoted 8 lines
Won't agree here. We're initializing PIRQ[n] routing in this fragment, it's better not to rely on any values but simply initialize all PIRQ[n]_ROUT registers, this makes it explicit.
Even if it is unnecessary due to defaults it's more obvious to set these registers to our own values than to force a reader to either look up their emulation in QEMU code or read the ICH9 pdf to confirm assumptions.
But if you start doing this, you should do it for all the registers. Why is PIRQE..PIRQH routing special that you need to re-write the default value? But not SIRQ_CNTL for example?
I think a comment noting that the default value for those registers is what we expect (0x80 - Interrupt Routing Disabled) would be better.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 44 lines
-----Original Message-----
From: Roger Pau Monne
Sent: 20 March 2018 08:51
To: Alexey G <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 8 lines · Alexey G
Quoted 5 lines · On Mon, 19 Mar 2018 15:58:02 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 4 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculationof the
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
number of buses according to results of the PCI devices enumeration.
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see
IOREQ_TYPE_PCI_CONFIG). With this changeyou
Quoted 6 lines
Quoted 6 lines
will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.
Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Adding Paul who knows more about all this.
In which use cases multiple PCI-device emulators are used for a single HVM domain? Is it a proprietary setup?
Likely. I think XenGT might be using it. It's a feature of the IOREQ implementation in Xen.
Multiple ioreq servers are a supported use-case for Xen, if only experimental at this point. And indeed xengt is one such use-case.
Quoted 11 lines
Traditional PCI config space accesses are not IO port space accesses. The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for example
xen_map_pcidevin the QEMU code.By treating MCFG accesses as MMIO you are bypassing the IOREQ PCI layer, and thus a IOREQ server could register a PCI device and only receive PCI configuration accesses from the IO port space, while MCFG accesses would be forwarded somewhere else.
Quoted 4 lines
I think you need to make the IOREQ code aware of the MCFG area and
XEN_DMOP_IO_RANGE_PCIneeds to forward both IO space and MCFG accesses
to the right IOREQ server.
Yes, Xen must intercept all accesses to PCI config space and route them accordingly.
Quoted 11 lines
Quoted 10 lines
I assume it is somehow related to this code in
xen-hvm.c:/* Fake a write to port 0xCF8 so that * the config space access will target the * correct device model. */ val = (1u << 31) | ((req->addr & 0x0f00) <...> do_outp(0xcf8, 4, val);if yes, similar thing can be made for
IOREQ_TYPE_COPYaccesses to the emulated MMCONFIG if needed.
Quoted 3 lines
I have to admit I don't know that much about QEMU, and I have no idea what the chunk above is supposed to accomplish.
The easiest way to make QEMU behave appropriately when dealing with a config space ioreq was indeed to make it appear as a write to cf8 followed by a read or write to cfc.
Quoted 6 lines
Quoted 5 lines
In HVM+QEMU case we are not limited to merely passed through devices, most of the observable PCI config space devices belong to one particular QEMU instance. This dictates the overall emulated MMCONFIG layout for a domain which should be in sync to what QEMU emulates via
CF8h/CFCh
Quoted 9 lines
Quoted 3 lines
accesses... and between multiple device model instances (if there are any, still not sure what multiple PCI-device emulators you mentioned really are).
In newer versions of Xen (>4.5 IIRC, Paul knows more), QEMU doesn't directly trap accesses to the 0xcf8/0xcfc IO ports, it's Xen instead the one that detects and decodes such accesses, and then forwards them to the IOREQ server that has been registered to handle them.
Correct.
Quoted 20 lines
You cannot simply forward all MCFG accesses to QEMU as MMIO accesses, Xen needs to decode them and they need to be handled as
IOREQ_TYPE_PCI_CONFIGrequests, notIOREQ_TYPE_COPYIMO.Quoted 9 lines
Basically, we have an emulated MMCONFIG area of 64/128/256MB size in the MMIO hole of the guest HVM domain. (BTW, this area itself can be considered a feature of the chipset the device model emulates.) It can be relocated to some other place in MMIO hole, this means that QEMU will trap accesses to the specific to the emulated chipset PCIEXBAR register and will issue same MMIO unmap/map calls as for any normal emulated MMIO range.
Quoted 6 lines
On the other hand, it won't be easy to provide emulated MMCONFIG translation into
IOREQ_TYPE_PCI_CONFIGfrom Xen side. Xen should know current emulated MMCONFIG area position and size in order to translate (or not) accesses to it into corresponding BDF/reg pair (+whether that area is enabled for decoding or not). This will likely require to introduce new hypercall(s).
Quoted 9 lines
Yes, you will have to introduce new hypercalls to tell Xen the position/size of the MCFG hole. Likely you want to tell it the start address, the pci segment, start bus and end bus. I know pci segment and start bus is always going to be 0 ATM, but it would be nice to have a complete interface.
By your comment above I think you want an interface that allows you to remove/add those MCFG areas at runtime.
We're going to want hotplug eventually so, yes, devices need to appear and disappear dynamically.
Quoted 7 lines
Quoted 1 line
The question is if there will be any difference or benefit at all.
IMO it's not about benefits or differences, it's about correctness. Xen currently detects accesses to the PCI configuration space from IO ports and for consistency it should also detect accesses to this space by any other means.
Yes, this is a 'must' rather than a 'should' though.
Quoted 14 lines
Quoted 8 lines
It's basically the same emulated MMIO range after all, but in one case we trap accesses to it in Xen and translate them into
IOREQ_TYPE_PCI_CONFIGrequests.
We have to provide some infrastructure to let Xen know where the device model/guest expects to use the MMCONFIG area (and its size). The device model will need to use this infrastructure, informing Xen of any changes. Also, due to MMCONFIG nature there might be some pitfalls like a necessity to send multipleIOREQ_TYPE_PCI_CONFIGioreqs causedby
Quoted 1 line
a single memory read/write operation.
This seems all fine. Why do you expect MCFG access to create multiple
IOREQ_TYPE_PCI_CONFIGbut not multipleIOREQ_TYPE_COPY?
Quoted 8 lines
Quoted 5 lines
In another case, we still have an emulated MMIO range, but Xen will send plain
IOREQ_TYPE_COPYrequests to QEMU which it handles itself. In such case, all code to work with MMCONFIG accesses is available for reuse right away (mmcfg -> pci_* translation in QEMU), no new functionality required neither in Xen or QEMU.As I tried to argument above, I think this is not correct, but I would also like that Paul expresses his opinion as the IOREQ maintainer.
Xen should handle MMCONFIG accesses. All PCI device emulators should register for PCI config space by SBDF and the mechanism by which the Xen intercepts the config access and routes it to the emulator should be none of the emulators concern. QEMU does not own the PCI bus topology; Xen does, and it's been this way for quite some time (even if the implementation is incomplete).
Paul
Quoted 47 lines
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127++++++++++++++++++++++++++++--------
tools/firmware/hvmloader/pci_regs.h| 2 + 3 files changed, 106
insertions(+), 27 deletions(-)(quoted diff) +0
-0
b/tools/firmware/hvmloader/config.hindex6fde6b7b60..5443ecd804100644 ---a/tools/firmware/hvmloader/config.h
+++b/tools/firmware/hvmloader/config.h(quoted diff) +0
-0@@ -53,10 +53,14 @@ extern uint8_t ioapic_version; #define PCI_ISA_DEVFN 0x08 /* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U /* ISA IRQs 5,10,11 are PCIconnected */ #define
PCI_ICH9_LPC_DEVFN0xf8/* dev 31, fn 0 */
+#definePCI_MCH_DEVFN0 /* bus 0, dev 0, func 0 */Quoted 8 lines
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 5 lines
/* MMIO hole: Hardcoded defaults, which can be dynamically
expanded. */ #definePCI_MEM_END0xfc000000tools/firmware/hvmloader/config.h +2
-0+/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64What the reasoning for this value? Do we know which devices need ECAM areas?
Quoted 3 lines
Yes, Xen is limited to bus 0 emulation currently, the description states "When multiple PCI buses support for Xen will be implemented,
PCI_MAX_MCFG_BUSESmay be changed to calculation of the number ofbuses
Quoted 10 lines
according to results of the PCI devices enumeration".
I think it might be better to replace 'switch (
PCI_MAX_MCFG_BUSES)' with the real code right away, i.e. change it to'switch (
max_bus_num, aligned up to 64/128/256 boundary)', wheremax_bus_numshould be set in PCI device enumeration code inpci_setup(). As we are limited to bus 0 currently, we'll just set it to 0 for now, before/after the PCI device enumeration loop (which should became multi-bus capable eventually).I guess this is all pretty much hardcoded to bus 0 in several places, so I'm not sure it's worth to add
PCI_MAX_MCFG_BUSES. IMO if something like this should be added it should bePCI_MAX_BUSES, and several places should be changed to make use of it. Or ideally we should find a way to detect this at runtime, without needed any hardcoded defines.
Quoted 4 lines
I think it would be good if you can add a note comment describing the different MCFG sizes supported by the Q35 chipset (64/128/256).
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Alexey GQuoted 18 lines
Quoted 1 line
Quoted 1 line · On 19.03.18 at 22:20, <[email protected]> wrote:
Quoted 7 lines · On Mon, 19 Mar 2018 17:49:09 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 6 lines · Alexey Gerasimenko
tools/firmware/hvmloader/util.c +0
-0@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; }Quoted 7 lines
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 789
Quoted 5 lines
tools/firmware/hvmloader/util.c +5
-0+#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1
PCIEXBAR_ENABLEmaybe?
Quoted 2 lines
PCIEXBAREN is just an official name of this bit from the Intel datasheet. :) OK, will rename it to
PCIEXBAR_ENABLE.
I think using names from the datasheet (where they exist) is preferable in cases like this one.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
tools/firmware/hvmloader/util.c +11
-0+switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;
Quoted 6 lines
Quoted 3 lines
Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
I think all these #defines should find a way to
pci_regs.h, it seems like an appropriate place for them.
I don't think device specific defines belong into pci_regs.h.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
Quoted 18 lines · On Tue, 20 Mar 2018 03:36:57 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 18 lines
Quoted 1 line
Quoted 1 line · On 19.03.18 at 22:20, <[email protected]> wrote:
Quoted 14 lines · On Mon, 19 Mar 2018 17:49:09 +0000 Roger Pau Monné <[email protected]> wrote:
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 789
Quoted 11 lines · Alexey Gerasimenko
tools/firmware/hvmloader/util.c +5
-0@@ -782,6 +782,69 @@ int get_pc_machine_type(void)return machine_type; } +#define PCIEXBAR_ADDR_MASK_64MB (~((1ULL << 26) - 1)) +#define PCIEXBAR_ADDR_MASK_128MB (~((1ULL << 27) - 1)) +#define PCIEXBAR_ADDR_MASK_256MB (~((1ULL << 28) - 1)) +#define PCIEXBAR_LENGTH_BITS (reg) (((reg) >> 1) & 3) +#define PCIEXBAREN 1
PCIEXBAR_ENABLEmaybe?
Quoted 5 lines
Quoted 2 lines
PCIEXBAREN is just an official name of this bit from the Intel datasheet. :) OK, will rename it to
PCIEXBAR_ENABLE.I think using names from the datasheet (where they exist) is preferable in cases like this one.
Leaving it intact then.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
tools/firmware/hvmloader/util.c +11
-0+switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
I think all these #defines should find a way to
pci_regs.h, it seems like an appropriate place for them.I don't think device specific defines belong into
pci_regs.h.
Will gather all these #defines and macros in the new pci_regs_q35.h file. It should not harm to include it from pci_regs.h I think, in order to include pci_regs.h only in *.c.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 10/12] libacpi: build ACPI MCFG table if requested
In reply to Roger Pau MonnéQuoted 73 lines · On Tue, 20 Mar 2018 09:03:56 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 57 lines · Alexey G
Quoted 42 lines · On Mon, 19 Mar 2018 17:33:34 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 27 lines · Alexey Gerasimenko
This adds
construct_mcfg()function to libacpi which allows to build MCFG table for a given mmconfig_addr/mmconfig_len pair if theACPI_HAS_MCFGflag was specified inacpi_configstruct.The maximum bus number is calculated from
mmconfig_lenusingMCFG_SIZE_TO_NUM_BUSESmacro (1MByte of MMIO space per bus).Signed-off-by Alexey Gerasimenko <[email protected]>--- tools/libacpi/acpi2_0.h | 21 +++++++++++++++++++++ tools/libacpi/build.c | 42++++++++++++++++++++++++++++++++++++++++++
tools/libacpi/libacpi.h| 4 ++++ 3 files changed, 67 insertions(+)tools/libacpi/acpi2_0.h +5
-0@@ -422,6 +422,25 @@ struct acpi_20_slit { };/* +* PCI Express Memory Mapped Configuration Description Table +*/ +struct mcfg_range_entry { +uint64_t base_address; +uint16_t pci_segment;tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 439
Quoted 10 lines
tools/libacpi/acpi2_0.h +10
-0+uint8_t start_pci_bus_num; +uint8_t end_pci_bus_num; +uint32_t reserved; +}; + +struct acpi_mcfg { +struct acpi_header header; +uint8_t reserved[8 ]; +struct mcfg_range_entry entries[1 ]; +};I would define this as:
Quoted 11 lines
struct acpi_10_mcfg { struct acpi_header header; uint8_t reserved[8]; struct acpi_10_mcfg_entry { uint64_t base_address; uint16_t pci_segment; uint8_t start_pci_bus; uint8_t end_pci_bus; uint32_t reserved; } entries[1]; };Quoted 1 line
Hmm, a choice of preference, but OK, will move it inside.
Note the name change also (
acpi_10_mcfg). Also I think you can drop theacpi_10_mcfg_entryname and just use an anonymous struct.Quoted 10 lines
Quoted 9 lines
tools/libacpi/acpi2_0.h struct acpi_20_slit { · line 445
Quoted 6 lines
tools/libacpi/acpi2_0.h +6
-0+ + mcfg->entries[0 ].base_address = config->mmconfig_addr; + mcfg->entries[0 ].pci_segment =0 ; + mcfg->entries[0 ].start_pci_bus_num =0 ; + mcfg->entries[0 ].end_pci_bus_num = +MCFG_SIZE_TO_NUM_BUSES (config->mmconfig_len) -1 ;Why not pass the
start_busandend_busvalues inacpi_configat least?
Quoted 8 lines
Quoted 4 lines
start_pci_bus_numwill be always 0.It will be kinda ugly to pass
config->mmconfig_addralong withconfig->end_pci_bus_num, baseaddr+size combo looks nicer I think.I'm not going to insist, but ACPI doesn't really care about the size, it just needs to know the start and end. Seems pointless to write a value here that later libacpi needs to convert to the value it actually needs. Also start/end buses are
uint8_t, size isuint32_t.
As the underlying implementation is limited to just one PCI segment and we need to pass only one MCFG range entry, I guess it will be ok to use the mmconfig_addr + mmconfig_num_buses pair (almost same as end_bus, but more size-descriptive).
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 06/12] hvmloader: add basic Q35 support
In reply to Roger Pau MonnéQuoted 11 lines · On Tue, 20 Mar 2018 09:20:01 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 10 lines · Alexey G
Quoted 6 lines · On Mon, 19 Mar 2018 15:30:14 +0000 Roger Pau Monné <[email protected]> wrote:
Review
Quoted 2 lines · Alexey Gerasimenko
(quoted diff) +2
-0+ { + case 0x0300:All this values need to be defines documented somewhere.
Quoted 7 lines
Quoted 3 lines
Agree... although it was not me who introduced all these hardcoded PCI class values. :) I'll change these numbers into newly added
pci_regs.h#defines in the non-functional patch.Right. I've realized that later. If you place this code moment in a separate patch without any other modifications I won't complain about the lack of defines (although it would be nice to have them :)).
OK, will do.
Quoted 21 lines
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
tools/firmware/hvmloader/pci.c +5
-0+ { +pci_writeb (PCI_ICH9_LPC_DEVFN ,0x60 + link, isa_irq); + +/* PIRQE..PIRQH are unused */ + pci_writeb(PCI_ICH9_LPC_DEVFN, 0x68 + link,
0x80);According to the spec
0x80is the default value for this registers, do you really need to write it?Is maybe QEMU not correctly setting the default value?
Quoted 8 lines
Won't agree here. We're initializing PIRQ[n] routing in this fragment, it's better not to rely on any values but simply initialize all PIRQ[n]_ROUT registers, this makes it explicit.
Even if it is unnecessary due to defaults it's more obvious to set these registers to our own values than to force a reader to either look up their emulation in QEMU code or read the ICH9 pdf to confirm assumptions.
Quoted 6 lines
But if you start doing this, you should do it for all the registers. Why is PIRQE..PIRQH routing special that you need to re-write the default value? But not
SIRQ_CNTLfor example?I think a comment noting that the default value for those registers is what we expect (
0x80- Interrupt Routing Disabled) would be better.
It will depend on future QEMU/hvmloader changes a bit, but I think switching to the comment about the default values instead of initialization should be good.
Quoted 2 lines
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 34 lines · On Tue, 20 Mar 2018 08:50:48 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 33 lines · Alexey G
Quoted 30 lines · On Mon, 19 Mar 2018 15:58:02 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 13 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration.Quoted 4 lines
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see
IOREQ_TYPE_PCI_CONFIG). With this change you will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Adding Paul who knows more about all this.
In which use cases multiple PCI-device emulators are used for a single HVM domain? Is it a proprietary setup?
Likely. I think XenGT might be using it. It's a feature of the IOREQ implementation in Xen.
According to public slides for the feature, both PCI conf and MMIO accesses can be routed to the designated device model. It looks like for this particular setup it doesn't really matter which particular ioreq type must be used for MMCONFIG accesses -- either IOREQ_TYPE_PCI_CONFIG or IOREQ_TYPE_COPY (MMIO accesses) should be acceptable. The only thing which matters is ioreq routing itself -- making decisions to which device model the PCI conf/MMIO ioreq should be sent.
Quoted 1 line
Traditional PCI config space accesses are not IO port space accesses.
(assuming 'not' mistyped here)
Quoted 3 lines
The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for
That's one of the reasons why current IOREQ_TYPE_PCI_CONFIG implementation is a bit inconvenient for MMCONFIG MMIO accesses -- it's too much CF8h/CFCh-centric in its implementation, might be painful to change something in the code which was intended for CF8h/CFCh handling (and not for MMIO processing).
Quoted 6 lines
example
xen_map_pcidevin the QEMU code.By treating MCFG accesses as MMIO you are bypassing the IOREQ PCI layer, and thus a IOREQ server could register a PCI device and only receive PCI configuration accesses from the IO port space, while MCFG accesses would be forwarded somewhere else.
It will be handled by IOREQ too, just using a different IOREQ type (MMIO one). The basic question is why do we have to stick to PCI conf space ioreqs for emulating MMIO accesses to MMCONFIG.
Quoted 3 lines
I think you need to make the IOREQ code aware of the MCFG area and
XEN_DMOP_IO_RANGE_PCIneeds to forward both IO space and MCFG accesses to the right IOREQ server.
Right now there is no way to inform Xen where the emulated MMCONFIG area is located in order to make this decision, based on the address within MMCONFIG range. A new dmop/hypercall is needed (with args similar to pci_mmcfg_reserved) along with its usage in QEMU.
I'll try to summarize two different approaches to MMCONFIG handling. For both approaches the final PCI config host interface for a passed through device in QEMU will remain same as at the moment -- xen_host_pci_* functions in /hw/xen.
Approach #1. Informing Xen about MMCONFIG area changes and letting Xen to translate MMIO accesses to _PCI_CONFIG ioreqs:
1. QEMU will trap accesses to PCIEXBAR, calling Xen via dmop/hypercall to let the latter know of any MMCONFIG area address/size/status changes
2. Xen will trap MMIO accesses to the current MMCONFIG location and convert memory accesses into one or several _PCI_CONFIG ioreqs and send them to a chosen device model
3. QEMU will receive _PCI_CONFIG ioreqs with SBDF and 12-bit offsets inside which it needs to somehow pass to pci_host_config_{read,write}_common() for emulation. It might require few hacks to make the gears turn (due to QEMU pci conf read/write model).
At the moment emulated CF8h/CFCh ports play a special role in all this -- xen-hvm.c writes an AMD-style value to the emulated CF8h port "so that the config space access will target the correct device model" (quote). Not sure about this and why it's is needed if Xen actually makes the decision to which DM the PCI conf ioreq should be sent.
One minor note: these new 'set_mmconfig_' dmops/hypercalls have to be triggered inside the chipset-specific emulation code in QEMU (PCIEXBAR handling in Q35 case). If there will be another machine which needs to emulate MMCONFIG control differently -- we have no choice but to insert these dmops/hypercalls into another chipset-specific emulation code as well, eg. inside HECBASE emulation code.
Approach #2. Handling MMCONFIG area inside QEMU using usual MMIO emulation:
1. QEMU will trap accesses to PCIEXBAR (or whatever else possibly supported in the future like HECBASE), eventually asking Xen to map the MMCONFIG MMIO range for ioreq servicing just like it does for any other emulated MMIO range, via map_io_range_to_ioreq_server(). All changes in MMCONFIG placement/status will lead to remapping/unmapping the MMIO range.
2. Xen will trap MMIO accesses to this area and forward them to QEMU as MMIO (IOREQ_TYPE_COPY) ioreqs
3. QEMU will receive these accesses and pass them to the existing MMCONFIG emulation -- pcie_mmcfg_data_read/write handlers, finally resulting in same xen_host_pci_* function calls as before.
This approach works "right out of the box", no changes needed for either Xen or QEMU. As both _PCI_CONFIG and MMIO type ioreqs are processed, either method can be used to access PCI/extended config space -- CF8/CFC port I/O or MMIO accesses to MMCONFIG.
IOREQ routing for multiple device emulators can be supported too. In fact, the same mmconfig dmops/hypercalls can be added to let Xen know where MMCONFIG area resides, Xen will use this information to forward MMCONFIG MMIO ioreqs accordingly to BDF of the address. The difference with the approach #1 is that these interfaces are now completely optional when we use MMIO ioreqs for MMCONFIG on vanilla Xen/QEMU.
The question is why IOREQ_TYPE_COPY -> IOREQ_TYPE_PCI_CONFIG translation is a must have thing at all? It won't make handling simpler. For current QEMU implementation IOREQ_TYPE_COPY (MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with 'set_mmconfig_' dmop/hypercall for the 'multiple device emulators' IOREQ_TYPE_COPY routing to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.
After all, all this is Q35-specific and won't harm the existing i440 emulation in any way.
Quoted 123 lines
Quoted 9 lines
I assume it is somehow related to this code in
xen-hvm.c:/* Fake a write to port 0xCF8 so that * the config space access will target the * correct device model. */ val = (1u << 31) | ((req->addr & 0x0f00) <...> do_outp(0xcf8, 4, val);if yes, similar thing can be made for
IOREQ_TYPE_COPYaccesses to the emulated MMCONFIG if needed.I have to admit I don't know that much about QEMU, and I have no idea what the chunk above is supposed to accomplish.
Quoted 8 lines
In HVM+QEMU case we are not limited to merely passed through devices, most of the observable PCI config space devices belong to one particular QEMU instance. This dictates the overall emulated MMCONFIG layout for a domain which should be in sync to what QEMU emulates via CF8h/CFCh accesses... and between multiple device model instances (if there are any, still not sure what multiple PCI-device emulators you mentioned really are).
In newer versions of Xen (>4.5 IIRC, Paul knows more), QEMU doesn't directly trap accesses to the 0xcf8/0xcfc IO ports, it's Xen instead the one that detects and decodes such accesses, and then forwards them to the IOREQ server that has been registered to handle them.
You cannot simply forward all MCFG accesses to QEMU as MMIO accesses, Xen needs to decode them and they need to be handled as
IOREQ_TYPE_PCI_CONFIGrequests, notIOREQ_TYPE_COPYIMO.Quoted 9 lines
Basically, we have an emulated MMCONFIG area of 64/128/256MB size in the MMIO hole of the guest HVM domain. (BTW, this area itself can be considered a feature of the chipset the device model emulates.) It can be relocated to some other place in MMIO hole, this means that QEMU will trap accesses to the specific to the emulated chipset PCIEXBAR register and will issue same MMIO unmap/map calls as for any normal emulated MMIO range.
Quoted 6 lines
On the other hand, it won't be easy to provide emulated MMCONFIG translation into
IOREQ_TYPE_PCI_CONFIGfrom Xen side. Xen should know current emulated MMCONFIG area position and size in order to translate (or not) accesses to it into corresponding BDF/reg pair (+whether that area is enabled for decoding or not). This will likely require to introduce new hypercall(s).Yes, you will have to introduce new hypercalls to tell Xen the position/size of the MCFG hole. Likely you want to tell it the start address, the pci segment, start bus and end bus. I know pci segment and start bus is always going to be 0 ATM, but it would be nice to have a complete interface.
By your comment above I think you want an interface that allows you to remove/add those MCFG areas at runtime.
Quoted 1 line
The question is if there will be any difference or benefit at all.
IMO it's not about benefits or differences, it's about correctness. Xen currently detects accesses to the PCI configuration space from IO ports and for consistency it should also detect accesses to this space by any other means.
Quoted 9 lines
It's basically the same emulated MMIO range after all, but in one case we trap accesses to it in Xen and translate them into
IOREQ_TYPE_PCI_CONFIGrequests.
We have to provide some infrastructure to let Xen know where the device model/guest expects to use the MMCONFIG area (and its size). The device model will need to use this infrastructure, informing Xen of any changes. Also, due to MMCONFIG nature there might be some pitfalls like a necessity to send multipleIOREQ_TYPE_PCI_CONFIGioreqs caused by a single memory read/write operation.This seems all fine. Why do you expect MCFG access to create multiple
IOREQ_TYPE_PCI_CONFIGbut not multipleIOREQ_TYPE_COPY?Quoted 5 lines
In another case, we still have an emulated MMIO range, but Xen will send plain
IOREQ_TYPE_COPYrequests to QEMU which it handles itself. In such case, all code to work with MMCONFIG accesses is available for reuse right away (mmcfg -> pci_* translation in QEMU), no new functionality required neither in Xen or QEMU.As I tried to argument above, I think this is not correct, but I would also like that Paul expresses his opinion as the IOREQ maintainer.
Quoted 30 lines
Quoted 16 lines
Quoted 16 lines
tools/firmware/hvmloader/config.h | 4 ++ tools/firmware/hvmloader/pci.c | 127++++++++++++++++++++++++++++--------
tools/firmware/hvmloader/pci_regs.h| 2 + 3 files changed, 106
insertions(+), 27 deletions(-)(quoted diff) +0
-0
b/tools/firmware/hvmloader/config.hindex6fde6b7b60..5443ecd804100644 ---a/tools/firmware/hvmloader/config.h
+++b/tools/firmware/hvmloader/config.h(quoted diff) +0
-0@@ -53,10 +53,14 @@ extern uint8_t ioapic_version; #define PCI_ISA_DEVFN 0x08 /* dev 1, fn 0 */ #define PCI_ISA_IRQ_MASK 0x0c20U /* ISA IRQs 5,10,11 are PCIconnected */ #define
PCI_ICH9_LPC_DEVFN0xf8/* dev 31, fn 0
*/ +#definePCI_MCH_DEVFN0 /* bus 0, dev 0, func 0 */Quoted 8 lines
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 5 lines
/* MMIO hole: Hardcoded defaults, which can be dynamically
expanded. */ #definePCI_MEM_END0xfc000000tools/firmware/hvmloader/config.h +2
-0+/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64What the reasoning for this value? Do we know which devices need ECAM areas?
Yes, Xen is limited to bus 0 emulation currently, the description states "When multiple PCI buses support for Xen will be implemented,
PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration".Quoted 8 lines
I think it might be better to replace 'switch (
PCI_MAX_MCFG_BUSES)' with the real code right away, i.e. change it to'switch (
max_bus_num, aligned up to 64/128/256 boundary)', wheremax_bus_numshould be set in PCI device enumeration code inpci_setup(). As we are limited to bus 0 currently, we'll just set it to 0 for now, before/after the PCI device enumeration loop (which should became multi-bus capable eventually).
Quoted 5 lines
I guess this is all pretty much hardcoded to bus 0 in several places, so I'm not sure it's worth to add
PCI_MAX_MCFG_BUSES. IMO if something like this should be added it should bePCI_MAX_BUSES, and several places should be changed to make use of it. Or ideally we should find a way to detect this at runtime, without needed any hardcoded defines.
Getting rid of bus 0 limitation should have high priority I'm afraid. It has become an obstacle for PCIe passthrough.
Quoted 2 lines
I think it would be good if you can add a note comment describing the different MCFG sizes supported by the Q35 chipset (64/128/256).
will add "...supported by Q35" here:
Quoted 2 lines
tools/firmware/hvmloader/config.h extern uint8_t ioapic_version; · line 62
Quoted 2 lines
tools/firmware/hvmloader/config.h +2
-0+/* possible values are: 64, 128, 256 */ +#define PCI_MAX_MCFG_BUSES 64
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 11/12] hvmloader: use libacpi to build MCFG table
In reply to Alexey GQuoted 23 lines
Quoted 1 line
Quoted 1 line · On 20.03.18 at 21:53, <[email protected]> wrote:
Quoted 20 lines · On Tue, 20 Mar 2018 03:36:57 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 19.03.18 at 22:20, <[email protected]> wrote:
Quoted 16 lines · On Mon, 19 Mar 2018 17:49:09 +0000 Roger Pau Monné <[email protected]> wrote:
tools/firmware/hvmloader/util.c int get_pc_machine_type(void) · line 808
Quoted 11 lines · Alexey Gerasimenko
tools/firmware/hvmloader/util.c +11
-0+switch (PCIEXBAR_LENGTH_BITS (reg)) + { +case 0 : + base &=PCIEXBAR_ADDR_MASK_256MB ; +break ; +case 1 : + base &=PCIEXBAR_ADDR_MASK_128MB ; +break ; +case 2 : + base &=PCIEXBAR_ADDR_MASK_64MB ; +break ;Missing newlines, plus this looks like it wants to use the defines introduced in patch 7 (PCIEXBAR_{64,128,256}_BUSES). Also any reason this patch and patch 7 cannot be put sequentially?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
I think all these #defines should find a way to
pci_regs.h, it seems like an appropriate place for them.I don't think device specific defines belong into
pci_regs.h.Will gather all these #defines and macros in the new
pci_regs_q35.hfile. It should not harm to include it frompci_regs.hI think, in order to includepci_regs.honly in *.c.
Well, no - no unnecessary dependencies please. If only a single file needs these definitions, only that file should include the respective header (if one is warranted in the first place).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 45 lines · Alexey G
Quoted 34 lines · On Tue, 20 Mar 2018 08:50:48 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 33 lines · Alexey G
Quoted 30 lines · On Mon, 19 Mar 2018 15:58:02 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 13 lines · Alexey Gerasimenko
Much like normal PCI BARs or other chipset-specific memory-mapped resources, MMCONFIG area needs space in MMIO hole, so we must allocate it manually.
The actual MMCONFIG size depends on a number of PCI buses available which should be covered by ECAM. Possible options are 64MB, 128MB and 256MB. As we are limited to the bus 0 currently, thus using lowest possible setting (64MB), #defined via
PCI_MAX_MCFG_BUSESinhvmloader/config.h. When multiple PCI buses support for Xen will be implemented,PCI_MAX_MCFG_BUSESmay be changed to calculation of the number of buses according to results of the PCI devices enumeration.Quoted 4 lines
The way to allocate MMCONFIG range in MMIO hole is similar to how other PCI BARs are allocated. The patch extends 'bars' structure to make it universal for any arbitrary BAR type -- either IO, MMIO, ROM or a chipset-specific resource.
I'm not sure this is fully correct. The IOREQ interface can differentiate PCI devices and forward config space accesses to different emulators (see
IOREQ_TYPE_PCI_CONFIG). With this change you will forward all MCFG accesses to QEMU, which will likely be wrong if there are multiple PCI-device emulators for the same domain.Ie: AFAICT Xen needs to know about the MCFG emulation and detect accesses to it in order to forward them to the right emulators.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Adding Paul who knows more about all this.
In which use cases multiple PCI-device emulators are used for a single HVM domain? Is it a proprietary setup?
Likely. I think XenGT might be using it. It's a feature of the IOREQ implementation in Xen.
Quoted 6 lines
According to public slides for the feature, both PCI conf and MMIO accesses can be routed to the designated device model. It looks like for this particular setup it doesn't really matter which particular ioreq type must be used for MMCONFIG accesses -- either
IOREQ_TYPE_PCI_CONFIGorIOREQ_TYPE_COPY(MMIO accesses) should be acceptable.
Isn't that going to be quite messy? How is the IOREQ server supposed to decode a MCFG access received as IOREQ_TYPE_COPY?
I don't think the IOREQ server needs to know the start of the MCFG region, in which case it won't be able to detect and decode the access if it's of type IOREQ_TYPE_COPY.
MCFG accesses need to be sent to the IOREQ server as IOREQ_TYPE_PCI_CONFIG, or else you are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding. In your case this would work because QEMU controls the position of the MCFG region, but there's no need for other IOREQ servers to know the position of the MCFG area.
Quoted 3 lines
The only thing which matters is ioreq routing itself -- making decisions to which device model the PCI conf/MMIO ioreq should be sent.
Hm, see above, but I'm fairly sure you need to forward those MCFG accesses as IOREQ_TYPE_PCI_CONFIG to the IOREQ server.
Quoted 3 lines
Quoted 1 line
Traditional PCI config space accesses are not IO port space accesses.
(assuming 'not' mistyped here)
Not really, this should instead be:
"Traditional PCI config space accesses are not forwarded to the IOREQ server as IO port space accesses (IOREQ_TYPE_PIO) but rather as PCI config space accesses (IOREQ_TYPE_PCI_CONFIG)."
Sorry for the confusion.
Quoted 9 lines
Quoted 3 lines
The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for
That's one of the reasons why current
IOREQ_TYPE_PCI_CONFIGimplementation is a bit inconvenient for MMCONFIG MMIO accesses -- it's too much CF8h/CFCh-centric in its implementation, might be painful to change something in the code which was intended for CF8h/CFCh handling (and not for MMIO processing).
I'm not sure I follow. Do you mean that changes should be made to the ioreq struct in order to forward MCFG accesses using IOREQ_TYPE_PCI_CONFIG as it's type?
Quoted 10 lines
Quoted 6 lines
example
xen_map_pcidevin the QEMU code.By treating MCFG accesses as MMIO you are bypassing the IOREQ PCI layer, and thus a IOREQ server could register a PCI device and only receive PCI configuration accesses from the IO port space, while MCFG accesses would be forwarded somewhere else.
It will be handled by IOREQ too, just using a different IOREQ type (MMIO one). The basic question is why do we have to stick to PCI conf space ioreqs for emulating MMIO accesses to MMCONFIG.
Because other IOREQ servers don't need to know about the position/size of the MCFG area, and cannot register MMIO ranges that cover their device's PCI configuration space in the MCFG region.
Not to mention that it would would be a terrible design flaw to force IOREQ servers to register PCI devices and MCFG areas belonging to those devices separately as MMIO in order to trap all possible PCI configuration space accesses.
Quoted 37 lines
Quoted 3 lines
I think you need to make the IOREQ code aware of the MCFG area and
XEN_DMOP_IO_RANGE_PCIneeds to forward both IO space and MCFG accesses to the right IOREQ server.Right now there is no way to inform Xen where the emulated MMCONFIG area is located in order to make this decision, based on the address within MMCONFIG range. A new dmop/hypercall is needed (with args similar to
pci_mmcfg_reserved) along with its usage in QEMU.I'll try to summarize two different approaches to MMCONFIG handling. For both approaches the final PCI config host interface for a passed through device in QEMU will remain same as at the moment -- xen_host_pci_* functions in /hw/xen.
Approach #1. Informing Xen about MMCONFIG area changes and letting Xen to translate MMIO accesses to
_PCI_CONFIGioreqs:1. QEMU will trap accesses to PCIEXBAR, calling Xen via dmop/hypercall to let the latter know of any MMCONFIG area address/size/status changes
2. Xen will trap MMIO accesses to the current MMCONFIG location and convert memory accesses into one or several
_PCI_CONFIGioreqs and send them to a chosen device model3. QEMU will receive
_PCI_CONFIGioreqs with SBDF and 12-bit offsets inside which it needs to somehow pass to pci_host_config_{read,write}_common() for emulation. It might require few hacks to make the gears turn (due to QEMU pci conf read/write model).
At the moment emulated CF8h/CFCh ports play a special role in all this --xen-hvm.cwrites an AMD-style value to the emulated CF8h port "so that the config space access will target the correct device model" (quote). Not sure about this and why it's is needed if Xen actually makes the decision to which DM the PCI conf ioreq should be sent.
Quoted 6 lines
One minor note: these new '
set_mmconfig_' dmops/hypercalls have to be triggered inside the chipset-specific emulation code in QEMU (PCIEXBAR handling in Q35 case). If there will be another machine which needs to emulate MMCONFIG control differently -- we have no choice but to insert these dmops/hypercalls into another chipset-specific emulation code as well, eg. inside HECBASE emulation code.
Maybe you could detect offsets >= 256 and replay them in QEMU like mmio accesses? Using the address_space_write or pcie_mmcfg_data_read/write functions?
I have to admit my knowledge of QEMU is quite limited, so I'm not sure of the best way to handle this.
Ideally we should find a way that doesn't involve having to modify each chipset to handle MCFG accesses from Xen. It would be nice to have some kind of interface inside of QEMU so all chipsets can register MCFG areas or modify them, but this is out of the scope of this work.
Regardless of how this ends up being implemented inside of QEMU I think the above approach is the right one from an architectural PoV.
AFAICT there are still some reserved bits in the ioreq struct that you could use to signal 'this is a MCFG PCI access' if required.
Quoted 22 lines
Approach #2. Handling MMCONFIG area inside QEMU using usual MMIO emulation:
1. QEMU will trap accesses to PCIEXBAR (or whatever else possibly supported in the future like HECBASE), eventually asking Xen to map the MMCONFIG MMIO range for ioreq servicing just like it does for any other emulated MMIO range, via
map_io_range_to_ioreq_server(). All changes in MMCONFIG placement/status will lead to remapping/unmapping the MMIO range.2. Xen will trap MMIO accesses to this area and forward them to QEMU as MMIO (
IOREQ_TYPE_COPY) ioreqs3. QEMU will receive these accesses and pass them to the existing MMCONFIG emulation -- pcie_mmcfg_data_read/write handlers, finally resulting in same xen_host_pci_* function calls as before.
This approach works "right out of the box", no changes needed for either Xen or QEMU. As both
_PCI_CONFIGand MMIO type ioreqs are processed, either method can be used to access PCI/extended config space -- CF8/CFC port I/O or MMIO accesses to MMCONFIG.
Quoted 6 lines
IOREQ routing for multiple device emulators can be supported too. In fact, the same mmconfig dmops/hypercalls can be added to let Xen know where MMCONFIG area resides, Xen will use this information to forward MMCONFIG MMIO ioreqs accordingly to BDF of the address. The difference with the approach #1 is that these interfaces are now completely optional when we use MMIO ioreqs for MMCONFIG on vanilla Xen/QEMU.
As said above, if you forward MCFG accesses as IOREQ_TYPE_COPY you are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding, this is not acceptable IMO.
Quoted 4 lines
The question is why
IOREQ_TYPE_COPY->IOREQ_TYPE_PCI_CONFIGtranslation is a must have thing at all? It won't make handling simpler. For current QEMU implementationIOREQ_TYPE_COPY(MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.
Granted it's likely easier to implement, but it's also incorrect. You seem to have in mind the picture of a single IOREQ server (QEMU) handling all the devices.
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Quoted 4 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.
I'm afraid this kind of issues would have been fairly easier to identify if a design document for this feature was sent to the list prior to it's implementation.
Regarding whether to accept something like this, I'm not really in favor, but IMO it depends on how much new code is added to handle this incorrect usage that would then go away (or would have to be changed) in order to handle the proper implementation.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 7 lines
-----Original Message-----
Quoted 4 lines
The question is why
IOREQ_TYPE_COPY->IOREQ_TYPE_PCI_CONFIGtranslation is a must have thing at all? It won't make handling simpler. For current QEMU implementationIOREQ_TYPE_COPY(MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.
Quoted 8 lines
Granted it's likely easier to implement, but it's also incorrect. You seem to have in mind the picture of a single IOREQ server (QEMU) handling all the devices.
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Indeed it will, and that is not acceptable even in the short term.
Quoted 5 lines
Quoted 4 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.
Introducing known breakage is not really on, particularly when it can be avoided with a reasonable amount of extra work.
Paul
Quoted 10 lines
I'm afraid this kind of issues would have been fairly easier to identify if a design document for this feature was sent to the list prior to it's implementation.
Regarding whether to accept something like this, I'm not really in favor, but IMO it depends on how much new code is added to handle this incorrect usage that would then go away (or would have to be changed) in order to handle the proper implementation.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 1 line · On Wed, 21 Mar 2018 09:09:11 +0000 Roger Pau Monné <[email protected]> wrote:
On Wed, Mar 21, 2018 at 10:58:40AM +1000, Alexey G wrote:
[...]
Quoted 9 lines
Quoted 6 lines
According to public slides for the feature, both PCI conf and MMIO accesses can be routed to the designated device model. It looks like for this particular setup it doesn't really matter which particular ioreq type must be used for MMCONFIG accesses -- either
IOREQ_TYPE_PCI_CONFIGorIOREQ_TYPE_COPY(MMIO accesses) should be acceptable.Isn't that going to be quite messy? How is the IOREQ server supposed to decode a MCFG access received as
IOREQ_TYPE_COPY?
This code is already available and in sync with QEMU legacy PCI conf emulation infrastructure.
Quoted 3 lines
I don't think the IOREQ server needs to know the start of the MCFG region, in which case it won't be able to detect and decode the access if it's of type
IOREQ_TYPE_COPY.
How do you think Xen will be able to know if arbitrary MMIO access targets MMCONFIG area and to which BDF the offset in this area belongs, without knowing where MMCONFIG is located and what PCI bus layout is? It's QEMU who emulate PCIEXBAR and can tell Xen where MMCONFIG is expected to be.
Quoted 7 lines
MCFG accesses need to be sent to the IOREQ server as
IOREQ_TYPE_PCI_CONFIG, or else you are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding. In your case this would work because QEMU controls the position of the MCFG region, but there's no need for other IOREQ servers to know the position of the MCFG area.
Quoted 6 lines
Quoted 3 lines
The only thing which matters is ioreq routing itself -- making decisions to which device model the PCI conf/MMIO ioreq should be sent.
Hm, see above, but I'm fairly sure you need to forward those MCFG accesses as
IOREQ_TYPE_PCI_CONFIGto the IOREQ server.
(a detailed answer below)
Quoted 24 lines
Quoted 4 lines
Quoted 2 lines
Traditional PCI config space accesses are not IO port space accesses.
(assuming 'not' mistyped here)
Not really, this should instead be:
"Traditional PCI config space accesses are not forwarded to the IOREQ server as IO port space accesses (
IOREQ_TYPE_PIO) but rather as PCI config space accesses (IOREQ_TYPE_PCI_CONFIG)."Sorry for the confusion.
Quoted 10 lines
Quoted 4 lines
The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for
That's one of the reasons why current
IOREQ_TYPE_PCI_CONFIGimplementation is a bit inconvenient for MMCONFIG MMIO accesses -- it's too much CF8h/CFCh-centric in its implementation, might be painful to change something in the code which was intended for CF8h/CFCh handling (and not for MMIO processing).
Quoted 3 lines
I'm not sure I follow. Do you mean that changes should be made to the ioreq struct in order to forward MCFG accesses using
IOREQ_TYPE_PCI_CONFIGas it's type?
No changes for ioreq structures needed for now.
Quoted 4 lines
Quoted 3 lines
It will be handled by IOREQ too, just using a different IOREQ type (MMIO one). The basic question is why do we have to stick to PCI conf space ioreqs for emulating MMIO accesses to MMCONFIG.
Quoted 8 lines
Because other IOREQ servers don't need to know about the position/size of the MCFG area, and cannot register MMIO ranges that cover their device's PCI configuration space in the MCFG region.
Not to mention that it would would be a terrible design flaw to force IOREQ servers to register PCI devices and MCFG areas belonging to those devices separately as MMIO in order to trap all possible PCI configuration space accesses.
PCI conf space layout is shared by the emulated machine. And MMCONFIG layout is mandated by this common PCI bus map.
Even if those 'multiple device models' see a different picture of PCI conf space, their visions of PCI bus must not overlap + MMCONFIG layout must be consistent between different device models.
Although it is a terrible mistake to think about the emulated PCI bus like it's a set of distinct PCI devices unrelated to each other. It's all coupled together. And this is especially true for PCIe. Many PCIe features rely on PCIe device interaction in PCIe fabric, eg. PCIe endpoints may interact with Root Complex in many ways. This cooperation may need to be emulated somehow, eg. to provide some support for PM features, link management or native hotplug facilities. Even if we have a real passed through device, we might need to provide an emulated PCIe Switch or a Root Port for it to function properly within the PCIe hierarchy.
Dedicating an isolated PCI device to some isolated device model -- that's what might be the design flaw, considering the PCIe world.
[...]
Quoted 7 lines
Maybe you could detect offsets >= 256 and replay them in QEMU like mmio accesses? Using the
address_space_writeor pcie_mmcfg_data_read/write functions?
I have to admit my knowledge of QEMU is quite limited, so I'm not sure of the best way to handle this.
Quoted 5 lines
Ideally we should find a way that doesn't involve having to modify each chipset to handle MCFG accesses from Xen. It would be nice to have some kind of interface inside of QEMU so all chipsets can register MCFG areas or modify them, but this is out of the scope of this work.
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.
2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it), xen-hvm.c should employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW, find_primary_bus() was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use in xen-hvm.c)
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Anything wrong/missing here?
(Adding Stefano and Anthony as xen-hvm.c mentioned)
Here is another suggestion:
1. QEMU use existing facilities to emulate PCIEXBAR for a Q35 machine, calling Xen's map_io_range_to_ioreq_server() API to mark MMIO range for emulation, just like for any other emulated MMIO range
2. All accesses to this area will be forwarded to QEMU as MMIO ioreqs and emulated flawlessly as everything is within QEMU architecture -- pci-host/PCIBus/PCIDevice machinery in place. No workarounds required for xen-hvm.c
3. CF8/CFC accesses will be forwarded as _PCI_CONFIG ioreqs, as usually. Both methods are in sync as they use common PCI emulation infrastructure in QEMU
4. At this point absolutely zero changes are required in both Xen and QEMU code. Only existing interfaces are used. In fact, no related code changes required at all except a bugfix for PCIEXBAR mask emulation (provided in this series)
5. But. Just to make the 'multiple device emulators' (no extra reasons so far) feature to work, we add the same hypercall/dmop usage to let Xen know where QEMU emulates MMCONFIG
6. Xen will continue to trap accesses to this range but instead of sending _COPY ioreq immediately, he will check the address against known MMCONFIG location (in the same manner as above), then convert the offset within it to BDF and he can proceed to usual BDF-based ioreq routing for those device emulator DMs, whatever they are
7. In fact, MMIO -> PCI conf ioreq translation can be freely used as well at this stage, if it is more convenient for 'multiple device emulators' feature users. It can be even made selectable.
So, the question which needs explanation is: why do you think MMIO->PCI conf ioreq translation is mandatory for MMCONFIG? Can't we just add new hypercall/dmop to make ioreq routing for 'multiple device emulators' to work while letting QEMU to use any API provided for him to do its tasks?
It's kinda funny to pretend that QEMU don't know anything about MMCONFIG being MMIO when it's QEMU who inform Xen about its memory address and size.
Quoted 63 lines
Regardless of how this ends up being implemented inside of QEMU I think the above approach is the right one from an architectural PoV.
AFAICT there are still some reserved bits in the ioreq struct that you could use to signal 'this is a MCFG PCI access' if required.
Quoted 22 lines
Approach #2. Handling MMCONFIG area inside QEMU using usual MMIO emulation:
1. QEMU will trap accesses to PCIEXBAR (or whatever else possibly supported in the future like HECBASE), eventually asking Xen to map the MMCONFIG MMIO range for ioreq servicing just like it does for any other emulated MMIO range, via
map_io_range_to_ioreq_server(). All changes in MMCONFIG placement/status will lead to remapping/unmapping the MMIO range.2. Xen will trap MMIO accesses to this area and forward them to QEMU as MMIO (
IOREQ_TYPE_COPY) ioreqs3. QEMU will receive these accesses and pass them to the existing MMCONFIG emulation -- pcie_mmcfg_data_read/write handlers, finally resulting in same xen_host_pci_* function calls as before.
This approach works "right out of the box", no changes needed for either Xen or QEMU. As both
_PCI_CONFIGand MMIO type ioreqs are processed, either method can be used to access PCI/extended config space -- CF8/CFC port I/O or MMIO accesses to MMCONFIG.Quoted 7 lines
IOREQ routing for multiple device emulators can be supported too. In fact, the same mmconfig dmops/hypercalls can be added to let Xen know where MMCONFIG area resides, Xen will use this information to forward MMCONFIG MMIO ioreqs accordingly to BDF of the address. The difference with the approach #1 is that these interfaces are now completely optional when we use MMIO ioreqs for MMCONFIG on vanilla Xen/QEMU.
As said above, if you forward MCFG accesses as
IOREQ_TYPE_COPYyou are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding, this is not acceptable IMO.Quoted 5 lines
The question is why
IOREQ_TYPE_COPY->IOREQ_TYPE_PCI_CONFIGtranslation is a must have thing at all? It won't make handling simpler. For current QEMU implementationIOREQ_TYPE_COPY(MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.Granted it's likely easier to implement, but it's also incorrect. You seem to have in mind the picture of a single IOREQ server (QEMU) handling all the devices.
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Quoted 4 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.I'm afraid this kind of issues would have been fairly easier to identify if a design document for this feature was sent to the list prior to it's implementation.
Quoted 6 lines
Regarding whether to accept something like this, I'm not really in favor, but IMO it depends on how much new code is added to handle this incorrect usage that would then go away (or would have to be changed) in order to handle the proper implementation.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantQuoted 6 lines · On Wed, 21 Mar 2018 09:36:04 +0000 Paul Durrant <[email protected]> wrote:
Quoted 4 lines
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Indeed it will, and that is not acceptable even in the short term.
Hmm, what exactly you are rejecting? QEMU's usage of established (and provided by Xen) interfaces for QEMU to use? Any particular reason why QEMU can use map_io_range_to_ioreq_server() in one case and can't in another? It's API available for QEMU after all.
If we actually switch to the emulated MMCONFIG range informing approach for Xen (via a new dmop/hypercall), who should prevent QEMU to actually map this range via map_io_range_to_ioreq_server? QEMU itself? Or Xen? How to will look, "QEMU asks us to map this range as emulated MMIO, but he previously told us that emulated PCIEXBAR register points there, so we won't allow him to do it"?
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.Introducing known breakage is not really on, particularly when it can be avoided with a reasonable amount of extra work.
It's hard to break something which doesn't exist. :) Multiple device emulators feature do not support translation/routing of MMCONFIG MMIO accesses currently, it must be designed first.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 118 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 21 March 2018 14:26
To: Roger Pau Monne <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Anthony Perard <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 1 line · On Wed, 21 Mar 2018 09:09:11 +0000 Roger Pau Monné <[email protected]> wrote:
On Wed, Mar 21, 2018 at 10:58:40AM +1000, Alexey G wrote:
[...]
Quoted 5 lines
Quoted 5 lines
According to public slides for the feature, both PCI conf and MMIO accesses can be routed to the designated device model. It looks like for this particular setup it doesn't really matter which particular ioreq type must be used for MMCONFIG accesses -- either
IOREQ_TYPE_PCI_CONFIGorIOREQ_TYPE_COPY(MMIO accesses) shouldbe
Quoted 4 lines
Quoted 1 line
acceptable.
Isn't that going to be quite messy? How is the IOREQ server supposed to decode a MCFG access received as
IOREQ_TYPE_COPY?This code is already available and in sync with QEMU legacy PCI conf emulation infrastructure.
Quoted 3 lines
I don't think the IOREQ server needs to know the start of the MCFG region, in which case it won't be able to detect and decode the access if it's of type
IOREQ_TYPE_COPY.How do you think Xen will be able to know if arbitrary MMIO access targets MMCONFIG area and to which BDF the offset in this area belongs, without knowing where MMCONFIG is located and what PCI bus layout is? It's QEMU who emulate PCIEXBAR and can tell Xen where MMCONFIG is expected to be.
Quoted 7 lines
MCFG accesses need to be sent to the IOREQ server as
IOREQ_TYPE_PCI_CONFIG, or else you are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding. In your case this would work because QEMU controls the position of the MCFG region, but there's no need for other IOREQ servers to know the position of the MCFG area.Quoted 6 lines
Quoted 3 lines
The only thing which matters is ioreq routing itself -- making decisions to which device model the PCI conf/MMIO ioreq should be sent.
Hm, see above, but I'm fairly sure you need to forward those MCFG accesses as
IOREQ_TYPE_PCI_CONFIGto the IOREQ server.(a detailed answer below)
Quoted 24 lines
Quoted 4 lines
Quoted 2 lines
Traditional PCI config space accesses are not IO port space accesses.
(assuming 'not' mistyped here)
Not really, this should instead be:
"Traditional PCI config space accesses are not forwarded to the IOREQ server as IO port space accesses (
IOREQ_TYPE_PIO) but rather as PCI config space accesses (IOREQ_TYPE_PCI_CONFIG)."Sorry for the confusion.
Quoted 10 lines
Quoted 4 lines
The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for
That's one of the reasons why current
IOREQ_TYPE_PCI_CONFIGimplementation is a bit inconvenient for MMCONFIG MMIO accesses -- it's too much CF8h/CFCh-centric in its implementation, might be painful to change something in the code which was intended for CF8h/CFCh handling (and not for MMIO processing).Quoted 3 lines
I'm not sure I follow. Do you mean that changes should be made to the ioreq struct in order to forward MCFG accesses using
IOREQ_TYPE_PCI_CONFIGas it's type?No changes for ioreq structures needed for now.
Quoted 4 lines
Quoted 3 lines
It will be handled by IOREQ too, just using a different IOREQ type (MMIO one). The basic question is why do we have to stick to PCI conf space ioreqs for emulating MMIO accesses to MMCONFIG.
Quoted 8 lines
Because other IOREQ servers don't need to know about the position/size of the MCFG area, and cannot register MMIO ranges that cover their device's PCI configuration space in the MCFG region.
Not to mention that it would would be a terrible design flaw to force IOREQ servers to register PCI devices and MCFG areas belonging to those devices separately as MMIO in order to trap all possible PCI configuration space accesses.
PCI conf space layout is shared by the emulated machine. And MMCONFIG layout is mandated by this common PCI bus map.
Even if those 'multiple device models' see a different picture of PCI conf space, their visions of PCI bus must not overlap + MMCONFIG layout must be consistent between different device models.
Although it is a terrible mistake to think about the emulated PCI bus like it's a set of distinct PCI devices unrelated to each other. It's all coupled together. And this is especially true for PCIe. Many PCIe features rely on PCIe device interaction in PCIe fabric, eg. PCIe endpoints may interact with Root Complex in many ways. This cooperation may need to be emulated somehow, eg. to provide some support for PM features, link management or native hotplug facilities. Even if we have a real passed through device, we might need to provide an emulated PCIe Switch or a Root Port for it to function properly within the PCIe hierarchy.
Quoted 3 lines
Dedicating an isolated PCI device to some isolated device model -- that's what might be the design flaw, considering the PCIe world.
I think that is the crux of the problem. The current multi-ioreq-server relies on being able to consider PCI devices as being isolated from each other... and that is basically fine because we only use a single PCI bus with no bridges. To move to PCIe will require more emulation in Xen, but I think that is the only way to do it properly.
Quoted 55 lines
[...]
Quoted 7 lines
Maybe you could detect offsets >= 256 and replay them in QEMU like mmio accesses? Using the
address_space_writeor pcie_mmcfg_data_read/write functions?
I have to admit my knowledge of QEMU is quite limited, so I'm not sure of the best way to handle this.Quoted 5 lines
Ideally we should find a way that doesn't involve having to modify each chipset to handle MCFG accesses from Xen. It would be nice to have some kind of interface inside of QEMU so all chipsets can register MCFG areas or modify them, but this is out of the scope of this work.
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation
of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)
Quoted 8 lines
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Anything wrong/missing here?
That all sounds plausible. All we essentially need to do is make sure the config space transactions make it to the right device model in QEMU. If the emulation in Xen is comprehensive then I guess there should not even be any reason for QEMU's idea of the bus topology and Xen's presentation of the bus topology to the guest to even match.
Paul
Quoted 57 lines
(Adding Stefano and Anthony as
xen-hvm.cmentioned)Here is another suggestion:
1. QEMU use existing facilities to emulate PCIEXBAR for a Q35 machine, calling Xen's
map_io_range_to_ioreq_server()API to mark MMIO range for emulation, just like for any other emulated MMIO range2. All accesses to this area will be forwarded to QEMU as MMIO ioreqs and emulated flawlessly as everything is within QEMU architecture -- pci-host/PCIBus/PCIDevice machinery in place. No workarounds required for
xen-hvm.c3. CF8/CFC accesses will be forwarded as
_PCI_CONFIGioreqs, as usually. Both methods are in sync as they use common PCI emulation infrastructure in QEMU4. At this point absolutely zero changes are required in both Xen and QEMU code. Only existing interfaces are used. In fact, no related code changes required at all except a bugfix for PCIEXBAR mask emulation (provided in this series)
5. But. Just to make the 'multiple device emulators' (no extra reasons so far) feature to work, we add the same hypercall/dmop usage to let Xen know where QEMU emulates MMCONFIG
6. Xen will continue to trap accesses to this range but instead of sending
_COPYioreq immediately, he will check the address against known MMCONFIG location (in the same manner as above), then convert the
offset within it to BDF and he can proceed to usual BDF-based ioreq routing for those device emulator DMs, whatever they are7. In fact, MMIO -> PCI conf ioreq translation can be freely used as well at this stage, if it is more convenient for 'multiple device emulators' feature users. It can be even made selectable.
So, the question which needs explanation is: why do you think
MMIO->PCIconf ioreq translation is mandatory for MMCONFIG? Can't we just add new hypercall/dmop to make ioreq routing for 'multiple device emulators' to work while letting QEMU to use any API provided for him to do its tasks?It's kinda funny to pretend that QEMU don't know anything about MMCONFIG being MMIO when it's QEMU who inform Xen about its memory address and size.
Quoted 9 lines
Regardless of how this ends up being implemented inside of QEMU I think the above approach is the right one from an architectural PoV.
AFAICT there are still some reserved bits in the ioreq struct that you could use to signal 'this is a MCFG PCI access' if required.
Quoted 3 lines
Approach #2. Handling MMCONFIG area inside QEMU using usual MMIO emulation:
Quoted 61 lines
Quoted 5 lines
Quoted 5 lines
1. QEMU will trap accesses to PCIEXBAR (or whatever else possibly supported in the future like HECBASE), eventually asking Xen to map the MMCONFIG MMIO range for ioreq servicing just like it does for any other emulated MMIO range, via
map_io_range_to_ioreq_server(). All changes in MMCONFIG placement/status will lead toremapping/unmapping
Quoted 55 lines
Quoted 14 lines
the MMIO range.
2. Xen will trap MMIO accesses to this area and forward them to QEMU as MMIO (
IOREQ_TYPE_COPY) ioreqs3. QEMU will receive these accesses and pass them to the existing MMCONFIG emulation -- pcie_mmcfg_data_read/write handlers, finally resulting in same xen_host_pci_* function calls as before.
This approach works "right out of the box", no changes needed for either Xen or QEMU. As both
_PCI_CONFIGand MMIO type ioreqs are processed, either method can be used to access PCI/extended config space -- CF8/CFC port I/O or MMIO accesses to MMCONFIG.Quoted 7 lines
IOREQ routing for multiple device emulators can be supported too. In fact, the same mmconfig dmops/hypercalls can be added to let Xen know where MMCONFIG area resides, Xen will use this information to forward MMCONFIG MMIO ioreqs accordingly to BDF of the address. The difference with the approach #1 is that these interfaces are now completely optional when we use MMIO ioreqs for MMCONFIG on vanilla Xen/QEMU.
As said above, if you forward MCFG accesses as
IOREQ_TYPE_COPYyou are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding, this is not acceptable IMO.Quoted 5 lines
The question is why
IOREQ_TYPE_COPY->IOREQ_TYPE_PCI_CONFIGtranslation is a must have thing at all? It won't make handling simpler. For current QEMU implementationIOREQ_TYPE_COPY(MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.Granted it's likely easier to implement, but it's also incorrect. You seem to have in mind the picture of a single IOREQ server (QEMU) handling all the devices.
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Quoted 4 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.I'm afraid this kind of issues would have been fairly easier to identify if a design document for this feature was sent to the list prior to it's implementation.
Regarding whether to accept something like this, I'm not really in favor, but IMO it depends on how much new code is added to handle this incorrect usage that would then go away (or would have to be changed) in order to handle the proper implementation.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 20 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 21 March 2018 14:35
To: Paul Durrant <[email protected]>
Cc: Roger Pau Monne <[email protected]>; xen- [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 6 lines · On Wed, 21 Mar 2018 09:36:04 +0000 Paul Durrant <[email protected]> wrote:
Quoted 4 lines
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Indeed it will, and that is not acceptable even in the short term.
Quoted 8 lines
Hmm, what exactly you are rejecting? QEMU's usage of established (and provided by Xen) interfaces for QEMU to use? Any particular reason why QEMU can use
map_io_range_to_ioreq_server()in one case and can't in another? It's API available for QEMU after all.If we actually switch to the emulated MMCONFIG range informing approach for Xen (via a new dmop/hypercall), who should prevent QEMU to actually map this range via
map_io_range_to_ioreq_server? QEMU itself? Or Xen?
Xen internal emulation always trumps any external emulator, so even if QEMU maps an MMIO range it will not see any accesses if Xen is handling emulation of that range.
Quoted 9 lines
How to will look, "QEMU asks us to map this range as emulated MMIO, but he previously told us that emulated PCIEXBAR register points there, so we won't allow him to do it"?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.
Quoted 6 lines
Quoted 2 lines
Introducing known breakage is not really on, particularly when it can be avoided with a reasonable amount of extra work.
It's hard to break something which doesn't exist. :) Multiple device emulators feature do not support translation/routing of MMCONFIG MMIO accesses currently, it must be designed first.
Indeed, but updating to a new chipset emulation that breaks existing functionality is not going to be helpful.
Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 16 lines · Alexey G
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.
2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
Quoted 8 lines
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
Why can't you just set size = 8 in that case in the ioreq?
QEMU should then reject those if the chipset doesn't support 64bit accesses. I cannot find in the spec any mention of whether this chipset supports 64bit MCFG accesses, and according to the PCIe spec 64bit accesses to MCFG should not be used unless the chipset is known to handle them correctly.
Quoted 4 lines
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
Quoted 10 lines
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)
That seems like a minor nit, but why not just use address_space_{read/write} to replay the MCFG accesses as memory read/writes?
Quoted 3 lines
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
If you replay the read/write I don't think so. In any case this is irrelevant. QEMU CPU emulation code is also unused when running under Xen.
Quoted 5 lines
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Anything wrong/missing here?
I think that's correct.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Alexey GQuoted 10 lines · Alexey G
Quoted 10 lines
Quoted 10 lines
if (b_info->u.hvm.mmio_hole_memkb) { uint64_t max_ram_below_4g = (1ULL << 32) -(quoted diff) +0
-0b/tools/libxl/libxl_types.idl index
35038120ca..f3ef3cbdde100644tools/libxl/libxl_types.idl +0
-0@@ -101,6 +101,12 @@ libxl_device_model_version =Enumeration("device_model_version", [ (2, "
QEMU_XEN"), #
Upstream based qemu-xen device model ])
Quoted 8 lines
Quoted 4 lines
tools/libxl/libxl_types.idl libxl_device_model_version = Enumeration("device_model_version", [ · line 105
Quoted 2 lines
tools/libxl/libxl_types.idl +2
-0+libxl_device_model_machine = Enumeration("device_model_machine", [ + (0, "UNKNOWN"),Shouldn't this be named DEFAULT?
"Unknown" here should be read as "unspecified", but I guess DEFAULT will be clearer anyway.
I'm afraid the ship has already sailed. There are far too many UNKNOWNs in libxl_types.idl so we might as well stick to it here.
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Roger Pau MonnéQuoted 21 lines · Roger Pau Monné
Quoted 12 lines · Alexey G
Quoted 8 lines · On Mon, 19 Mar 2018 17:01:18 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 5 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.I would rather name this
machine_chipsetordevice_model_chipset.Quoted 7 lines
device_model_prefix is a must I think -- multiple device model related options have names starting withdevice_model_.
device_model_chipset... well, maybe, but we're actually specifying a QEMU machine here. In QEMU mailing list there was even a suggestion to allow to pass a machine version number here, like "pc-q35-2.10". I think some opinions are needed here.
Quoted 5 lines
I'm not sure what a 'machine' is in QEMU speak, but in my mind I would consider PC a machine (vs ARM for example).
I think 'chipset' is clearer, but again others should express their opinion.
AIUI machine is a collection of chipset and peripherals, i.e. it covers more than the chipset alone.
Cc Anthony for correction.
Wei.
Quoted 2 lines
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 09/12] libxl: Xen Platform device support for Q35
In reply to Alexey GQuoted 36 lines · Alexey G
Quoted 33 lines · On Tue, 13 Mar 2018 04:33:54 +1000 Alexey Gerasimenko <[email protected]> wrote:
Current Xen/QEMU method to control Xen Platform device is a bit odd -- changing '
xen_platform_device' option value actually modifies QEMU emulated machine type, namely xenfv <--> pc.In order to avoid multiplying machine types, use the new way to control Xen Platform device for QEMU -- xen-platform-dev property. To maintain backward compatibility with existing Xen/QEMU setups, this is only applicable to q35 machine currently. i440 emulation uses the old method (xenfv/pc machine) to control Xen Platform device, this may be changed later to xen-platform-dev property as well.
Signed-off-by Alexey Gerasimenko <[email protected]>---
tools/libxl/libxl_dm.c| 6 +++++-1 file changed, 5 insertions(+), 1 deletion(-)tools/libxl/libxl_dm.c +5
-1@@ -1444,7 +1444,11 @@ static intlibxl__build_device_model_args_new (libxl__gc *gc, break;case LIBXL_DOMAIN_TYPE_HVM :if (b_info->device_model_machine ==LIBXL_DEVICE_MODEL_MACHINE_Q35 ) { - machinearg =libxl__sprintf (gc,"q35,accel=xen" ); +if (!libxl_defbool_val (b_info->u.hvm.xen_platform_pci)) { + machinearg =libxl__sprintf (gc,"q35,accel=xen" ); + }else { + machinearg =libxl__sprintf (gc,"q35,accel=xen,xen-platform-dev=on" ); + }
Quoted 8 lines
Quoted 4 lines
} else { if (!libxl_defbool_val(b_info->u.hvm.xen_platform_pci)) { /* Switching here to the machine "pc" which does notadd
Regarding this one -- QEMU maintainers suggested that supplying '-device xen-platform' directly should be a better approach than a machine property, so this patch is kinda obsolete.
I agree with QEMU maintainers.
Quoted 7 lines
Right now "xenfv" machine usage for qemu-xen seems to be limited to controlling the Xen platform device and applying the
HVM_MAX_VCPUSvalue to maxcpus + minor changes related to IGD passthrough. Both should be applicable for a "pc,accel=xen" machine as well I think, which in fact currently lacks theHVM_MAX_VCPUScheck for some reason.
Quoted 7 lines
Adding a distinct method to control Xen platform device for the q35 machine suggests to propagate the same approach to i440 machine types, but... it depends on who else can use xenfv for qemu-xen (not to be confused with xenfv usage on qemu-traditional).
Is there any other toolstacks/code which use xenfv machine solely to turn on/off Xen platform device?
Check libvirt?
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 27 lines · On Wed, 21 Mar 2018 15:20:17 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 17 lines · Alexey G
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.
2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
Quoted 8 lines
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
Quoted 7 lines
Why can't you just set size = 8 in that case in the ioreq?
QEMU should then reject those if the chipset doesn't support 64bit accesses. I cannot find in the spec any mention of whether this chipset supports 64bit MCFG accesses, and according to the PCIe spec 64bit accesses to MCFG should not be used unless the chipset is known to handle them correctly.
Yes, uint64_t should be enough in this particular case in fact, though memory nature of MMCONFIG accesses might still require to provide specific handling.
All right then, so it will be a dmop/hypercall to tell Xen where to trap MMIO accesses to MMCONFIG as you propose.
The primary device model (QEMU) will be emulating chipset-specific PCIEXBAR/etc and issuing this new dmop to tell Xen which area it needs to trap for MMIO MMCONFIG acceses. It's basically what map_io_range_to_ioreq_server does currently, but I guess a new dedicated dmop/hypercall is bearable.
Quoted 15 lines
Quoted 4 lines
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
Quoted 10 lines
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)
Quoted 3 lines
That seems like a minor nit, but why not just use address_space_{read/write} to replay the MCFG accesses as memory read/writes?
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for _PCI_CONFIG ioreqs. Here is how it will look:
QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is accessing this area -> Xen intercepts this MMIO access
But here's what happens next:
Xen translates MMIO access into PCI_CONFIG and sends it to DM -> DM receives _PCI_CONFIG ioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulation
I tnink some parts of this equation can be collapsed, isn't it?
Above scenario makes it obvious that at least for QEMU the MMIO->PCI conf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs? We can still route either ioreq type to multiple device emulators accordingly.
This will be the most universal and consistent approach -- either _COPY or _PCI_CONFIG-type ioreqs can be sent to DM, whatever it likes more.
Quoted 11 lines
Quoted 2 lines
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
If you replay the read/write I don't think so. In any case this is irrelevant. QEMU CPU emulation code is also unused when running under Xen.
Quoted 4 lines
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Quoted 5 lines
Quoted 1 line
Anything wrong/missing here?
I think that's correct.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 91 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 21 March 2018 16:57
To: Roger Pau Monne <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Anthony Perard <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 27 lines · On Wed, 21 Mar 2018 15:20:17 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 17 lines · Alexey G
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.
2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
Quoted 8 lines
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
Quoted 7 lines
Why can't you just set size = 8 in that case in the ioreq?
QEMU should then reject those if the chipset doesn't support 64bit accesses. I cannot find in the spec any mention of whether this chipset supports 64bit MCFG accesses, and according to the PCIe spec 64bit accesses to MCFG should not be used unless the chipset is known to handle them correctly.
Yes,
uint64_tshould be enough in this particular case in fact, though memory nature of MMCONFIG accesses might still require to provide specific handling.All right then, so it will be a dmop/hypercall to tell Xen where to trap MMIO accesses to MMCONFIG as you propose.
The primary device model (QEMU) will be emulating chipset-specific PCIEXBAR/etc and issuing this new dmop to tell Xen which area it needs to trap for MMIO MMCONFIG acceses. It's basically what
map_io_range_to_ioreq_serverdoes currently, but I guess a new dedicated dmop/hypercall is bearable.Quoted 3 lines
Quoted 3 lines
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf
reads/writes
Quoted 12 lines
Quoted 11 lines
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)Quoted 3 lines
That seems like a minor nit, but why not just use address_space_{read/write} to replay the MCFG accesses as memory read/writes?
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is
accessing this area -> Xen intercepts this MMIO access
Quoted 7 lines
But here's what happens next:
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulation
That would only be true of a dm that cannot handle PCI config ioreqs directly.
Paul
Quoted 27 lines
I tnink some parts of this equation can be collapsed, isn't it?
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs? We can still route either ioreq type to multiple device emulators accordingly.This will be the most universal and consistent approach -- either
_COPYor _PCI_CONFIG-type ioreqs can be sent to DM, whatever it likes more.Quoted 16 lines
Quoted 2 lines
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
If you replay the read/write I don't think so. In any case this is irrelevant. QEMU CPU emulation code is also unused when running under Xen.
Quoted 5 lines
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Anything wrong/missing here?
I think that's correct.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 08/12] libxl: Q35 support (new option device_model_machine)
In reply to Wei LiuQuoted 22 lines · Wei Liu
Quoted 21 lines · Roger Pau Monné
Quoted 19 lines · Alexey G
Quoted 8 lines · On Mon, 19 Mar 2018 17:01:18 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 5 lines · Alexey Gerasimenko
Provide a new domain config option to select the emulated machine type,
device_model_machine. It has following possible values:
- "i440" - i440 emulation (default)
- "q35" - emulate a Q35 machine. By default, the storage interface is AHCI.I would rather name this
machine_chipsetordevice_model_chipset.
device_model_prefix is a must I think -- multiple device model related options have names starting withdevice_model_.
device_model_chipset... well, maybe, but we're actually specifying a QEMU machine here. In QEMU mailing list there was even a suggestion to allow to pass a machine version number here, like "pc-q35-2.10". I think some opinions are needed here.
Quoted 8 lines
Quoted 5 lines
I'm not sure what a 'machine' is in QEMU speak, but in my mind I would consider PC a machine (vs ARM for example).
I think 'chipset' is clearer, but again others should express their opinion.
AIUI machine is a collection of chipset and peripherals, i.e. it covers more than the chipset alone.
The description of the QEMU machine "q35" is
"Standard PC (Q35 + ICH9, 2009)". So right in the description, Q35 is not enought to describe what -machine=q35 is about. And FYI "pc" or "pc_piix" description is "Standard PC (i440FX + PIIX, 1996)".
Also, we could expand the option to actually allow a user to select the exact version of the QEMU machine to use. Having "pc-i440fx-2.12" in the xl config file instead of just "pc" could prevent compatibility issue for an existing virtual machine.
I don't know what a chipset is when related to QEMU, beside being a piece of silicon in hardware. I think a QEMU machine is closer to a motherboard than just a chipset. Feel free to read "qemu.git/hw/i386/pc_piix.c" and "qemu.git/hw/i386/pc_q35.c" to read the difference between both machine.
Anyway, I think "device_model_machine" is better that ".._chipset". "machine" better describe the change involve when selecting q35.
Signature · 6 lines
Anthony PERARD _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 33 lines · Alexey G
Quoted 12 lines · On Wed, 21 Mar 2018 15:20:17 +0000 Roger Pau Monné <[email protected]> wrote:
Quoted 10 lines · Alexey G
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)Quoted 3 lines
That seems like a minor nit, but why not just use address_space_{read/write} to replay the MCFG accesses as memory read/writes?
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is accessing this area -> Xen intercepts this MMIO access
But here's what happens next:
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulation
Quoted 6 lines
I tnink some parts of this equation can be collapsed, isn't it?
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs?
You are just adding an extra level of complexity to an interface that's fairly simple. You register a PCI device using XEN_DMOP_IO_RANGE_PCI and you get IOREQ_TYPE_PCI_CONFIG ioreqs. Getting both IOREQ_TYPE_PCI_CONFIG and IOREQ_TYPE_COPY for PCI config space access is misleading.
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the IOREQ_TYPE_PCI_CONFIG which is trivial.
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.
It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an IOREQ_TYPE_PCI_CONFIG and it replays the IO space access using do_outp and cpu_ioreq_pio.
If you think using IOREQ_TYPE_COPY for MCFG accesses is such a benefit for QEMU, why not just translate the IOREQ_TYPE_PCI_CONFIG into IOREQ_TYPE_COPY in handle_ioreq and dispatch it using cpu_ioreq_move?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantQuoted 124 lines · On Wed, 21 Mar 2018 14:54:16 +0000 Paul Durrant <[email protected]> wrote:
Quoted 120 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 21 March 2018 14:26
To: Roger Pau Monne <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Anthony Perard <[email protected]>; Stefano Stabellini <[email protected]> Subject: Re: [Xen-devel] [RFC PATCH 07/12]
hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 1 line · On Wed, 21 Mar 2018 09:09:11 +0000 Roger Pau Monné <[email protected]> wrote:
On Wed, Mar 21, 2018 at 10:58:40AM +1000, Alexey G wrote:
[...]
Quoted 5 lines
Quoted 5 lines
According to public slides for the feature, both PCI conf and MMIO accesses can be routed to the designated device model. It looks like for this particular setup it doesn't really matter which particular ioreq type must be used for MMCONFIG accesses -- either
IOREQ_TYPE_PCI_CONFIGorIOREQ_TYPE_COPY(MMIO accesses) shouldbe
Quoted 4 lines
Quoted 1 line
acceptable.
Isn't that going to be quite messy? How is the IOREQ server supposed to decode a MCFG access received as
IOREQ_TYPE_COPY?This code is already available and in sync with QEMU legacy PCI conf emulation infrastructure.
Quoted 3 lines
I don't think the IOREQ server needs to know the start of the MCFG region, in which case it won't be able to detect and decode the access if it's of type
IOREQ_TYPE_COPY.How do you think Xen will be able to know if arbitrary MMIO access targets MMCONFIG area and to which BDF the offset in this area belongs, without knowing where MMCONFIG is located and what PCI bus layout is? It's QEMU who emulate PCIEXBAR and can tell Xen where MMCONFIG is expected to be.
Quoted 7 lines
MCFG accesses need to be sent to the IOREQ server as
IOREQ_TYPE_PCI_CONFIG, or else you are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding. In your case this would work because QEMU controls the position of the MCFG region, but there's no need for other IOREQ servers to know the position of the MCFG area.Quoted 6 lines
Quoted 3 lines
The only thing which matters is ioreq routing itself -- making decisions to which device model the PCI conf/MMIO ioreq should be sent.
Hm, see above, but I'm fairly sure you need to forward those MCFG accesses as
IOREQ_TYPE_PCI_CONFIGto the IOREQ server.(a detailed answer below)
Quoted 24 lines
Quoted 4 lines
Quoted 2 lines
Traditional PCI config space accesses are not IO port space accesses.
(assuming 'not' mistyped here)
Not really, this should instead be:
"Traditional PCI config space accesses are not forwarded to the IOREQ server as IO port space accesses (
IOREQ_TYPE_PIO) but rather as PCI config space accesses (IOREQ_TYPE_PCI_CONFIG)."Sorry for the confusion.
Quoted 10 lines
Quoted 4 lines
The IOREQ code in Xen detects accesses to ports 0xcf8/0xcfc and IOREQ servers can register devices they would like to receive configuration space accesses for. QEMU is already making use of this, see for
That's one of the reasons why current
IOREQ_TYPE_PCI_CONFIGimplementation is a bit inconvenient for MMCONFIG MMIO accesses -- it's too much CF8h/CFCh-centric in its implementation, might be painful to change something in the code which was intended for CF8h/CFCh handling (and not for MMIO processing).Quoted 3 lines
I'm not sure I follow. Do you mean that changes should be made to the ioreq struct in order to forward MCFG accesses using
IOREQ_TYPE_PCI_CONFIGas it's type?No changes for ioreq structures needed for now.
Quoted 9 lines
Quoted 3 lines
It will be handled by IOREQ too, just using a different IOREQ type (MMIO one). The basic question is why do we have to stick to PCI conf space ioreqs for emulating MMIO accesses to MMCONFIG.
Because other IOREQ servers don't need to know about the position/size of the MCFG area, and cannot register MMIO ranges that cover their device's PCI configuration space in the MCFG region.
Quoted 4 lines
Not to mention that it would would be a terrible design flaw to force IOREQ servers to register PCI devices and MCFG areas belonging to those devices separately as MMIO in order to trap all possible PCI configuration space accesses.
PCI conf space layout is shared by the emulated machine. And MMCONFIG layout is mandated by this common PCI bus map.
Even if those 'multiple device models' see a different picture of PCI conf space, their visions of PCI bus must not overlap + MMCONFIG layout must be consistent between different device models.
Although it is a terrible mistake to think about the emulated PCI bus like it's a set of distinct PCI devices unrelated to each other. It's all coupled together. And this is especially true for PCIe. Many PCIe features rely on PCIe device interaction in PCIe fabric, eg. PCIe endpoints may interact with Root Complex in many ways. This cooperation may need to be emulated somehow, eg. to provide some support for PM features, link management or native hotplug facilities. Even if we have a real passed through device, we might need to provide an emulated PCIe Switch or a Root Port for it to function properly within the PCIe hierarchy.
Quoted 3 lines
Dedicating an isolated PCI device to some isolated device model -- that's what might be the design flaw, considering the PCIe world.
Quoted 6 lines
I think that is the crux of the problem. The current multi-ioreq-server relies on being able to consider PCI devices as being isolated from each other... and that is basically fine because we only use a single PCI bus with no bridges. To move to PCIe will require more emulation in Xen, but I think that is the only way to do it properly.
Unfortunately, this approach won't work anymore for PCIe. We can't just separate PCI bus emulation with the chipset-specific emulation. Even MMCONFIG is a chipset-specific feature. In order to do this, Xen should emulate many device model features itself, probably to the point when QEMU can be safely dropped as DM.
The single bus has become a major issue for passthrough already:
http://lists.gnu.org/archive/html/qemu-devel/2018-03/msg03593.html
I want to replace this workaround with actual multiple bus support, although now I'm not sure if I can make it through -- there will be a lot of resistance likely. Although this is a mandatory feature for PCIe PT.
Quoted 63 lines
Quoted 63 lines
[...]
Quoted 7 lines
Maybe you could detect offsets >= 256 and replay them in QEMU like mmio accesses? Using the
address_space_writeor pcie_mmcfg_data_read/write functions?
I have to admit my knowledge of QEMU is quite limited, so I'm not sure of the best way to handle this.Quoted 5 lines
Ideally we should find a way that doesn't involve having to modify each chipset to handle MCFG accesses from Xen. It would be nice to have some kind of interface inside of QEMU so all chipsets can register MCFG areas or modify them, but this is out of the scope of this work.
Roger, Paul,
Here is what you suggest, just to clarify:
1. Add to Xen a new hypercall (+corresponding dmop) so QEMU can tell Xen where QEMU emulates machine's MMCONFIG (chipset-specific emulation
of PCIEXBAR/HECBASE/etc mmcfg relocation). Xen will rely on this information to know to which PCI device the address within MMCONFIG belong.2. Xen will trap this area + remap its trapping to other address if QEMU will inform Xen about emulated PCIEXBAR value change
3. Every MMIO access to the current MMCONFIG range will be converted into BDF first (by offset within this range, knowing where the range is)
4. Target device model is selected using calculated BDF
5. MMIO read/write accesses are converted into PCI config space ioreqs (like it was a CF8/CFCh operation instead of MMIO access). At this point ioreq structure allows to specify extended PCI conf offset (12-bit), so it will fit into PCI conf ioreq. For now let's assume that eg. a 64-bit memory operation is either aborted or workarounded by splitting this operation into multiple PCI conf ioreqs.
6. PCI conf read/write ioreqs are sent to the chosen device model
7. QEMU receive MMCONFIG memory reads/writes as PCI conf reads/writes
8. As these MMCONFIG PCI conf reads occur out of context (just address/len/data without any emulated device attached to it),
xen-hvm.cshould employ special logic to make it QEMU-friendly -- eg. right now it sends received PCI conf access into (emulated by QEMU) CF8h/CFCh ports.
There is a real problem to embed these "naked" accesses into QEMU infrastructure, workarounds are required. BTW,find_primary_bus()was dropped from QEMU code -- it could've been useful here. Let's assume some workaround is employed (like storing a required object pointers in global variables for later use inxen-hvm.c)9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Quoted 131 lines
Quoted 1 line
Anything wrong/missing here?
That all sounds plausible. All we essentially need to do is make sure the config space transactions make it to the right device model in QEMU. If the emulation in Xen is comprehensive then I guess there should not even be any reason for QEMU's idea of the bus topology and Xen's presentation of the bus topology to the guest to even match.
PaulQuoted 121 lines
(Adding Stefano and Anthony as
xen-hvm.cmentioned)Here is another suggestion:
1. QEMU use existing facilities to emulate PCIEXBAR for a Q35 machine, calling Xen's
map_io_range_to_ioreq_server()API to mark MMIO range for emulation, just like for any other emulated MMIO range2. All accesses to this area will be forwarded to QEMU as MMIO ioreqs and emulated flawlessly as everything is within QEMU architecture -- pci-host/PCIBus/PCIDevice machinery in place. No workarounds required for
xen-hvm.c3. CF8/CFC accesses will be forwarded as
_PCI_CONFIGioreqs, as usually. Both methods are in sync as they use common PCI emulation infrastructure in QEMU4. At this point absolutely zero changes are required in both Xen and QEMU code. Only existing interfaces are used. In fact, no related code changes required at all except a bugfix for PCIEXBAR mask emulation (provided in this series)
5. But. Just to make the 'multiple device emulators' (no extra reasons so far) feature to work, we add the same hypercall/dmop usage to let Xen know where QEMU emulates MMCONFIG
6. Xen will continue to trap accesses to this range but instead of sending
_COPYioreq immediately, he will check the address against known MMCONFIG location (in the same manner as above), then convert the
offset within it to BDF and he can proceed to usual BDF-based ioreq routing for those device emulator DMs, whatever they are7. In fact, MMIO -> PCI conf ioreq translation can be freely used as well at this stage, if it is more convenient for 'multiple device emulators' feature users. It can be even made selectable.
So, the question which needs explanation is: why do you think
MMIO->PCIconf ioreq translation is mandatory for MMCONFIG? Can't we just add new hypercall/dmop to make ioreq routing for 'multiple device emulators' to work while letting QEMU to use any API provided for him to do its tasks?It's kinda funny to pretend that QEMU don't know anything about MMCONFIG being MMIO when it's QEMU who inform Xen about its memory address and size.
Quoted 15 lines
Regardless of how this ends up being implemented inside of QEMU I think the above approach is the right one from an architectural PoV.
AFAICT there are still some reserved bits in the ioreq struct that you could use to signal 'this is a MCFG PCI access' if required.
Quoted 9 lines
Approach #2. Handling MMCONFIG area inside QEMU using usual MMIO emulation:
1. QEMU will trap accesses to PCIEXBAR (or whatever else possibly supported in the future like HECBASE), eventually asking Xen to map the MMCONFIG MMIO range for ioreq servicing just like it does for any other emulated MMIO range, via
map_io_range_to_ioreq_server(). All changes in MMCONFIG placement/status will lead toremapping/unmapping
Quoted 56 lines
Quoted 14 lines
the MMIO range.
2. Xen will trap MMIO accesses to this area and forward them to QEMU as MMIO (
IOREQ_TYPE_COPY) ioreqs3. QEMU will receive these accesses and pass them to the existing MMCONFIG emulation -- pcie_mmcfg_data_read/write handlers, finally resulting in same xen_host_pci_* function calls as before.
This approach works "right out of the box", no changes needed for either Xen or QEMU. As both
_PCI_CONFIGand MMIO type ioreqs are processed, either method can be used to access PCI/extended config space -- CF8/CFC port I/O or MMIO accesses to MMCONFIG.Quoted 7 lines
IOREQ routing for multiple device emulators can be supported too. In fact, the same mmconfig dmops/hypercalls can be added to let Xen know where MMCONFIG area resides, Xen will use this information to forward MMCONFIG MMIO ioreqs accordingly to BDF of the address. The difference with the approach #1 is that these interfaces are now completely optional when we use MMIO ioreqs for MMCONFIG on vanilla Xen/QEMU.
As said above, if you forward MCFG accesses as
IOREQ_TYPE_COPYyou are forcing each IOREQ server to know the position of the MCFG area in order to do the decoding, this is not acceptable IMO.Quoted 5 lines
The question is why
IOREQ_TYPE_COPY->IOREQ_TYPE_PCI_CONFIGtranslation is a must have thing at all? It won't make handling simpler. For current QEMU implementationIOREQ_TYPE_COPY(MMIO accesses for MMCONFIG) would be preferable as it allows to use the existing code.Granted it's likely easier to implement, but it's also incorrect. You seem to have in mind the picture of a single IOREQ server (QEMU) handling all the devices.
Although this is the most common scenario, it's not the only one supported by Xen. Your proposed solution breaks the usage of multiple IOREQ servers as PCI device emulators.
Quoted 5 lines
I think it will be safe to use MMCONFIG emulation on MMIO level for now and later extend it with '
set_mmconfig_' dmop/hypercall for the 'multiple device emulators'IOREQ_TYPE_COPYrouting to work same as for PCI conf, so it can be used by XenGT etc on Q35 as well.I'm afraid this kind of issues would have been fairly easier to identify if a design document for this feature was sent to the list prior to it's implementation.
Regarding whether to accept something like this, I'm not really in favor, but IMO it depends on how much new code is added to handle this incorrect usage that would then go away (or would have to be changed) in order to handle the proper implementation.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéOn Wed, 21 Mar 2018 17:15:04 +0000
Roger Pau Monné <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 4 lines
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs?You are just adding an extra level of complexity to an interface that's fairly simple. You register a PCI device using
XEN_DMOP_IO_RANGE_PCIand you getIOREQ_TYPE_PCI_CONFIGioreqs.
Yes, and it is still needed as we have two distinct (and not equal) interfaces to PCI conf space. Apart from 0..FFh range overlapping they can be considered very different interfaces. And whether it is a real system or emulated -- we can use either one of these two interfaces or both.
For QEMU zero changes are needed to support MMCONFIG MMIO accesses if they come as MMIO ioreqs. It's just what its MMCONFIG emulation code expects.
Anyway, for (kind of vague) users of the multiple ioreq servers capability we can enable MMIO translation to PCI conf ioreqs. Note that actually this is an extra step, not forwarding trapped MMCONFIG MMIO accesses to the selected device model as is.
Quoted 2 lines
Getting both
IOREQ_TYPE_PCI_CONFIGandIOREQ_TYPE_COPYfor PCI config space access is misleading.
These are very different accesses, both in transport and capabilities.
Quoted 4 lines
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the
IOREQ_TYPE_PCI_CONFIGwhich is trivial.
The "reconstruction of the memory access" you mentioned won't be easy actually. The thing is, address_space_read/write is not all what we need.
In order to translate PCI conf ioreqs back to emulated MMIO ops, we need to be an involved party, mainly to know where MMCONFIG area is located so we can construct the address within its range from BDF. This piece of information is destroyed in the process of MMIO ioreq translation to PCI conf type.
The code which parse PCI conf ioreqs in xen-hvm.c doesn't know anything about the current emulated MMCONFIG state. The correct way to have this info is to participate in its emulation. As we don't participate, we have no other way than trying to gain backdoor access to PCIHost fields via things like object_resolve_*(). This solution is cumbersome and ugly but will work... and may break anytime due to changes in QEMU.
QEMU maintainers will grin while looking at all this I'm afraid -- trapped MMIO accesses which are translated to PCI conf accesses which in turn translated back to emulated MMIO accesses upon receiving, along with tedious attempts to gain access to MMCONFIG-related info as we're not invited to the MMCONFIG emulation party.
The more I think about it, the more I like the existing map_io_range_to_ioreq_server() approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
Quoted 6 lines
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an
IOREQ_TYPE_PCI_CONFIGand it replays the IO space access usingdo_outpandcpu_ioreq_pio.
...And it is completely limited to basic PCI conf space. I don't know the context of this line in xen-hvm.c:
val = (1u << 31) | ((req->addr & 0x0f00) << 16) | ((sbdf & 0xffff) << 8)
| (req->addr & 0xfc);
but seems like current QEMU versions do not expect anything similar to AMD ECS-style accesses for 0CF8h. It is limited to basic PCI conf only.
Quoted 4 lines
If you think using
IOREQ_TYPE_COPYfor MCFG accesses is such a benefit for QEMU, why not just translate theIOREQ_TYPE_PCI_CONFIGintoIOREQ_TYPE_COPYinhandle_ioreqand dispatch it usingcpu_ioreq_move?
Answered above, we need to somehow have access to the info which don't belong to us for this step.
Quoted 1 line
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantOn Wed, 21 Mar 2018 17:06:28 +0000
Paul Durrant <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is
accessing this area -> Xen intercepts this MMIO accessBut here's what happens next:
Quoted 8 lines
Quoted 5 lines
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulationThat would only be true of a dm that cannot handle PCI config ioreqs directly.
It's just a bit problematic for xen-hvm.c (Xen ioreq processor in QEMU).
It receives these PCI conf ioreqs out of any context. To workaround this, existing code issues I/O to emulated CF8h/CFCh ports in order to allow QEMU to find their target. But we can't use the same method for MMCONFIG accesses -- this works for basic PCI conf space only.
We need to either locate PCIBus/PCIDevice manually via object lookups and then proceed to something like pci_host_config_read_common(), or to convert the PCI conf access into the emulated MMIO access... again, a required piece of information is missing -- we need to somehow learn the current MMCONFIG address to recreate the memory address to be emulated.
Let's put it simply -- the goal to make PCI conf ioreqs to reach their MMCONFIG targets in xen-hvm.c is easily achievable but it will look like a hack. MMIO ioreqs are preferable for MMCONFIG -- no extra logic needed for them, we can directly pass them for emulation in a way somewhat reminiscent of CF8h/CFCh replay, except for memory.
Ideally it would be PCI conf ioreq translation for supplemental device emulators while skipping this translation for QEMU. QEMU expects PCI config ioreqs only for CF8/CFC accesses. I assume it's DEMU/VGPU which of primary concern here, not experimental users like XenGT.
Quoted 30 lines
Paul
Quoted 27 lines
I tnink some parts of this equation can be collapsed, isn't it?
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs? We can still route either ioreq type to multiple device emulators accordingly.This will be the most universal and consistent approach -- either
_COPYor _PCI_CONFIG-type ioreqs can be sent to DM, whatever it likes more.Quoted 16 lines
Quoted 2 lines
9. Existing MMCONFIG-handling code in QEMU will be unused in this scenario
If you replay the read/write I don't think so. In any case this is irrelevant. QEMU CPU emulation code is also unused when running under Xen.
Quoted 5 lines
10. All this needed primarily to make the specific "Multiple device emulators" feature to work (XenGT was mentioned as its user) on Q35 with MMCONFIG.
Anything wrong/missing here?
I think that's correct.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 24 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 01:31, <[email protected]> wrote:
On Wed, 21 Mar 2018 17:06:28 +0000
Paul Durrant <[email protected]> wrote:
[...]Quoted 11 lines
Quoted 11 lines
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is
accessing this area -> Xen intercepts this MMIO accessBut here's what happens next:
Quoted 8 lines
Quoted 5 lines
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulationThat would only be true of a dm that cannot handle PCI config ioreqs directly.
Quoted 6 lines
It's just a bit problematic for
xen-hvm.c(Xen ioreq processor in QEMU).It receives these PCI conf ioreqs out of any context. To workaround this, existing code issues I/O to emulated CF8h/CFCh ports in order to allow QEMU to find their target. But we can't use the same method for MMCONFIG accesses -- this works for basic PCI conf space only.
I think you want to view this the other way around: No physical device would ever get to see MMCFG accesses (or CF8/CFC port ones). This same layering is what we should have in the virtualized case.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 73 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 21 March 2018 22:50
To: Roger Pau Monne <[email protected]>
Cc: [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Anthony Perard <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringOn Wed, 21 Mar 2018 17:15:04 +0000
Roger Pau Monné <[email protected]> wrote:
[...]Quoted 8 lines
Quoted 4 lines
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs?You are just adding an extra level of complexity to an interface that's fairly simple. You register a PCI device using
XEN_DMOP_IO_RANGE_PCIand you getIOREQ_TYPE_PCI_CONFIGioreqs.Yes, and it is still needed as we have two distinct (and not equal) interfaces to PCI conf space. Apart from 0..FFh range overlapping they can be considered very different interfaces. And whether it is a real system or emulated -- we can use either one of these two interfaces or both.
For QEMU zero changes are needed to support MMCONFIG MMIO accesses if
they come as MMIO ioreqs. It's just what its MMCONFIG emulation code expects.
Anyway, for (kind of vague) users of the multiple ioreq servers capability we can enable MMIO translation to PCI conf ioreqs. Note that actually this is an extra step, not forwarding trapped MMCONFIG MMIO accesses to the selected device model as is.Quoted 1 line
Getting both
IOREQ_TYPE_PCI_CONFIGandIOREQ_TYPE_COPYfor PCIconfig
Quoted 1 line
space access is misleading.
These are very different accesses, both in transport and capabilities.
Quoted 4 lines
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the
IOREQ_TYPE_PCI_CONFIGwhich is trivial.The "reconstruction of the memory access" you mentioned won't be easy actually. The thing is, address_space_read/write is not all what we need.
In order to translate PCI conf ioreqs back to emulated MMIO ops, we need to be an involved party, mainly to know where MMCONFIG area is located so we can construct the address within its range from BDF. This piece of information is destroyed in the process of MMIO ioreq translation to PCI conf type.
The code which parse PCI conf ioreqs in
xen-hvm.cdoesn't know anything about the current emulated MMCONFIG state. The correct way to have this info is to participate in its emulation. As we don't participate, we have no other way than trying to gain backdoor access to PCIHost fields via things like object_resolve_*(). This solution is cumbersome and ugly but will work... and may break anytime due to changes in QEMU.QEMU maintainers will grin while looking at all this I'm afraid -- trapped MMIO accesses which are translated to PCI conf accesses which in turn translated back to emulated MMIO accesses upon receiving, along with tedious attempts to gain access to MMCONFIG-related info as we're not invited to the MMCONFIG emulation party.
Quoted 10 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
I don't think we even want QEMU to have the freedom to say where the MMCONFIG areas are located, do we? QEMU is not in charge of the guest memory map and it is not responsible for the building the MCFG table, Xen is. So it should be Xen that decides where the MMCONFIG area goes for each registered PCI device and it should be Xen that adds that to the MCFG table. It should be Xen that handles the MMCONFIG MMIO accesses and these should be forwarded to QEMU as PCI config IOREQs. Now, it may be that we need to introduce a Xen specific mechanism into QEMU to then route those config space transactions to the device models but that would be an improvement over the current cf8/cfc hackery anyway.
Paul
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an
IOREQ_TYPE_PCI_CONFIGand it replays the IO space access usingdo_outpandcpu_ioreq_pio....And it is completely limited to basic PCI conf space. I don't know the context of this line in
xen-hvm.c:val = (1u << 31) | ((
req->addr&0x0f00) << 16) | ((sbdf &0xffff) << 8)| (req->addr & 0xfc);but seems like current QEMU versions do not expect anything similar to AMD ECS-style accesses for 0CF8h. It is limited to basic PCI conf only.
Quoted 9 lines
Quoted 4 lines
If you think using
IOREQ_TYPE_COPYfor MCFG accesses is such a benefit for QEMU, why not just translate theIOREQ_TYPE_PCI_CONFIGintoIOREQ_TYPE_COPYinhandle_ioreqand dispatch it usingcpu_ioreq_move?Answered above, we need to somehow have access to the info which don't belong to us for this step.
Quoted 1 line
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Jan BeulichQuoted 32 lines · On Thu, 22 Mar 2018 03:04:16 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 24 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 01:31, <[email protected]> wrote:
On Wed, 21 Mar 2018 17:06:28 +0000
Paul Durrant <[email protected]> wrote:
[...]Quoted 11 lines
Quoted 11 lines
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen new MMCONFIG address/size -> Xen (re)maps MMIO trapping area -> someone is
accessing this area -> Xen intercepts this MMIO accessBut here's what happens next:
Quoted 8 lines
Quoted 5 lines
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulationThat would only be true of a dm that cannot handle PCI config ioreqs directly.
Quoted 7 lines
It's just a bit problematic for
xen-hvm.c(Xen ioreq processor in QEMU).It receives these PCI conf ioreqs out of any context. To workaround this, existing code issues I/O to emulated CF8h/CFCh ports in order to allow QEMU to find their target. But we can't use the same method for MMCONFIG accesses -- this works for basic PCI conf space only.
Quoted 4 lines
I think you want to view this the other way around: No physical device would ever get to see MMCFG accesses (or CF8/CFC port ones). This same layering is what we should have in the virtualized case.
We have purely virtual layout of the PCI bus along with virtual, emulated and completely unrelated to host's MMCONFIG -- so what's exposed? This emulated MMCONFIG simply a supplement to virtual PCI bus and its layout correspond to the virtual PCI bus guest/QEMU see.
It's QEMU who controls chipset-specific PCIEXBAR emulation and knows about MMCONFIG position and size. QEMU informs Xen about where it is, in order to receive events about R/W accesses to this emulated area -- so, why he should receive these events in a form of PCI conf BDF/reg and not simply as MMCONFIG offset directly if it is basically the same thing?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 12 lines · Alexey G
On Wed, 21 Mar 2018 17:15:04 +0000
Roger Pau Monné <[email protected]> wrote:
[...]Quoted 8 lines
Quoted 4 lines
Above scenario makes it obvious that at least for QEMU the
MMIO->PCIconf translation is a redundant step. Why not to allow specifying for DM whether it prefers to receive MMCONFIG accesses as native (MMIO ones) or as translated PCI conf ioreqs?You are just adding an extra level of complexity to an interface that's fairly simple. You register a PCI device using
XEN_DMOP_IO_RANGE_PCIand you getIOREQ_TYPE_PCI_CONFIGioreqs.
Quoted 5 lines
Yes, and it is still needed as we have two distinct (and not equal) interfaces to PCI conf space. Apart from 0..FFh range overlapping they can be considered very different interfaces. And whether it is a real system or emulated -- we can use either one of these two interfaces or both.
The legacy PCI config space accesses and the MCFG config space access are just different methods of accessing the PCI configuration space, but the data _must_ be exactly the same. I don't see how a device would care about where the access to the config space originated.
Quoted 3 lines
For QEMU zero changes are needed to support MMCONFIG MMIO accesses if they come as MMIO ioreqs. It's just what its MMCONFIG emulation code expects.
As I said many times in this thread, you seem to be focused around what's best for QEMU only, and this is wrong. The IOREQ interface is used by QEMU, but it's also used by other device emulators.
I get the feeling that you assume that the correct solution is the one that involves less changes to Xen and QEMU. This is simply not true.
Quoted 19 lines
Anyway, for (kind of vague) users of the multiple ioreq servers capability we can enable MMIO translation to PCI conf ioreqs. Note that actually this is an extra step, not forwarding trapped MMCONFIG MMIO accesses to the selected device model as is.
Quoted 2 lines
Getting both
IOREQ_TYPE_PCI_CONFIGandIOREQ_TYPE_COPYfor PCI config space access is misleading.These are very different accesses, both in transport and capabilities.
Quoted 4 lines
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the
IOREQ_TYPE_PCI_CONFIGwhich is trivial.The "reconstruction of the memory access" you mentioned won't be easy actually. The thing is, address_space_read/write is not all what we need.
Quoted 5 lines
In order to translate PCI conf ioreqs back to emulated MMIO ops, we need to be an involved party, mainly to know where MMCONFIG area is located so we can construct the address within its range from BDF. This piece of information is destroyed in the process of MMIO ioreq translation to PCI conf type.
QEMU certainly knows the position of the MCFG area (because it's the one that tells Xen about it), so I don't understand your concerns above.
Quoted 6 lines
The code which parse PCI conf ioreqs in
xen-hvm.cdoesn't know anything about the current emulated MMCONFIG state. The correct way to have this info is to participate in its emulation. As we don't participate, we have no other way than trying to gain backdoor access to PCIHost fields via things like object_resolve_*(). This solution is cumbersome and ugly but will work... and may break anytime due to changes in QEMU.
OK, so you don't want to reconstruct the access, fine.
Then just inject it using pcie_mmcfg_data_{read/write} or some similar wrapper. My suggestion was just to try to use the easier way to get this injected into QEMU.
Quoted 6 lines
QEMU maintainers will grin while looking at all this I'm afraid -- trapped MMIO accesses which are translated to PCI conf accesses which in turn translated back to emulated MMIO accesses upon receiving, along with tedious attempts to gain access to MMCONFIG-related info as we're not invited to the MMCONFIG emulation party.
Quoted 9 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
I'm sorry, but I'm getting lost.
You complain that using IOREQ_TYPE_PCI_CONFIG is not a good approach because QEMU needs to know the position of the MCFG area if we want to reconstruct and forward the MMIO access. And then you are proposing to use IOREQ_TYPE_COPY which _requires_ QEMU to know the position of the MCFG area in order to do the decoding of the PCI config space access.
Quoted 16 lines
Quoted 6 lines
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an
IOREQ_TYPE_PCI_CONFIGand it replays the IO space access usingdo_outpandcpu_ioreq_pio....And it is completely limited to basic PCI conf space. I don't know the context of this line in
xen-hvm.c:val = (1u << 31) | ((
req->addr&0x0f00) << 16) | ((sbdf &0xffff) << 8)| (req->addr & 0xfc);but seems like current QEMU versions do not expect anything similar to AMD ECS-style accesses for 0CF8h. It is limited to basic PCI conf only.
Quoted 7 lines
Quoted 4 lines
If you think using
IOREQ_TYPE_COPYfor MCFG accesses is such a benefit for QEMU, why not just translate theIOREQ_TYPE_PCI_CONFIGintoIOREQ_TYPE_COPYinhandle_ioreqand dispatch it usingcpu_ioreq_move?Answered above, we need to somehow have access to the info which don't belong to us for this step.
Why not? QEMU tells Xen the position of the MCFG area but then you complain that QEMU doesn't know the position of the MCFG area?
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantQuoted 11 lines · Paul Durrant
Quoted 10 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
Quoted 2 lines
I don't think we even want QEMU to have the freedom to say where the MMCONFIG areas are located, do we?
Sadly this how the chipset works. The PCIEXBAR register contains the position of the MCFG area. And this is emulated by QEMU.
Quoted 3 lines
QEMU is not in charge of the
guest memory map and it is not responsible for the building the MCFG table, Xen is.
Well, the one that builds the MCFG table is hvmloader actually, which is the one that initially sets the value of PCIEXBAR and thus the initial position of the MCFG.
Quoted 8 lines
So it should be Xen that decides where the MMCONFIG
area goes for each registered PCI device and it should be Xen that adds that to the MCFG table. It should be Xen that handles the MMCONFIG MMIO accesses and these should be forwarded to QEMU as PCI config IOREQs. Now, it may be that we need to introduce a Xen specific mechanism into QEMU to then route those config space transactions to the device models but that would be an improvement over the current cf8/cfc hackery anyway.
I think we need a way for QEMU to tell Xen the position of the MCFG area, and any changes to it.
I don't think we want to emulate the PCIEXBAR register inside of Xen, if we do that then we would likely have to emulate the full Express Chipset inside of Xen.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Alexey GQuoted 54 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 22 March 2018 09:55
To: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>; Anthony Perard <[email protected]>; Ian Jackson <[email protected]>; Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>; Wei Liu <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 9 lines · On Thu, 22 Mar 2018 03:04:16 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 01:31, <[email protected]> wrote:
On Wed, 21 Mar 2018 17:06:28 +0000
Paul Durrant <[email protected]> wrote:
[...]Quoted 5 lines
Quoted 5 lines
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen
new
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMCONFIG address/size -> Xen (re)maps MMIO trapping area ->
someone
Quoted 22 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
is
accessing this area -> Xen intercepts this MMIO accessBut here's what happens next:
Quoted 8 lines
Quoted 5 lines
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulationThat would only be true of a dm that cannot handle PCI config ioreqs directly.
Quoted 7 lines
It's just a bit problematic for
xen-hvm.c(Xen ioreq processor in QEMU).It receives these PCI conf ioreqs out of any context. To workaround this, existing code issues I/O to emulated CF8h/CFCh ports in order to allow QEMU to find their target. But we can't use the same method for MMCONFIG accesses -- this works for basic PCI conf space only.
Quoted 4 lines
I think you want to view this the other way around: No physical device would ever get to see MMCFG accesses (or CF8/CFC port ones). This same layering is what we should have in the virtualized case.
Quoted 7 lines
We have purely virtual layout of the PCI bus along with virtual, emulated and completely unrelated to host's MMCONFIG -- so what's exposed? This emulated MMCONFIG simply a supplement to virtual PCI bus and its layout correspond to the virtual PCI bus guest/QEMU see.
It's QEMU who controls chipset-specific PCIEXBAR emulation and knows about MMCONFIG position and size.
...and I think that it the wrong solution for Xen. We only use QEMU as an emulator for peripheral devices; we should not be using it for this kind of emulation... that should be brought into the hypervisor.
Quoted 1 line
QEMU informs Xen about where it is,
No. Xen should not care where QEMU wants to put it because the MMIO emulations should not even read QEMU.
Paul
Quoted 4 lines
in order to receive events about R/W accesses to this emulated area -- so, why he should receive these events in a form of PCI conf BDF/reg and not simply as MMCONFIG offset directly if it is basically the same thing?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéQuoted 25 lines
-----Original Message-----
From: Roger Pau Monne
Sent: 22 March 2018 10:06
To: Paul Durrant <[email protected]>
Cc: 'Alexey G' <[email protected]>; [email protected]; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Anthony Perard <[email protected]>; Stefano Stabellini <[email protected]>
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 6 lines · Paul Durrant
Quoted 6 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trapMMCONFIG
Quoted 5 lines
Quoted 4 lines
MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
Quoted 5 lines
Quoted 2 lines
I don't think we even want QEMU to have the freedom to say where the MMCONFIG areas are located, do we?
Sadly this how the chipset works. The PCIEXBAR register contains the position of the MCFG area. And this is emulated by QEMU.
So we should be emulating that in Xen, not handing it off to QEMU. Our integration with QEMU is already terrible and using QEMU to emulate the PCIe chipset will only make it worse.
Quoted 19 lines
Quoted 4 lines
QEMU is not in charge of the
guest memory map and it is not responsible for the building the MCFG table, Xen is.Well, the one that builds the MCFG table is hvmloader actually, which is the one that initially sets the value of PCIEXBAR and thus the initial position of the MCFG.
Quoted 4 lines
So it should be Xen that decides where the MMCONFIG area goes for each registered PCI device and it should be Xen that adds that to the MCFG table. It should be Xen that handles the MMCONFIG MMIO accesses and these should be forwarded to QEMU as
PCI
Quoted 4 lines
config IOREQs. Now, it may be that we need to introduce a Xen specific mechanism into QEMU to then route those config space transactions to the device models but that would be an improvement over the current cf8/cfc hackery anyway.
Quoted 7 lines
I think we need a way for QEMU to tell Xen the position of the MCFG area, and any changes to it.
I don't think we want to emulate the PCIEXBAR register inside of Xen, if we do that then we would likely have to emulate the full Express Chipset inside of Xen.
No, that's *exactly* what we should be doing. We should only be using QEMU for emulation of discrete peripheral devices.
Paul
Quoted 1 line
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantQuoted 1 line · On Thu, 22 Mar 2018 09:29:44 +0000 Paul Durrant <[email protected]> wrote:
Quoted 1 line
-----Original Message-----
[...]
Quoted 40 lines
Quoted 29 lines
Quoted 4 lines
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the
IOREQ_TYPE_PCI_CONFIGwhich is trivial.The "reconstruction of the memory access" you mentioned won't be easy actually. The thing is, address_space_read/write is not all what we need.
In order to translate PCI conf ioreqs back to emulated MMIO ops, we need to be an involved party, mainly to know where MMCONFIG area is located so we can construct the address within its range from BDF. This piece of information is destroyed in the process of MMIO ioreq translation to PCI conf type.
The code which parse PCI conf ioreqs in
xen-hvm.cdoesn't know anything about the current emulated MMCONFIG state. The correct way to have this info is to participate in its emulation. As we don't participate, we have no other way than trying to gain backdoor access to PCIHost fields via things like object_resolve_*(). This solution is cumbersome and ugly but will work... and may break anytime due to changes in QEMU.QEMU maintainers will grin while looking at all this I'm afraid -- trapped MMIO accesses which are translated to PCI conf accesses which in turn translated back to emulated MMIO accesses upon receiving, along with tedious attempts to gain access to MMCONFIG-related info as we're not invited to the MMCONFIG emulation party.
Quoted 10 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.
Quoted 11 lines
I don't think we even want QEMU to have the freedom to say where the MMCONFIG areas are located, do we? QEMU is not in charge of the guest memory map and it is not responsible for the building the MCFG table, Xen is. So it should be Xen that decides where the MMCONFIG area goes for each registered PCI device and it should be Xen that adds that to the MCFG table. It should be Xen that handles the MMCONFIG MMIO accesses and these should be forwarded to QEMU as PCI config IOREQs. Now, it may be that we need to introduce a Xen specific mechanism into QEMU to then route those config space transactions to the device models but that would be an improvement over the current cf8/cfc hackery anyway.
Well, MMCONFIG is a chipset-specific thing. We probably can't simply abstract its usage, merely providing ACPI MCFG table for it.
Its layout must correspond to the emulated PCI conf space where the majority of devices belong to QEMU. Although we can track all QEMU's usage of emulated/PT PCI devices and build this layout themselves, this design may introduce multiple issues. For QEMU handling of such PCI conf ioreq without knowing anything about MMCONFIG will become worse -- previously he at least knew that those belong to the MMCONFIG range he emulates, but in case of PCI conf ioreqs situation gets a bit more complicated -- either CF8/CFC workaround or manual lookup of the target device from rather isolated xen-hvm.c. Feasible, yes, but will look like a dirty hack -- doing part of QEMU's internal job.
These are merely inconveniences, main problem here at the moment is OVMF. OVMF does MMCONFIG relocation by writing to PCIEXBAR he knows about on Q35, followed by using it at the address he expects. This is something I want to address in subsequent patches.
Quoted 27 lines
Paul
Quoted 25 lines
Quoted 6 lines
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an
IOREQ_TYPE_PCI_CONFIGand it replays the IO space access usingdo_outpandcpu_ioreq_pio....And it is completely limited to basic PCI conf space. I don't know the context of this line in
xen-hvm.c:val = (1u << 31) | ((
req->addr&0x0f00) << 16) | ((sbdf &0xffff) << 8) | (req->addr&0xfc);but seems like current QEMU versions do not expect anything similar to AMD ECS-style accesses for 0CF8h. It is limited to basic PCI conf only.
Quoted 4 lines
If you think using
IOREQ_TYPE_COPYfor MCFG accesses is such a benefit for QEMU, why not just translate theIOREQ_TYPE_PCI_CONFIGintoIOREQ_TYPE_COPYinhandle_ioreqand dispatch it usingcpu_ioreq_move?Answered above, we need to somehow have access to the info which don't belong to us for this step.
Quoted 1 line
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantOn Thu, 22 Mar 2018 10:09:16 +0000
Paul Durrant <[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
I don't think we even want QEMU to have the freedom to say where the MMCONFIG areas are located, do we?
Sadly this how the chipset works. The PCIEXBAR register contains the position of the MCFG area. And this is emulated by QEMU.
So we should be emulating that in Xen, not handing it off to QEMU. Our integration with QEMU is already terrible and using QEMU to emulate the PCIe chipset will only make it worse.
I guess QEMU guys will tell that it will actually improve. :) One of the very first observation I made while learning Xen/QEMU was that Xen and QEMU behave sort of like stepmother and stepdaughter -- dislike each other but have to live together in one house for now. I think a better interaction will benefit both.
There are some architectural issues (MMIO hole control for passthrough needs is one of them) which can be solved by actually improving coordination with QEMU, while not sacrificing the security in any way.
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
QEMU is not in charge of the
guest memory map and it is not responsible for the building the MCFG table, Xen is.Well, the one that builds the MCFG table is hvmloader actually, which is the one that initially sets the value of PCIEXBAR and thus the initial position of the MCFG.
Quoted 4 lines
So it should be Xen that decides where the MMCONFIG area goes for each registered PCI device and it should be Xen that adds that to the MCFG table. It should be Xen that handles the MMCONFIG MMIO accesses and these should be forwarded to QEMU as
PCI
Quoted 4 lines
config IOREQs. Now, it may be that we need to introduce a Xen specific mechanism into QEMU to then route those config space transactions to the device models but that would be an improvement over the current cf8/cfc hackery anyway.
I think we need a way for QEMU to tell Xen the position of the MCFG area, and any changes to it.
Quoted 6 lines
Quoted 4 lines
I don't think we want to emulate the PCIEXBAR register inside of Xen, if we do that then we would likely have to emulate the full Express Chipset inside of Xen.
No, that's *exactly* what we should be doing. We should only be using QEMU for emulation of discrete peripheral devices.
Emulated PCIe Switch (PCI-PCI bridge basically) can be considered a discrete peripheral device which can function alone?
If we should emulate the whole PCIe bus, where will be the dividing line between chipset emulation and PCIe hierarchy emulation?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Paul DurrantQuoted 64 lines · On Thu, 22 Mar 2018 10:06:09 +0000 Paul Durrant <[email protected]> wrote:
Quoted 56 lines
-----Original Message-----
From: Alexey G [mailto:[email protected]]
Sent: 22 March 2018 09:55
To: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>; Anthony Perard <[email protected]>; Ian Jackson <[email protected]>; Paul Durrant <[email protected]>; Roger Pau Monne <[email protected]>; Wei Liu <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [Xen-devel] [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoringQuoted 10 lines · On Thu, 22 Mar 2018 03:04:16 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 01:31, <[email protected]> wrote:
On Wed, 21 Mar 2018 17:06:28 +0000
Paul Durrant <[email protected]> wrote:
[...]Quoted 6 lines
Quoted 6 lines
Well, this might work actually. Although the overall scenario will be overcomplicated a bit for
_PCI_CONFIGioreqs. Here is how it will look:QEMU receives PCIEXBAR update -> calls the new dmop to tell Xen
new
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
MMCONFIG address/size -> Xen (re)maps MMIO trapping area ->
someone
Quoted 23 lines
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
is
accessing this area -> Xen intercepts this MMIO accessBut here's what happens next:
Quoted 8 lines
Quoted 5 lines
Xen translates MMIO access into
PCI_CONFIGand sends it to DM -> DM receives_PCI_CONFIGioreq -> DM translates BDF/addr info back to the offset in emulated MMCONFIG range -> DM calls address_space_read/write to trigger MMIO emulationThat would only be true of a dm that cannot handle PCI config ioreqs directly.
Quoted 8 lines
It's just a bit problematic for
xen-hvm.c(Xen ioreq processor in QEMU).It receives these PCI conf ioreqs out of any context. To workaround this, existing code issues I/O to emulated CF8h/CFCh ports in order to allow QEMU to find their target. But we can't use the same method for MMCONFIG accesses -- this works for basic PCI conf space only.
Quoted 4 lines
I think you want to view this the other way around: No physical device would ever get to see MMCFG accesses (or CF8/CFC port ones). This same layering is what we should have in the virtualized case.
Quoted 7 lines
We have purely virtual layout of the PCI bus along with virtual, emulated and completely unrelated to host's MMCONFIG -- so what's exposed? This emulated MMCONFIG simply a supplement to virtual PCI bus and its layout correspond to the virtual PCI bus guest/QEMU see.
It's QEMU who controls chipset-specific PCIEXBAR emulation and knows about MMCONFIG position and size.
Quoted 8 lines
...and I think that it the wrong solution for Xen. We only use QEMU as an emulator for peripheral devices; we should not be using it for this kind of emulation... that should be brought into the hypervisor.
Quoted 1 line
QEMU informs Xen about where it is,
No. Xen should not care where QEMU wants to put it because the MMIO emulations should not even read QEMU.
QEMU does a lot of MMIO emulation, what's so special in the emulated MMCONFIG? It has absolutely nothing to do with host's MMCONFIG, neither in address/size or the internal layout. None of the host MMCONFIG-related facilities touched in any way. It is purely virtual thing.
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.
Quoted 6 lines
Paul
Quoted 4 lines
in order to receive events about R/W accesses to this emulated area -- so, why he should receive these events in a form of PCI conf BDF/reg and not simply as MMCONFIG offset directly if it is basically the same thing?
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 12:56, <[email protected]> wrote:
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via
map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.
You continue to ignore the routing requirement multiple ioreq servers impose.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Roger Pau MonnéOn Thu, 22 Mar 2018 09:57:16 +0000
Roger Pau Monné <[email protected]> wrote:
[...]
Quoted 10 lines
Quoted 5 lines
Yes, and it is still needed as we have two distinct (and not equal) interfaces to PCI conf space. Apart from 0..FFh range overlapping they can be considered very different interfaces. And whether it is a real system or emulated -- we can use either one of these two interfaces or both.
The legacy PCI config space accesses and the MCFG config space access are just different methods of accessing the PCI configuration space, but the data _must_ be exactly the same. I don't see how a device would care about where the access to the config space originated.
If they were different methods of accessing the same thing, they could've been used interchangeably. When we've got a PCI conf ioreq which has offset>100h we know we cannot just pass it to emulated CF8/CFC but have to emulate this specifically.
Quoted 47 lines
Quoted 3 lines
For QEMU zero changes are needed to support MMCONFIG MMIO accesses if they come as MMIO ioreqs. It's just what its MMCONFIG emulation code expects.
As I said many times in this thread, you seem to be focused around what's best for QEMU only, and this is wrong. The IOREQ interface is used by QEMU, but it's also used by other device emulators.
I get the feeling that you assume that the correct solution is the one that involves less changes to Xen and QEMU. This is simply not true.
Quoted 19 lines
Anyway, for (kind of vague) users of the multiple ioreq servers capability we can enable MMIO translation to PCI conf ioreqs. Note that actually this is an extra step, not forwarding trapped MMCONFIG MMIO accesses to the selected device model as is.
Quoted 2 lines
Getting both
IOREQ_TYPE_PCI_CONFIGandIOREQ_TYPE_COPYfor PCI config space access is misleading.These are very different accesses, both in transport and capabilities.
Quoted 4 lines
In both cases Xen would have to do the MCFG access decoding in order to figure out which IOREQ server will handle the request. At which point the only step that you avoid is the reconstruction of the memory access from the
IOREQ_TYPE_PCI_CONFIGwhich is trivial.The "reconstruction of the memory access" you mentioned won't be easy actually. The thing is, address_space_read/write is not all what we need.
Quoted 5 lines
In order to translate PCI conf ioreqs back to emulated MMIO ops, we need to be an involved party, mainly to know where MMCONFIG area is located so we can construct the address within its range from BDF. This piece of information is destroyed in the process of MMIO ioreq translation to PCI conf type.
QEMU certainly knows the position of the MCFG area (because it's the one that tells Xen about it), so I don't understand your concerns above.
Quoted 7 lines
The code which parse PCI conf ioreqs in
xen-hvm.cdoesn't know anything about the current emulated MMCONFIG state. The correct way to have this info is to participate in its emulation. As we don't participate, we have no other way than trying to gain backdoor access to PCIHost fields via things like object_resolve_*(). This solution is cumbersome and ugly but will work... and may break anytime due to changes in QEMU.
Quoted 5 lines
OK, so you don't want to reconstruct the access, fine.
Then just inject it using pcie_mmcfg_data_{read/write} or some similar wrapper. My suggestion was just to try to use the easier way to get this injected into QEMU.
QEMU knows its position, the problem it that xen-hvm.c (ioreq processor) is rather isolated from MMCONFIG emulation.
If you check the pcie_mmcfg_data_read/write MMCONFIG handlers in QEMU, you can see this:
static uint64_t pcie_mmcfg_data_read(void *opaque, <...>
{
PCIExpressHost *e = opaque;...
We know this 'opaque' when we do MMIO-style MMCONFIG handling as pcie_mmcfg_data_read/write are actual handlers.
But xen-hvm.c needs to gain access to PCIExpressHost out of nowhere, which is possible but considered a hack by QEMU. We can also insert some code to MMCONFIG emulation which will store info we need to some global variables to be used across wildly different and unrelated modules. It will work, but anyone who see it will have bad thoughts on his mind.
Quoted 44 lines
Quoted 6 lines
QEMU maintainers will grin while looking at all this I'm afraid -- trapped MMIO accesses which are translated to PCI conf accesses which in turn translated back to emulated MMIO accesses upon receiving, along with tedious attempts to gain access to MMCONFIG-related info as we're not invited to the MMCONFIG emulation party.
Quoted 9 lines
The more I think about it, the more I like the existing
map_io_range_to_ioreq_server()approach. :( It works without doing anything, no hacks, no new interfaces, both MMCONFIG and CF8/CFC are working as expected. There is a problem to make it compatible with the specific multiple ioreq servers feature, but providing a new dmop/hypercall (which you suggest is a must have thing to trap MMCONFIG MMIO to give QEMU only the freedom to tell where it is located) allows to solve this problem in any possible way, either MMIO -> PCI conf translation or anything else.I'm sorry, but I'm getting lost.
You complain that using
IOREQ_TYPE_PCI_CONFIGis not a good approach because QEMU needs to know the position of the MCFG area if we want to reconstruct and forward the MMIO access. And then you are proposing to useIOREQ_TYPE_COPYwhich _requires_ QEMU to know the position of the MCFG area in order to do the decoding of the PCI config space access.Quoted 21 lines
Quoted 6 lines
Quoted 2 lines
We can still route either ioreq
type to multiple device emulators accordingly.It's exactly the same that's done for IO space PCI config space addresses. QEMU gets an
IOREQ_TYPE_PCI_CONFIGand it replays the IO space access usingdo_outpandcpu_ioreq_pio....And it is completely limited to basic PCI conf space. I don't know the context of this line in
xen-hvm.c:val = (1u << 31) | ((
req->addr&0x0f00) << 16) | ((sbdf &0xffff) << 8) | (req->addr&0xfc);but seems like current QEMU versions do not expect anything similar to AMD ECS-style accesses for 0CF8h. It is limited to basic PCI conf only.
Quoted 4 lines
If you think using
IOREQ_TYPE_COPYfor MCFG accesses is such a benefit for QEMU, why not just translate theIOREQ_TYPE_PCI_CONFIGintoIOREQ_TYPE_COPYinhandle_ioreqand dispatch it usingcpu_ioreq_move?
Quoted 5 lines
Quoted 2 lines
Answered above, we need to somehow have access to the info which don't belong to us for this step.
Why not? QEMU tells Xen the position of the MCFG area but then you complain that QEMU doesn't know the position of the MCFG area?
Answered above.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 14 lines · Alexey G
On Thu, 22 Mar 2018 09:57:16 +0000
Roger Pau Monné <[email protected]> wrote:
[...]Quoted 10 lines
Quoted 5 lines
Yes, and it is still needed as we have two distinct (and not equal) interfaces to PCI conf space. Apart from 0..FFh range overlapping they can be considered very different interfaces. And whether it is a real system or emulated -- we can use either one of these two interfaces or both.
The legacy PCI config space accesses and the MCFG config space access are just different methods of accessing the PCI configuration space, but the data _must_ be exactly the same. I don't see how a device would care about where the access to the config space originated.
Quoted 4 lines
If they were different methods of accessing the same thing, they could've been used interchangeably. When we've got a PCI conf ioreq which has offset>100h we know we cannot just pass it to emulated CF8/CFC but have to emulate this specifically.
This is already not the best approach to dispatch PCI config space access in QEMU. I think the interface in QEMU should be:
pci_conf_space_{read/write}(sbdf, register, size , data)
And this would go directly into the device. But I assume this involves a non-trivial amount of work to be implemented. Hence xen-hvm.c usage of the IO port access replay.
Quoted 20 lines
Quoted 5 lines
OK, so you don't want to reconstruct the access, fine.
Then just inject it using pcie_mmcfg_data_{read/write} or some similar wrapper. My suggestion was just to try to use the easier way to get this injected into QEMU.
QEMU knows its position, the problem it that
xen-hvm.c(ioreq processor) is rather isolated from MMCONFIG emulation.If you check the pcie_mmcfg_data_read/write MMCONFIG handlers in QEMU, you can see this:
static
uint64_tpcie_mmcfg_data_read(void *opaque, <...>{ PCIExpressHost *e = opaque;...
We know this 'opaque' when we do MMIO-style MMCONFIG handling as pcie_mmcfg_data_read/write are actual handlers.
Quoted 6 lines
But
xen-hvm.cneeds to gain access to PCIExpressHost out of nowhere, which is possible but considered a hack by QEMU. We can also insert some code to MMCONFIG emulation which will store info we need to some global variables to be used across wildly different and unrelated modules. It will work, but anyone who see it will have bad thoughts on his mind.
Since you need to notify Xen the MCFG area address, why not just store the MCFG address while doing this operation? You could do this with a helper in xen-hvm.c, and keep the variable locally to that file.
In any case, this is a QEMU implementation detail. IMO the IOREQ interface is clear and should not be bended like this just because 'this is easier to implement in QEMU'.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
In reply to Jan BeulichQuoted 10 lines · On Thu, 22 Mar 2018 06:09:44 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 12:56, <[email protected]> wrote:
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via
map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.You continue to ignore the routing requirement multiple ioreq servers impose.
If the emulated MMCONFIG approach will be modified to become fully compatible with multiple ioreq servers (whatever they used for), I assume there will be no objections that emulated MMCONFIG can't be used?
I just want to clarify this moment -- why people think that a completely emulated MMIO range, not related in any
way to host's MMCONFIG may compromise something.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 14:05, <[email protected]> wrote:
Quoted 10 lines · On Thu, 22 Mar 2018 06:09:44 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 12:56, <[email protected]> wrote:
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via
map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.You continue to ignore the routing requirement multiple ioreq servers impose.
Quoted 7 lines
If the emulated MMCONFIG approach will be modified to become fully compatible with multiple ioreq servers (whatever they used for), I assume there will be no objections that emulated MMCONFIG can't be used?
I just want to clarify this moment -- why people think that a completely emulated MMIO range, not related in any
way to host's MMCONFIG may compromise something.
Compromise? All that was said so far - afair - was that this is the wrong way round design wise.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 23 lines · On Thu, 22 Mar 2018 07:20:00 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 14:05, <[email protected]> wrote:
Quoted 10 lines · On Thu, 22 Mar 2018 06:09:44 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 12:56, <[email protected]> wrote:
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via
map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.You continue to ignore the routing requirement multiple ioreq servers impose.
Quoted 7 lines
If the emulated MMCONFIG approach will be modified to become fully compatible with multiple ioreq servers (whatever they used for), I assume there will be no objections that emulated MMCONFIG can't be used?
I just want to clarify this moment -- why people think that a completely emulated MMIO range, not related in any way to host's MMCONFIG may compromise something.
Quoted 2 lines
Compromise? All that was said so far - afair - was that this is the wrong way round design wise.
I assume it's all about emulating some real system for HVM, for other goals PV/PVH are available. What is a proper, design-wise way to emulate the MMIO-based MMCONFIG range Q35 provides you think of?
Here is what I've heard so far in this thread:
1. Add a completely new dmop/hypercall so that QEMU can tell Xen where emulated MMCONFIG MMIO area is located and in the same time map it for MMIO trapping to intercept accesses. Latter action is the same what map_io_range_to_ioreq_server() does, but let's ignore it for now because there was opinion that we need to stick to a distinct hypercall.
2. Upon trapping accesses to this emulated range, Xen will pretend that QEMU didn't just told him about MMCONFIG location and size and instead convert MMIO access into PCI conf one and send the ioreq to QEMU or some other DM.
3. If there will be a PCIEXBAR relocation (OVMF does it currently for MMCONFIG usage, but we must later teach him non-QEMU manners), QEMU must immediately inform Xen about any changes in MMCONFIG location/status.
4. QEMU receives PCI conf access while expecting the MMIO address, so xen-hvm.c has to deal with it somehow, either obtaining MMCONFIG base and recreating emulated MMIO access from BDF/reg or doing the dirty work of finding PCIBus/PCIDevice target itself as it cannot use emulated CF8/CFC ports due to legacy PCI conf size limitation.
Please confirm that it is a preferable solution or if something missing.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [RFC PATCH 07/12] hvmloader: allocate MMCONFIG area in the MMIO hole + minor code refactoring
Quoted 51 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 15:34, <[email protected]> wrote:
Quoted 23 lines · On Thu, 22 Mar 2018 07:20:00 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 14:05, <[email protected]> wrote:
Quoted 10 lines · On Thu, 22 Mar 2018 06:09:44 -0600 "Jan Beulich" <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 22.03.18 at 12:56, <[email protected]> wrote:
I really don't understand why some people have that fear of emulated MMCONFIG -- it's really the same thing as any other MMIO range QEMU already emulates via
map_io_range_to_ioreq_server(). No sensitive information exposed. It is related only to emulated PCI conf space which QEMU already knows about and use, providing emulated PCI devices for it.You continue to ignore the routing requirement multiple ioreq servers impose.
Quoted 7 lines
If the emulated MMCONFIG approach will be modified to become fully compatible with multiple ioreq servers (whatever they used for), I assume there will be no objections that emulated MMCONFIG can't be used?
I just want to clarify this moment -- why people think that a completely emulated MMIO range, not related in any way to host's MMCONFIG may compromise something.Quoted 2 lines
Compromise? All that was said so far - afair - was that this is the wrong way round design wise.
I assume it's all about emulating some real system for HVM, for other goals PV/PVH are available. What is a proper, design-wise way to emulate the MMIO-based MMCONFIG range Q35 provides you think of?
Here is what I've heard so far in this thread:
1. Add a completely new dmop/hypercall so that QEMU can tell Xen where emulated MMCONFIG MMIO area is located and in the same time map it for MMIO trapping to intercept accesses. Latter action is the same what
map_io_range_to_ioreq_server()does, but let's ignore it for now because there was opinion that we need to stick to a distinct hypercall.2. Upon trapping accesses to this emulated range, Xen will pretend that QEMU didn't just told him about MMCONFIG location and size and instead convert MMIO access into PCI conf one and send the ioreq to QEMU or some other DM.
3. If there will be a PCIEXBAR relocation (OVMF does it currently for MMCONFIG usage, but we must later teach him non-QEMU manners), QEMU must immediately inform Xen about any changes in MMCONFIG location/status.
Quoted 7 lines
4. QEMU receives PCI conf access while expecting the MMIO address, so
xen-hvm.chas to deal with it somehow, either obtaining MMCONFIG base and recreating emulated MMIO access from BDF/reg or doing the dirty work of finding PCIBus/PCIDevice target itself as it cannot use emulated CF8/CFC ports due to legacy PCI conf size limitation.Please confirm that it is a preferable solution or if something missing.
I'm afraid this is only part of the picture, as you've been told by others before. We first of all need to settle on who emulates the core chipset registers. Depending on that will be how Xen would learn about the MCFG location inside the guest.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel