xen: RMRR fix

v8:

A brief summary:

* Rebased on the latest
* We skip creating p2m table associated to RMRR range as Jan and Tim commented And especially, this is identified to setting these tables as INVALID since as you know all p2m tables already are initialized as INVALID. Actually I also tried to reset them as INVALID again, everything is still fine but some warning logs show we're resetting these Already-Invalided p2m entries so just keep skipping these entries.
* Add something into mem_access to skip these RMRR ranges to make sure we're in the same page.
* Provide a new approach to control if
1. we want to check/reserve all RMRR ranges
2. Or just some RMRR ranges specific to those devices we want to PT
This brings a new parameter to enable/disable this, and new domCTL to get SBDF associated to those PT devices.
* Add a new check point in mem_hole_alloc() since we may use this to allocate some ranges living in runtime cycle, like igd_opregion_pgbase.
* Some code itself improvements/corrections.

The details:

0001-tools-hvmloader-link-errno.h-from-xen-internal.patch

This is a new patch but Acked by Jan.

0002-introduce-XEN_DOMCTL_set_rdm.patch

This is a new patch to introduce that parameter and domCTL. Please refer to the patch head description.

0003-introduce-XENMEM_reserved_device_memory_map.patch

This is slightly rebased to make sure all codes should be in PT case, and this is from Jan and Acked by Kevin. To me, I just rebase on the latest.

0004-update-the-existing-hypercall-to-support-XEN_DOMCTL_.patch

This is a new patch since after we introduce that parameter and domCTL, we need to rework that existing hypercall from #4 patch.

Most should be discussed between Jan and me, but I changed something slightly again since I found we were missing some scenarios, like the hypercall caller don't know construct how much buffers to carry forward all necessary entries at first, and additionally, we also need to expose a return value from iommu ops as some caller expects.

So I have to ask Jan take a look at this again. And maybe we can squash this into #4 eventually.

0005-tools-libxc-introduce-hypercall-for-xc_reserved_devi.patch 0006-tools-libxc-check-if-modules-space-is-overlapping-wi.patch 0007-hvmloader-util-get-reserved-device-memory-maps.patch

Just rebase and cleanup.

0008-hvmloader-mmio-reconcile-guest-mmio-with-reserved-de.patch

Just rebase and cleanup, and some code corrections as Jan comments.

0009-hvmloader-ram-check-if-guest-memory-is-out-of-reserv.patch

Just rebase and cleanup, and some code improvements.

0010-hvmloader-mem_hole_alloc-skip-any-overlap-with-reser.patch

This is a new used to address igd_opregion_pgbase as Yang mentioned to me.

0011-xen-x86-p2m-reject-populating-for-reserved-device-me.patch

Refactor some codes to handle such a case. Note please refer to a description above.

0012-xen-x86-ept-handle-reserved-device-memory-in-ept_han.patch

Just improve comments.

0013-xen-mem_access-don-t-allow-accessing-reserved-device.patch

This is a new to sync mem_access after our action in memory populating.

0014-xen-x86-p2m-introduce-set_identity_p2m_entry.patch 0015-xen-vtd-create-RMRR-mapping.patch 0016-xen-vtd-group-assigned-device-with-RMRR.patch

There's nothing to change.

0017-xen-vtd-re-enable-USB-device-assignment-if-enable-pc.patch

Just rebase.

How to reproduce this issu:

* In shared-ept case with Xen.
* Target owns RMRR.
* Do IGD passthrough with Windows guest OS: gfx_passthru=1 pci=["00:02.0"]
* Please use qemu-xen-traditional.

My test machine is BDW with Windows 7.

v7:

This series of patches try to reconcile those remaining problems but just post as RFC to ask for any comments to refine everything.

The current whole scheme is as follows:

1. Reconcile guest mmio with RMRR in pci_setup
2. Reconcile guest RAM with RMRR in e820 table

Then in theory guest wouldn't access any RMRR range.

3. Just initialize all RMRR ranges as p2m_access_n in p2m table:
gfn:mfn:p2m_access_n

Here I think we shouldn't set 1:1 to expose RMRR to guest if guest may never have a device assignment. It can prevent from leaking RMRR.

4. We reset those mappings as 1:1:p2m_mmio_direct:p2m_ram_rw once we have a device assignment.

5. Before we take real device assignment, any access to RMRR may issue ept_handle_violation because of p2m_access_n. Then we just call update_guest_eip() to return.

6. After a device assignment, guest may maliciously access RMRR ranges although we already reserve in e820 table. In the worst-case scenario just that device can't work well. But this behavior should be same as native so I think we shouldn't do anything here.

7. Its not necessary to introduce any flag in ept_set_entry.

First of all, hypervisor/dom0 should be trusted. Any user should make sure they never override any valid RMRR tables without any check. So our original set_identity_p2m_entry() tries to set as follows:

- gfn space unoccupied -> insert mapping; success.
- gfn space already occupied by 1:1 RMRR mapping -> do nothing; success.
- gfn space already occupied by other mapping -> fail.

Now in our case we add a rule:
- if p2m_access_n is set we also set this mapping.

Another reason is that ept_set_entry is called in many scenarios to support its own management, I think we shouldn't corrupt this mechanism and its also difficult to cover all points.

8. We need to take a consideration grouping all devices that have same RMRR range to make sure they're just assigned to one VM.

----------------------------------------------------------------
Jan Beulich (1):
introduce XENMEM_reserved_device_memory_map

Tiejun Chen (16):
tools/hvmloader: link errno.h from xen internal
introduce XEN_DOMCTL_set_rdm
update the existing hypercall to support XEN_DOMCTL_set_rdm tools/libxc: introduce hypercall for xc_reserved_device_memory_map tools/libxc: check if modules space is overlapping with reserved device memory hvmloader/util: get reserved device memory maps
hvmloader/mmio: reconcile guest mmio with reserved device memory hvmloader/ram: check if guest memory is out of reserved device memory maps hvmloader/mem_hole_alloc: skip any overlap with reserved device memory xen/x86/p2m: reject populating for reserved device memory mapping xen/x86/ept: handle reserved device memory in ept_handle_violation xen/mem_access: don't allow accessing reserved device memory xen/x86/p2m: introduce set_identity_p2m_entry
xen:vtd: create RMRR mapping
xen/vtd: group assigned device with RMRR
xen/vtd: re-enable USB device assignment if enable pci_force

 .gitignore                           |   1 +
 docs/man/xl.cfg.pod.5                |   6 ++++++
 docs/misc/vtd.txt                    |  15 ++++++++++++++
 tools/firmware/hvmloader/Makefile    |   7 ++++++-
 tools/firmware/hvmloader/e820.c      | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/firmware/hvmloader/pci.c       |  54 +++++++++++++++++++++++++++++++++++++++++++++++++-
 tools/firmware/hvmloader/util.c      |  90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 tools/firmware/hvmloader/util.h      |   4 ++++
 tools/libxc/include/xenctrl.h        |  11 +++++++++++
 tools/libxc/xc_domain.c              |  58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/libxc/xc_hvm_build_x86.c       |  94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 tools/libxl/libxl_create.c           |   3 +++
 tools/libxl/libxl_dm.c               |  47 ++++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h         |   4 ++++
 tools/libxl/libxl_types.idl          |   2 ++
 tools/libxl/libxlu_pci.c             |   2 ++
 tools/libxl/xl_cmdimpl.c             |  10 ++++++++++
 xen/arch/x86/hvm/vmx/vmx.c           |  18 +++++++++++++++++
 xen/arch/x86/mm/p2m.c                |  87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 xen/common/compat/memory.c           |  97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 xen/common/mem_access.c              |  41 ++++++++++++++++++++++++++++++++++++++
 xen/common/memory.c                  |  94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/iommu.c      |  10 ++++++++++
 xen/drivers/passthrough/pci.c        |  39 ++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/vtd/dmar.c   |  69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 xen/drivers/passthrough/vtd/dmar.h   |   3 +++
 xen/drivers/passthrough/vtd/extern.h |   1 +
 xen/drivers/passthrough/vtd/iommu.c  |  94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------
 xen/drivers/passthrough/vtd/utils.c  |  18 +++++++++++++++++
 xen/include/asm-x86/hvm/domain.h     |   4 ++++
 xen/include/asm-x86/p2m.h            |  13 ++++++++++++
 xen/include/public/domctl.h          |  21 ++++++++++++++++++++
 xen/include/public/memory.h          |  29 ++++++++++++++++++++++++++-
 xen/include/xen/iommu.h              |   4 ++++
 xen/include/xen/pci.h                |   2 ++
 xen/include/xlat.lst                 |   3 ++-
 xen/xsm/flask/hooks.c                |   1 +
 37 files changed, 1187 insertions(+), 37 deletions(-)

Thanks
Tiejun