xen: RMRR fix

RFC PATCH Superseded Tiejun Chen 7 participants 180 messages Oct 24, 2014 – Nov 21, 2014 Page 1 of 2

Share & tools

Tiejun Chen

author

2014-10-24 07:34 UTC #0 lore raw

RFC PATCH v7 1/13 xen: RMRR fix

This series of patches try to reconcile those remaining problems but just post as RFC to ask for any comments to refine everything.

The current whole scheme is as follows:

1. Reconcile guest mmio with RMRR in pci_setup
2. Reconcile guest RAM with RMRR in e820 table

Then in theory guest wouldn't access any RMRR range.

3. Just initialize all RMRR ranges as p2m_access_n in p2m table:
gfn:mfn:p2m_access_n

Here I think we shouldn't set 1:1 to expose RMRR to guest if guest may never have a device assignment. It can prevent from leaking RMRR.

4. We reset those mappings as 1:1:p2m_mmio_direct:p2m_ram_rw once we have a device assignment.

5. Before we take real device assignment, any access to RMRR may issue ept_handle_violation because of p2m_access_n. Then we just call update_guest_eip() to return.

6. After a device assignment, guest may maliciously access RMRR ranges although we already reserve in e820 table. In the worst-case scenario just that device can't work well. But this behavior should be same as native so I think we shouldn't do anything here.

7. Its not necessary to introduce any flag in ept_set_entry.

First of all, hypervisor/dom0 should be trusted. Any user should make sure they never override any valid RMRR tables without any check. So our original set_identity_p2m_entry() tries to set as follows:

- gfn space unoccupied -> insert mapping; success.
- gfn space already occupied by 1:1 RMRR mapping -> do nothing; success.
- gfn space already occupied by other mapping -> fail.

Now in our case we add a rule:
- if p2m_access_n is set we also set this mapping.

Another reason is that ept_set_entry is called in many scenarios to support its own management, I think we shouldn't corrupt this mechanism and its also difficult to cover all points.

8. We need to take a consideration grouping all devices that have same RMRR range to make sure they're just assigned to one VM.

----------------------------------------------------------------
Jan Beulich (1):
introduce XENMEM_reserved_device_memory_map

Tiejun Chen (12):
tools/libxc: introduce hypercall for xc_reserved_device_memory_map tools/libxc: check if modules space is overlapping with reserved device memory hvmloader/util: get reserved device memory maps
hvmloader/mmio: reconcile guest mmio with reserved device memory hvmloader/ram: check if guest memory is out of reserved device memory maps xen/x86/p2m: introduce p2m_check_reserved_device_memory xen/x86/p2m: set p2m_access_n for reserved device memory mapping xen/x86/ept: handle reserved device memory in ept_handle_violation xen/x86/p2m: introduce set_identity_p2m_entry
xen:vtd: create RMRR mapping
xen/vtd: re-enable USB device assignment
xen/vtd: group assigned device with RMRR

 tools/firmware/hvmloader/e820.c      | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/firmware/hvmloader/pci.c       |  68 +++++++++++++++++++++++++++++++++++++++++++++++++-
 tools/firmware/hvmloader/util.c      |  66 ++++++++++++++++++++++++++++++++++++++++++++++++
 tools/firmware/hvmloader/util.h      |   6 +++++
 tools/libxc/include/xenctrl.h        |   4 +++
 tools/libxc/xc_domain.c              |  29 +++++++++++++++++++++
 tools/libxc/xc_hvm_build_x86.c       | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------
 xen/arch/x86/hvm/vmx/vmx.c           |  14 +++++++++++
 xen/arch/x86/mm/p2m.c                |  52 ++++++++++++++++++++++++++++++++++++++
 xen/common/compat/memory.c           |  52 ++++++++++++++++++++++++++++++++++++++
 xen/common/memory.c                  |  49 ++++++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/iommu.c      |  10 ++++++++
 xen/drivers/passthrough/vtd/dmar.c   |  46 +++++++++++++++++++++++++++++++++-
 xen/drivers/passthrough/vtd/dmar.h   |   3 ++-
 xen/drivers/passthrough/vtd/extern.h |   1 +
 xen/drivers/passthrough/vtd/iommu.c  |  93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------
 xen/drivers/passthrough/vtd/utils.c  |   7 ------
 xen/include/asm-x86/p2m.h            |  17 +++++++++++++
 xen/include/public/memory.h          |  24 +++++++++++++++++-
 xen/include/xen/iommu.h              |   4 +++
 xen/include/xlat.lst                 |   3 ++-
 21 files changed, 828 insertions(+), 46 deletions(-)

Thanks
Tiejun

Tiejun Chen

author

2014-10-24 07:34 UTC #1 lore raw

RFC PATCH v7 1/13 introduce XENMEM_reserved_device_memory_map

From: Jan Beulich <[email protected]>

This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.

Signed-off-by Jan Beulich <[email protected]>

Signed-off-by Tiejun Chen <[email protected]>

---
 xen/common/compat/memory.c           | 52 ++++++++++++++++++++++++++++++++++++
 xen/common/memory.c                  | 49 +++++++++++++++++++++++++++++++++
 xen/drivers/passthrough/iommu.c      | 10 +++++++
 xen/drivers/passthrough/vtd/dmar.c   | 17 ++++++++++++
 xen/drivers/passthrough/vtd/extern.h |  1 +
 xen/drivers/passthrough/vtd/iommu.c  |  1 +
 xen/include/public/memory.h          | 24 ++++++++++++++++-
 xen/include/xen/iommu.h              |  4 +++
 xen/include/xlat.lst                 |  3 ++-
 9 files changed, 159 insertions(+), 2 deletions(-)

xen/common/compat/memory.c +52 -0

index 43d02bc..01b9f6e 100644
@@ -16,6 +16,35 @@ CHECK_TYPE(domid);
 
 CHECK_mem_access_op;
 
+struct get_reserved_device_memory {
+    struct compat_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
+{
+    struct get_reserved_device_memory *grdm = ctxt;
+
+    if ( grdm->used_entries < grdm->map.nr_entries )
+    {
+        struct compat_mem_reserved_device_memory rdm = {
+            .start_pfn = start, .nr_pages = nr
+        };
+
+        if ( rdm.start_pfn != start || rdm.nr_pages != nr )
+            return -ERANGE;
+
+        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                     &rdm, 1) )
+            return -EFAULT;
+    }
+
+    ++grdm->used_entries;
+
+    return 0;
+}
+
 int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
 {
     int split, op = cmd & MEMOP_CMD_MASK;

@@ -273,6 +302,29 @@ int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
             break;
         }
 
+#ifdef HAS_PASSTHROUGH
+        case XENMEM_reserved_device_memory_map:
+        {
+            struct get_reserved_device_memory grdm;
+
+            if ( copy_from_guest(&grdm.map, compat, 1) ||
+                 !compat_handle_okay(grdm.map.buffer, grdm.map.nr_entries) )
+                return -EFAULT;
+
+            grdm.used_entries = 0;
+            rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
+                                                  &grdm);
+
+            if ( !rc && grdm.map.nr_entries < grdm.used_entries )
+                rc = -ENOBUFS;
+            grdm.map.nr_entries = grdm.used_entries;
+            if ( __copy_to_guest(compat, &grdm.map, 1) )
+                rc = -EFAULT;
+
+            return rc;
+        }
+#endif
+
         default:
             return compat_arch_memory_op(cmd, compat);
         }

xen/common/memory.c +49 -0

61 lines not on this page · raw message

xen/drivers/passthrough/iommu.c +10 -0

index cc12735..7c17e8d 100644
@@ -344,6 +344,16 @@ void iommu_crash_shutdown(void)
     iommu_enabled = iommu_intremap = 0;
 }
 
+int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+
+    if ( !iommu_enabled || !ops->get_reserved_device_memory )
+        return 0;
+
+    return ops->get_reserved_device_memory(func, ctxt);
+}
+
 bool_t iommu_has_feature(struct domain *d, enum iommu_feature feature)
 {
     const struct hvm_iommu *hd = domain_hvm_iommu(d);

xen/drivers/passthrough/vtd/dmar.c +17 -0

index 1152c3a..141e735 100644
@@ -893,3 +893,20 @@ int platform_supports_x2apic(void)
     unsigned int mask = ACPI_DMAR_INTR_REMAP | ACPI_DMAR_X2APIC_OPT_OUT;
     return cpu_has_x2apic && ((dmar_flags & mask) == ACPI_DMAR_INTR_REMAP);
 }
+
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    struct acpi_rmrr_unit *rmrr;
+    int rc = 0;
+
+    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    {
+        rc = func(PFN_DOWN(rmrr->base_address),
+                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
+                  ctxt);
+        if ( rc )
+            break;
+    }
+
+    return rc;
+}

xen/drivers/passthrough/vtd/extern.h +1 -0

index 5524dba..f9ee9b0 100644
@@ -75,6 +75,7 @@ int domain_context_mapping_one(struct domain *domain, struct iommu *iommu,
                                u8 bus, u8 devfn, const struct pci_dev *);
 int domain_context_unmap_one(struct domain *domain, struct iommu *iommu,
                              u8 bus, u8 devfn);
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt);
 
 unsigned int io_apic_read_remap_rte(unsigned int apic, unsigned int reg);
 void io_apic_write_remap_rte(unsigned int apic,

xen/drivers/passthrough/vtd/iommu.c +1 -0

index 1c52981..efd3390 100644
@@ -2491,6 +2491,7 @@ const struct iommu_ops intel_iommu_ops = {
     .crash_shutdown = vtd_crash_shutdown,
     .iotlb_flush = intel_iommu_iotlb_flush,
     .iotlb_flush_all = intel_iommu_iotlb_flush_all,
+    .get_reserved_device_memory = intel_iommu_get_reserved_device_memory,
     .dump_p2m_table = vtd_dump_p2m_table,
 };

xen/include/public/memory.h +23 -1

index f021958..4534215 100644
@@ -573,7 +573,29 @@ struct vnuma_topology_info {
 typedef struct vnuma_topology_info vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */
+/*
+ * For legacy reasons, some devices must be configured with special memory
+ * regions to function correctly.  The guest must avoid using any of these
+ * regions.
+ */
+#define XENMEM_reserved_device_memory_map   27
+struct xen_mem_reserved_device_memory {
+    xen_pfn_t start_pfn;
+    xen_ulong_t nr_pages;
+};
+typedef struct xen_mem_reserved_device_memory xen_mem_reserved_device_memory_t;
+DEFINE_XEN_GUEST_HANDLE(xen_mem_reserved_device_memory_t);
+
+struct xen_mem_reserved_device_memory_map {
+    /* IN/OUT */
+    unsigned int nr_entries;
+    /* OUT */
+    XEN_GUEST_HANDLE(xen_mem_reserved_device_memory_t) buffer;
+};
+typedef struct xen_mem_reserved_device_memory_map xen_mem_reserved_device_memory_map_t;
+DEFINE_XEN_GUEST_HANDLE(xen_mem_reserved_device_memory_map_t);
+
+/* Next available subop number is 28 */
 
 #endif /* __XEN_PUBLIC_MEMORY_H__ */

xen/include/xen/iommu.h +4 -0

index 8eb764a..409f6f8 100644
@@ -120,6 +120,8 @@ void iommu_dt_domain_destroy(struct domain *d);
 
 struct page_info;
 
+typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, void *ctxt);
+
 struct iommu_ops {
     int (*init)(struct domain *d);
     void (*hwdom_init)(struct domain *d);

@@ -156,12 +158,14 @@ struct iommu_ops {
     void (*crash_shutdown)(void);
     void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
     void (*iotlb_flush_all)(struct domain *d);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
     void (*dump_p2m_table)(struct domain *d);
 };
 
 void iommu_suspend(void);
 void iommu_resume(void);
 void iommu_crash_shutdown(void);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
 
 void iommu_share_p2m_table(struct domain *d);

xen/include/xlat.lst +2 -1

index 234b668..2e60a0d 100644
@@ -60,7 +60,8 @@
 !	memory_exchange			memory.h
 !	memory_map			memory.h
 !	memory_reservation		memory.h
-?	mem_access_op		memory.h
+?	mem_access_op			memory.h
+!	mem_reserved_device_memory_map	memory.h
 !	pod_target			memory.h
 !	remove_from_physmap		memory.h
 ?	physdev_eoi			physdev.h

Signature · 1 line

1.9.1

Jan Beulich

Acked-by Kevin Tian on v7 1/13

2014-10-24 14:11 UTC #2 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 9 lines

Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
From: Jan Beulich <[email protected]>
This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.
Signed-off-by Jan Beulich <[email protected]>
Signed-off-by Tiejun Chen <[email protected]>

I'm confused - you dropped Kevins ack and instead added you S-o-b despite you not having changed anything in the patch.

That said - in my local version I did a little bit of renaming. I'm attaching the most recent version for your reference.

Jan

introduce XENMEM_reserved_device_memory_map

This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.

Signed-off-by Jan Beulich <[email protected]>

Acked-by Kevin Tian <[email protected]>

xen/common/compat/memory.c +52 -0

@@ -16,6 +16,35 @@ CHECK_TYPE(domid);
 
 CHECK_mem_access_op;
 
+struct get_reserved_device_memory {
+    struct compat_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
+{
+    struct get_reserved_device_memory *grdm = ctxt;
+
+    if ( grdm->used_entries < grdm->map.nr_entries )
+    {
+        struct compat_reserved_device_memory rdm = {
+            .start_pfn = start, .nr_pages = nr
+        };
+
+        if ( rdm.start_pfn != start || rdm.nr_pages != nr )
+            return -ERANGE;
+
+        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                     &rdm, 1) )
+            return -EFAULT;
+    }
+
+    ++grdm->used_entries;
+
+    return 0;
+}
+
 int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
 {
     int split, op = cmd & MEMOP_CMD_MASK;

@@ -273,6 +302,29 @@ int compat_memory_op(unsigned int cmd, X
             break;
         }
 
+#ifdef HAS_PASSTHROUGH
+        case XENMEM_reserved_device_memory_map:
+        {
+            struct get_reserved_device_memory grdm;
+
+            if ( copy_from_guest(&grdm.map, compat, 1) ||
+                 !compat_handle_okay(grdm.map.buffer, grdm.map.nr_entries) )
+                return -EFAULT;
+
+            grdm.used_entries = 0;
+            rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
+                                                  &grdm);
+
+            if ( !rc && grdm.map.nr_entries < grdm.used_entries )
+                rc = -ENOBUFS;
+            grdm.map.nr_entries = grdm.used_entries;
+            if ( __copy_to_guest(compat, &grdm.map, 1) )
+                rc = -EFAULT;
+
+            return rc;
+        }
+#endif
+
         default:
             return compat_arch_memory_op(cmd, compat);
         }

xen/common/memory.c +49 -0

61 lines not on this page · raw message

xen/drivers/passthrough/iommu.c +10 -0

@@ -344,6 +344,16 @@ void iommu_crash_shutdown(void)
     iommu_enabled = iommu_intremap = 0;
 }
 
+int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+
+    if ( !iommu_enabled || !ops->get_reserved_device_memory )
+        return 0;
+
+    return ops->get_reserved_device_memory(func, ctxt);
+}
+
 bool_t iommu_has_feature(struct domain *d, enum iommu_feature feature)
 {
     const struct hvm_iommu *hd = domain_hvm_iommu(d);

xen/drivers/passthrough/vtd/dmar.c +17 -0

@@ -893,3 +893,20 @@ int platform_supports_x2apic(void)
     unsigned int mask = ACPI_DMAR_INTR_REMAP | ACPI_DMAR_X2APIC_OPT_OUT;
     return cpu_has_x2apic && ((dmar_flags & mask) == ACPI_DMAR_INTR_REMAP);
 }
+
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    struct acpi_rmrr_unit *rmrr;
+    int rc = 0;
+
+    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    {
+        rc = func(PFN_DOWN(rmrr->base_address),
+                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
+                  ctxt);
+        if ( rc )
+            break;
+    }
+
+    return rc;
+}

xen/drivers/passthrough/vtd/extern.h +1 -0

@@ -75,6 +75,7 @@ int domain_context_mapping_one(struct do
                                u8 bus, u8 devfn, const struct pci_dev *);
 int domain_context_unmap_one(struct domain *domain, struct iommu *iommu,
                              u8 bus, u8 devfn);
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt);
 
 unsigned int io_apic_read_remap_rte(unsigned int apic, unsigned int reg);
 void io_apic_write_remap_rte(unsigned int apic,

xen/drivers/passthrough/vtd/iommu.c +1 -0

@@ -2491,6 +2491,7 @@ const struct iommu_ops intel_iommu_ops =
     .crash_shutdown = vtd_crash_shutdown,
     .iotlb_flush = intel_iommu_iotlb_flush,
     .iotlb_flush_all = intel_iommu_iotlb_flush_all,
+    .get_reserved_device_memory = intel_iommu_get_reserved_device_memory,
     .dump_p2m_table = vtd_dump_p2m_table,
 };

xen/include/public/memory.h +23 -1

@@ -573,7 +573,29 @@ struct vnuma_topology_info {
 typedef struct vnuma_topology_info vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */
+/*
+ * For legacy reasons, some devices must be configured with special memory
+ * regions to function correctly.  The guest must avoid using any of these
+ * regions.
+ */
+#define XENMEM_reserved_device_memory_map   27
+struct xen_reserved_device_memory {
+    xen_pfn_t start_pfn;
+    xen_ulong_t nr_pages;
+};
+typedef struct xen_reserved_device_memory xen_reserved_device_memory_t;
+DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_t);
+
+struct xen_reserved_device_memory_map {
+    /* IN/OUT */
+    unsigned int nr_entries;
+    /* OUT */
+    XEN_GUEST_HANDLE(xen_reserved_device_memory_t) buffer;
+};
+typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
+DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
+
+/* Next available subop number is 28 */
 
 #endif /* __XEN_PUBLIC_MEMORY_H__ */

xen/include/xen/iommu.h +4 -0

@@ -120,6 +120,8 @@ void iommu_dt_domain_destroy(struct doma
 
 struct page_info;
 
+typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, void *ctxt);
+
 struct iommu_ops {
     int (*init)(struct domain *d);
     void (*hwdom_init)(struct domain *d);

@@ -156,12 +158,14 @@ struct iommu_ops {
     void (*crash_shutdown)(void);
     void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
     void (*iotlb_flush_all)(struct domain *d);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
     void (*dump_p2m_table)(struct domain *d);
 };
 
 void iommu_suspend(void);
 void iommu_resume(void);
 void iommu_crash_shutdown(void);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
 
 void iommu_share_p2m_table(struct domain *d);

xen/include/xlat.lst +2 -1

@@ -61,9 +61,10 @@
 !	memory_exchange			memory.h
 !	memory_map			memory.h
 !	memory_reservation		memory.h
-?	mem_access_op		memory.h
+?	mem_access_op			memory.h
 !	pod_target			memory.h
 !	remove_from_physmap		memory.h
+!	reserved_device_memory_map	memory.h
 ?	physdev_eoi			physdev.h
 ?	physdev_get_free_pirq		physdev.h
 ?	physdev_irq			physdev.h

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Chen, Tiejun

author

2014-10-27 02:11 UTC #3 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 11 lines · Jan Beulich

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
From: Jan Beulich <[email protected]>
This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.
Signed-off-by Jan Beulich <[email protected]>
Signed-off-by Tiejun Chen <[email protected]>
I'm confused - you dropped Kevins ack and instead added you S-o-b

I need to add this ACK.

Quoted 1 line

despite you not having changed anything in the patch.

The original you sent to me previously is needed to rebase on the latest. Please check xen/include/public/memory.h.

The original:

xen/include/public/memory.h +0 -1

@@ -573,7 +573,29 @@ struct vnuma_topology_info {
 typedef struct vnuma_topology_info vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */

Mine:

xen/include/public/memory.h +0 -1

@@ -523,7 +523,29 @@ DEFINE_XEN_GUEST_HANDLE(xen_mem_sharing_
 
 #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
-/* Next available subop number is 26 */

Unless you're saying this kind of rebase shouldn't introduce my SOB.

Quoted 3 lines

That said - in my local version I did a little bit of renaming. I'm attaching the most recent version for your reference.

I will pick this to try.

Thanks
Tiejun

Chen, Tiejun

author

2014-10-27 02:18 UTC #4 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 13 lines · Tiejun

Quoted 11 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
From: Jan Beulich <[email protected]>
This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.
Signed-off-by Jan Beulich <[email protected]>
Signed-off-by Tiejun Chen <[email protected]>
I'm confused - you dropped Kevins ack and instead added you S-o-b

Quoted 8 lines

I need to add this ACK.
Quoted 1 line
despite you not having changed anything in the patch.
The original you sent to me previously is needed to rebase on the latest. Please check xen/include/public/memory.h.
The original:

s/The original/Mine

Quoted 10 lines

xen/include/public/memory.h +0 -1

@@ -573,7 +573,29 @@ struct vnuma_topology_info {
  typedef struct vnuma_topology_info vnuma_topology_info_t;
  DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */

Mine:

s/Mine/The original

Quoted 26 lines

xen/include/public/memory.h +0 -1
@@ -523,7 +523,29 @@ DEFINE_XEN_GUEST_HANDLE(xen_mem_sharing_
 
  #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
-/* Next available subop number is 26 */
 
Unless you're saying this kind of rebase shouldn't introduce my SOB.
Quoted 3 lines
That said - in my local version I did a little bit of renaming. I'm attaching the most recent version for your reference.
I will pick this to try.
Thanks
Tiejun
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Jan Beulich

2014-10-27 09:42 UTC #5 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

In reply to Chen, Tiejun

Quoted 41 lines

Quoted 1 line
Quoted 1 line · On 27.10.14 at 03:11, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
From: Jan Beulich <[email protected]>
This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.
Signed-off-by Jan Beulich <[email protected]>
Signed-off-by Tiejun Chen <[email protected]>
I'm confused - you dropped Kevins ack and instead added you S-o-b
I need to add this ACK.
Quoted 1 line
despite you not having changed anything in the patch.
The original you sent to me previously is needed to rebase on the latest. Please check xen/include/public/memory.h.
The original:
xen/include/public/memory.h +0 -1
@@ -573,7 +573,29 @@ struct vnuma_topology_info {
  typedef struct vnuma_topology_info vnuma_topology_info_t;
  DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */
 
Mine:
xen/include/public/memory.h +0 -1
@@ -523,7 +523,29 @@ DEFINE_XEN_GUEST_HANDLE(xen_mem_sharing_
 
  #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
-/* Next available subop number is 26 */
 

Quoted 1 line

Unless you're saying this kind of rebase shouldn't introduce my SOB.

I indeed don't think rebasing counts as you actively doing any changes.

Jan

Chen, Tiejun

author

2014-10-28 02:22 UTC #6 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 37 lines · Jan Beulich

Quoted 37 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 03:11, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
From: Jan Beulich <[email protected]>
This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.
Signed-off-by Jan Beulich <[email protected]>
Signed-off-by Tiejun Chen <[email protected]>
I'm confused - you dropped Kevins ack and instead added you S-o-b
I need to add this ACK.
Quoted 1 line
despite you not having changed anything in the patch.
The original you sent to me previously is needed to rebase on the latest. Please check xen/include/public/memory.h.
The original:
xen/include/public/memory.h +0 -1
@@ -573,7 +573,29 @@ struct vnuma_topology_info {
   typedef struct vnuma_topology_info vnuma_topology_info_t;
   DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */
 
Mine:
xen/include/public/memory.h +0 -0
@@ -523,7 +523,29 @@ DEFINE_XEN_GUEST_HANDLE(xen_mem_sharing_
 

Quoted 9 lines

Quoted 5 lines
xen/include/public/memory.h +0 -1
   #endif /* defined(__XEN__) || defined(__XEN_TOOLS__) */
 
-/* Next available subop number is 26 */
 
Unless you're saying this kind of rebase shouldn't introduce my SOB.
I indeed don't think rebasing counts as you actively doing any changes.

Fine. I think this patch has a long history between us.

Thanks
Tiejun

Julien Grall

2014-10-27 13:35 UTC #7 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

In reply to Tiejun Chen

Hi,

Quoted 5 lines · Tiejun Chen

xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644

xen/common/memory.c out: · line 701

Quoted 10 lines

xen/common/memory.c +7 -0

@@ -692,6 +692,32 @@ out:
     return rc;
 }
 
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)

This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.

Regards,

Signature · 1 line

Julien Grall

Chen, Tiejun

author Acked-by Kevin Tian

2014-10-28 02:35 UTC #8 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 11 lines · Julien Grall

Hi,
Quoted 8 lines · Tiejun Chen
xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644
@@ -692,6 +692,32 @@ out:
      return rc;
  }
 

Quoted 11 lines

xen/common/memory.c out: · line 701
Quoted 7 lines
xen/common/memory.c +7 -0
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.

I guess you mean we need to do this,

xen/common/memory.c +2 -0

index 1449c10..2177c56 100644
@@ -692,6 +692,7 @@ out:
     return rc;
 }
 
+#ifdef HAS_PASSTHROUGH
 struct get_reserved_device_memory {
     struct xen_reserved_device_memory_map map;
     unsigned int used_entries;

@@ -717,6 +718,7 @@ static int get_reserved_device_memory(xen_pfn_t start,
 
     return 0;
 }
+#endif
 
 long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {

Jan,

With this above change, is the following working for you?

Signed-off-by Jan Beulich <[email protected]>

Acked-by Kevin Tian <[email protected]>

CC Julien Grall <[email protected]>

[Julien: Protect some definitions by an #ifdef HAS_PASSTHROUGH.]

Signed-off-by Tiejun Chen <[email protected]>

Thanks
Tiejun

Jan Beulich

2014-10-28 10:36 UTC #9 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 48 lines

Quoted 1 line

Quoted 1 line · On 28.10.14 at 03:35, <[email protected]> wrote:

Quoted 11 lines · Julien Grall

Hi,
Quoted 8 lines · Tiejun Chen
xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644
@@ -692,6 +692,32 @@ out:
      return rc;
  }
 

Quoted 11 lines

xen/common/memory.c out: · line 701
Quoted 7 lines
xen/common/memory.c +7 -0
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.

I guess you mean we need to do this,

xen/common/memory.c +2 -0

index 1449c10..2177c56 100644
@@ -692,6 +692,7 @@ out:
      return rc;
  }
 
+#ifdef HAS_PASSTHROUGH
  struct get_reserved_device_memory {
      struct xen_reserved_device_memory_map map;
      unsigned int used_entries;

@@ -717,6 +718,7 @@ static int get_reserved_device_memory(xen_pfn_t start,
 
      return 0;
  }
+#endif
 
  long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
  {

Quoted 3 lines

Jan,
With this above change, is the following working for you?

I already fixed this in my version (which I view to be the canonical one).

Jan

Chen, Tiejun

author

2014-10-29 00:40 UTC #10 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 48 lines · Jan Beulich

Quoted 48 lines

Quoted 1 line

Quoted 1 line · On 28.10.14 at 03:35, <[email protected]> wrote:

Quoted 11 lines · Julien Grall

Hi,
Quoted 8 lines · Tiejun Chen
xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644
@@ -692,6 +692,32 @@ out:
       return rc;
   }
 

Quoted 11 lines

xen/common/memory.c out: · line 701
Quoted 7 lines
xen/common/memory.c +7 -0
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.

I guess you mean we need to do this,

xen/common/memory.c +2 -0

index 1449c10..2177c56 100644
@@ -692,6 +692,7 @@ out:
       return rc;
   }
 
+#ifdef HAS_PASSTHROUGH
   struct get_reserved_device_memory {
       struct xen_reserved_device_memory_map map;
       unsigned int used_entries;

@@ -717,6 +718,7 @@ static int get_reserved_device_memory(xen_pfn_t start,
 
       return 0;
   }
+#endif
 
   long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
   {

Quoted 7 lines

Quoted 3 lines
Jan,
With this above change, is the following working for you?
I already fixed this in my version (which I view to be the canonical one).

Are you point that attached patch? Are you sure? Here I pick some code fragments from your latest patch,

xen/common/memory.c +49 -0

@@ -692,6 +692,32 @@ out:
     return rc;
 }
 
+struct get_reserved_device_memory {
+    struct xen_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
+{
+    struct get_reserved_device_memory *grdm = ctxt;
+
+    if ( grdm->used_entries < grdm->map.nr_entries )
+    {
+        struct xen_reserved_device_memory rdm = {
+            .start_pfn = start, .nr_pages = nr
+        };
+
+        if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
+                                    &rdm, 1) )
+            return -EFAULT;
+    }
+
+    ++grdm->used_entries;
+
+    return 0;
+}
+
 long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
     struct domain *d;

@@ -1101,6 +1127,29 @@ long do_memory_op(unsigned long cmd, XEN
         break;
     }
 
+#ifdef HAS_PASSTHROUGH
+    case XENMEM_reserved_device_memory_map:
+    {
+        struct get_reserved_device_memory grdm;
+
+        if ( copy_from_guest(&grdm.map, arg, 1) ||
+             !guest_handle_okay(grdm.map.buffer, grdm.map.nr_entries) )
+            return -EFAULT;
+
+        grdm.used_entries = 0;
+        rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
+                                              &grdm);
+
+        if ( !rc && grdm.map.nr_entries < grdm.used_entries )
+            rc = -ENOBUFS;
+        grdm.map.nr_entries = grdm.used_entries;
+        if ( __copy_to_guest(arg, &grdm.map, 1) )
+            rc = -EFAULT;
+
+        break;
+    }
+#endif
+
     default:
         rc = arch_memory_op(cmd, arg);
         break;

--- a/xen/drivers/passthrough/iommu.c

Thanks
Tiejun

Quoted 3 lines

Jan

Jan Beulich

2014-10-29 08:53 UTC #11 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 52 lines

Quoted 1 line
Quoted 1 line · On 29.10.14 at 01:40, <[email protected]> wrote:
Quoted 50 lines · Jan Beulich
Quoted 50 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 03:35, <[email protected]> wrote:
Quoted 11 lines · Julien Grall
Hi,
Quoted 8 lines · Tiejun Chen
xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644
@@ -692,6 +692,32 @@ out:
       return rc;
   }
 
Quoted 11 lines
xen/common/memory.c out: · line 701
Quoted 7 lines
xen/common/memory.c +7 -0
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.
I guess you mean we need to do this,
xen/common/memory.c +2 -0
index 1449c10..2177c56 100644
@@ -692,6 +692,7 @@ out:
       return rc;
   }
 
+#ifdef HAS_PASSTHROUGH
   struct get_reserved_device_memory {
       struct xen_reserved_device_memory_map map;
       unsigned int used_entries;
@@ -717,6 +718,7 @@ static int get_reserved_device_memory(xen_pfn_t start,
 
       return 0;
   }
+#endif
 
   long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
   {
Jan,

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
With this above change, is the following working for you?
I already fixed this in my version (which I view to be the canonical one).
Are you point that attached patch? Are you sure? Here I pick some code fragments from your latest patch,

No, that fixup was in response to Julien's comment, which I think was given after I had forwarded you my version. In any event, what's going to get applied is what I have here unless you find a need to do substantial changes to it.

Jan

Chen, Tiejun

author

2014-10-30 02:53 UTC #12 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 58 lines · Jan Beulich

Quoted 58 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 01:40, <[email protected]> wrote:
Quoted 55 lines · Jan Beulich
Quoted 48 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 03:35, <[email protected]> wrote:
Quoted 11 lines · Julien Grall
Hi,
Quoted 8 lines · Tiejun Chen
xen/common/memory.c +0 -0
index cc36e39..51a32a8 100644
@@ -692,6 +692,32 @@ out:
        return rc;
    }
 
Quoted 11 lines
xen/common/memory.c out: · line 701
Quoted 7 lines
xen/common/memory.c +7 -0
+struct get_reserved_device_memory {
+    struct xen_mem_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
This function is only used when HAS_PASSTHROUGH is defined. You have to protected by an #ifdef HAS_PASSTHROUGH.
I guess you mean we need to do this,
xen/common/memory.c +2 -0
index 1449c10..2177c56 100644
@@ -692,6 +692,7 @@ out:
        return rc;
    }
 
+#ifdef HAS_PASSTHROUGH
    struct get_reserved_device_memory {
        struct xen_reserved_device_memory_map map;
        unsigned int used_entries;
@@ -717,6 +718,7 @@ static int get_reserved_device_memory(xen_pfn_t start,
 
        return 0;
    }
+#endif
 
    long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
    {
Quoted 3 lines
Jan,
With this above change, is the following working for you?
I already fixed this in my version (which I view to be the canonical one).

Quoted 5 lines

Quoted 2 lines
Are you point that attached patch? Are you sure? Here I pick some code fragments from your latest patch,
No, that fixup was in response to Julien's comment, which I think was given after I had forwarded you my version. In any event,

So could you send me the latest? I'd like to replace it in my tree.

Thanks
Tiejun

Quoted 7 lines

what's going to get applied is what I have here unless you find a need to do substantial changes to it.
Jan

Jan Beulich

Acked-by Kevin Tian

2014-10-30 09:10 UTC #13 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 2 lines

Quoted 1 line
Quoted 1 line · On 30.10.14 at 03:53, <[email protected]> wrote:
So could you send me the latest? I'd like to replace it in my tree.

Here you go.

Jan

introduce XENMEM_reserved_device_memory_map

This is a prerequisite for punching holes into HVM and PVH guests' P2M to allow passing through devices that are associated with (on VT-d) RMRRs.

Signed-off-by Jan Beulich <[email protected]>

Acked-by Kevin Tian <[email protected]>

xen/common/compat/memory.c +54 -0

@@ -16,6 +16,37 @@ CHECK_TYPE(domid);
 
 CHECK_mem_access_op;
 
+#ifdef HAS_PASSTHROUGH
+struct get_reserved_device_memory {
+    struct compat_reserved_device_memory_map map;
+    unsigned int used_entries;
+};
+
+static int get_reserved_device_memory(xen_pfn_t start,
+                                      xen_ulong_t nr, void *ctxt)
+{
+    struct get_reserved_device_memory *grdm = ctxt;
+
+    if ( grdm->used_entries < grdm->map.nr_entries )
+    {
+        struct compat_reserved_device_memory rdm = {
+            .start_pfn = start, .nr_pages = nr
+        };
+
+        if ( rdm.start_pfn != start || rdm.nr_pages != nr )
+            return -ERANGE;
+
+        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                     &rdm, 1) )
+            return -EFAULT;
+    }
+
+    ++grdm->used_entries;
+
+    return 0;
+}
+#endif
+
 int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
 {
     int split, op = cmd & MEMOP_CMD_MASK;

@@ -273,6 +304,29 @@ int compat_memory_op(unsigned int cmd, X
             break;
         }
 
+#ifdef HAS_PASSTHROUGH
+        case XENMEM_reserved_device_memory_map:
+        {
+            struct get_reserved_device_memory grdm;
+
+            if ( copy_from_guest(&grdm.map, compat, 1) ||
+                 !compat_handle_okay(grdm.map.buffer, grdm.map.nr_entries) )
+                return -EFAULT;
+
+            grdm.used_entries = 0;
+            rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
+                                                  &grdm);
+
+            if ( !rc && grdm.map.nr_entries < grdm.used_entries )
+                rc = -ENOBUFS;
+            grdm.map.nr_entries = grdm.used_entries;
+            if ( __copy_to_guest(compat, &grdm.map, 1) )
+                rc = -EFAULT;
+
+            return rc;
+        }
+#endif
+
         default:
             return compat_arch_memory_op(cmd, compat);
         }

xen/common/memory.c +51 -0

63 lines not on this page · raw message

xen/drivers/passthrough/iommu.c +10 -0

@@ -344,6 +344,16 @@ void iommu_crash_shutdown(void)
     iommu_enabled = iommu_intremap = 0;
 }
 
+int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+
+    if ( !iommu_enabled || !ops->get_reserved_device_memory )
+        return 0;
+
+    return ops->get_reserved_device_memory(func, ctxt);
+}
+
 bool_t iommu_has_feature(struct domain *d, enum iommu_feature feature)
 {
     const struct hvm_iommu *hd = domain_hvm_iommu(d);

xen/drivers/passthrough/vtd/dmar.c +17 -0

@@ -893,3 +893,20 @@ int platform_supports_x2apic(void)
     unsigned int mask = ACPI_DMAR_INTR_REMAP | ACPI_DMAR_X2APIC_OPT_OUT;
     return cpu_has_x2apic && ((dmar_flags & mask) == ACPI_DMAR_INTR_REMAP);
 }
+
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    struct acpi_rmrr_unit *rmrr;
+    int rc = 0;
+
+    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    {
+        rc = func(PFN_DOWN(rmrr->base_address),
+                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
+                  ctxt);
+        if ( rc )
+            break;
+    }
+
+    return rc;
+}

xen/drivers/passthrough/vtd/extern.h +1 -0

@@ -75,6 +75,7 @@ int domain_context_mapping_one(struct do
                                u8 bus, u8 devfn, const struct pci_dev *);
 int domain_context_unmap_one(struct domain *domain, struct iommu *iommu,
                              u8 bus, u8 devfn);
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt);
 
 unsigned int io_apic_read_remap_rte(unsigned int apic, unsigned int reg);
 void io_apic_write_remap_rte(unsigned int apic,

xen/drivers/passthrough/vtd/iommu.c +1 -0

@@ -2491,6 +2491,7 @@ const struct iommu_ops intel_iommu_ops =
     .crash_shutdown = vtd_crash_shutdown,
     .iotlb_flush = intel_iommu_iotlb_flush,
     .iotlb_flush_all = intel_iommu_iotlb_flush_all,
+    .get_reserved_device_memory = intel_iommu_get_reserved_device_memory,
     .dump_p2m_table = vtd_dump_p2m_table,
 };

xen/include/public/memory.h +23 -1

@@ -573,7 +573,29 @@ struct vnuma_topology_info {
 typedef struct vnuma_topology_info vnuma_topology_info_t;
 DEFINE_XEN_GUEST_HANDLE(vnuma_topology_info_t);
 
-/* Next available subop number is 27 */
+/*
+ * For legacy reasons, some devices must be configured with special memory
+ * regions to function correctly.  The guest must avoid using any of these
+ * regions.
+ */
+#define XENMEM_reserved_device_memory_map   27
+struct xen_reserved_device_memory {
+    xen_pfn_t start_pfn;
+    xen_ulong_t nr_pages;
+};
+typedef struct xen_reserved_device_memory xen_reserved_device_memory_t;
+DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_t);
+
+struct xen_reserved_device_memory_map {
+    /* IN/OUT */
+    unsigned int nr_entries;
+    /* OUT */
+    XEN_GUEST_HANDLE(xen_reserved_device_memory_t) buffer;
+};
+typedef struct xen_reserved_device_memory_map xen_reserved_device_memory_map_t;
+DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_map_t);
+
+/* Next available subop number is 28 */
 
 #endif /* __XEN_PUBLIC_MEMORY_H__ */

xen/include/xen/iommu.h +4 -0

@@ -120,6 +120,8 @@ void iommu_dt_domain_destroy(struct doma
 
 struct page_info;
 
+typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, void *ctxt);
+
 struct iommu_ops {
     int (*init)(struct domain *d);
     void (*hwdom_init)(struct domain *d);

@@ -156,12 +158,14 @@ struct iommu_ops {
     void (*crash_shutdown)(void);
     void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
     void (*iotlb_flush_all)(struct domain *d);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
     void (*dump_p2m_table)(struct domain *d);
 };
 
 void iommu_suspend(void);
 void iommu_resume(void);
 void iommu_crash_shutdown(void);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
 
 void iommu_share_p2m_table(struct domain *d);

xen/include/xlat.lst +2 -1

@@ -61,9 +61,10 @@
 !	memory_exchange			memory.h
 !	memory_map			memory.h
 !	memory_reservation		memory.h
-?	mem_access_op		memory.h
+?	mem_access_op			memory.h
 !	pod_target			memory.h
 !	remove_from_physmap		memory.h
+!	reserved_device_memory_map	memory.h
 ?	physdev_eoi			physdev.h
 ?	physdev_get_free_pirq		physdev.h
 ?	physdev_irq			physdev.h

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Chen, Tiejun

author

2014-10-31 01:03 UTC #14 lore raw

Re: [v7][RFC][PATCH 01/13] introduce XENMEM_reserved_device_memory_map

Quoted 4 lines · Jan Beulich

Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 03:53, <[email protected]> wrote:
So could you send me the latest? I'd like to replace it in my tree.
Here you go.

Thanks a lot.

Tiejun

Tiejun Chen

author

2014-10-24 07:34 UTC #15 lore raw

RFC PATCH v7 2/13 tools/libxc: introduce hypercall for xc_reserved_device_memory_map

In reply to Tiejun Chen

We will introduce that hypercall xc_reserved_device_memory_map approach to libxc.

Signed-off-by Tiejun Chen <[email protected]>

---
 tools/libxc/include/xenctrl.h |  4 ++++
 tools/libxc/xc_domain.c       | 29 +++++++++++++++++++++++++++++
 2 files changed, 33 insertions(+)

tools/libxc/include/xenctrl.h +4 -0

index 564e187..b4a60ba 100644
@@ -1288,6 +1288,10 @@ int xc_domain_set_memory_map(xc_interface *xch,
 int xc_get_machine_memory_map(xc_interface *xch,
                               struct e820entry entries[],
                               uint32_t max_entries);
+
+int xc_reserved_device_memory_map(xc_interface *xch,
+                                  struct xen_mem_reserved_device_memory entries[],
+                                  uint32_t *max_entries);
 #endif
 int xc_domain_set_time_offset(xc_interface *xch,
                               uint32_t domid,

tools/libxc/xc_domain.c +29 -0

index a9bcd4a..5f63b6c 100644
@@ -659,6 +659,35 @@ int xc_domain_set_memory_map(xc_interface *xch,
 
     return rc;
 }
+
+int xc_reserved_device_memory_map(xc_interface *xch,
+                                  struct xen_mem_reserved_device_memory entries[],
+                                  uint32_t *max_entries)
+{
+    int rc;
+    struct xen_mem_reserved_device_memory_map memmap = {
+        .nr_entries = *max_entries
+    };
+    DECLARE_HYPERCALL_BOUNCE(entries,
+                             sizeof(struct xen_mem_reserved_device_memory) *
+                             *max_entries, XC_HYPERCALL_BUFFER_BOUNCE_OUT);
+
+    if ( xc_hypercall_bounce_pre(xch, entries) )
+        return -1;
+
+    set_xen_guest_handle(memmap.buffer, entries);
+
+    rc = do_memory_op(xch, XENMEM_reserved_device_memory_map,
+                      &memmap, sizeof(memmap));
+
+    xc_hypercall_bounce_post(xch, entries);
+
+    if ( errno == ENOBUFS )
+        *max_entries = memmap.nr_entries;
+
+    return rc ? rc : memmap.nr_entries;
+}
+
 int xc_get_machine_memory_map(xc_interface *xch,
                               struct e820entry entries[],
                               uint32_t max_entries)

Signature · 1 line

1.9.1

Tiejun Chen

author

2014-10-24 07:34 UTC #16 lore raw

RFC PATCH v7 3/13 tools/libxc: check if modules space is overlapping with reserved device memory

In reply to Tiejun Chen

In case of reserved device memory overlapping with ram, it also probably overlap with modules space so we need to check these reserved device memory as well.

Signed-off-by Tiejun Chen <[email protected]>

---
 tools/libxc/xc_hvm_build_x86.c | 111 +++++++++++++++++++++++++++++++++++------
 1 file changed, 96 insertions(+), 15 deletions(-)

tools/libxc/xc_hvm_build_x86.c +96 ~~-15~~

index c81a25b..cb590ee 100644
@@ -54,9 +54,99 @@
 
 #define VGA_HOLE_SIZE (0x20)
 
+/* Record reserved device memory. */
+static struct xen_mem_reserved_device_memory *xmrdm = NULL;
+
+/*
+ * Check whether there exists mmio hole in the specified memory range.
+ * Returns 1 if exists, else returns 0.
+ */
+static int check_mmio_hole(uint64_t start, uint64_t memsize,
+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;
+}
+
+/* Getting all reserved device memory map info. */
+static int xc_get_reserved_device_memory_map(xc_interface *xch)
+{
+    static uint32_t nr_entries = 0;
+    int rc = 0;
+
+    if ( !xmrdm )
+    {
+        /* Assume we have one entry if not enough we'll expand.*/
+        nr_entries = 1;
+        if ( (xmrdm = malloc(nr_entries *
+                             sizeof(xen_mem_reserved_device_memory_t))) == NULL )
+        {
+            PERROR("Could not allocate memory for map.");
+            return -1;
+        }
+
+        rc = xc_reserved_device_memory_map(xch, xmrdm, &nr_entries);
+        if ( rc < 0 )
+        {
+            switch ( errno )
+            {
+            case ENOENT: /* reserved device memory doesn't exist. */
+                rc = 0;
+                break;
+            case ENOBUFS: /* Need more space */
+                free(xmrdm);
+                /* Now we need more space to map all reserved device memory maps. */
+                if ( (xmrdm = malloc(nr_entries *
+                                     sizeof(xen_mem_reserved_device_memory_t))) == NULL )
+                {
+                    PERROR("Could not allocate memory.");
+                    return -1;
+                }
+                rc = xc_reserved_device_memory_map(xch, xmrdm, &nr_entries);
+                if ( rc )
+                {
+                    PERROR("Could not get reserved device memory maps.");
+                    free(xmrdm);
+                    return rc;
+                }
+                break;
+            default: /* Failed to get reserved device memory maps. */
+                PERROR("Could not get reserved device memory maps.");
+                return rc;
+            }
+        }
+    }
+
+    return nr_entries;
+}
+
+static int xc_check_modules_space(xc_interface *xch, uint64_t *mstart_out,
+                                  uint64_t *mend_out)
+{
+    unsigned int i = 0;
+    uint64_t rdm_start = 0, rdm_end = 0;
+    int nr_entries = xc_get_reserved_device_memory_map(xch);
+
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = xmrdm[i].start_pfn << XC_PAGE_SHIFT;
+        rdm_end = rdm_start + (xmrdm[i].nr_pages << XC_PAGE_SHIFT);
+
+        /* Just use check_mmio_hole() to check modules ranges. */
+        if ( check_mmio_hole(rdm_start, xmrdm[i].nr_pages << XC_PAGE_SHIFT,
+                             *mstart_out, *mend_out) )
+            return -1;
+    }
+
+    return 0;
+}
+
 static int modules_init(struct xc_hvm_build_args *args,
                         uint64_t vend, struct elf_binary *elf,
-                        uint64_t *mstart_out, uint64_t *mend_out)
+                        uint64_t *mstart_out, uint64_t *mend_out,
+                        xc_interface *xch)
 {
 #define MODULE_ALIGN 1UL << 7
 #define MB_ALIGN     1UL << 20

@@ -80,6 +170,10 @@ static int modules_init(struct xc_hvm_build_args *args,
     if ( *mend_out > vend )    
         return -1;
 
+    /* Is it overlapping with reserved device memory? */
+    if ( xc_check_modules_space(xch, mstart_out, mend_out) )
+        return -1;
+
     if ( args->acpi_module.length != 0 )
         args->acpi_module.guest_addr_out = *mstart_out;
     if ( args->smbios_module.length != 0 )

@@ -226,19 +320,6 @@ static int loadmodules(xc_interface *xch,
     return rc;
 }
 
-/*
- * Check whether there exists mmio hole in the specified memory range.
- * Returns 1 if exists, else returns 0.
- */
-static int check_mmio_hole(uint64_t start, uint64_t memsize,
-                           uint64_t mmio_start, uint64_t mmio_size)
-{
-    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
-        return 0;
-    else
-        return 1;
-}
-
 static int setup_guest(xc_interface *xch,
                        uint32_t dom, struct xc_hvm_build_args *args,
                        char *image, unsigned long image_size)

@@ -282,7 +363,7 @@ static int setup_guest(xc_interface *xch,
         goto error_out;
     }
 
-    if ( modules_init(args, v_end, &elf, &m_start, &m_end) != 0 )
+    if ( modules_init(args, v_end, &elf, &m_start, &m_end, xch) != 0 )
     {
         ERROR("Insufficient space to load modules.");
         goto error_out;

Signature · 1 line

1.9.1

Tiejun Chen

author

2014-10-24 07:34 UTC #17 lore raw

RFC PATCH v7 4/13 hvmloader/util: get reserved device memory maps

In reply to Tiejun Chen

We need to use reserved device memory maps with multiple times, so provide just one common function should be friend.

Signed-off-by Tiejun Chen <[email protected]>

---
 tools/firmware/hvmloader/util.c | 66 +++++++++++++++++++++++++++++++++++++++++
 tools/firmware/hvmloader/util.h |  6 ++++
 2 files changed, 72 insertions(+)

tools/firmware/hvmloader/util.c +66 -0

index 80d822f..aa2c17c 100644
@@ -828,6 +828,72 @@ int hpet_exists(unsigned long hpet_base)
     return ((hpet_id >> 16) == 0x8086);
 }
 
+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries)
+{
+    int rc;
+    struct xen_mem_reserved_device_memory_map memmap = {
+        .nr_entries = *max_entries
+    };
+
+    set_xen_guest_handle(memmap.buffer, entries);
+
+    rc = hypercall_memory_op(XENMEM_reserved_device_memory_map, &memmap);
+    if (rc == -ENOBUFS)
+        *max_entries = memmap.nr_entries;
+
+    return rc;
+}
+
+/* Getting all reserved device memory map info in case of hvmloader. */
+int hvm_get_reserved_device_memory_map(void)
+{
+    static uint32_t nr_entries = 0;
+    int rc = 0;
+
+    if ( !rdm_map )
+    {
+        /* Assume we have one entry if not enough we'll expand.*/
+        nr_entries = 1;
+        rdm_map = mem_alloc(nr_entries *
+                            sizeof(struct xen_mem_reserved_device_memory), 0);
+        if ( !rdm_map )
+        {
+            printf("No space to get reserved dev memory maps!\n");
+            return rc;
+        }
+
+        rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+        if ( rc == -ENOBUFS )
+        {
+            rdm_map = mem_alloc(nr_entries *
+                                sizeof(struct xen_mem_reserved_device_memory),
+                                0);
+            if ( rdm_map )
+            {
+                rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+                if ( rc )
+                {
+                    printf("Could not get reserved dev memory info on domain");
+                    return rc;
+                }
+            }
+            else
+            {
+                printf("No space to get reserved dev memory maps!\n");
+                return rc;
+            }
+        }
+        else if ( rc )
+        {
+            printf("Could not get reserved dev memory info on domain");
+            return rc;
+        }
+    }
+
+    return nr_entries;
+}
+
 /*
  * Local variables:
  * mode: C

tools/firmware/hvmloader/util.h +6 -0

index a70e4aa..81abc44 100644
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                      unsigned int bios_image_base);
 void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
+struct xen_mem_reserved_device_memory *rdm_map;
+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries);
+int hvm_get_reserved_device_memory_map(void);
 #ifndef NDEBUG
 void perform_tests(void);
 #else

Signature · 1 line

1.9.1

Jan Beulich

2014-10-24 14:22 UTC #18 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:

tools/firmware/hvmloader/util.c +2 -0

@@ -828,6 +828,72 @@ int hpet_exists(unsigned long hpet_base)
     return ((hpet_id >> 16) == 0x8086);
 }
 
+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries)

tools/firmware/hvmloader/util.c int hpet_exists(unsigned long hpet_base) · line 842

Quoted 10 lines

tools/firmware/hvmloader/util.c +10 -0

+{
+    int rc;
+    struct xen_mem_reserved_device_memory_map memmap = {
+        .nr_entries = *max_entries
+    };
+
+    set_xen_guest_handle(memmap.buffer, entries);
+
+    rc = hypercall_memory_op(XENMEM_reserved_device_memory_map, &memmap);
+    if (rc == -ENOBUFS)

Coding style.

Quoted 43 lines

tools/firmware/hvmloader/util.c +43 -0

+        *max_entries = memmap.nr_entries;
+
+    return rc;
+}
+
+/* Getting all reserved device memory map info in case of hvmloader. */
+int hvm_get_reserved_device_memory_map(void)
+{
+    static uint32_t nr_entries = 0;
+    int rc = 0;
+
+    if ( !rdm_map )
+    {
+        /* Assume we have one entry if not enough we'll expand.*/
+        nr_entries = 1;
+        rdm_map = mem_alloc(nr_entries *
+                            sizeof(struct xen_mem_reserved_device_memory), 0);
+        if ( !rdm_map )
+        {
+            printf("No space to get reserved dev memory maps!\n");
+            return rc;
+        }
+
+        rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+        if ( rc == -ENOBUFS )
+        {
+            rdm_map = mem_alloc(nr_entries *
+                                sizeof(struct xen_mem_reserved_device_memory),
+                                0);
+            if ( rdm_map )
+            {
+                rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+                if ( rc )
+                {
+                    printf("Could not get reserved dev memory info on domain");
+                    return rc;
+                }
+            }
+            else
+            {
+                printf("No space to get reserved dev memory maps!\n");
+                return rc;
+            }

tools/firmware/hvmloader/util.c int hpet_exists(unsigned long hpet_base) · line 895

Quoted 10 lines

tools/firmware/hvmloader/util.c +10 -0

+        }
+        else if ( rc )
+        {
+            printf("Could not get reserved dev memory info on domain");
+            return rc;
+        }
+    }
+
+    return nr_entries;
+}

I continue to think that adding these functions without user isn't really worthwhile in a separate patch.

tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245

Quoted 8 lines

tools/firmware/hvmloader/util.h +2 -0

@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                      unsigned int bios_image_base);
 void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */

This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having the Makefile symlink the hypervisor header here.

tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 250

Quoted 5 lines

tools/firmware/hvmloader/util.h +4 -0

+struct xen_mem_reserved_device_memory *rdm_map;
+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries);
+int hvm_get_reserved_device_memory_map(void);
 #ifndef NDEBUG

Blank line missing after your additions.

Jan

Chen, Tiejun

author

2014-10-27 03:12 UTC #19 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 7 lines · Jan Beulich

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.c +0 -0
@@ -828,6 +828,72 @@ int hpet_exists(unsigned long hpet_base)
      return ((hpet_id >> 16) == 0x8086);
  }
 

Quoted 14 lines

tools/firmware/hvmloader/util.c int hpet_exists(unsigned long hpet_base) · line 842

Quoted 12 lines

tools/firmware/hvmloader/util.c +12 -0

+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries)
+{
+    int rc;
+    struct xen_mem_reserved_device_memory_map memmap = {
+        .nr_entries = *max_entries
+    };
+
+    set_xen_guest_handle(memmap.buffer, entries);
+
+    rc = hypercall_memory_op(XENMEM_reserved_device_memory_map, &memmap);
+    if (rc == -ENOBUFS)

Coding style.

    if ( rc == -ENOBUFS )

Quoted 49 lines

tools/firmware/hvmloader/util.c +48 -0

+        *max_entries = memmap.nr_entries;
+
+    return rc;
+}
+
+/* Getting all reserved device memory map info in case of hvmloader. */
+int hvm_get_reserved_device_memory_map(void)
+{
+    static uint32_t nr_entries = 0;
+    int rc = 0;
+
+    if ( !rdm_map )
+    {
+        /* Assume we have one entry if not enough we'll expand.*/
+        nr_entries = 1;
+        rdm_map = mem_alloc(nr_entries *
+                            sizeof(struct xen_mem_reserved_device_memory), 0);
+        if ( !rdm_map )
+        {
+            printf("No space to get reserved dev memory maps!\n");
+            return rc;
+        }
+
+        rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+        if ( rc == -ENOBUFS )
+        {
+            rdm_map = mem_alloc(nr_entries *
+                                sizeof(struct xen_mem_reserved_device_memory),
+                                0);
+            if ( rdm_map )
+            {
+                rc = get_reserved_device_memory_map(rdm_map, &nr_entries);
+                if ( rc )
+                {
+                    printf("Could not get reserved dev memory info on domain");
+                    return rc;
+                }
+            }
+            else
+            {
+                printf("No space to get reserved dev memory maps!\n");
+                return rc;
+            }
+        }
+        else if ( rc )
+        {
+            printf("Could not get reserved dev memory info on domain");
+            return rc;

Quoted 8 lines

tools/firmware/hvmloader/util.c int hpet_exists(unsigned long hpet_base) · line 895
Quoted 5 lines
tools/firmware/hvmloader/util.c +5 -0
+        }
+    }
+
+    return nr_entries;
+}
I continue to think that adding these functions without user isn't really worthwhile in a separate patch.

These functions will be called in pci.c:pci_setup() and e820.c:build_e820_table(). If you really like one big patch I can squash these three patches as one.

Quoted 7 lines

tools/firmware/hvmloader/util.h +0 -0

@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                       unsigned int bios_image_base);
  void dump_e820_table(struct e820entry *e820, unsigned int nr);

Quoted 5 lines

tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely

If without this line, #include <xen/memory.h>,

In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],

make[8]: *** [build.o] Error 1

Quoted 3 lines

misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having

Seems we just need to include this,

#include <errno.h>

Quoted 10 lines

the Makefile symlink the hypervisor header here.
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 250
Quoted 5 lines
tools/firmware/hvmloader/util.h +4 -0
+struct xen_mem_reserved_device_memory *rdm_map;
+int get_reserved_device_memory_map(struct xen_mem_reserved_device_memory entries[],
+                                   uint32_t *max_entries);
+int hvm_get_reserved_device_memory_map(void);
  #ifndef NDEBUG
Blank line missing after your additions.

Added.

Thanks
Tiejun

Jan Beulich

2014-10-27 09:45 UTC #20 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 15 lines

Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                       unsigned int bios_image_base);
  void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely

Quoted 8 lines

If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                       ^
make[8]: *** [build.o] Error 1

So just forward declare the structure ahead of the function declaration.

Quoted 7 lines

Quoted 3 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having
Seems we just need to include this,
#include <errno.h>

You shouldn't include system headers here - what if the build system's -E... values differ from Xen's? Please remember that what your making changes to is not arbitrary application code.

Jan

Chen, Tiejun

author

2014-10-28 05:21 UTC #21 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 24 lines · Jan Beulich

Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                        unsigned int bios_image_base);
   void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                        ^
make[8]: *** [build.o] Error 1

Quoted 2 lines

So just forward declare the structure ahead of the function declaration.

tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>

So any reason I can't do such a same thing?

Quoted 11 lines

Quoted 8 lines
Quoted 4 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having
Seems we just need to include this,
#include <errno.h>
You shouldn't include system headers here - what if the build system's -E... values differ from Xen's? Please remember that what your making

tools/firmware/hvmloader/xenbus.c:30:#include <errno.h>

And why will Xen define this different?

Thanks
Tiejun

Quoted 6 lines

changes to is not arbitrary application code.
Jan

Jan Beulich

2014-10-28 09:48 UTC #22 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 26 lines

Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                        unsigned int bios_image_base);
   void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                        ^
make[8]: *** [build.o] Error 1

Quoted 7 lines

Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>
So any reason I can't do such a same thing?

You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
Seems we just need to include this,
#include <errno.h>
You shouldn't include system headers here - what if the build system's -E... values differ from Xen's? Please remember that what your making
tools/firmware/hvmloader/xenbus.c:30:#include <errno.h>

This is a completely different case: For one, no-one really looks at the error codes generated here. And even if someone would, these would be error value purely internal to hvmloader. Whereas in your case you want to interpret a value you get back from the hypervisor.

Quoted 1 line

And why will Xen define this different?

Why would Linux, *BSD, Solaris, and whatever else OS usable as a build host for building Xen all be required to use exactly the same -E... definitions when already Linux has variations for some of them depending on host architecture (i.e. when doing cross builds you'd risk running into problems even on Linux)?

Once again - please always keep in mind that you're modifying hypervisor code, not some simple user mode application.

Jan

Chen, Tiejun

author

2014-10-29 06:54 UTC #23 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 32 lines · Jan Beulich

Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                         unsigned int bios_image_base);
    void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                         ^
make[8]: *** [build.o] Error 1
Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>

Quoted 7 lines

Quoted 1 line
So any reason I can't do such a same thing?
You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).

Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having

So is the following is a way "via having the Makefile symlink the hypervisor header here."?

tools/include/Makefile +0 -0

@@ -17,6 +17,7 @@ xen/.dir:
        ln -sf ../xen-sys/$(XEN_OS) xen/sys
        ln -sf $(addprefix $(XEN_ROOT)/xen/include/xen/,libelf.h

elfstructs.h) xen/libelf/
ln -s ../xen-foreign xen/foreign
+ ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen
touch $@

.PHONY: install

Then we just need include this in util.c:

tools/firmware/hvmloader/util.c +1 -0

@@ -26,6 +26,7 @@
 #include <xen/xen.h>
 #include <xen/memory.h>
 #include <xen/sched.h>
+#include <xen/errno.h>
 
 void wrmsr(uint32_t idx, uint64_t v)
 {

Thanks
Tiejun

Quoted 30 lines

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Seems we just need to include this,
#include <errno.h>
You shouldn't include system headers here - what if the build system's -E... values differ from Xen's? Please remember that what your making
tools/firmware/hvmloader/xenbus.c:30:#include <errno.h>
This is a completely different case: For one, no-one really looks at the error codes generated here. And even if someone would, these would be error value purely internal to hvmloader. Whereas in your case you want to interpret a value you get back from the hypervisor.
Quoted 1 line
And why will Xen define this different?
Why would Linux, *BSD, Solaris, and whatever else OS usable as a build host for building Xen all be required to use exactly the same -E... definitions when already Linux has variations for some of them depending on host architecture (i.e. when doing cross builds you'd risk running into problems even on Linux)?
Once again - please always keep in mind that you're modifying hypervisor code, not some simple user mode application.
Jan

Jan Beulich

2014-10-29 09:05 UTC #24 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 36 lines

Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                         unsigned int bios_image_base);
    void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                         ^
make[8]: *** [build.o] Error 1
Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>
So any reason I can't do such a same thing?

Quoted 8 lines

Quoted 5 lines
You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.

Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.

Quoted 10 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having
So is the following is a way "via having the Makefile symlink the hypervisor header here."?

tools/include/Makefile xen/.dir: · line 20

Quoted 4 lines

tools/include/Makefile +1 -0

@@ -17,6 +17,7 @@ xen/.dir:
         ln -sf ../xen-sys/$(XEN_OS) xen/sys
         ln -sf $(addprefix $(XEN_ROOT)/xen/include/xen/,libelf.h elfstructs.h) xen/libelf/
         ln -s ../xen-foreign xen/foreign
+       ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen

Along those lines at least. Consult with the tools maintainers as to whether this would better go into a special subdirectory (and perhaps not even under tools/include, but somewhere beneath tools/firmware/hvmloader/, since it's not meant to be used by anyone else - as one of the ones de facto looking after hvmloader irrespective of what ./MAINTAINERS says, I'd strongly recommend limiting the visibility scope of this header as much as possible).

tools/firmware/hvmloader/util.c line 29

Quoted 9 lines

Then we just need include this in util.c:
tools/firmware/hvmloader/util.c +1 -0
@@ -26,6 +26,7 @@
  #include <xen/xen.h>
  #include <xen/memory.h>
  #include <xen/sched.h>
+#include <xen/errno.h>

Exactly.

Jan

Chen, Tiejun

author

2014-10-30 05:55 UTC #25 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 45 lines · Jan Beulich

Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                          unsigned int bios_image_base);
     void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                          ^
make[8]: *** [build.o] Error 1
Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>
So any reason I can't do such a same thing?
Quoted 8 lines
Quoted 5 lines
You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.

Quoted 11 lines

Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.

Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,

#include "util.h"
...
#include <xen/memory.h>

We can't redefine struct xen_reserved_device_memory in util.h.

Quoted 8 lines

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
misplaced here. While I generally wouldn't recommend doing this, I think in the case here including the hypervisor header that defines them would be okay. Perhaps not via relative path, but via having
So is the following is a way "via having the Makefile symlink the hypervisor header here."?

Quoted 12 lines

tools/include/Makefile xen/.dir: · line 20
Quoted 7 lines
tools/include/Makefile +1 -0
@@ -17,6 +17,7 @@ xen/.dir:
          ln -sf ../xen-sys/$(XEN_OS) xen/sys
          ln -sf $(addprefix $(XEN_ROOT)/xen/include/xen/,libelf.h elfstructs.h) xen/libelf/
          ln -s ../xen-foreign xen/foreign
+       ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen
Along those lines at least. Consult with the tools maintainers as to whether this would better go into a special subdirectory (and perhaps not even under tools/include, but somewhere beneath tools/firmware/hvmloader/, since it's not meant to be used by

Agree with you.

Quoted 3 lines

anyone else - as one of the ones de facto looking after hvmloader irrespective of what ./MAINTAINERS says, I'd strongly recommend limiting the visibility scope of this header as much as possible).

So this time this email is Toed the following maintainers

Ian Jackson <[email protected]> Stefano Stabellini <[email protected]> Ian Campbell <[email protected]>
Wei Liu <[email protected]>

So all tools maintainers,

Could you give us such a comment?

'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen' versus
'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h $(XEN_ROOT)/tools/firmware/hvmloader/,

Thanks
Tiejun

Quoted 16 lines

tools/firmware/hvmloader/util.c line 29
Quoted 10 lines
Then we just need include this in util.c:
tools/firmware/hvmloader/util.c +1 -0
@@ -26,6 +26,7 @@
   #include <xen/xen.h>
   #include <xen/memory.h>
   #include <xen/sched.h>
+#include <xen/errno.h>
Exactly.
Jan

Jan Beulich

2014-10-30 09:13 UTC #26 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 59 lines

Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 45 lines · Jan Beulich
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                          unsigned int bios_image_base);
     void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                          ^
make[8]: *** [build.o] Error 1
Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>
So any reason I can't do such a same thing?
Quoted 8 lines
Quoted 5 lines
You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.

Quoted 8 lines

Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in util.h.

Redefine? I said forward declare.

Quoted 7 lines

So all tools maintainers,
Could you give us such a comment?
'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen' versus 'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h $(XEN_ROOT)/tools/firmware/hvmloader/,

I think if it is (as suggested) to be limited to hvmloader, the symlinking also should be done in its Makefile rather than along with other tools stuff.

Jan

Chen, Tiejun

author

2014-10-31 02:20 UTC #27 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 62 lines · Jan Beulich

Quoted 62 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 45 lines · Jan Beulich
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 06:21, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 04:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                           unsigned int bios_image_base);
      void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
Quoted 5 lines
tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 245
Quoted 2 lines
tools/firmware/hvmloader/util.h +2 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
This is a joke I hope? The #include belongs at the top (albeit afaict you don't really need it here), and the #define is completely
Quoted 8 lines
If without this line, #include <xen/memory.h>,
In file included from build.c:25:0:
../util.h:246:70: error: array type has incomplete element type
int get_reserved_device_memory_map(struct xen_reserved_device_memory
entries[],
                                                                           ^
make[8]: *** [build.o] Error 1
Quoted 2 lines
So just forward declare the structure ahead of the function declaration.
tools/firmware/hvmloader/pci.c:28:#include <xen/memory.h> tools/firmware/hvmloader/ovmf.c:36:#include <xen/memory.h>
So any reason I can't do such a same thing?
Quoted 8 lines
Quoted 5 lines
You can, but it's undesirable. You're wanting this in a header, i.e. you'll make everyone consuming that header also implicitly depend on the new header you would include. We shouldn't pointlessly add build dependencies (and we should really try to reduce them where possible).
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,

Quoted 7 lines

Quoted 5 lines
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.

Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,

unsigned int hvm_get_reserved_device_memory_map(void);

since this is enough to pci_setup() and construct_rdm_e820_maps().

tools/firmware/hvmloader/util.c:

struct xen_reserved_device_memory *rdm_map;
static int
get_reserved_device_memory_map(struct xen_reserved_device_memory entries[], uint32_t *max_entries)

{
	...
}
unsigned int hvm_get_reserved_device_memory_map(void)
{
	...
}

Quoted 5 lines

Quoted 5 lines
So all tools maintainers,
Could you give us such a comment?

Quoted 7 lines

Quoted 3 lines
'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h xen' versus 'ln -sf $(XEN_ROOT)/xen/include/xen/errno.h $(XEN_ROOT)/tools/firmware/hvmloader/,
I think if it is (as suggested) to be limited to hvmloader, the symlinking also should be done in its Makefile rather than along with other tools stuff.

You're right.

I just send out a separate patch to ask a review from tools side, and CCed you.

Thanks
Tiejun

Jan Beulich

2014-10-31 08:14 UTC #28 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 27 lines

Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>

Quoted 8 lines

Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);

To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is generally possible.

Jan

Chen, Tiejun

author

2014-11-03 02:22 UTC #29 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 31 lines · Jan Beulich

Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 18 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Quoted 8 lines
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.

Quoted 8 lines

Quoted 4 lines
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is

Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.

Quoted 1 line

generally possible.

The following is what I did:

+struct xen_reserved_device_memory *rdm_map;
+static int
+get_reserved_device_memory_map(struct xen_reserved_device_memory entries[],
+ uint32_t *max_entries)
+{
+ int rc;
+ struct xen_reserved_device_memory_map xrdmmap = {
+ .nr_entries = *max_entries
+ };
+
+ set_xen_guest_handle(xrdmmap.buffer, entries);
+
+ rc = hypercall_memory_op(XENMEM_reserved_device_memory_map, &xrdmmap);
+ if ( rc == -ENOBUFS )
+ *max_entries = xrdmmap.nr_entries;
+
+ return rc;
+}
+
+/*
+ * Getting all reserved device memory map info in case of hvmloader.
+ * We just return zero for any failed cases, and this means we
+ * can't further handle any reserved device memory.
+ */
+unsigned int hvm_get_reserved_device_memory_map(void)
+{
+ ...
+}

So if you think they're not good, just please define these prototypes then I can finish them.

Thanks
Tiejun

Jan Beulich

2014-11-03 08:53 UTC #30 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 36 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
Quoted 7 lines
Quoted 5 lines
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.

This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.

Jan

Chen, Tiejun

author

2014-11-03 09:32 UTC #31 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 42 lines · Jan Beulich

Quoted 42 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 39 lines · Jan Beulich
Quoted 27 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is

Quoted 8 lines

Quoted 2 lines
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.

Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some duplicated codes, unless you mean I should define that as static inside in local.

Thanks
Tiejun

Jan Beulich

2014-11-03 09:45 UTC #32 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 53 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
Quoted 42 lines · Jan Beulich
Quoted 42 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 39 lines · Jan Beulich
Quoted 27 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Quoted 8 lines
Quoted 2 lines
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.

Quoted 3 lines

Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some duplicated codes, unless you mean I should define that as static inside in local.

Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?

Jan

Chen, Tiejun

author

2014-11-03 09:55 UTC #33 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 53 lines · Jan Beulich

Quoted 53 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
Quoted 50 lines · Jan Beulich
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
Quoted 7 lines
Quoted 5 lines
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.

Quoted 6 lines

Quoted 3 lines
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some duplicated codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?

Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.

Thanks
Tiejun

Jan Beulich

2014-11-03 10:02 UTC #34 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 59 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 57 lines · Jan Beulich
Quoted 56 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
Quoted 42 lines · Jan Beulich
Quoted 42 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 39 lines · Jan Beulich
Quoted 27 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Quoted 8 lines
Quoted 2 lines
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some duplicated codes, unless you mean I should define that as static inside in local.

Quoted 7 lines

Quoted 2 lines
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.

Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.

Jan

Chen, Tiejun

author

2014-11-21 06:26 UTC #35 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 62 lines · Jan Beulich

Quoted 62 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 59 lines · Jan Beulich
Quoted 53 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
Quoted 42 lines · Jan Beulich
Quoted 42 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 39 lines · Jan Beulich
Quoted 27 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern' to them when we really need them.
Quoted 11 lines
Please stop thinking this way. Declarations for things defined in .c files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are and remain in sync). I hate having to again repeat my remark that you shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the hypervisor (and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't also apply to application code. It's just that in the hypervisor and kernel (and certain other code system components) the consequences of being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare hvm_get_reserved_device_memory_map() in the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Quoted 8 lines
Quoted 2 lines
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of hvmloader.
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Quoted 3 lines
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some duplicated codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?

Quoted 8 lines

Quoted 4 lines
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.
Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.

Just go back here since I realize we always use mem_alloc(), which is pick from RESERVED_MEMORY, to allocate all buffer inside this hypercall caller in hvmloader, but unfortunately we have no any associated free function implementation in hvmloader, so if we call this multiple times this means it really waster more memory in RESERVED_MEMORY. So I still think one global variable should be fine.

Thanks
Tiejun

Tian, Kevin

2014-11-21 07:43 UTC #36 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 86 lines

From: Chen, Tiejun
Sent: Friday, November 21, 2014 2:26 PM
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
On 2014/11/3 16:53, Jan Beulich wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern'
to them
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
when we really need them.
Please stop thinking this way. Declarations for things defined
in .c
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
files are to be present in headers, and the defining .c file has to include that header (making sure declaration and definition are
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
remain in sync). I hate having to again repeat my remark that
you
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
shouldn't forget it's not application code that you're modifying. Robust and maintainable code are a requirement in the
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(and, as said it being an extension of it, hvmloader). Which - just to avoid any misunderstanding - isn't to say that this shouldn't
also
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
apply to application code. It's just that in the hypervisor and
kernel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(and certain other code system components) the consequences
of
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare
hvm_get_reserved_device_memory_map() in
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of
hvmloader.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some
duplicated
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?
Quoted 8 lines
Quoted 4 lines
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.
Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.

Quoted 7 lines

Just go back here since I realize we always use mem_alloc(), which is pick from RESERVED_MEMORY, to allocate all buffer inside this hypercall caller in hvmloader, but unfortunately we have no any associated free function implementation in hvmloader, so if we call this multiple times this means it really waster more memory in RESERVED_MEMORY. So I still think one global variable should be fine.

it's natural to get reserved information once, and then saved for either one-time or multiple-time references.

Thanks
Kevin

Jan Beulich

2014-11-21 07:54 UTC #37 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 94 lines

Quoted 87 lines
Quoted 1 line · On 21.11.14 at 08:43, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, November 21, 2014 2:26 PM
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
On 2014/11/3 16:53, Jan Beulich wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
On 30.10.14 at 06:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern'
to them
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
when we really need them.
Please stop thinking this way. Declarations for things defined
in .c
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
files are to be present in headers, and the defining .c file has to
include that header (making sure declaration and definition are
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
remain in sync). I hate having to again repeat my remark that
you
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
shouldn't forget it's not application code that you're modifying.
Robust and maintainable code are a requirement in the
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(and, as said it being an extension of it, hvmloader). Which - just
to avoid any misunderstanding - isn't to say that this shouldn't
also
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
apply to application code. It's just that in the hypervisor and
kernel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(and certain other code system components) the consequences
of
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and '<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare
hvm_get_reserved_device_memory_map() in
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of
hvmloader.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some
duplicated
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?
Quoted 8 lines
Quoted 4 lines
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.
Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.
Quoted 6 lines
Just go back here since I realize we always use mem_alloc(), which is pick from RESERVED_MEMORY, to allocate all buffer inside this hypercall caller in hvmloader, but unfortunately we have no any associated free function implementation in hvmloader, so if we call this multiple times this means it really waster more memory in RESERVED_MEMORY. So I still think one global variable should be fine.

Quoted 2 lines

it's natural to get reserved information once, and then saved for either one-time or multiple-time references.

Not really natural, but possible. Yet if done this way, then the interface shouldn't give the appearance of retrieving it every time, i.e. the global should be set up separately and the users of the data should access the data rather than calling a (fake) function.

Jan

Tian, Kevin

2014-11-21 08:01 UTC #38 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 107 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Friday, November 21, 2014 3:54 PM
Quoted 9 lines
Quoted 9 lines
Quoted 1 line · On 21.11.14 at 08:43, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, November 21, 2014 2:26 PM
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
On 2014/11/3 16:53, Jan Beulich wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
On 30.10.14 at 06:55, <[email protected]> wrote:
On 2014/10/29 17:05, Jan Beulich wrote:
On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add
'extern'
Quoted 6 lines
Quoted 6 lines
to them
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
when we really need them.
Please stop thinking this way. Declarations for things defined
in .c
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
files are to be present in headers, and the defining .c file has
to
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include that header (making sure declaration and definition
are
Quoted 2 lines
Quoted 2 lines
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
remain in sync). I hate having to again repeat my remark
that
Quoted 2 lines
Quoted 2 lines
you
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
shouldn't forget it's not application code that you're
modifying.
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Robust and maintainable code are a requirement in the
hypervisor
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(and, as said it being an extension of it, hvmloader). Which -
just
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
to avoid any misunderstanding - isn't to say that this shouldn't
also
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
apply to application code. It's just that in the hypervisor and
kernel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(and certain other code system components) the
consequences
Quoted 11 lines
Quoted 11 lines
of
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and
'<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
We can't redefine struct xen_reserved_device_memory in
util.h.
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Redefine? I said forward declare.
Seems we just need to declare
hvm_get_reserved_device_memory_map() in
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the head file, tools/firmware/hvmloader/util.h,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to
get
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
RDM without more hypercall or function call in the context of
hvmloader.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This argument which you brought up before, and which we
commented
Quoted 29 lines
Quoted 28 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some
duplicated
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?
Quoted 8 lines
Quoted 4 lines
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.
Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.
Just go back here since I realize we always use mem_alloc(), which is pick from RESERVED_MEMORY, to allocate all buffer inside this hypercall caller in hvmloader, but unfortunately we have no any associated free function implementation in hvmloader, so if we call this multiple times this means it really waster more memory in RESERVED_MEMORY. So I still think one global variable should be fine.

Quoted 8 lines

Quoted 2 lines
it's natural to get reserved information once, and then saved for either one-time or multiple-time references.
Not really natural, but possible. Yet if done this way, then the interface shouldn't give the appearance of retrieving it every time, i.e. the global should be set up separately and the users of the data should access the data rather than calling a (fake) function.

that's what I meant.

Thanks
Kevin

Chen, Tiejun

author

2014-11-21 08:54 UTC #39 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

In reply to Jan Beulich

Quoted 94 lines · Jan Beulich

Quoted 94 lines
Quoted 93 lines
Quoted 1 line · On 21.11.14 at 08:43, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, November 21, 2014 2:26 PM
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:55, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:32, <[email protected]> wrote:
On 2014/11/3 16:53, Jan Beulich wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 03:22, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:20, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
On 30.10.14 at 06:55, <[email protected]> wrote:
On 2014/10/29 17:05, Jan Beulich wrote:
On 29.10.14 at 07:54, <[email protected]> wrote:
Looks I can remove those stuff from util.h and just add 'extern'
to them
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
when we really need them.
Please stop thinking this way. Declarations for things defined
in .c
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
files are to be present in headers, and the defining .c file has to
include that header (making sure declaration and definition are
and
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
remain in sync). I hate having to again repeat my remark that
you
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
shouldn't forget it's not application code that you're modifying.
Robust and maintainable code are a requirement in the
hypervisor
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(and, as said it being an extension of it, hvmloader). Which - just
to avoid any misunderstanding - isn't to say that this shouldn't
also
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
apply to application code. It's just that in the hypervisor and
kernel
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(and certain other code system components) the consequences
of
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
being lax are much more severe.
Okay. But currently, the pci.c file already include 'util.h' and
'<xen/memory.h>,
#include "util.h"
...
#include <xen/memory.h>
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
We can't redefine struct xen_reserved_device_memory in util.h.
Redefine? I said forward declare.
Seems we just need to declare
hvm_get_reserved_device_memory_map() in
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
the head file, tools/firmware/hvmloader/util.h,
unsigned int hvm_get_reserved_device_memory_map(void);
To me this looks very much like poor programming style, even if in the context of hvmloader communicating information via global variables rather than function arguments and return values is
Do you mean you don't like a global variable? But it can be use to get RDM without more hypercall or function call in the context of
hvmloader.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
This argument which you brought up before, and which we commented on before, is pretty pointless. We don't really care much about doing one or two more hypercalls from hvmloader, unless these would be long-running ones.
Another benefit to use a global variable is that we wouldn't allocate xen_reserved_device_memory * N each time, and reduce some
duplicated
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
codes, unless you mean I should define that as static inside in local.
Now this reason is indeed worth a consideration. How many times is the data being needed/retrieved?
Quoted 8 lines
Quoted 4 lines
Currently there are two cases in tools/hvmloader, setup pci and build e820 table. Each time, as you know we don't know how may entries we should require, so we always allocate one instance then according to the return value to allocate the proper instances to get that.
Hmm, two uses isn't really that bad, i.e. I'd then still be in favor of a more "normal" interface.
Just go back here since I realize we always use mem_alloc(), which is pick from RESERVED_MEMORY, to allocate all buffer inside this hypercall caller in hvmloader, but unfortunately we have no any associated free function implementation in hvmloader, so if we call this multiple times this means it really waster more memory in RESERVED_MEMORY. So I still think one global variable should be fine.

Quoted 6 lines

Quoted 2 lines
it's natural to get reserved information once, and then saved for either one-time or multiple-time references.
Not really natural, but possible. Yet if done this way, then the interface shouldn't give the appearance of retrieving it every time, i.e. the global should be set up separately and the users of the

Shouldn't we exactly implemented this previously?

+struct xen_mem_reserved_device_memory *rdm_map;

As a global variable, any caller should check if this is !NULL before they call that function.

Thanks
Tiejun

Quoted 5 lines

data should access the data rather than calling a (fake) function.
Jan

Jan Beulich

2014-11-21 09:33 UTC #40 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 11 lines

Quoted 1 line
Quoted 1 line · On 21.11.14 at 09:54, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 21.11.14 at 08:43, <[email protected]> wrote:
it's natural to get reserved information once, and then saved for either one-time or multiple-time references.
Not really natural, but possible. Yet if done this way, then the interface shouldn't give the appearance of retrieving it every time, i.e. the global should be set up separately and the users of the
Shouldn't we exactly implemented this previously?

Not afair (the mention of how it should not be done above was specifically targeting what I recall you did so far).

Quoted 2 lines

As a global variable, any caller should check if this is !NULL before they call that function.

Of course.

Jan

Jan Beulich

2014-10-24 14:27 UTC #41 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

In reply to Tiejun Chen

tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 246

Quoted 10 lines

Quoted 1 line

Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:

tools/firmware/hvmloader/util.h +3 -0

@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                      unsigned int bios_image_base);
 void dump_e820_table(struct e820entry *e820, unsigned int nr);
 
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
+struct xen_mem_reserved_device_memory *rdm_map;

Oh, and - without "extern" this creates an instance in each translation unit (and would cause build failures the moment someone passed -fno-common to the compiler).

Jan

Chen, Tiejun

author

2014-10-27 05:07 UTC #42 lore raw

Re: [v7][RFC][PATCH 04/13] hvmloader/util: get reserved device memory maps

Quoted 7 lines · Jan Beulich

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/util.h +0 -0
@@ -241,6 +241,12 @@ int build_e820_table(struct e820entry *e820,
                       unsigned int bios_image_base);
  void dump_e820_table(struct e820entry *e820, unsigned int nr);
 

Quoted 8 lines

tools/firmware/hvmloader/util.h int build_e820_table(struct e820entry *e820, · line 246
Quoted 3 lines
tools/firmware/hvmloader/util.h +3 -0
+#include <xen/memory.h>
+#define ENOBUFS     105 /* No buffer space available */
+struct xen_mem_reserved_device_memory *rdm_map;
Oh, and - without "extern" this creates an instance in each translation unit (and would cause build failures the moment someone passed -fno-common to the compiler).

So instead, just define this in util.c then use this with 'extern'.

Thanks
Tiejun

Tiejun Chen

author

2014-10-24 07:34 UTC #43 lore raw

RFC PATCH v7 5/13 hvmloader/mmio: reconcile guest mmio with reserved device memory

In reply to Tiejun Chen

We need to make sure all mmio allocation don't overlap any rdm, reserved device memory. Here we just skip all reserved device memory range in mmio space.

Signed-off-by Tiejun Chen <[email protected]>

---
 tools/firmware/hvmloader/pci.c | 68 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 67 insertions(+), 1 deletion(-)

tools/firmware/hvmloader/pci.c +67 -1

index 3712988..e26481c 100644
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
 enum virtual_vga virtual_vga = VGA_none;
 unsigned long igd_opregion_pgbase = 0;
 
+unsigned int need_skip_rmrr = 0;
+
+/*
+ * Check whether there exists mmio hole in the specified memory range.
+ * Returns 1 if exists, else returns 0.
+ */
+static int check_mmio_hole_confliction(uint64_t start, uint64_t memsize,
+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;
+}
+
+static int check_reserved_device_memory_map(uint64_t mmio_base,
+                                            uint64_t mmio_max)
+{
+    uint32_t i = 0;
+    uint64_t rdm_start, rdm_end;
+    int nr_entries = -1;
+
+    nr_entries = hvm_get_reserved_device_memory_map();
+
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
+        if ( check_mmio_hole_confliction(rdm_start, (rdm_end - rdm_start),
+                                         mmio_base, mmio_max - mmio_base) )
+        {
+            need_skip_rmrr++;
+        }
+    }
+
+    return nr_entries;
+}
+
 void pci_setup(void)
 {
     uint8_t is_64bar, using_64bar, bar64_relocate = 0;

@@ -58,7 +96,9 @@ void pci_setup(void)
         uint32_t bar_reg;
         uint64_t bar_sz;
     } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
+    uint64_t rdm_start, rdm_end;
 
     const char *s;
     /*

@@ -309,6 +349,14 @@ void pci_setup(void)
     io_resource.base = 0xc000;
     io_resource.max = 0x10000;
 
+    /* Check low mmio range. */
+    nr_entries = check_reserved_device_memory_map(mem_resource.base,
+                                                  mem_resource.max);
+    /* Check high mmio range. */
+    if ( nr_entries > 0 )
+        nr_entries = check_reserved_device_memory_map(high_mem_resource.base,
+                                                      high_mem_resource.max);
+
     /* Assign iomem and ioport resources in descending order of size. */
     for ( i = 0; i < nr_bars; i++ )
     {

@@ -363,11 +411,29 @@ void pci_setup(void)
             bar_data &= ~PCI_BASE_ADDRESS_IO_MASK;
         }
 
+ reallocate_mmio:
         base = (resource->base  + bar_sz - 1) & ~(uint64_t)(bar_sz - 1);
         bar_data |= (uint32_t)base;
         bar_data_upper = (uint32_t)(base >> 32);
         base += bar_sz;
 
+        if ( need_skip_rmrr )
+        {
+            for ( j = 0; j < nr_entries; j++ )
+            {
+                rdm_start = rdm_map[j].start_pfn << PAGE_SHIFT;
+                rdm_end = rdm_start + (rdm_map[j].nr_pages << PAGE_SHIFT);
+                if ( check_mmio_hole_confliction(rdm_start,
+                                                 (rdm_end - rdm_start),
+                                                 base, bar_sz) )
+                {
+                    resource->base = rdm_end;
+                    need_skip_rmrr--;
+                    goto reallocate_mmio;
+                }
+            }
+        }
+
         if ( (base < resource->base) || (base > resource->max) )
         {
             printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "

Signature · 1 line

1.9.1

Jan Beulich

2014-10-24 14:42 UTC #44 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40

Quoted 8 lines

Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +1 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
 enum virtual_vga virtual_vga = VGA_none;
 unsigned long igd_opregion_pgbase = 0;
 
+unsigned int need_skip_rmrr = 0;

Static (and without initializer)?

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 46

Quoted 6 lines

tools/firmware/hvmloader/pci.c +6 -0

+
+/*
+ * Check whether there exists mmio hole in the specified memory range.
+ * Returns 1 if exists, else returns 0.
+ */
+static int check_mmio_hole_confliction(uint64_t start, uint64_t memsize,

I don't think the word "confliction" exists. "conflict" please.

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 52

Quoted 6 lines

tools/firmware/hvmloader/pci.c +6 -0

+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;

Make this a simple single return statement?

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 58

Quoted 6 lines

tools/firmware/hvmloader/pci.c +6 -0

+}
+
+static int check_reserved_device_memory_map(uint64_t mmio_base,
+                                            uint64_t mmio_max)
+{
+    uint32_t i = 0;

Pointless initializer.

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 60

Quoted 2 lines

tools/firmware/hvmloader/pci.c +2 -0

+    uint64_t rdm_start, rdm_end;
+    int nr_entries = -1;

And again.

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 62

Quoted 2 lines

tools/firmware/hvmloader/pci.c +2 -0
+
+    nr_entries = hvm_get_reserved_device_memory_map();

It's completely unclear why this can't be the variable's initializer.

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67

Quoted 5 lines

tools/firmware/hvmloader/pci.c +5 -0

+
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);

I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 68

Quoted 1 line

tools/firmware/hvmloader/pci.c +1 -0
+        if ( check_mmio_hole_confliction(rdm_start, (rdm_end - rdm_start),

Pointless parentheses.

Quoted 13 lines

tools/firmware/hvmloader/pci.c +9 -0

+                                         mmio_base, mmio_max - mmio_base) )
+        {
+            need_skip_rmrr++;
+        }
+    }
+
+    return nr_entries;
+}
+
 void pci_setup(void)
 {
     uint8_t is_64bar, using_64bar, bar64_relocate = 0;

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100

Quoted 6 lines

tools/firmware/hvmloader/pci.c +2 -1

@@ -58,7 +96,9 @@ void pci_setup(void)
         uint32_t bar_reg;
         uint64_t bar_sz;
     } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;

And another pointless initializer. Plus as a count of something this surely wants to be "unsigned int". Also I guess the variable name is too generic - nr_rdm_entries perhaps?

Quoted 9 lines

tools/firmware/hvmloader/pci.c +1 -0

@@ -363,11 +411,29 @@ void pci_setup(void)
             bar_data &= ~PCI_BASE_ADDRESS_IO_MASK;
         }
 
+ reallocate_mmio:
         base = (resource->base  + bar_sz - 1) & ~(uint64_t)(bar_sz - 1);
         bar_data |= (uint32_t)base;
         bar_data_upper = (uint32_t)(base >> 32);
         base += bar_sz;

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 428

Quoted 10 lines

tools/firmware/hvmloader/pci.c +9 -0

 
+        if ( need_skip_rmrr )
+        {
+            for ( j = 0; j < nr_entries; j++ )
+            {
+                rdm_start = rdm_map[j].start_pfn << PAGE_SHIFT;
+                rdm_end = rdm_start + (rdm_map[j].nr_pages << PAGE_SHIFT);
+                if ( check_mmio_hole_confliction(rdm_start,
+                                                 (rdm_end - rdm_start),
+                                                 base, bar_sz) )

"base" was already updated by this point.

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 432

Quoted 4 lines

tools/firmware/hvmloader/pci.c +4 -0

+                {
+                    resource->base = rdm_end;
+                    need_skip_rmrr--;
+                    goto reallocate_mmio;

If you ever get here, the earlier determination of whether the MMIO hole is large enough may get invalidated. I.e. I'm afraid this approach is still to simplistic. Also to way you do the retry, the resulting bar_data and bar_data_upper will likely end up being garbage.

Did you actually _test_ this code?

Jan

Chen, Tiejun

author

2014-10-27 07:12 UTC #45 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 10 lines · Jan Beulich

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +1 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
  enum virtual_vga virtual_vga = VGA_none;
  unsigned long igd_opregion_pgbase = 0;
 
+unsigned int need_skip_rmrr = 0;
Static (and without initializer)?

static unsigned int need_skip_rmrr;

Quoted 9 lines

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 46
Quoted 7 lines
tools/firmware/hvmloader/pci.c +6 -0
+
+/*
+ * Check whether there exists mmio hole in the specified memory range.
+ * Returns 1 if exists, else returns 0.
+ */
+static int check_mmio_hole_confliction(uint64_t start, uint64_t memsize,
I don't think the word "confliction" exists. "conflict" please.

s/check_mmio_hole_confliction/check_mmio_hole_conflict/g

Quoted 9 lines

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 52
Quoted 7 lines
tools/firmware/hvmloader/pci.c +6 -0
+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;
Make this a simple single return statement?

static int check_mmio_hole_conflict(uint64_t start, uint64_t memsize,

                                    uint64_t mmio_start, uint64_t 
mmio_size)
{
    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
        return 0;

    return 1;
}

Quoted 13 lines

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 58
Quoted 7 lines
tools/firmware/hvmloader/pci.c +6 -0
+}
+
+static int check_reserved_device_memory_map(uint64_t mmio_base,
+                                            uint64_t mmio_max)
+{
+    uint32_t i = 0;
Pointless initializer.
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 60
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+    uint64_t rdm_start, rdm_end;
+    int nr_entries = -1;

Quoted 6 lines

And again.
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 62
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+
+    nr_entries = hvm_get_reserved_device_memory_map();
It's completely unclear why this can't be the variable's initializer.

    uint32_t i;
    uint64_t rdm_start, rdm_end;
    int nr_rdm_entries = hvm_get_reserved_device_memory_map();

Quoted 9 lines

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67
Quoted 6 lines
tools/firmware/hvmloader/pci.c +5 -0
+
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.

I tried to go back looking into something but just found you were saying I shouldn't use PAGE_SHIFT and PAGE_SIZE at the same time. If I'm still missing could you show me what you expect?

Quoted 4 lines

tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 68
Quoted 2 lines
tools/firmware/hvmloader/pci.c +1 -0
+        if ( check_mmio_hole_confliction(rdm_start, (rdm_end - rdm_start),
Pointless parentheses.

if ( check_mmio_hole_conflict(rdm_start, rdm_end - rdm_start,

Quoted 14 lines

tools/firmware/hvmloader/pci.c +9 -0

+                                         mmio_base, mmio_max - mmio_base) )
+        {
+            need_skip_rmrr++;
+        }
+    }
+
+    return nr_entries;
+}
+
  void pci_setup(void)
  {
      uint8_t is_64bar, using_64bar, bar64_relocate = 0;

@@ -58,7 +96,9 @@ void pci_setup(void)

Quoted 8 lines

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100
Quoted 6 lines
tools/firmware/hvmloader/pci.c +2 -1
          uint32_t bar_reg;
          uint64_t bar_sz;
      } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
And another pointless initializer. Plus as a count of something this

int nr_rdm_entries;

Quoted 1 line

surely wants to be "unsigned int". Also I guess the variable name

nr_rdm_entries should be literally unsigned int but this value always be set from hvm_get_reserved_device_memory_map(),

nr_rdm_entries = hvm_get_reserved_device_memory_map()

I hope that return value can be negative value in some failed case

Quoted 1 line

is too generic - nr_rdm_entries perhaps?

Okay, s/nr_entries/nr_rdm_entries/g

Quoted 11 lines

tools/firmware/hvmloader/pci.c +1 -0

@@ -363,11 +411,29 @@ void pci_setup(void)
              bar_data &= ~PCI_BASE_ADDRESS_IO_MASK;
          }
 
+ reallocate_mmio:
          base = (resource->base  + bar_sz - 1) & ~(uint64_t)(bar_sz - 1);
          bar_data |= (uint32_t)base;
          bar_data_upper = (uint32_t)(base >> 32);
          base += bar_sz;

Quoted 11 lines

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 428

Quoted 9 lines

tools/firmware/hvmloader/pci.c +9 -0

+        if ( need_skip_rmrr )
+        {
+            for ( j = 0; j < nr_entries; j++ )
+            {
+                rdm_start = rdm_map[j].start_pfn << PAGE_SHIFT;
+                rdm_end = rdm_start + (rdm_map[j].nr_pages << PAGE_SHIFT);
+                if ( check_mmio_hole_confliction(rdm_start,
+                                                 (rdm_end - rdm_start),
+                                                 base, bar_sz) )

"base" was already updated by this point.

Looks I should insert these code fragments between

bar_data_upper = (uint32_t)(base >> 32);

and

base += bar_sz;

So (See below)

Quoted 9 lines

tools/firmware/hvmloader/pci.c void pci_setup(void) · line 432
Quoted 5 lines
tools/firmware/hvmloader/pci.c +4 -0
+                {
+                    resource->base = rdm_end;
+                    need_skip_rmrr--;
+                    goto reallocate_mmio;
If you ever get here, the earlier determination of whether the MMIO hole is large enough may get invalidated. I.e. I'm afraid this approach is still to simplistic. Also to way you do the retry, the resulting bar_data

Here move 'reallocate_mmio' downward one line, and

s/resource->base = rdm_end;/base = (rdm_end + bar_sz - 1) & ~(uint64_t)(bar_sz - 1);

Then,

	...
        base = (resource->base  + bar_sz - 1) & ~(uint64_t)(bar_sz - 1);
 reallocate_mmio:
        bar_data |= (uint32_t)base;
        bar_data_upper = (uint32_t)(base >> 32);

        if ( need_skip_rmrr )
        {
            for ( j = 0; j < nr_rdm_entries; j++ )
            {
                rdm_start = rdm_map[j].start_pfn << PAGE_SHIFT;
                rdm_end = rdm_start + (rdm_map[j].nr_pages << PAGE_SHIFT);
                if ( check_mmio_hole_conflict(rdm_start, rdm_end - 
rdm_start,
                                              base, bar_sz) )
                {
                    base = (rdm_end  + bar_sz - 1) & ~(uint64_t)(bar_sz 
- 1);
                    need_skip_rmrr--;
                    goto reallocate_mmio;
                }
            }
        }

        base += bar_sz;

Additionally, actually there are some original codes just following my codes:

        if ( need_skip_rmrr )
        {
		...
        }

	base += bar_sz;

        if ( (base < resource->base) || (base > resource->max) )
        {
            printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                   "resource!\n", devfn>>3, devfn&7, bar_reg,
                   PRIllx_arg(bar_sz));
            continue;
        }

This can guarantee we don't overwhelm the previous mmio range.

Quoted 3 lines

and bar_data_upper will likely end up being garbage.
Did you actually _test_ this code?

Actually in my real case those RMRR ranges are always below MMIO.

Thanks
Tiejun

Jan Beulich

2014-10-27 09:56 UTC #46 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 9 lines

Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +0 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
  enum virtual_vga virtual_vga = VGA_none;
  unsigned long igd_opregion_pgbase = 0;
 

Quoted 5 lines

Quoted 3 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40
Quoted 1 line
tools/firmware/hvmloader/pci.c +1 -0
+unsigned int need_skip_rmrr = 0;
Static (and without initializer)?
static unsigned int need_skip_rmrr;

Please stop echoing back what was requested.

Quoted 10 lines

Quoted 8 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 52
Quoted 6 lines
tools/firmware/hvmloader/pci.c +6 -0
+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;
Make this a simple single return statement?
static int check_mmio_hole_conflict(uint64_t start, uint64_t memsize,

Quoted 8 lines

                                     uint64_t mmio_start, uint64_t 
mmio_size)
{
     if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
         return 0;

     return 1;
}

Is "a simple single return statement" ambiguous in any way? This

static bool check_mmio_hole_conflict(uint64_t start, uint64_t memsize,
uint64_t mmio_start, uint64_t mmio_size)

{
     return start + memsize > mmio_start && start < mmio_start + mmio_size;
}

is how I think this should be.

Quoted 14 lines

Quoted 14 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 58
Quoted 6 lines
tools/firmware/hvmloader/pci.c +6 -0
+}
+
+static int check_reserved_device_memory_map(uint64_t mmio_base,
+                                            uint64_t mmio_max)
+{
+    uint32_t i = 0;
Pointless initializer.
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 60
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+    uint64_t rdm_start, rdm_end;
+    int nr_entries = -1;
And again.

Quoted 8 lines

Quoted 4 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 56
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+
+    nr_entries = hvm_get_reserved_device_memory_map();
It's completely unclear why this can't be the variable's initializer.
     uint32_t i;
     uint64_t rdm_start, rdm_end;
     int nr_rdm_entries = hvm_get_reserved_device_memory_map();

And (see also below) "unsigned int". It's bogus anyway to have the function return the count by normal means by the actual array via a global variable. I think you ought to switch to a consistent model.

Quoted 11 lines

Quoted 7 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4 -0
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.
I tried to go back looking into something but just found you were saying I shouldn't use PAGE_SHIFT and PAGE_SIZE at the same time. If I'm still missing could you show me what you expect?

Shifting a 32-bit quantity left still yields a 32-bit quantity, no matter whether the result is then stored in a 64-bit variable. You need to up-cast the left side of the shift first.

Quoted 12 lines

Quoted 9 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100
Quoted 7 lines
tools/firmware/hvmloader/pci.c +2 -1
@@ -58,7 +96,9 @@ void pci_setup(void)
          uint32_t bar_reg;
          uint64_t bar_sz;
      } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
And another pointless initializer. Plus as a count of something this
int nr_rdm_entries;

Quoted 8 lines

Quoted 1 line
surely wants to be "unsigned int". Also I guess the variable name
nr_rdm_entries should be literally unsigned int but this value always be set from hvm_get_reserved_device_memory_map(),
nr_rdm_entries = hvm_get_reserved_device_memory_map()
I hope that return value can be negative value in some failed case

If only you checked for these negative values...

Quoted 18 lines

Additionally, actually there are some original codes just following my codes:

         if ( need_skip_rmrr )
         {
		...
         }

	base += bar_sz;

         if ( (base < resource->base) || (base > resource->max) )
         {
             printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                    "resource!\n", devfn>>3, devfn&7, bar_reg,
                    PRIllx_arg(bar_sz));
             continue;
         }

Quoted 1 line

This can guarantee we don't overwhelm the previous mmio range.

Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.

Quoted 5 lines

Quoted 3 lines
and bar_data_upper will likely end up being garbage.
Did you actually _test_ this code?
Actually in my real case those RMRR ranges are always below MMIO.

Below whose MMIO? The host's or the guest's? In the latter case, just (in order to test your code) increase the range reserved for MMIO enough to cover the RMRR range.

Jan

Chen, Tiejun

author

2014-10-28 07:11 UTC #47 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 9 lines · Jan Beulich

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +0 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
   enum virtual_vga virtual_vga = VGA_none;
   unsigned long igd_opregion_pgbase = 0;
 

Quoted 7 lines

Quoted 5 lines
Quoted 3 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40
Quoted 1 line
tools/firmware/hvmloader/pci.c +1 -0
+unsigned int need_skip_rmrr = 0;
Static (and without initializer)?
static unsigned int need_skip_rmrr;
Please stop echoing back what was requested.

I try to fix inline to make sure I'm addressing all comments inline properly. If you think this is correct please just ignore that.

Quoted 22 lines

Quoted 11 lines
Quoted 9 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 52
Quoted 7 lines
tools/firmware/hvmloader/pci.c +6 -0
+                           uint64_t mmio_start, uint64_t mmio_size)
+{
+    if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
+        return 0;
+    else
+        return 1;
Make this a simple single return statement?
static int check_mmio_hole_conflict(uint64_t start, uint64_t memsize,
Quoted 8 lines
                                      uint64_t mmio_start, uint64_t
mmio_size)
{
      if ( start + memsize <= mmio_start || start >= mmio_start + mmio_size )
          return 0;

      return 1;
}
Is "a simple single return statement" ambiguous in any way? This

Quoted 7 lines

static bool check_mmio_hole_conflict(uint64_t start, uint64_t memsize,
uint64_t mmio_start, uint64_t mmio_size)
{
      return start + memsize > mmio_start && start < mmio_start + mmio_size;
}
is how I think this should be.

Thanks for your show.

Quoted 36 lines

Quoted 15 lines
Quoted 15 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 58
Quoted 7 lines
tools/firmware/hvmloader/pci.c +6 -0
+}
+
+static int check_reserved_device_memory_map(uint64_t mmio_base,
+                                            uint64_t mmio_max)
+{
+    uint32_t i = 0;
Pointless initializer.
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 60
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+    uint64_t rdm_start, rdm_end;
+    int nr_entries = -1;
And again.
Quoted 8 lines
Quoted 4 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 56
Quoted 2 lines
tools/firmware/hvmloader/pci.c +2 -0
+
+    nr_entries = hvm_get_reserved_device_memory_map();
It's completely unclear why this can't be the variable's initializer.
      uint32_t i;
      uint64_t rdm_start, rdm_end;
      int nr_rdm_entries = hvm_get_reserved_device_memory_map();
And (see also below) "unsigned int". It's bogus anyway to have the function return the count by normal means by the actual array via a global variable. I think you ought to switch to a consistent model.
Quoted 8 lines
Quoted 7 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4 -0
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.

Quoted 7 lines

Quoted 3 lines
I tried to go back looking into something but just found you were saying I shouldn't use PAGE_SHIFT and PAGE_SIZE at the same time. If I'm still missing could you show me what you expect?
Shifting a 32-bit quantity left still yields a 32-bit quantity, no matter whether the result is then stored in a 64-bit variable. You need to up-cast the left side of the shift first.

Do you mean this?

rdm_start = (uint64_t)rdm_map[j].start_pfn << PAGE_SHIFT; rdm_end = rdm_start + ((uint64_t)rdm_map[j].nr_pages << PAGE_SHIFT);

Quoted 15 lines

Quoted 15 lines
Quoted 10 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100
Quoted 8 lines
tools/firmware/hvmloader/pci.c +2 -1
@@ -58,7 +96,9 @@ void pci_setup(void)
           uint32_t bar_reg;
           uint64_t bar_sz;
       } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
And another pointless initializer. Plus as a count of something this
int nr_rdm_entries;
Quoted 1 line
surely wants to be "unsigned int". Also I guess the variable name

Quoted 8 lines

Quoted 6 lines
nr_rdm_entries should be literally unsigned int but this value always be set from hvm_get_reserved_device_memory_map(),
nr_rdm_entries = hvm_get_reserved_device_memory_map()
I hope that return value can be negative value in some failed case
If only you checked for these negative values...

May I can simplify these failed cases handle with within hvm_get_reserved_device_memory_map() like,

	if ( rc )
		return 0;

Because actually we don't need any negative return value again. So '0' is always fine. So here,

unsigned long nr_rdm_entries = hvm_get_reserved_device_memory_map();

Quoted 19 lines

Additionally, actually there are some original codes just following my codes:

          if ( need_skip_rmrr )
          {
		...
          }

	base += bar_sz;

          if ( (base < resource->base) || (base > resource->max) )
          {
              printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                     "resource!\n", devfn>>3, devfn&7, bar_reg,
                     PRIllx_arg(bar_sz));
              continue;
          }

Quoted 4 lines

Quoted 1 line
This can guarantee we don't overwhelm the previous mmio range.
Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.

I don't understand what a side effect is. I just to try to make sure BAR space skip any conflict range but they are still in these resource ranges.

Quoted 10 lines

Quoted 6 lines
Quoted 4 lines
and bar_data_upper will likely end up being garbage.
Did you actually _test_ this code?
Actually in my real case those RMRR ranges are always below MMIO.
Below whose MMIO? The host's or the guest's? In the latter case, just (in order to test your code) increase the range reserved for MMIO enough to cover the RMRR range.

In my platform,

RMRR region: base_addr ab80a000 end_address ab81dfff RMRR region: base_addr ad000000 end_address af7fffff

So I guess you hope I change this

#define PCI_MEM_START 0xf0000000

#define PCI_MEM_START 0xa0000000

right?

But what test you want to see? Just boot?

Thanks
Tiejun

Jan Beulich

2014-10-28 09:56 UTC #48 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 15 lines

Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:11, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +1 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
   enum virtual_vga virtual_vga = VGA_none;
   unsigned long igd_opregion_pgbase = 0;
 
+unsigned int need_skip_rmrr = 0;
Static (and without initializer)?

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
static unsigned int need_skip_rmrr;
Please stop echoing back what was requested.
I try to fix inline to make sure I'm addressing all comments inline properly. If you think this is correct please just ignore that.

But it still takes people reading you reply time to read that. Please ask back only when you think an earlier reply was ambiguous.

Quoted 12 lines

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4 -0
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.
I tried to go back looking into something but just found you were saying I shouldn't use PAGE_SHIFT and PAGE_SIZE at the same time. If I'm still missing could you show me what you expect?

Quoted 8 lines

Quoted 3 lines
Shifting a 32-bit quantity left still yields a 32-bit quantity, no matter whether the result is then stored in a 64-bit variable. You need to up-cast the left side of the shift first.
Do you mean this?
rdm_start = (uint64_t)rdm_map[j].start_pfn << PAGE_SHIFT; rdm_end = rdm_start + ((uint64_t)rdm_map[j].nr_pages << PAGE_SHIFT);

Yes. Finally.

Quoted 25 lines

Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100
Quoted 7 lines
tools/firmware/hvmloader/pci.c +2 -1
@@ -58,7 +96,9 @@ void pci_setup(void)
           uint32_t bar_reg;
           uint64_t bar_sz;
       } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
And another pointless initializer. Plus as a count of something this
int nr_rdm_entries;
Quoted 1 line
surely wants to be "unsigned int". Also I guess the variable name
Quoted 8 lines
Quoted 6 lines
nr_rdm_entries should be literally unsigned int but this value always be set from hvm_get_reserved_device_memory_map(),
nr_rdm_entries = hvm_get_reserved_device_memory_map()
I hope that return value can be negative value in some failed case
If only you checked for these negative values...
May I can simplify these failed cases handle with within hvm_get_reserved_device_memory_map() like,

Quoted 7 lines

	if ( rc )
		return 0;
Because actually we don't need any negative return value again. So '0' is always fine. So here,
unsigned long nr_rdm_entries = hvm_get_reserved_device_memory_map();

Except that "unsigned int" would seem more consistent.

Quoted 18 lines

Additionally, actually there are some original codes just following my codes:

          if ( need_skip_rmrr )
          {
		...
          }

	base += bar_sz;

          if ( (base < resource->base) || (base > resource->max) )
          {
              printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                     "resource!\n", devfn>>3, devfn&7, bar_reg,
                     PRIllx_arg(bar_sz));
              continue;
          }

Quoted 7 lines

Quoted 4 lines
Quoted 1 line
This can guarantee we don't overwhelm the previous mmio range.
Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.
I don't understand what a side effect is. I just to try to make sure BAR space skip any conflict range but they are still in these resource ranges.

A side effect is an effect you don't primarily intend with your change (or more generally, with any particular operation). In the case here, a BAR that previously got a value assigned may not anymore with your change in place. An acceptable effect of your change would be if the value it gets assigned is now different, but not assigning a value at all is not acceptable.

Quoted 17 lines

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
and bar_data_upper will likely end up being garbage.
Did you actually _test_ this code?
Actually in my real case those RMRR ranges are always below MMIO.
Below whose MMIO? The host's or the guest's? In the latter case, just (in order to test your code) increase the range reserved for MMIO enough to cover the RMRR range.
In my platform,
RMRR region: base_addr ab80a000 end_address ab81dfff RMRR region: base_addr ad000000 end_address af7fffff
So I guess you hope I change this

Quoted 7 lines

#define PCI_MEM_START 0xf0000000
to
#define PCI_MEM_START 0xa0000000
right?

Almost. You'd have to use 0x80000000, or other logic would break.

Quoted 1 line

But what test you want to see? Just boot?

Yes, guest boot, but including proper inspection that what your code does is really how it should be.

Jan

Chen, Tiejun

author

2014-10-29 07:03 UTC #49 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 76 lines · Jan Beulich

Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:11, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 40
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
tools/firmware/hvmloader/pci.c +1 -0
@@ -37,6 +37,44 @@ uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0;
    enum virtual_vga virtual_vga = VGA_none;
    unsigned long igd_opregion_pgbase = 0;
 
+unsigned int need_skip_rmrr = 0;
Static (and without initializer)?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
static unsigned int need_skip_rmrr;
Please stop echoing back what was requested.
I try to fix inline to make sure I'm addressing all comments inline properly. If you think this is correct please just ignore that.
But it still takes people reading you reply time to read that. Please ask back only when you think an earlier reply was ambiguous.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 7 lines
tools/firmware/hvmloader/pci.c uint64_t pci_hi_mem_start = 0, pci_hi_mem_end = 0; · line 67
Quoted 4 lines
tools/firmware/hvmloader/pci.c +4 -0
+    for ( i = 0; i < nr_entries; i++ )
+    {
+        rdm_start = rdm_map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (rdm_map[i].nr_pages << PAGE_SHIFT);
I'm pretty certain I pointed out before that you can't simply shift these fields - you risk losing significant bits.
I tried to go back looking into something but just found you were saying I shouldn't use PAGE_SHIFT and PAGE_SIZE at the same time. If I'm still missing could you show me what you expect?
Quoted 8 lines
Quoted 3 lines
Shifting a 32-bit quantity left still yields a 32-bit quantity, no matter whether the result is then stored in a 64-bit variable. You need to up-cast the left side of the shift first.
Do you mean this?
rdm_start = (uint64_t)rdm_map[j].start_pfn << PAGE_SHIFT; rdm_end = rdm_start + ((uint64_t)rdm_map[j].nr_pages << PAGE_SHIFT);
Yes. Finally.
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
Quoted 9 lines
tools/firmware/hvmloader/pci.c void pci_setup(void) · line 100
Quoted 7 lines
tools/firmware/hvmloader/pci.c +2 -1
@@ -58,7 +96,9 @@ void pci_setup(void)
            uint32_t bar_reg;
            uint64_t bar_sz;
        } *bars = (struct bars *)scratch_start;
-    unsigned int i, nr_bars = 0;
+    unsigned int i, j, nr_bars = 0;
+    int nr_entries = 0;
And another pointless initializer. Plus as a count of something this
int nr_rdm_entries;
Quoted 1 line
surely wants to be "unsigned int". Also I guess the variable name
Quoted 8 lines
Quoted 6 lines
nr_rdm_entries should be literally unsigned int but this value always be set from hvm_get_reserved_device_memory_map(),
nr_rdm_entries = hvm_get_reserved_device_memory_map()
I hope that return value can be negative value in some failed case
If only you checked for these negative values...
May I can simplify these failed cases handle with within hvm_get_reserved_device_memory_map() like,
	if ( rc )
		return 0;

Quoted 6 lines

Quoted 4 lines
Because actually we don't need any negative return value again. So '0' is always fine. So here,
unsigned long nr_rdm_entries = hvm_get_reserved_device_memory_map();
Except that "unsigned int" would seem more consistent.

Fixed.

Quoted 27 lines

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Additionally, actually there are some original codes just following my codes:
           if ( need_skip_rmrr )
           {
		...
           }

	base += bar_sz;

           if ( (base < resource->base) || (base > resource->max) )
           {
               printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                      "resource!\n", devfn>>3, devfn&7, bar_reg,
                      PRIllx_arg(bar_sz));
               continue;
           }
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This can guarantee we don't overwhelm the previous mmio range.
Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.
I don't understand what a side effect is. I just to try to make sure BAR space skip any conflict range but they are still in these resource ranges.

Quoted 6 lines

A side effect is an effect you don't primarily intend with your change (or more generally, with any particular operation). In the case here, a BAR that previously got a value assigned may not anymore with your change in place. An acceptable effect of your change would be if the value it gets assigned is now different, but not assigning a value at all is not acceptable.

As I understand that value just need to align with BAR and size. Then any range should be fine. Here I think its not necessary to consider any space restriction, i.e, some device may just access under 4G.

Quoted 20 lines

Quoted 20 lines
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
and bar_data_upper will likely end up being garbage.
Did you actually _test_ this code?
Actually in my real case those RMRR ranges are always below MMIO.
Below whose MMIO? The host's or the guest's? In the latter case, just (in order to test your code) increase the range reserved for MMIO enough to cover the RMRR range.
In my platform,
RMRR region: base_addr ab80a000 end_address ab81dfff RMRR region: base_addr ad000000 end_address af7fffff
So I guess you hope I change this
#define PCI_MEM_START 0xf0000000

Quoted 9 lines

Quoted 5 lines
to
#define PCI_MEM_START 0xa0000000
right?
Almost. You'd have to use 0x80000000, or other logic would break.
Quoted 1 line
But what test you want to see? Just boot?

I don't see any strong error from boot.

Quoted 4 lines

Yes, guest boot, but including proper inspection that what your code does is really how it should be.

Okay, I will go into details.

Thanks
Tiejun

Quoted 2 lines

Jan

Jan Beulich

2014-10-29 09:08 UTC #50 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 40 lines

Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:03, <[email protected]> wrote:
Quoted 31 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:11, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 1 line · Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
Additionally, actually there are some original codes just following my codes:
           if ( need_skip_rmrr )
           {
		...
           }

	base += bar_sz;

           if ( (base < resource->base) || (base > resource->max) )
           {
               printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                      "resource!\n", devfn>>3, devfn&7, bar_reg,
                      PRIllx_arg(bar_sz));
               continue;
           }
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This can guarantee we don't overwhelm the previous mmio range.
Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.
I don't understand what a side effect is. I just to try to make sure BAR space skip any conflict range but they are still in these resource ranges.
Quoted 6 lines
A side effect is an effect you don't primarily intend with your change (or more generally, with any particular operation). In the case here, a BAR that previously got a value assigned may not anymore with your change in place. An acceptable effect of your change would be if the value it gets assigned is now different, but not assigning a value at all is not acceptable.

Quoted 3 lines

As I understand that value just need to align with BAR and size. Then any range should be fine. Here I think its not necessary to consider any space restriction, i.e, some device may just access under 4G.

No. The code determining where to put the lower boundary of the MMIO range doesn't (with your present patch) consider the regions the actual assignment code now skips. Hence the lower boundary may not be low enough to accommodate all BARs.

Jan

Chen, Tiejun

author

2014-10-30 03:18 UTC #51 lore raw

Re: [v7][RFC][PATCH 05/13] hvmloader/mmio: reconcile guest mmio with reserved device memory

Quoted 40 lines · Jan Beulich

Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:03, <[email protected]> wrote:
Quoted 37 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:11, <[email protected]> wrote:
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 08:12, <[email protected]> wrote:
Quoted 1 line · Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
Additionally, actually there are some original codes just following my codes:
            if ( need_skip_rmrr )
            {
		...
            }

	base += bar_sz;

            if ( (base < resource->base) || (base > resource->max) )
            {
                printf("pci dev %02x:%x bar %02x size "PRIllx": no space for "
                       "resource!\n", devfn>>3, devfn&7, bar_reg,
                       PRIllx_arg(bar_sz));
                continue;
            }
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
This can guarantee we don't overwhelm the previous mmio range.
Resulting in the BAR not getting a value assigned afaict. Certainly not what we want as a side effect of your changes.
I don't understand what a side effect is. I just to try to make sure BAR space skip any conflict range but they are still in these resource ranges.
A side effect is an effect you don't primarily intend with your change (or more generally, with any particular operation). In the case here, a BAR that previously got a value assigned may not anymore with your change in place. An acceptable effect of your change would be if the value it gets assigned is now different, but not assigning a value at all is not acceptable.

Quoted 8 lines

Quoted 3 lines
As I understand that value just need to align with BAR and size. Then any range should be fine. Here I think its not necessary to consider any space restriction, i.e, some device may just access under 4G.
No. The code determining where to put the lower boundary of the MMIO range doesn't (with your present patch) consider the regions the actual assignment code now skips. Hence the lower boundary may not be low enough to accommodate all BARs.

Right, so maybe before any mmio allocation, we can expand mmio resource to own sufficient space covering RMRR and all BARs.

Thanks
Tiejun

Quoted 5 lines

Jan

Tiejun Chen

author

2014-10-24 07:34 UTC #52 lore raw

RFC PATCH v7 6/13 hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Tiejun Chen

We need to check to reserve all reserved device memory maps in e820 to avoid any potential guest memory conflict.

Currently, if we can't insert RDM entries directly, we may need to handle several ranges as follows:
a. Fixed Ranges --> BUG()
lowmem_reserved_base-0xA0000: reserved by BIOS implementation, BIOS region,
RESERVED_MEMBASE ~ 0x100000000,
b. RAM or RAM:Hole -> Try to reserve

Signed-off-by Tiejun Chen <[email protected]>

---
 tools/firmware/hvmloader/e820.c | 215 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 215 insertions(+)

tools/firmware/hvmloader/e820.c +215 -0

index 2e05e93..d188e02 100644
@@ -68,12 +68,218 @@ void dump_e820_table(struct e820entry *e820, unsigned int nr)
     }
 }
 
+static unsigned int construct_rdm_e820_maps(unsigned int next_e820_entry_index,
+                                            uint32_t nr_map,
+                                            struct xen_mem_reserved_device_memory *map,
+                                            struct e820entry *e820,
+                                            unsigned int lowmem_reserved_base,
+                                            unsigned int bios_image_base)
+{
+    unsigned int i, j, sum_nr = next_e820_entry_index + nr_map;
+    uint64_t start, end, next_start, rdm_start, rdm_end;
+    uint32_t type;
+    unsigned int insert = 0, do_insert = 0;
+    int err = 0;
+
+ do_real_construct:
+    for ( i = 0; i < nr_map; i++ )
+    {
+        rdm_start = map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (map[i].nr_pages << PAGE_SHIFT);
+
+        for ( j = 0; j < next_e820_entry_index - 1; j++ )
+        {
+            start = e820[j].addr;
+            end = e820[j].addr + e820[j].size;
+            type = e820[j].type;
+            next_start = e820[j+1].addr;
+
+            /* lowmem_reserved_base-0xA0000: reserved by BIOS implementation. */
+            if ( lowmem_reserved_base < 0xA0000 &&
+                 start == lowmem_reserved_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /*
+             * BIOS region.
+             */
+            if ( start == bios_image_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /* The default memory map always occupy one fixed reserved
+             * range: RESERVED_MEMBASE ~ 0x100000000
+             */
+            if ( rdm_start >= RESERVED_MEMBASE &&
+                      rdm_start <= ((uint64_t)1 << 32) )
+            {
+                err = -1;
+                break;
+            }
+
+            /* Just amid those remaining e820 entries. */
+            if ( (rdm_start > end) && (rdm_end < next_start) )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j+1],
+                            (sum_nr - j - 1) * sizeof(struct e820entry));
+
+                    /* Then fill RMRR into that entry. */
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* Already at the end. */
+            else if ( (rdm_start > end) && !next_start )
+            {
+                if ( do_insert )
+                {
+                    e820[next_e820_entry_index].addr = rdm_start;
+                    e820[next_e820_entry_index].size = rdm_end - rdm_start;
+                    e820[next_e820_entry_index].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If completely overlap with one RAM range. */
+            else if ( rdm_start == start && rdm_end == end && type == E820_RAM )
+            {
+                if ( do_insert )
+                    e820[j].type = E820_RESERVED;
+                insert++;
+            }
+            /* If we're just alligned with start of one RAM range. */
+            else if ( rdm_start == start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+1].addr = rdm_end;
+                    e820[j+1].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+1].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j].addr = rdm_start;
+                    e820[j].size = rdm_end - rdm_start;
+                    e820[j].type = E820_RESERVED;
+                }
+                insert++;
+            }
+            /* If we're just alligned with end of one RAM range. */
+            else if ( rdm_start > start && rdm_end == end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If we're just in of one RAM range */
+            else if ( rdm_start > start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+2].addr = rdm_end;
+                    e820[j+2].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+2].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+                }
+                insert++;
+            }
+            /* If we're going last RAM:Hole range */
+            else if ( end < next_start &&
+                      rdm_start > start &&
+                      rdm_end < next_start &&
+                      type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+        }
+    }
+
+    /* These overlap may issue guest can't work well. */
+    if ( err )
+    {
+        printf("Guest can't work with some reserved device memory overlap!\n");
+        BUG();
+    }
+
+    /* Just return if done. */
+    if ( do_insert )
+        return next_e820_entry_index;
+
+    /* Fine to construct RDM mappings into e820. */
+    if ( insert == nr_map )
+    {
+        do_insert = 1;
+        goto do_real_construct;
+    }
+    /* Overlap. */
+    else
+    {
+        printf("RDM overlap with those existing e820 entries!\n");
+        printf("So we don't construct RDM mapping in e820!\n");
+    }
+
+    return next_e820_entry_index;
+}
+
 /* Create an E820 table based on memory parameters provided in hvm_info. */
 int build_e820_table(struct e820entry *e820,
                      unsigned int lowmem_reserved_base,
                      unsigned int bios_image_base)
 {
     unsigned int nr = 0;
+    int nr_entries = 0;
 
     if ( !lowmem_reserved_base )
             lowmem_reserved_base = 0xA0000;

@@ -169,6 +375,15 @@ int build_e820_table(struct e820entry *e820,
         nr++;
     }
 
+    if ( rdm_map )
+    {
+        nr_entries = hvm_get_reserved_device_memory_map();
+        if ( nr_entries > 0 )
+            nr = construct_rdm_e820_maps(nr, nr_entries, rdm_map, e820,
+                                         lowmem_reserved_base,
+                                         bios_image_base);
+    }
+
     return nr;
 }

Signature · 1 line

1.9.1

Jan Beulich

2014-10-24 14:56 UTC #53 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 10 lines

Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
We need to check to reserve all reserved device memory maps in e820 to avoid any potential guest memory conflict.
Currently, if we can't insert RDM entries directly, we may need to handle several ranges as follows:
a. Fixed Ranges --> BUG()
lowmem_reserved_base-0xA0000: reserved by BIOS implementation, BIOS region,
RESERVED_MEMBASE ~ 0x100000000,

This seems conceptually wrong to me, and I said so before:
Depending on host characteristics this approach may mean you're going to be unable to build any HVM guests. Minimally there needs to be a way to avoid these checks (resulting in devices associated with RMRRs not being assignable to such a guest). I'm therefore only going to briefly look at the rest of this patch.

Quoted 80 lines

tools/firmware/hvmloader/e820.c +3 -0

+static unsigned int construct_rdm_e820_maps(unsigned int next_e820_entry_index,
+                                            uint32_t nr_map,
+                                            struct

xen_mem_reserved_device_memory *map,

tools/firmware/hvmloader/e820.c +76 -0

+                                            struct e820entry *e820,
+                                            unsigned int lowmem_reserved_base,
+                                            unsigned int bios_image_base)
+{
+    unsigned int i, j, sum_nr = next_e820_entry_index + nr_map;
+    uint64_t start, end, next_start, rdm_start, rdm_end;
+    uint32_t type;
+    unsigned int insert = 0, do_insert = 0;
+    int err = 0;
+
+ do_real_construct:
+    for ( i = 0; i < nr_map; i++ )
+    {
+        rdm_start = map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (map[i].nr_pages << PAGE_SHIFT);
+
+        for ( j = 0; j < next_e820_entry_index - 1; j++ )
+        {
+            start = e820[j].addr;
+            end = e820[j].addr + e820[j].size;
+            type = e820[j].type;
+            next_start = e820[j+1].addr;
+
+            /* lowmem_reserved_base-0xA0000: reserved by BIOS implementation. */
+            if ( lowmem_reserved_base < 0xA0000 &&
+                 start == lowmem_reserved_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /*
+             * BIOS region.
+             */
+            if ( start == bios_image_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /* The default memory map always occupy one fixed reserved
+             * range: RESERVED_MEMBASE ~ 0x100000000
+             */
+            if ( rdm_start >= RESERVED_MEMBASE &&
+                      rdm_start <= ((uint64_t)1 << 32) )
+            {
+                err = -1;
+                break;
+            }
+
+            /* Just amid those remaining e820 entries. */
+            if ( (rdm_start > end) && (rdm_end < next_start) )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j+1],
+                            (sum_nr - j - 1) * sizeof(struct e820entry));
+
+                    /* Then fill RMRR into that entry. */
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* Already at the end. */
+            else if ( (rdm_start > end) && !next_start )
+            {
+                if ( do_insert )

tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 159

Quoted 10 lines

tools/firmware/hvmloader/e820.c +10 -0

+                {
+                    e820[next_e820_entry_index].addr = rdm_start;
+                    e820[next_e820_entry_index].size = rdm_end - rdm_start;
+                    e820[next_e820_entry_index].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If completely overlap with one RAM range. */
+            else if ( rdm_start == start && rdm_end == end && type == E820_RAM )

Comment and expression disagree.

Quoted 76 lines

tools/firmware/hvmloader/e820.c +76 -0

+            {
+                if ( do_insert )
+                    e820[j].type = E820_RESERVED;
+                insert++;
+            }
+            /* If we're just alligned with start of one RAM range. */
+            else if ( rdm_start == start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+1].addr = rdm_end;
+                    e820[j+1].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+1].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j].addr = rdm_start;
+                    e820[j].size = rdm_end - rdm_start;
+                    e820[j].type = E820_RESERVED;
+                }
+                insert++;
+            }
+            /* If we're just alligned with end of one RAM range. */
+            else if ( rdm_start > start && rdm_end == end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If we're just in of one RAM range */
+            else if ( rdm_start > start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+2].addr = rdm_end;
+                    e820[j+2].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+2].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+                }
+                insert++;
+            }
+            /* If we're going last RAM:Hole range */
+            else if ( end < next_start &&
+                      rdm_start > start &&
+                      rdm_end < next_start &&
+                      type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+

tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 245

Quoted 10 lines

tools/firmware/hvmloader/e820.c +10 -0

+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }

This if-else-if series looks horrible - is there really no way to consolidate it? Also, other than punching holes in the E820 map you don't seem to be doing anything here. And the earlier tools side patches didn't do anything about this either. Consequently, at the time where it may become necessary to establish the 1:1 mapping in the P2M, there'll be the RAM mapping still there, causing the device assignment to fail. Again - did you _test_ this scenario with a big enough guest and your gfx card passed through?

Jan

Chen, Tiejun

author

2014-10-27 08:09 UTC #54 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 11 lines · Jan Beulich

Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
We need to check to reserve all reserved device memory maps in e820 to avoid any potential guest memory conflict.
Currently, if we can't insert RDM entries directly, we may need to handle several ranges as follows:
a. Fixed Ranges --> BUG()
lowmem_reserved_base-0xA0000: reserved by BIOS implementation, BIOS region,
RESERVED_MEMBASE ~ 0x100000000,

Quoted 5 lines

This seems conceptually wrong to me, and I said so before:
Depending on host characteristics this approach may mean you're going to be unable to build any HVM guests. Minimally there needs to be a way to avoid these checks (resulting in devices associated with RMRRs not being assignable to such a guest). I'm therefore

I just use 'err' to indicate if these fixed range overlaps RMRR,

+    /* These overlap may issue guest can't work well. */
+    if ( err )
+    {

+ printf("Guest can't work with some reserved device memory overlap!\n");
+ BUG();
+ }

As I understand, these fixed ranges don't like RAM that we can move safely out any RMRR overlap. And actually its rare to overlap with those fixed ranges.

But I can remove BUG if you insist on this point.

Quoted 86 lines

only going to briefly look at the rest of this patch.

Quoted 84 lines

tools/firmware/hvmloader/e820.c +3 -0

+static unsigned int construct_rdm_e820_maps(unsigned int next_e820_entry_index,
+                                            uint32_t nr_map,
+                                            struct

xen_mem_reserved_device_memory *map,

tools/firmware/hvmloader/e820.c +80 -0

+                                            struct e820entry *e820,
+                                            unsigned int lowmem_reserved_base,
+                                            unsigned int bios_image_base)
+{
+    unsigned int i, j, sum_nr = next_e820_entry_index + nr_map;
+    uint64_t start, end, next_start, rdm_start, rdm_end;
+    uint32_t type;
+    unsigned int insert = 0, do_insert = 0;
+    int err = 0;
+
+ do_real_construct:
+    for ( i = 0; i < nr_map; i++ )
+    {
+        rdm_start = map[i].start_pfn << PAGE_SHIFT;
+        rdm_end = rdm_start + (map[i].nr_pages << PAGE_SHIFT);
+
+        for ( j = 0; j < next_e820_entry_index - 1; j++ )
+        {
+            start = e820[j].addr;
+            end = e820[j].addr + e820[j].size;
+            type = e820[j].type;
+            next_start = e820[j+1].addr;
+
+            /* lowmem_reserved_base-0xA0000: reserved by BIOS implementation. */
+            if ( lowmem_reserved_base < 0xA0000 &&
+                 start == lowmem_reserved_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /*
+             * BIOS region.
+             */
+            if ( start == bios_image_base )
+            {
+                if ( rdm_start >= start && rdm_start <= end )
+                {
+                    err = -1;
+                    break;
+                }
+            }
+
+            /* The default memory map always occupy one fixed reserved
+             * range: RESERVED_MEMBASE ~ 0x100000000
+             */
+            if ( rdm_start >= RESERVED_MEMBASE &&
+                      rdm_start <= ((uint64_t)1 << 32) )
+            {
+                err = -1;
+                break;
+            }
+
+            /* Just amid those remaining e820 entries. */
+            if ( (rdm_start > end) && (rdm_end < next_start) )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j+1],
+                            (sum_nr - j - 1) * sizeof(struct e820entry));
+
+                    /* Then fill RMRR into that entry. */
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* Already at the end. */
+            else if ( (rdm_start > end) && !next_start )
+            {
+                if ( do_insert )
+                {
+                    e820[next_e820_entry_index].addr = rdm_start;
+                    e820[next_e820_entry_index].size = rdm_end - rdm_start;
+                    e820[next_e820_entry_index].type = E820_RESERVED;

Quoted 8 lines

tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 159
Quoted 6 lines
tools/firmware/hvmloader/e820.c +6 -0
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If completely overlap with one RAM range. */
+            else if ( rdm_start == start && rdm_end == end && type == E820_RAM )
Comment and expression disagree.

What about this?

/* If coincide with one RAM range. */

Quoted 82 lines

tools/firmware/hvmloader/e820.c +81 -0

+            {
+                if ( do_insert )
+                    e820[j].type = E820_RESERVED;
+                insert++;
+            }
+            /* If we're just alligned with start of one RAM range. */
+            else if ( rdm_start == start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+1].addr = rdm_end;
+                    e820[j+1].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+1].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j].addr = rdm_start;
+                    e820[j].size = rdm_end - rdm_start;
+                    e820[j].type = E820_RESERVED;
+                }
+                insert++;
+            }
+            /* If we're just alligned with end of one RAM range. */
+            else if ( rdm_start > start && rdm_end == end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
+            /* If we're just in of one RAM range */
+            else if ( rdm_start > start && rdm_end < end && type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+2], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j+2].addr = rdm_end;
+                    e820[j+2].size = e820[j].addr + e820[j].size - rdm_end;
+                    e820[j+2].type = E820_RAM;
+                    next_e820_entry_index++;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+                }
+                insert++;
+            }
+            /* If we're going last RAM:Hole range */
+            else if ( end < next_start &&
+                      rdm_start > start &&
+                      rdm_end < next_start &&
+                      type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;

Quoted 8 lines

tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 145
Quoted 5 lines
tools/firmware/hvmloader/e820.c +5 -0
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
This if-else-if series looks horrible - is there really no way to consolidate it? Also, other than punching holes in the E820 map you don't seem to

I know this is ugly but as you know there's no any rule we can make good use of this case. RMRR can start anywhere so We have to assume any scenarios,

1. Just amid those remaining e820 entries.
2. Already at the end.
3. If coincide with one RAM range.
4. If we're just aligned with start of one RAM range.
5. If we're just aligned with end of one RAM range.
6. If we're just in of one RAM range.
7. If we're going last RAM:Hole range.

So if you think we're handling correctly, maybe we can continue optimizing this way once we have a better idea.

Quoted 4 lines

be doing anything here. And the earlier tools side patches didn't do anything about this either. Consequently, at the time where it may become necessary to establish the 1:1 mapping in the P2M, there'll be the RAM mapping still there, causing the device assignment to fail.

But I already set these range as p2m_access_n, and as you see I also reserved these range in e820 table. So although the RAM mapping still is still there but no any actual access.

Then when we assign device, we will override these p2m entry as 1:1 mapping if p2m_access_n is set.

Quoted 2 lines

Again - did you _test_ this scenario with a big enough guest and your gfx card passed through?

Yes. I validate these patches as following configuration:

memory = 2816
gfx_passthru=1
pci=["00:02.0"]

And I also perform 'xl dmesg' to check e820 table. Here I can provide an instance with overlap,

RMRR range:

root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace# xl dmesg | grep RMRR
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ab80a000 end_address ab81dfff
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ad000000 end_address af7fffff
root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace#

Without my patch:

(d4) E820 table:

(d4)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d4)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d4)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d4)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d4)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d4)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d4)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d4)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED
(d4)  HOLE: 00000000:af800000 - 00000000:fc000000
(d4)  [07]: 00000000:fc000000 - 00000001:00000000: RESERVED

With my patch:

(d2) f0000-fffff: Main BIOS
(d2) E820 table:

(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d2)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d2)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d2)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED
(d2)  HOLE: 00000000:af800000 - 00000000:fc000000
(d2)  [07]: 00000000:fc000000 - 00000000:fdffc000: RESERVED
(d2)  [08]: 00000000:fdffc000 - 00000000:fdfff000: NVS
(d2)  [09]: 00000000:fdfff000 - 00000001:00000000: RESERVED

(d2) Invoking ROMBIOS ...

Thanks
Tiejun

Jan Beulich

2014-10-27 10:17 UTC #55 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 28 lines

Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
We need to check to reserve all reserved device memory maps in e820 to avoid any potential guest memory conflict.
Currently, if we can't insert RDM entries directly, we may need to handle several ranges as follows:
a. Fixed Ranges --> BUG()
lowmem_reserved_base-0xA0000: reserved by BIOS implementation, BIOS region,
RESERVED_MEMBASE ~ 0x100000000,
Quoted 5 lines
This seems conceptually wrong to me, and I said so before:
Depending on host characteristics this approach may mean you're going to be unable to build any HVM guests. Minimally there needs to be a way to avoid these checks (resulting in devices associated with RMRRs not being assignable to such a guest). I'm therefore
I just use 'err' to indicate if these fixed range overlaps RMRR,
tools/firmware/hvmloader/e820.c +6 -0
+    /* These overlap may issue guest can't work well. */
+    if ( err )
+    {
+        printf("Guest can't work with some reserved device memory overlap!\n");
+        BUG();
+    }

Quoted 3 lines

As I understand, these fixed ranges don't like RAM that we can move safely out any RMRR overlap. And actually its rare to overlap with those fixed ranges.

Again - one of my systems has RMRRs in the Ex000 range, which certainly risks overlapping with the BIOS image should that one be larger than 64k. Plus with RMRRs being in that region, I can certainly see (physical) systems with small enough BIOS images to place RMRRs even in the low Fx000 range, which then quite certainly would overlap with the (virtual) BIOS range.

Quoted 1 line

But I can remove BUG if you insist on this point.

Whether removing the BUG() here is correct and/or sufficient to address my concern I can't immediately tell. What I insist on is that _no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).

Quoted 37 lines

Quoted 16 lines
Quoted 16 lines
tools/firmware/hvmloader/e820.c +16 -0
+            /* If we're going last RAM:Hole range */
+            else if ( end < next_start &&
+                      rdm_start > start &&
+                      rdm_end < next_start &&
+                      type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
Quoted 8 lines
tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 145
Quoted 5 lines
tools/firmware/hvmloader/e820.c +5 -0
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
This if-else-if series looks horrible - is there really no way to consolidate it? Also, other than punching holes in the E820 map you don't seem to
I know this is ugly but as you know there's no any rule we can make good use of this case. RMRR can start anywhere so We have to assume any scenarios,
1. Just amid those remaining e820 entries.
2. Already at the end.
3. If coincide with one RAM range.
4. If we're just aligned with start of one RAM range.
5. If we're just aligned with end of one RAM range.
6. If we're just in of one RAM range.
7. If we're going last RAM:Hole range.

Quoted 2 lines

So if you think we're handling correctly, maybe we can continue optimizing this way once we have a better idea.

I understand that there are various cases to be considered, but that's no different elsewhere. For example, look at xen/arch/x86/e820.c:e820_change_range_type() which gets away with quite a bit shorter an if/else-if sequence.

Quoted 8 lines

Quoted 4 lines
be doing anything here. And the earlier tools side patches didn't do anything about this either. Consequently, at the time where it may become necessary to establish the 1:1 mapping in the P2M, there'll be the RAM mapping still there, causing the device assignment to fail.
But I already set these range as p2m_access_n, and as you see I also reserved these range in e820 table. So although the RAM mapping still is still there but no any actual access.

That's being done in patch 8, but we're talking about patch 6 here. Also - what size are the RMRRs in your case? The USB ones I know of are typical single or very few page ones, so having the guest lose that amount of memory may be tolerable. But if the ranges can get any larger than a couple of pages, or if there can reasonably be a larger amount of them (like could be the case on e.g. multi-node systems), simply hiding that memory may not be well received by our users.

Quoted 13 lines

RMRR range:

root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace# xl dmesg | 
grep RMRR
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ab80a000 end_address ab81dfff
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ad000000 end_address af7fffff
root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace#

Without my patch:

(d4) E820 table:

Quoted 8 lines

(d4)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d4)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d4)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d4)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d4)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d4)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d4)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d4)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED

Where would this reserved range come from when you patches aren't in place?

Quoted 8 lines

(d4) HOLE: 00000000:af800000 - 00000000:fc000000 (d4) [07]: 00000000:fc000000 - 00000001:00000000: RESERVED
With my patch:
(d2) f0000-fffff: Main BIOS
(d2) E820 table:

Quoted 8 lines

(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d2)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d2)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d2)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED

And this already answers what I asked above: You shouldn't be blindly hiding 40Mb from the guest.

Jan

Chen, Tiejun

author

2014-10-28 07:47 UTC #56 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 39 lines · Jan Beulich

Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
We need to check to reserve all reserved device memory maps in e820 to avoid any potential guest memory conflict.
Currently, if we can't insert RDM entries directly, we may need to handle several ranges as follows:
a. Fixed Ranges --> BUG()
lowmem_reserved_base-0xA0000: reserved by BIOS implementation, BIOS region,
RESERVED_MEMBASE ~ 0x100000000,
Quoted 5 lines
This seems conceptually wrong to me, and I said so before:
Depending on host characteristics this approach may mean you're going to be unable to build any HVM guests. Minimally there needs to be a way to avoid these checks (resulting in devices associated with RMRRs not being assignable to such a guest). I'm therefore
I just use 'err' to indicate if these fixed range overlaps RMRR,
tools/firmware/hvmloader/e820.c +6 -0
+    /* These overlap may issue guest can't work well. */
+    if ( err )
+    {
+        printf("Guest can't work with some reserved device memory overlap!\n");
+        BUG();
+    }
Quoted 3 lines
As I understand, these fixed ranges don't like RAM that we can move safely out any RMRR overlap. And actually its rare to overlap with those fixed ranges.
Again - one of my systems has RMRRs in the Ex000 range, which certainly risks overlapping with the BIOS image should that one be larger than 64k. Plus with RMRRs being in that region, I can certainly see (physical) systems with small enough BIOS images to place RMRRs even in the low Fx000 range, which then quite certainly would overlap with the (virtual) BIOS range.

Quoted 4 lines

Quoted 1 line
But I can remove BUG if you insist on this point.
Whether removing the BUG() here is correct and/or sufficient to address my concern I can't immediately tell. What I insist on is that

Okay.

Quoted 6 lines

_no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).

As we know just legacy USB and GFX need these RMRR ranges. Especially, I believe just USB need << 1M space, so it may be possible to be placed below 1M. But I think we can ask BIOS to reallocate them upwards like my real platform,

RMRR region: base_addr ab80a000 end_address ab81dfff

I don't know what platform you're using, maybe its a legacy machine? But anyway it should be feasible to update BIOS. And even we can ask BIOS do this as a normal rule in the future.

For GFX, oftentimes it need dozens of MB,

RMRR region: base_addr ad000000 end_address af7fffff

So it shouldn't be overlapped with <1M.

Quoted 38 lines

Quoted 38 lines
Quoted 17 lines
Quoted 17 lines
tools/firmware/hvmloader/e820.c +16 -0
+            /* If we're going last RAM:Hole range */
+            else if ( end < next_start &&
+                      rdm_start > start &&
+                      rdm_end < next_start &&
+                      type == E820_RAM )
+            {
+                if ( do_insert )
+                {
+                    memmove(&e820[j+1], &e820[j],
+                            (sum_nr - j) * sizeof(struct e820entry));
+
+                    e820[j].size = rdm_start - e820[j].addr;
+                    e820[j].type = E820_RAM;
+
+                    e820[j+1].addr = rdm_start;
+                    e820[j+1].size = rdm_end - rdm_start;
Quoted 8 lines
tools/firmware/hvmloader/e820.c void dump_e820_table(struct e820entry *e820, unsigned int nr) · line 145
Quoted 5 lines
tools/firmware/hvmloader/e820.c +5 -0
+                    e820[j+1].type = E820_RESERVED;
+                    next_e820_entry_index++;
+                }
+                insert++;
+            }
This if-else-if series looks horrible - is there really no way to consolidate it? Also, other than punching holes in the E820 map you don't seem to
I know this is ugly but as you know there's no any rule we can make good use of this case. RMRR can start anywhere so We have to assume any scenarios,
1. Just amid those remaining e820 entries.
2. Already at the end.
3. If coincide with one RAM range.
4. If we're just aligned with start of one RAM range.
5. If we're just aligned with end of one RAM range.
6. If we're just in of one RAM range.
7. If we're going last RAM:Hole range.

Quoted 6 lines

Quoted 2 lines
So if you think we're handling correctly, maybe we can continue optimizing this way once we have a better idea.
I understand that there are various cases to be considered, but that's no different elsewhere. For example, look at xen/arch/x86/e820.c:e820_change_range_type() which gets

I don't think this circumstance is same as our requirement.

Here we are trying to insert different multiple entries that they have different range.

Anyway, I can take a further look at if we can improve this.

Quoted 7 lines

away with quite a bit shorter an if/else-if sequence.
Quoted 5 lines
Quoted 4 lines
be doing anything here. And the earlier tools side patches didn't do anything about this either. Consequently, at the time where it may become necessary to establish the 1:1 mapping in the P2M, there'll be the RAM mapping still there, causing the device assignment to fail.

Quoted 6 lines

Quoted 3 lines
But I already set these range as p2m_access_n, and as you see I also reserved these range in e820 table. So although the RAM mapping still is still there but no any actual access.
That's being done in patch 8, but we're talking about patch 6 here. Also - what size are the RMRRs in your case? The USB ones I know

RMRR region: base_addr ab80a000 end_address ab81dfff RMRR region: base_addr ad000000 end_address af7fffff

Quoted 6 lines

of are typical single or very few page ones, so having the guest lose that amount of memory may be tolerable. But if the ranges can get any larger than a couple of pages, or if there can reasonably be a larger amount of them (like could be the case on e.g. multi-node systems), simply hiding that memory may not be well received by our users.

Customers may accept dozens of MB but I'm not sure.

Quoted 43 lines

Quoted 14 lines

RMRR range:

root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace# xl dmesg |
grep RMRR
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ab80a000 end_address ab81dfff
(XEN) [VT-D]dmar.c:834: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:679:   RMRR region: base_addr ad000000 end_address af7fffff
root@tchen0-Shark-Bay-Client-platform:/home/tchen0/workspace#

Without my patch:

(d4) E820 table:

Quoted 8 lines

(d4)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d4)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d4)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d4)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d4)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d4)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d4)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d4)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED

Where would this reserved range come from when you patches aren't in place?

Quoted 8 lines

(d4) HOLE: 00000000:af800000 - 00000000:fc000000 (d4) [07]: 00000000:fc000000 - 00000001:00000000: RESERVED
With my patch:
(d2) f0000-fffff: Main BIOS
(d2) E820 table:

Quoted 8 lines

(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d2)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d2)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d2)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED

Quoted 2 lines

And this already answers what I asked above: You shouldn't be blindly hiding 40Mb from the guest.

If we don't reserve these RMRR ranges, so guest may create 1:1 mapping. Then it will affect a device usage in other VM, or a device usage may corrupt these ranges in other VM.

Yes, we really need a policy to do this. So please tell me what you expect.

Thanks
Tiejun

Jan Beulich

2014-10-28 10:06 UTC #57 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 4 lines

Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:47, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
On 2014/10/24 22:56, Jan Beulich wrote:

Quoted 8 lines

Quoted 6 lines
_no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).
As we know just legacy USB and GFX need these RMRR ranges.

This is specified where?

Quoted 8 lines

Especially, I
believe just USB need << 1M space, so it may be possible to be placed below 1M. But I think we can ask BIOS to reallocate them upwards like my real platform,
RMRR region: base_addr ab80a000 end_address ab81dfff
I don't know what platform you're using, maybe its a legacy machine?

A Westmere one.

Quoted 9 lines

But
anyway it should be feasible to update BIOS. And even we can ask BIOS do this as a normal rule in the future.
For GFX, oftentimes it need dozens of MB,
RMRR region: base_addr ad000000 end_address af7fffff
So it shouldn't be overlapped with <1M.

These "I believe" and "shouldn't" are a real problem here: Please make claims only based on the specification, not on observations on a particular system. In the real world, you have to be prepared for implementations of a specification to be taking more liberties than allowed for; you should never assume people don't even make use off the full scope a specification provides for. I know I'm repeating myself, but again - remember you're changing the hypervisor here (and view hvmloader as an extension of the hypervisor inside the guest).

Quoted 15 lines

Quoted 15 lines
Quoted 14 lines
I know this is ugly but as you know there's no any rule we can make good use of this case. RMRR can start anywhere so We have to assume any scenarios,
1. Just amid those remaining e820 entries.
2. Already at the end.
3. If coincide with one RAM range.
4. If we're just aligned with start of one RAM range.
5. If we're just aligned with end of one RAM range.
6. If we're just in of one RAM range.
7. If we're going last RAM:Hole range.
So if you think we're handling correctly, maybe we can continue optimizing this way once we have a better idea.

Quoted 8 lines

Quoted 3 lines
I understand that there are various cases to be considered, but that's no different elsewhere. For example, look at xen/arch/x86/e820.c:e820_change_range_type() which gets
I don't think this circumstance is same as our requirement.
Here we are trying to insert different multiple entries that they have different range.

Inserting multiple entries can always be done by inserting one entry at a time. If that yields better (easier to understand and maintain) code, and if the code path isn't a hot one, that should be the route to go.

Quoted 13 lines

Quoted 12 lines

With my patch:

(d2) f0000-fffff: Main BIOS
(d2) E820 table:

(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d2)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d2)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d2)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED

Quoted 8 lines

Quoted 2 lines
And this already answers what I asked above: You shouldn't be blindly hiding 40Mb from the guest.
If we don't reserve these RMRR ranges, so guest may create 1:1 mapping. Then it will affect a device usage in other VM, or a device usage may corrupt these ranges in other VM.
Yes, we really need a policy to do this. So please tell me what you expect.

In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.

Jan

Chen, Tiejun

author

2014-10-29 07:43 UTC #58 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 11 lines · Jan Beulich

Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:47, <[email protected]> wrote:
Quoted 8 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
On 2014/10/24 22:56, Jan Beulich wrote:
_no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).

Quoted 3 lines

Quoted 1 line
As we know just legacy USB and GFX need these RMRR ranges.
This is specified where?

In VT-D specification, I just see,

"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform designers should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."

Quoted 22 lines

Quoted 9 lines
Especially, I
believe just USB need << 1M space, so it may be possible to be placed below 1M. But I think we can ask BIOS to reallocate them upwards like my real platform,
RMRR region: base_addr ab80a000 end_address ab81dfff
I don't know what platform you're using, maybe its a legacy machine?
A Westmere one.
Quoted 9 lines
But
anyway it should be feasible to update BIOS. And even we can ask BIOS do this as a normal rule in the future.
For GFX, oftentimes it need dozens of MB,
RMRR region: base_addr ad000000 end_address af7fffff
So it shouldn't be overlapped with <1M.

Quoted 9 lines

These "I believe" and "shouldn't" are a real problem here: Please make claims only based on the specification, not on observations on a particular system. In the real world, you have to be prepared for implementations of a specification to be taking more liberties than allowed for; you should never assume people don't even make use off the full scope a specification provides for. I know I'm repeating myself, but again - remember you're changing the hypervisor here (and view hvmloader as an extension of the hypervisor inside the guest).

RMRR really is very troublesome.

The legacy usage of USB just cover ps2 emulation as I know. And as you see these address are different in different platforms so this mean they're not redistricted somewhere specific. And GFX need more space so its not possible to be placed under 1M.

So maybe I can drop patch #12, xen/vtd: re-enable USB device assignment, to leave USB out our scope. Or a little improvement is to check if its own range is below 1M.

Quoted 20 lines

Quoted 20 lines
Quoted 19 lines
Quoted 13 lines
I know this is ugly but as you know there's no any rule we can make good use of this case. RMRR can start anywhere so We have to assume any scenarios,
1. Just amid those remaining e820 entries.
2. Already at the end.
3. If coincide with one RAM range.
4. If we're just aligned with start of one RAM range.
5. If we're just aligned with end of one RAM range.
6. If we're just in of one RAM range.
7. If we're going last RAM:Hole range.
Quoted 2 lines
So if you think we're handling correctly, maybe we can continue optimizing this way once we have a better idea.
I understand that there are various cases to be considered, but that's no different elsewhere. For example, look at xen/arch/x86/e820.c:e820_change_range_type() which gets

Quoted 7 lines

Quoted 4 lines
I don't think this circumstance is same as our requirement.
Here we are trying to insert different multiple entries that they have different range.
Inserting multiple entries can always be done by inserting one entry at a time. If that yields better (easier to understand and

Insert means you have to split one existing entry. Even it may overlap with two entries at the same time, and these two entries may not be continuous.

Quoted 2 lines

maintain) code, and if the code path isn't a hot one, that should be the route to go.

I promise I can take a further look but it may be difficult to me.

Quoted 17 lines

Quoted 17 lines
Quoted 16 lines
Quoted 5 lines
With my patch:
(d2) f0000-fffff: Main BIOS
(d2) E820 table:
Quoted 8 lines
(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:ab80a000: RAM
(d2)  [04]: 00000000:ab80a000 - 00000000:ab81e000: RESERVED
(d2)  [05]: 00000000:ab81e000 - 00000000:ad000000: RAM
(d2)  [06]: 00000000:ad000000 - 00000000:af800000: RESERVED
And this already answers what I asked above: You shouldn't be blindly hiding 40Mb from the guest.

Quoted 8 lines

Quoted 5 lines
If we don't reserve these RMRR ranges, so guest may create 1:1 mapping. Then it will affect a device usage in other VM, or a device usage may corrupt these ranges in other VM.
Yes, we really need a policy to do this. So please tell me what you expect.
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.

Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.

Thanks
Tiejun

Quoted 3 lines

Jan

Jan Beulich

2014-10-29 09:15 UTC #59 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 17 lines

Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:47, <[email protected]> wrote:
Quoted 8 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
On 2014/10/24 22:56, Jan Beulich wrote:
_no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).
Quoted 3 lines
Quoted 1 line
As we know just legacy USB and GFX need these RMRR ranges.
This is specified where?

Quoted 7 lines

In VT-D specification, I just see,
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform designers should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."

Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...

Quoted 10 lines

RMRR really is very troublesome.
The legacy usage of USB just cover ps2 emulation as I know. And as you see these address are different in different platforms so this mean they're not redistricted somewhere specific. And GFX need more space so its not possible to be placed under 1M.
So maybe I can drop patch #12, xen/vtd: re-enable USB device assignment, to leave USB out our scope. Or a little improvement is to check if its own range is below 1M.

I think we made clear a number of iterations ago that rather than aiming for another half baked solution, it should be done right this time. No excuses. It's bad enough that this half broken code made it into the tree originally.

Quoted 5 lines

Quoted 2 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.

Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really happy, but could probably be talked into accepting this.

Jan

Chen, Tiejun

author

2014-10-30 03:11 UTC #60 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 19 lines · Jan Beulich

Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 08:47, <[email protected]> wrote:
Quoted 8 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 09:09, <[email protected]> wrote:
On 2014/10/24 22:56, Jan Beulich wrote:
_no matter_ what RMRRs a physical host has, it should not prevent the creation of guests (the worst that may result is that passing through certain devices doesn't work anymore, and even then the operator needs to be given a way of circumventing this if (s)he knows that the device won't access the range post-boot, or if it's being deemed acceptable for it to do so).
Quoted 3 lines
Quoted 1 line
As we know just legacy USB and GFX need these RMRR ranges.
This is specified where?
In VT-D specification, I just see,

Quoted 8 lines

Quoted 5 lines
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform designers should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."
Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...

Yes. But this already clarify RMRR "used for legacy usage" and "avoid or limit use of reserved memory regions", so RMRR would be gone finally. So I mean it may be acceptable to assume something based our known info.

Quoted 8 lines

Quoted 8 lines
RMRR really is very troublesome.
The legacy usage of USB just cover ps2 emulation as I know. And as you see these address are different in different platforms so this mean they're not redistricted somewhere specific. And GFX need more space so its not possible to be placed under 1M.

Quoted 8 lines

Quoted 3 lines
So maybe I can drop patch #12, xen/vtd: re-enable USB device assignment, to leave USB out our scope. Or a little improvement is to check if its own range is below 1M.
I think we made clear a number of iterations ago that rather than aiming for another half baked solution, it should be done right this time. No excuses. It's bad enough that this half broken code made it into the tree originally.

Okay. But if USB is really overlapping with BIOS...

Furthermore, maybe I can ask some guys to look into if we can move RMRR ranges in the hypervisor. If yes, things could be better.

Quoted 11 lines

Quoted 6 lines
Quoted 3 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.
Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really

I don't think so. If you're considering a VM, this case should be same under native circumstance. And in native case, all RMRR ranges are marked as reserved in e820 table.

Thanks
Tiejun

Quoted 6 lines

happy, but could probably be talked into accepting this.
Jan

Jan Beulich

2014-10-30 09:20 UTC #61 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 11 lines

Quoted 1 line
Quoted 1 line · On 30.10.14 at 04:11, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
In VT-D specification, I just see,
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform designers should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."

Quoted 6 lines

Quoted 2 lines
Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...
Yes. But this already clarify RMRR "used for legacy usage" and "avoid or limit use of reserved memory regions", so RMRR would be gone finally. So I mean it may be acceptable to assume something based our known info.

No. Making assumption on observed broken behavior is okay (to work around it), but making assumptions for not (yet) observed correct behavior to be absent should never be done.

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.

Quoted 8 lines

Quoted 4 lines
Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really
I don't think so. If you're considering a VM, this case should be same under native circumstance. And in native case, all RMRR ranges are marked as reserved in e820 table.

That would only be a valid comparison if all devices associated with RMRRs would also be passed to that particular VM. But since you're doing the E820 adjustment to all VMs (no matter whether they would ever get _any_ device passed through to them) this is not comparing like entities.

Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)

Jan

Chen, Tiejun

author

2014-10-31 05:41 UTC #62 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 43 lines · Jan Beulich

Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 04:11, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
In VT-D specification, I just see,
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform designers should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."
Quoted 6 lines
Quoted 2 lines
Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...
Yes. But this already clarify RMRR "used for legacy usage" and "avoid or limit use of reserved memory regions", so RMRR would be gone finally. So I mean it may be acceptable to assume something based our known info.
No. Making assumption on observed broken behavior is okay (to work around it), but making assumptions for not (yet) observed correct behavior to be absent should never be done.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.
Quoted 8 lines
Quoted 4 lines
Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really
I don't think so. If you're considering a VM, this case should be same under native circumstance. And in native case, all RMRR ranges are marked as reserved in e820 table.
That would only be a valid comparison if all devices associated with RMRRs would also be passed to that particular VM. But since you're doing the E820 adjustment to all VMs (no matter whether they would ever get _any_ device passed through to them) this is not comparing like entities.

Quoted 6 lines

Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)

Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?

If yes, could you give us some rules we should follow?

Thanks
Tiejun

Tian, Kevin

2014-10-31 06:21 UTC #63 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 55 lines

From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 04:11, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
In VT-D specification, I just see,
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform
designers
Quoted 36 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."
Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...
Yes. But this already clarify RMRR "used for legacy usage" and "avoid or limit use of reserved memory regions", so RMRR would be gone finally. So I mean it may be acceptable to assume something based our known info.
No. Making assumption on observed broken behavior is okay (to work around it), but making assumptions for not (yet) observed correct behavior to be absent should never be done.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.
Quoted 8 lines
Quoted 4 lines
Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really
I don't think so. If you're considering a VM, this case should be same under native circumstance. And in native case, all RMRR ranges are marked as reserved in e820 table.
That would only be a valid comparison if all devices associated with RMRRs would also be passed to that particular VM. But since you're doing the E820 adjustment to all VMs (no matter whether they would ever get _any_ device passed through to them) this is not comparing like entities.
Quoted 6 lines
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)

Quoted 4 lines

Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?

Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.

If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.

Thanks
Kevin

Chen, Tiejun

author

2014-10-31 07:02 UTC #64 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 55 lines · Kevin

Quoted 55 lines
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 04:11, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 29.10.14 at 08:43, <[email protected]> wrote:
In VT-D specification, I just see,
"The RMRR regions are expected to be used for legacy usages (such as USB, UMA Graphics, etc.) requiring reserved memory. Platform
designers
Quoted 42 lines
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
should avoid or limit use of reserved memory regions since these require system software to create holes in the DMA virtual address range available to system software and its driver."
Nice that you quote it, but did you also read it properly? There's this little "etc" following the explicit naming of USB and UMA...
Yes. But this already clarify RMRR "used for legacy usage" and "avoid or limit use of reserved memory regions", so RMRR would be gone finally. So I mean it may be acceptable to assume something based our known info.
No. Making assumption on observed broken behavior is okay (to work around it), but making assumptions for not (yet) observed correct behavior to be absent should never be done.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
In the tool stack, don't even populate these holes with RAM. This will then lead to RAM getting populated further up at the upper end.
Shouldn't populate RAM still with guest_physmap_add_entry()? If yes, we already be there to mark them as p2m_access_n.
Quoted 8 lines
Quoted 4 lines
Marking them with p2m_access_n is not the same as not populating the regions in the first place. Again - hiding multiple megabytes of memory (and who knows if it can't grow into the gigabyte range) is just not acceptable. Even for just a few pages I wouldn't be really
I don't think so. If you're considering a VM, this case should be same under native circumstance. And in native case, all RMRR ranges are marked as reserved in e820 table.
That would only be a valid comparison if all devices associated with RMRRs would also be passed to that particular VM. But since you're doing the E820 adjustment to all VMs (no matter whether they would ever get _any_ device passed through to them) this is not comparing like entities.
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)

Quoted 7 lines

Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is

Could we just extend a sub option as one subset of 'pci' parameter? like pci = ["00:02.0@passthrough"] or something else, I think it should make sense.

Quoted 1 line

activated at VM creation. Otherwise we just keep existing behavior.

Yes, VM creation should be failed from any overlap. If without this option, just make device assignment failed in case of conflict, but VM can still step forward.

Quoted 5 lines

If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.

When we check if any RMRR exists, we should post this recommended message while parsing ACPI for RMRR.

Thanks
Tiejun

Quoted 5 lines

Thanks
Kevin

Jan Beulich

2014-10-31 08:20 UTC #65 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Tian, Kevin

Quoted 16 lines

Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?

Quoted 8 lines

Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.

Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.

Jan

Chen, Tiejun

author

2014-11-03 05:49 UTC #66 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 25 lines · Jan Beulich

Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.

Quoted 6 lines

Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.

Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...

So anyway, let me be clear something here before we step next. Firstly I'm wondering if you will refine that patch to achieve our requirement by yourself since looks you'd like to maintain that patch, or I should do this? And of course we can walk another way, like xenstore as Yang suggested previously.

Thanks
Tiejun

Jan Beulich

2014-11-03 08:56 UTC #67 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 34 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.

Quoted 3 lines

Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...

Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.

Jan

Chen, Tiejun

author

2014-11-03 09:40 UTC #68 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 38 lines · Jan Beulich

Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.
Quoted 3 lines
Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...

Quoted 6 lines

Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.

If we really need to work with a hypercall, maybe we can introduce a little bit to construct that to callback with multiple entries like this, for instance,

RMRR entry0 have three devices, and entry1 have two devices,

[start0, nr_pages0, bdf0],
[start0, nr_pages0, bdf1],
[start0, nr_pages0, bdf2],
[start1, nr_pages1, bdf3],
[start1, nr_pages1, bdf4],

Although its cost more buffers, actually as you know this actual case is really rare. So maybe this way can be feasible. Then we don't need additional hypercall or xenstore.

Thanks
Tiejun

Jan Beulich

2014-11-03 09:51 UTC #69 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 59 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:40, <[email protected]> wrote:
Quoted 38 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.
Quoted 3 lines
Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...
Quoted 6 lines
Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.
If we really need to work with a hypercall, maybe we can introduce a little bit to construct that to callback with multiple entries like this, for instance,
RMRR entry0 have three devices, and entry1 have two devices,
[start0, nr_pages0, bdf0],
[start0, nr_pages0, bdf1],
[start0, nr_pages0, bdf2],
[start1, nr_pages1, bdf3],
[start1, nr_pages1, bdf4],

Quoted 3 lines

Although its cost more buffers, actually as you know this actual case is really rare. So maybe this way can be feasible. Then we don't need additional hypercall or xenstore.

Conceptually, as a MEMOP, it has no business reporting BDFs. And then rather than returning the same address range more than once, having the caller supply a handle to an array and storing all of the SBDFs (or perhaps a single segment would suffice along with all the BDFs) there would seem to be an approach more consistent with what we do elsewhere.

Jan

Chen, Tiejun

author

2014-11-03 11:32 UTC #70 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 63 lines · Jan Beulich

Quoted 59 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:40, <[email protected]> wrote:
Quoted 38 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.
Quoted 3 lines
Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...
Quoted 6 lines
Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.
If we really need to work with a hypercall, maybe we can introduce a little bit to construct that to callback with multiple entries like this, for instance,
RMRR entry0 have three devices, and entry1 have two devices,
[start0, nr_pages0, bdf0],
[start0, nr_pages0, bdf1],
[start0, nr_pages0, bdf2],
[start1, nr_pages1, bdf3],
[start1, nr_pages1, bdf4],
Quoted 3 lines
Although its cost more buffers, actually as you know this actual case is really rare. So maybe this way can be feasible. Then we don't need additional hypercall or xenstore.

Quoted 6 lines

Conceptually, as a MEMOP, it has no business reporting BDFs. And then rather than returning the same address range more than once, having the caller supply a handle to an array and storing all of the SBDFs (or perhaps a single segment would suffice along with all the BDFs) there would seem to be an approach more consistent with what we do elsewhere.

Here I'm wondering if we really need to expose BDFs to tools. Actually tools just want to know those range no matter who owns these entries. I mean we can do this in Xen.

When we try to assign device as passthrough, Xen can get that bdf so Xen can pre-check everything inside that hypercall, and Xen can return something like this,

#1 If this device has RMRR, we return that rmrr buffer. This is similar with our current implementation. #2 If not, we return 'nr_entries' as '0' to notify hvmloader it has nothing to do.

Thanks
Tiejun

Jan Beulich

2014-11-03 11:43 UTC #71 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 72 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:32, <[email protected]> wrote:
Quoted 63 lines · Jan Beulich
Quoted 59 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:40, <[email protected]> wrote:
Quoted 38 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.
Quoted 3 lines
Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...
Quoted 6 lines
Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.
If we really need to work with a hypercall, maybe we can introduce a little bit to construct that to callback with multiple entries like this, for instance,
RMRR entry0 have three devices, and entry1 have two devices,
[start0, nr_pages0, bdf0],
[start0, nr_pages0, bdf1],
[start0, nr_pages0, bdf2],
[start1, nr_pages1, bdf3],
[start1, nr_pages1, bdf4],
Quoted 3 lines
Although its cost more buffers, actually as you know this actual case is really rare. So maybe this way can be feasible. Then we don't need additional hypercall or xenstore.
Quoted 6 lines
Conceptually, as a MEMOP, it has no business reporting BDFs. And then rather than returning the same address range more than once, having the caller supply a handle to an array and storing all of the SBDFs (or perhaps a single segment would suffice along with all the BDFs) there would seem to be an approach more consistent with what we do elsewhere.

Quoted 3 lines

Here I'm wondering if we really need to expose BDFs to tools. Actually tools just want to know those range no matter who owns these entries. I mean we can do this in Xen.

As pointed out before, in order for the tools to (a) avoid punching _all possible_ holes when not asked to but (b) punch holes for all devices assigned right at guest boot time, I don't think the tools can get away without having some way of identifying the _specific_ reserved memory regions a guest needs. Whether this works via SBDF or some other means is tbd.

Quoted 6 lines

When we try to assign device as passthrough, Xen can get that bdf so Xen can pre-check everything inside that hypercall, and Xen can return something like this,
#1 If this device has RMRR, we return that rmrr buffer. This is similar with our current implementation.

"That rmrr buffer" being which? Remember that there may be multiple devices associated with RMRRs and multiple devices assigned to a guest.

Quoted 2 lines

#2 If not, we return 'nr_entries' as '0' to notify hvmloader it has nothing to do.

This (as vague as it is) hints in the same domain-specific reserved region determination that I was already hinting at.

Jan

Chen, Tiejun

author

2014-11-03 11:58 UTC #72 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 87 lines · Jan Beulich

Quoted 72 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:32, <[email protected]> wrote:
Quoted 63 lines · Jan Beulich
Quoted 59 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:40, <[email protected]> wrote:
Quoted 38 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 06:49, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 16 lines
Quoted 11 lines
Quoted 1 line · On 31.10.14 at 07:21, <[email protected]> wrote:
From: Chen, Tiejun
Sent: Friday, October 31, 2014 1:41 PM
Quoted 6 lines · Jan Beulich
Thinking about this some more, this odd universal hole punching in the E820 is very likely to end up causing problems. Hence I think this really should be optional behavior, with pass through of devices associated with RMRRs failing if not used. (This ought to include punching holes for _just_ the devices passed through to a guest upon creation when the option is not enabled.)
Quoted 4 lines
Yeah, we had a similar discussion internal to add a parameter to force reserving RMRR. In this case we can't create a VM if these ranges conflict with anything. So what about this idea?
Quoted 8 lines
Adding a new parameter (e.g. 'check-passthrough') looks the right approach. When the parameter is on, RMRR check/hole punch is activated at VM creation. Otherwise we just keep existing behavior.
If user configures device pass-through at creation time, this parameter will be set by default. If user wants the VM capable of device hot-plug, an explicit parameter can be added in the config file to enforce RMRR check at creation time.
Quoted 6 lines
Not exactly, I specifically described it slightly differently above. When devices get passed through and the option is absent, holes should be punched only for the RMRRs associated with those devices (i.e. ideally none). Of course this means we'll need a way to associate RMRRs with devices in the tool stack and hvmloader, i.e. the current XENMEM_reserved_device_memory_map alone won't suffice.
Quoted 3 lines
Yeah, current hypercall just provide RMRR entries without that associated BDF. And especially, in some cases one range may be shared by multiple devices...
Quoted 6 lines
Before we decide who's going to do an eventual change we need to determine what behavior we want, and whether this hypercall is really the right one. Quite possibly we'd need a per-domain view along with the global view, and hence rather than modifying this one we may need to introduce e.g. a new domctl.
If we really need to work with a hypercall, maybe we can introduce a little bit to construct that to callback with multiple entries like this, for instance,
RMRR entry0 have three devices, and entry1 have two devices,
[start0, nr_pages0, bdf0],
[start0, nr_pages0, bdf1],
[start0, nr_pages0, bdf2],
[start1, nr_pages1, bdf3],
[start1, nr_pages1, bdf4],
Quoted 3 lines
Although its cost more buffers, actually as you know this actual case is really rare. So maybe this way can be feasible. Then we don't need additional hypercall or xenstore.
Quoted 6 lines
Conceptually, as a MEMOP, it has no business reporting BDFs. And then rather than returning the same address range more than once, having the caller supply a handle to an array and storing all of the SBDFs (or perhaps a single segment would suffice along with all the BDFs) there would seem to be an approach more consistent with what we do elsewhere.
Quoted 3 lines
Here I'm wondering if we really need to expose BDFs to tools. Actually tools just want to know those range no matter who owns these entries. I mean we can do this in Xen.
As pointed out before, in order for the tools to (a) avoid punching _all possible_ holes when not asked to but (b) punch holes for all devices assigned right at guest boot time, I don't think the tools can get away without having some way of identifying the _specific_ reserved memory regions a guest needs. Whether this works via SBDF or some other means is tbd.
Quoted 4 lines
When we try to assign device as passthrough, Xen can get that bdf so Xen can pre-check everything inside that hypercall, and Xen can return something like this,

Quoted 6 lines

Quoted 2 lines
#1 If this device has RMRR, we return that rmrr buffer. This is similar with our current implementation.
"That rmrr buffer" being which? Remember that there may be multiple devices associated with RMRRs and multiple devices assigned to a guest.

Maybe I don't describe what I want to do.

I know there are multiple device associated one RMRR, and multiple device can be assigned one guest.

Firstly we have a rule that we just allow all devices associated one RMRR to be assign same VM, right? So I mean while we create VM, we always call current hypercall but inside hypercall, Xen can know which devices will be assigned to this VM. So Xen still lookup that RMRR list but now Xen would check if these RMRR belongs to that device we want to assign this domain. If yes, we just let that callback go through these RMRR info from that list but exclude other unrelated RMRR. If not, we don't go through any RMRR info so that 'nr_entries' is also zero.

Thanks
Tiejun

Quoted 10 lines

Quoted 3 lines
#2 If not, we return 'nr_entries' as '0' to notify hvmloader it has nothing to do.
This (as vague as it is) hints in the same domain-specific reserved region determination that I was already hinting at.
Jan

Jan Beulich

2014-11-03 12:34 UTC #73 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 5 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:58, <[email protected]> wrote:
Firstly we have a rule that we just allow all devices associated one RMRR to be assign same VM, right? So I mean while we create VM, we always call current hypercall but inside hypercall, Xen can know which devices will be assigned to this VM.

I.e. the hypercall (at least optionally) becomes per-domain rather than global. And you imply that device assignment happens before memory getting populated (which likely can be arranged for in the tool stack if that's not already the case, but which isn't currently mandated by the hypervisor).

Jan

Quoted 8 lines

So Xen still lookup that RMRR list
but now Xen would check if these RMRR belongs to that device we want to assign this domain. If yes, we just let that callback go through these RMRR info from that list but exclude other unrelated RMRR. If not, we don't go through any RMRR info so that 'nr_entries' is also zero.
Thanks
Tiejun

Chen, Tiejun

author

2014-11-04 05:05 UTC #74 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 9 lines · Jan Beulich

Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:58, <[email protected]> wrote:
Firstly we have a rule that we just allow all devices associated one RMRR to be assign same VM, right? So I mean while we create VM, we always call current hypercall but inside hypercall, Xen can know which devices will be assigned to this VM.
I.e. the hypercall (at least optionally) becomes per-domain rather than global. And you imply that device assignment happens before memory getting populated (which likely can be arranged

I tried to find a clue about this point but unfortunately I can't trace when we assign device exactly. But in theory, based on your hint I prefer the device assignment should follow memory getting populated. Because when we add a device, we need to create iommu map so this means at this moment the guest should already finish populating memory, right?

Thanks
Tiejun

Quoted 16 lines

for in the tool stack if that's not already the case, but which isn't currently mandated by the hypervisor).
Jan
Quoted 8 lines
So Xen still lookup that RMRR list
but now Xen would check if these RMRR belongs to that device we want to assign this domain. If yes, we just let that callback go through these RMRR info from that list but exclude other unrelated RMRR. If not, we don't go through any RMRR info so that 'nr_entries' is also zero.
Thanks
Tiejun

Jan Beulich

2014-11-04 07:54 UTC #75 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 12 lines

Quoted 1 line
Quoted 1 line · On 04.11.14 at 06:05, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:58, <[email protected]> wrote:
Firstly we have a rule that we just allow all devices associated one RMRR to be assign same VM, right? So I mean while we create VM, we always call current hypercall but inside hypercall, Xen can know which devices will be assigned to this VM.
I.e. the hypercall (at least optionally) becomes per-domain rather than global. And you imply that device assignment happens before memory getting populated (which likely can be arranged

Quoted 5 lines

I tried to find a clue about this point but unfortunately I can't trace when we assign device exactly. But in theory, based on your hint I prefer the device assignment should follow memory getting populated. Because when we add a device, we need to create iommu map so this means at this moment the guest should already finish populating memory, right?

There's no such strong connection: When a device gets assigned, IOMMU mappings get created for all memory the guest already has assigned (which at least in theory can include the "none" case). When (more) memory gets assigned after a device was already assigned to the guest, the IOMMU mappings would simply get updated.

While I think you're right in that memory assignment happens before device assignment, for your specific purpose it might have been easier the other way around, since when memory gets populated first you'll need special peeking into which devices will get assigned later in order to avoid the respective RMRR areas, or you'll need to modify device assignment code to move the RAM populated there out of the way.

Jan

Chen, Tiejun

author

2014-11-05 02:59 UTC #76 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 25 lines · Jan Beulich

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 04.11.14 at 06:05, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 12:58, <[email protected]> wrote:
Firstly we have a rule that we just allow all devices associated one RMRR to be assign same VM, right? So I mean while we create VM, we always call current hypercall but inside hypercall, Xen can know which devices will be assigned to this VM.
I.e. the hypercall (at least optionally) becomes per-domain rather than global. And you imply that device assignment happens before memory getting populated (which likely can be arranged
Quoted 5 lines
I tried to find a clue about this point but unfortunately I can't trace when we assign device exactly. But in theory, based on your hint I prefer the device assignment should follow memory getting populated. Because when we add a device, we need to create iommu map so this means at this moment the guest should already finish populating memory, right?
There's no such strong connection: When a device gets assigned, IOMMU mappings get created for all memory the guest already has assigned (which at least in theory can include the "none" case). When (more) memory gets assigned after a device was already assigned to the guest, the IOMMU mappings would simply get updated.

Quoted 2 lines

While I think you're right in that memory assignment happens before device assignment, for your specific purpose it might have

Yeah, I also double checked this sequence with printk.

Quoted 6 lines

been easier the other way around, since when memory gets populated first you'll need special peeking into which devices will get assigned later in order to avoid the respective RMRR areas, or you'll need to modify device assignment code to move the RAM populated there out of the way.

Although I really don't grab these device assignment codes explicitly, but looks it may corrupt some existing frameworks to move device assignment codes.

I'm not sure if you already have a definitive solution. If yes, please guide me and just ignore the reset.

But if not, I have a preliminary picture as follows:

#1 Policy

We will introduce a parameter globally, 'pci_rdmforce'. Then we can pass that in the config file, like

pci_rdmforce = 1 => Of course this should be 0 by default.

'1' means we should force check to reserve all ranges. If failed VM wouldn't be created successfully. This also can give user a chance to work well with later hotplug, even if not a device assignment while creating VM.

But we can override that by one specific pci device:

pci = ['AA:BB.CC,rdmforce=0/1]

But this 'rdmforce' should be 1 by default since obviously any passthrough device always need to do this. Actually no one really want to set as '0' so it may be unnecessary but I'd like to leave this as a potential approach.

So if 'pci_rdmforce = 0' and 'rdmforce = 1', we just check to reserve the ranges associated to this device.

And in any case of 'pci_rdmforce = 0' which means we have nothing to do, so we will have a higher likelihood of failure to assign a device with RMRR in both case of adding and attaching it. So we have to reject that in case of overlapping. Actually this just depends on whether we can set these identified p2m mapping.

#2 How-to

Firstly we should post actively a recommendation message to indicate we'd better set 'pci_rdmforce' to work passthrough, if any RMRR exists while parsing ACPI.

Next, we need a new domctl to provide these info, 'pci_rdmforce' and 'rdm_force' when parse the config file. Certainly we may need to introduce and set two new fields in strcut domain, and since then we just use these fields to modify our current existing iommu callback inside to support our policy. I mean we just expose those associated RMRR entry. I think its easy to implement this since inside Xen we can know which entry is owned by which device. So this can benefit us to avoid modifying any tools codes and most Xen codes we already addressed.

So is it feasible to you? Or Anything else I'm missing?

Thanks
Tiejun

Jan Beulich

2014-11-05 17:00 UTC #77 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 1 line

Quoted 1 line
Quoted 1 line
"Chen, Tiejun" <[email protected]> 11/05/14 3:59 AM >>>

Everything up to here sounded reasonable.

Quoted 8 lines

Next, we need a new domctl to provide these info, 'pci_rdmforce' and 'rdm_force' when parse the config file. Certainly we may need to introduce and set two new fields in strcut domain, and since then we just use these fields to modify our current existing iommu callback inside to support our policy. I mean we just expose those associated RMRR entry. I think its easy to implement this since inside Xen we can know which entry is owned by which device. So this can benefit us to avoid modifying any tools codes and most Xen codes we already addressed.

Whether a domctl is the right approach I can't really tell with this somewhat vague description.

Jan

Chen, Tiejun

author

2014-11-06 09:28 UTC #78 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 13 lines · Jan Beulich

Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chen, Tiejun" <[email protected]> 11/05/14 3:59 AM >>>
Everything up to here sounded reasonable.
Quoted 8 lines
Next, we need a new domctl to provide these info, 'pci_rdmforce' and 'rdm_force' when parse the config file. Certainly we may need to introduce and set two new fields in strcut domain, and since then we just use these fields to modify our current existing iommu callback inside to support our policy. I mean we just expose those associated RMRR entry. I think its easy to implement this since inside Xen we can know which entry is owned by which device. So this can benefit us to avoid modifying any tools codes and most Xen codes we already addressed.

Quoted 3 lines

Whether a domctl is the right approach I can't really tell with this somewhat vague description.

So based on our current patch, I generate a draft patch as follows to show my idea. Note I just tried to compile this.

tools/libxc/xc_domain.c +15 -0

index 009e351..ff22228 100644
@@ -1642,6 +1642,21 @@ int xc_assign_device(
     return do_domctl(xch, &domctl);
 }
 
+int xc_domain_device_setrdm(xc_interface *xch,
+                            uint32_t domid,
+                            uint32_t pci_rdmforce,
+                            uint32_t pci_dev_bitmap)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_set_rdm;
+    domctl.domain = domid;
+    domctl.u.set_rdm.pci_rdmforce = pci_rdmforce;
+    domctl.u.set_rdm.pci_dev_bitmap = pci_dev_bitmap;
+
+    return do_domctl(xch, &domctl);
+}
+
 int xc_get_device_group(
     xc_interface *xch,
     uint32_t domid,

tools/libxl/libxl_dm.c +13 -0

index 3e191c3..ca8b754 100644
@@ -90,6 +90,19 @@ const char *libxl__domain_device_model(libxl__gc *gc,
     return dm;
 }
 
+int libxl__domain_device_setrdm(libxl__gc *gc,
+                                const libxl_domain_build_info *info,
+                                uint32_t dm_domid)
+{
+    int ret;
+    libxl_ctx *ctx = libxl__gc_owner(gc);
+
+    ret = xc_domain_device_setrdm(ctx->xch, dm_domid, info->pci_rdmforce,
+                                  info->pci_dev_bitmap);
+
+    return ret;
+}
+
 const libxl_vnc_info *libxl__dm_vnc(const libxl_domain_config

*guest_config)

 {
     const libxl_vnc_info *vnc = NULL;

tools/libxl/libxl_internal.h +0 -0

index 4361421..06938ee 100644
@@ -1471,6 +1471,11 @@ _hidden int libxl__domain_build(libxl__gc *gc,
 /* for device model creation */
 _hidden const char *libxl__domain_device_model(libxl__gc *gc,
                                         const libxl_domain_build_info

*info);
+
+_hidden int libxl__domain_device_setrdm(libxl__gc *gc,
+ const libxl_domain_build_info *info,
+ uint32_t domid);
+
_hidden int libxl__need_xenpv_qemu(libxl__gc *gc,
int nr_consoles, libxl__device_console *consoles, int nr_vfbs, libxl_device_vfb *vfbs,

tools/libxl/libxl_types.idl +2 -0

index ca3f724..ed20823 100644
@@ -398,6 +398,8 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("kernel",           string),
     ("cmdline",          string),
     ("ramdisk",          string),
+    ("pci_rdmforce",   uint32),
+    ("pci_dev_bitmap",   uint32),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
                                        ("bios",

libxl_bios_type),
@@ -518,6 +520,7 @@ libxl_device_pci = Struct("device_pci", [

     ("power_mgmt", bool),
     ("permissive", bool),
     ("seize", bool),
+    ("rdmforce", bool),
     ])

libxl_device_vtpm = Struct("device_vtpm", [

tools/libxl/libxlu_pci.c +2 -0

index 26fb143..989eac8 100644
@@ -143,6 +143,8 @@ int xlu_pci_parse_bdf(XLU_Config *cfg, libxl_device_pci *pcidev, const char *str
                     pcidev->permissive = atoi(tok);
                 }else if ( !strcmp(optkey, "seize") ) {
                     pcidev->seize = atoi(tok);
+                }else if ( !strcmp(optkey, "rdmforce") ) {
+                    pcidev->rdmforce = atoi(tok);
                 }else{
                     XLU__PCI_ERR(cfg, "Unknown PCI BDF option: %s",

optkey);
                 }

tools/libxl/xl_cmdimpl.c +12 -0

index 3c9f146..64a1e51 100644
@@ -904,6 +904,7 @@ static void replace_string(char **str, const char *val)
     *str = xstrdup(val);
 }
 
+#define PCI_BDF(b,d,f) ((((b) & 0xff) << 8) | PCI_DEVFN(d,f))
 static void parse_config_data(const char *config_source,
                               const char *config_data,
                               int config_len,

@@ -919,6 +920,7 @@ static void parse_config_data(const char *config_source,
     int pci_msitranslate = 0;
     int pci_permissive = 0;
     int pci_seize = 0;
+    int pci_rdmforce = 0;
     int i, e;
 
     libxl_domain_create_info *c_info = &d_config->c_info;

@@ -1699,6 +1701,9 @@ skip_vfb:
     if (!xlu_cfg_get_long (config, "pci_seize", &l, 0))
         pci_seize = l;
 
+    if (!xlu_cfg_get_long (config, "pci_rdmforce", &l, 0))
+        pci_rdmforce = l;
+
     /* To be reworked (automatically enabled) once the auto ballooning
      * after guest starts is done (with PCI devices passed in). */
     if (c_info->type == LIBXL_DOMAIN_TYPE_PV) {

@@ -1719,9 +1724,16 @@ skip_vfb:
             pcidev->power_mgmt = pci_power_mgmt;
             pcidev->permissive = pci_permissive;
             pcidev->seize = pci_seize;
+            pcidev->rdmforce = pci_rdmforce;
             if (!xlu_pci_parse_bdf(config, pcidev, buf))
+            {
+                /* Just fake this wit a bitmap. */
+                b_info.pci_dev_bitmap |=  1 << PCI_BDF(pcidev->bus, pcidev->dev,
+                                                       pcidev->func);
                 d_config->num_pcidevs++;
+            }
         }
+        b_info.pci_rdmforce = pci_rdmforce;
         if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
             libxl_defbool_set(&b_info->u.pv.e820_host, true);
     }

xen/arch/x86/hvm/vmx/vmx.c +0 -0

index dc18f1b..3bbd28f 100644
@@ -2434,6 +2434,7 @@ static void ept_handle_violation(unsigned long qualification, paddr_t gpa)
      * handle such a scenario as its own logic.
      */
     ret =

iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+ d,

                                            &gfn);
     if ( ret )
     {

xen/arch/x86/mm/p2m.c +0 -0

index 113d996..ba489ce 100644
@@ -691,6 +691,7 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
         if ( !is_hardware_domain(d) )
         {
             rc =

iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+ d,

                                                   &gfn);
             /* We always avoid populating reserved device memory. */
             if ( rc == 1 )

xen/common/compat/memory.c +0 -0

index af613b9..cd99ba7 100644
@@ -315,6 +315,7 @@ int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
 
             grdm.used_entries = 0;
             rc =

iommu_get_reserved_device_memory(get_reserved_device_memory,
+ current->domain,

                                                   &grdm);

             if ( !rc && grdm.map.nr_entries < grdm.used_entries )

xen/common/mem_access.c +0 -0

index 4c84f88..e7973ee 100644
@@ -69,6 +69,7 @@ static int mem_access_check_rdm(struct domain *d, uint64_aligned_t start,
         {
             gfn = start + i;
             rc =

iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+ d,

                                                   &gfn);
             if ( rc < 0 )
                 printk(XENLOG_WARNING

xen/common/memory.c +1 -0

index 2177c56..21db828 100644
@@ -1140,6 +1140,7 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 
         grdm.used_entries = 0;
         rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
+                                              current->domain,
                                               &grdm);
 
         if ( !rc && grdm.map.nr_entries < grdm.used_entries )

xen/drivers/passthrough/iommu.c +3 -2

index 7c17e8d..7a5c66d 100644
@@ -344,14 +344,15 @@ void iommu_crash_shutdown(void)
     iommu_enabled = iommu_intremap = 0;
 }
 
-int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+int iommu_get_reserved_device_memory(iommu_grdm_t *func, struct domain* d,
+                                     void *ctxt)
 {
     const struct iommu_ops *ops = iommu_get_ops();
 
     if ( !iommu_enabled || !ops->get_reserved_device_memory )
         return 0;
 
-    return ops->get_reserved_device_memory(func, ctxt);
+    return ops->get_reserved_device_memory(func, d, ctxt);
 }
 
 bool_t iommu_has_feature(struct domain *d, enum iommu_feature feature)

xen/drivers/passthrough/pci.c +12 -0

index 1eba833..ee689ff 100644
@@ -1540,6 +1540,18 @@ int iommu_do_pci_domctl(
         }
         break;
 
+    case XEN_DOMCTL_set_rdm:
+        if ( unlikely(d->is_dying) )
+        {
+            ret = -EINVAL;
+            break;
+        }
+
+        d->arch.hvm_domain.pci_rdmforce = domctl->u.set_rdm.pci_rdmforce;
+        d->arch.hvm_domain.pci_dev_bitmap = domctl->u.set_rdm.pci_dev_bitmap;
+
+        break;
+
     case XEN_DOMCTL_assign_device:
         if ( unlikely(d->is_dying) )
         {

xen/drivers/passthrough/vtd/dmar.c +1 -0

index 546eca9..138840c 100644
@@ -28,6 +28,7 @@
 #include <xen/xmalloc.h>
 #include <xen/pci.h>
 #include <xen/pci_regs.h>
+#include <xen/sched.h>
 #include <asm/string.h>
 #include "dmar.h"
 #include "iommu.h"

@@ -921,18 +922,28 @@ int platform_supports_x2apic(void)
     return cpu_has_x2apic && ((dmar_flags & mask) ==

ACPI_DMAR_INTR_REMAP);
}

-int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt) +int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, struct domain *d,
+ void *ctxt)

 {
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;

+ int i;
+ u16 bdf;

- list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+ for_each_rmrr_device ( rmrr, bdf, i )

- rc = func(PFN_DOWN(rmrr->base_address),
- PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
- ctxt);
- if ( rc )
- break;
+ if ( d->arch.hvm_domain.pci_rdmforce )
+ {
+ if ( d->arch.hvm_domain.pci_dev_bitmap & (uint32_t)(1 << bdf) )
+ {
+ rc = func(PFN_DOWN(rmrr->base_address),

+                                   PFN_UP(rmrr->end_address) -
+                                   PFN_DOWN(rmrr->base_address),
+                                   ctxt);

+ if ( rc )
+ break;
+ }
+ }

     }

     return rc;

xen/drivers/passthrough/vtd/extern.h +2 -1

index f9ee9b0..df9fed3 100644
@@ -75,7 +75,8 @@ int domain_context_mapping_one(struct domain *domain, struct iommu *iommu,
                                u8 bus, u8 devfn, const struct pci_dev *);
 int domain_context_unmap_one(struct domain *domain, struct iommu *iommu,
                              u8 bus, u8 devfn);
-int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt);
+int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, struct domain *d,
+                                           void *ctxt);
 
 unsigned int io_apic_read_remap_rte(unsigned int apic, unsigned int reg);
 void io_apic_write_remap_rte(unsigned int apic,

xen/include/asm-x86/hvm/domain.h +3 -0

index 2757c7f..5dab8dd 100644
@@ -90,6 +90,9 @@ struct hvm_domain {
     /* Cached CF8 for guest PCI config cycles */
     uint32_t                pci_cf8;
 
+    uint32_t                pci_rdmforce;
+    uint32_t                pci_dev_bitmap;
+
     struct pl_time         pl_time;
 
     struct hvm_io_handler *io_handler;

xen/include/public/domctl.h +11 -1

index 58b19e7..8b7cd5b 100644
@@ -484,6 +484,14 @@ struct xen_domctl_assign_device {
 typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
 
+/* Control whether/how we check and reserve device memory. */
+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;
+    uint32_t  pci_dev_bitmap;
+};
+typedef struct xen_domctl_set_rdm xen_domctl_set_rdm_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_rdm_t);
+
 /* Retrieve sibling devices infomation of machine_sbdf */
 /* XEN_DOMCTL_get_device_group */
 struct xen_domctl_get_device_group {

@@ -1056,6 +1064,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_set_vcpu_msrs                 73
 #define XEN_DOMCTL_setvnumainfo                  74
 #define XEN_DOMCTL_psr_cmt_op                    75
+#define XEN_DOMCTL_set_rdm                       76
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002

@@ -1118,7 +1127,8 @@ struct xen_domctl {
         struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
         struct xen_domctl_vnuma             vnuma;
         struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[120];
     } u;
 };
 typedef struct xen_domctl xen_domctl_t;

xen/include/xen/iommu.h +0 -0

index 409f6f8..adf3d52 100644
@@ -158,14 +158,14 @@ struct iommu_ops {
     void (*crash_shutdown)(void);
     void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned

int page_count);
     void (*iotlb_flush_all)(struct domain *d);
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct domain *, void *);
     void (*dump_p2m_table)(struct domain *d);

};

void iommu_suspend(void);
void iommu_resume(void);
void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *); +int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain *, void *);

void iommu_share_p2m_table(struct domain *d);

Thanks
Tiejun

Jan Beulich

2014-11-06 10:06 UTC #79 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 490

Quoted 11 lines

Quoted 1 line

Quoted 1 line · On 06.11.14 at 10:28, <[email protected]> wrote:

xen/include/public/domctl.h +4 -0

@@ -484,6 +484,14 @@ struct xen_domctl_assign_device {
  typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
  DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
 
+/* Control whether/how we check and reserve device memory. */
+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;
+    uint32_t  pci_dev_bitmap;

So this allows for 32 devices; considering that you index the bitmap by BDF, all this covers are 0000:00:00.0 through 0000:00:03.7. Hardly enough to cover even a single pass through device (iirc the range above is fully used by emulated devices). And of course a much larger bitmap isn't a good solution either.

Nor am I really clear what you need the 32 bits of "pci_rdmforce" for, nor why this field isn't just being named "force". Without a comment alongside the fields, it's remaining guesswork anyway when and how this domctl is to be used. Looking at your change to intel_iommu_get_reserved_device_memory() the field appears to be simply redundant.

Quoted 6 lines

xen/include/xen/iommu.h +0 -0

@@ -158,14 +158,14 @@ struct iommu_ops {
      void (*crash_shutdown)(void);
      void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
      void (*iotlb_flush_all)(struct domain *d);

xen/include/xen/iommu.h struct iommu_ops { · line 168

Quoted 10 lines

xen/include/xen/iommu.h +2 -2

-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct domain *, void *);
      void (*dump_p2m_table)(struct domain *d);
  };
 
  void iommu_suspend(void);
  void iommu_resume(void);
  void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain *, void *);

I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).

Jan

Chen, Tiejun

author

2014-11-07 10:27 UTC #80 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 7 lines · Jan Beulich

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 06.11.14 at 10:28, <[email protected]> wrote:
xen/include/public/domctl.h +0 -0
@@ -484,6 +484,14 @@ struct xen_domctl_assign_device {
   typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
   DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
 

Quoted 6 lines

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 490
Quoted 4 lines
xen/include/public/domctl.h +4 -0
+/* Control whether/how we check and reserve device memory. */
+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;
+    uint32_t  pci_dev_bitmap;
So this allows for 32 devices; considering that you index the bitmap

Sorry its my fault when I just focus on figuring out a doable way. We need to cover 256 x 32 x 8 = 65536.

Quoted 3 lines

by BDF, all this covers are 0000:00:00.0 through 0000:00:03.7. Hardly enough to cover even a single pass through device (iirc the range above is fully used by emulated devices). And of course a

Are you saying Xen restrict some BDFs specific to emulate some devices? But I don't see these associated codes.

Quoted 1 line

much larger bitmap isn't a good solution either.

So I guess I need to reconstruct something new, please see the new draft codes.

Quoted 3 lines

Nor am I really clear what you need the 32 bits of "pci_rdmforce" for, nor why this field isn't just being named "force". Without a

This variable can tell Xen to force check and reserve all RMRR ranges in that case the user is sure he's going to hotplug a device later but indeed, he really don't assign any device while creating a VM.

Please see the new draft codes as well.

Quoted 16 lines

comment alongside the fields, it's remaining guesswork anyway when and how this domctl is to be used. Looking at your change to intel_iommu_get_reserved_device_memory() the field appears to be simply redundant.

Quoted 11 lines

xen/include/xen/iommu.h +1 -1

@@ -158,14 +158,14 @@ struct iommu_ops {
       void (*crash_shutdown)(void);
       void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
       void (*iotlb_flush_all)(struct domain *d);
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct domain *, void *);
       void (*dump_p2m_table)(struct domain *d);
   };

Quoted 10 lines

xen/include/xen/iommu.h struct iommu_ops { · line 168
Quoted 5 lines
xen/include/xen/iommu.h +1 -1
   void iommu_suspend(void);
   void iommu_resume(void);
   void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain *, void *);
I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).

I'm not 100% sure if we can call current->domain in all scenarios. If you can help me confirm this I'd really like to remove this change :) Now I assume this should be true as follows:

tools/libxc/include/xenctrl.h +6 -0

index 30764b4..5957d2e 100644
@@ -2036,6 +2036,12 @@ int xc_assign_device(xc_interface *xch,
                      uint32_t domid,
                      uint32_t machine_bdf);
 
+int xc_domain_device_setrdm(xc_interface *xch,
+                            uint32_t domid,
+                            uint32_t num_pcidevs,
+                            uint32_t pci_rdmforce,
+                            struct xen_guest_pcidev_info *pcidevs);
+
 int xc_get_device_group(xc_interface *xch,
                      uint32_t domid,
                      uint32_t machine_bdf,

tools/libxc/xc_domain.c +28 -0

index 009e351..f38b400 100644
@@ -1642,6 +1642,34 @@ int xc_assign_device(
     return do_domctl(xch, &domctl);
 }
 
+int xc_domain_device_setrdm(xc_interface *xch,
+                            uint32_t domid,
+                            uint32_t num_pcidevs,
+                            uint32_t pci_rdmforce,
+                            struct xen_guest_pcidev_info *pcidevs)
+{
+    int ret;
+    DECLARE_DOMCTL;
+    DECLARE_HYPERCALL_BOUNCE(pcidevs,
+                             num_pcidevs*sizeof(xen_guest_pcidev_info_t),
+                             XC_HYPERCALL_BUFFER_BOUNCE_IN);
+
+    if ( xc_hypercall_bounce_pre(xch, pcidevs) )
+        return -1;
+
+    domctl.cmd = XEN_DOMCTL_set_rdm;
+    domctl.domain = domid;
+    domctl.u.set_rdm.pci_rdmforce = pci_rdmforce;
+    domctl.u.set_rdm.num_pcidevs = num_pcidevs;
+    set_xen_guest_handle(domctl.u.set_rdm.pcidevs, pcidevs);
+
+    ret = do_domctl(xch, &domctl);
+
+    xc_hypercall_bounce_post(xch, pcidevs);
+
+    return ret;
+}
+
 int xc_get_device_group(
     xc_interface *xch,
     uint32_t domid,

tools/libxc/xc_domain_restore.c +2 -2

index d8bd9b3..ac48a82 100644
@@ -2165,8 +2165,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
 
         if ( !ext_vcpucontext )
             goto vcpu_ext_state_restore;
-        memcpy(&domctl.u.ext_vcpucontext, vcpup, 128);
-        vcpup += 128;
+        memcpy(&domctl.u.ext_vcpucontext, vcpup, 112);
+        vcpup += 112;
         domctl.cmd = XEN_DOMCTL_set_ext_vcpucontext;
         domctl.domain = dom;
         frc = xc_domctl(xch, &domctl);

tools/libxl/libxl_create.c +3 -0

index b1ff5ae..2429416 100644
@@ -860,6 +860,9 @@ static void initiate_domain_create(libxl__egc *egc,
     ret = libxl__domain_build_info_setdefault(gc, &d_config->b_info);
     if (ret) goto error_out;
 
+    ret = libxl__domain_device_setrdm(gc, d_config, domid);
+    if (ret) goto error_out;
+
     if (!sched_params_valid(gc, domid, &d_config->b_info.sched_params)) {
         LOG(ERROR, "Invalid scheduling parameters\n");
         ret = ERROR_INVAL;

tools/libxl/libxl_dm.c +36 -0

index 3e191c3..9e402d1 100644
@@ -90,6 +90,42 @@ const char *libxl__domain_device_model(libxl__gc *gc,
     return dm;
 }
 
+int libxl__domain_device_setrdm(libxl__gc *gc,
+                                libxl_domain_config *d_config,
+                                uint32_t dm_domid)
+{
+    int i, ret;
+    libxl_ctx *ctx = libxl__gc_owner(gc);
+    struct xen_guest_pcidev_info *pcidevs = NULL;
+
+    if ( d_config->num_pcidevs )
+    {
+        pcidevs = malloc(d_config->num_pcidevs*sizeof(xen_guest_pcidev_info_t));
+        if ( pcidevs )
+        {
+            for (i = 0; i < d_config->num_pcidevs; i++)
+            {
+                pcidevs[i].func = d_config->pcidevs[i].func;
+                pcidevs[i].dev = d_config->pcidevs[i].dev;
+                pcidevs[i].bus = d_config->pcidevs[i].bus;
+                pcidevs[i].domain = d_config->pcidevs[i].domain;
+                pcidevs[i].rdmforce = d_config->pcidevs[i].rdmforce;
+            }
+        }
+        else
+            LIBXL__LOG(CTX, LIBXL__LOG_ERROR,
+                               "Can't allocate for pcidevs.");
+    }
+
+    ret = xc_domain_device_setrdm(ctx->xch, dm_domid,
+                                  (uint32_t)d_config->num_pcidevs,
+                                  d_config->b_info.pci_rdmforce,
+                                  pcidevs);
+    free(pcidevs);
+
+    return ret;
+}
+
 const libxl_vnc_info *libxl__dm_vnc(const libxl_domain_config

*guest_config)

 {
     const libxl_vnc_info *vnc = NULL;

tools/libxl/libxl_internal.h +0 -0

index 4361421..c48a0e6 100644
@@ -1471,6 +1471,11 @@ _hidden int libxl__domain_build(libxl__gc *gc,
 /* for device model creation */
 _hidden const char *libxl__domain_device_model(libxl__gc *gc,
                                         const libxl_domain_build_info

*info);
+
+_hidden int libxl__domain_device_setrdm(libxl__gc *gc,
+ libxl_domain_config *d_config,
+ uint32_t domid);
+
_hidden int libxl__need_xenpv_qemu(libxl__gc *gc,
int nr_consoles, libxl__device_console *consoles, int nr_vfbs, libxl_device_vfb *vfbs,

tools/libxl/libxl_types.idl +1 -0

index ca3f724..b700263 100644
@@ -398,6 +398,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("kernel",           string),
     ("cmdline",          string),
     ("ramdisk",          string),
+    ("pci_rdmforce",   uint32),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
                                        ("bios",

libxl_bios_type),
@@ -518,6 +519,7 @@ libxl_device_pci = Struct("device_pci", [

     ("power_mgmt", bool),
     ("permissive", bool),
     ("seize", bool),
+    ("rdmforce", bool),
     ])

libxl_device_vtpm = Struct("device_vtpm", [

tools/libxl/libxlu_pci.c +2 -0

index 26fb143..989eac8 100644
@@ -143,6 +143,8 @@ int xlu_pci_parse_bdf(XLU_Config *cfg, libxl_device_pci *pcidev, const char *str
                     pcidev->permissive = atoi(tok);
                 }else if ( !strcmp(optkey, "seize") ) {
                     pcidev->seize = atoi(tok);
+                }else if ( !strcmp(optkey, "rdmforce") ) {
+                    pcidev->rdmforce = atoi(tok);
                 }else{
                     XLU__PCI_ERR(cfg, "Unknown PCI BDF option: %s",

optkey);
                 }

tools/libxl/xl_cmdimpl.c +7 -0

index 3c9f146..436b4f3 100644
@@ -904,6 +904,7 @@ static void replace_string(char **str, const char *val)
     *str = xstrdup(val);
 }
 
+#define PCI_BDF(b,d,f) ((((b) & 0xff) << 8) | PCI_DEVFN(d,f))
 static void parse_config_data(const char *config_source,
                               const char *config_data,
                               int config_len,

@@ -919,6 +920,7 @@ static void parse_config_data(const char *config_source,
     int pci_msitranslate = 0;
     int pci_permissive = 0;
     int pci_seize = 0;
+    int pci_rdmforce = 0;
     int i, e;
 
     libxl_domain_create_info *c_info = &d_config->c_info;

@@ -1699,6 +1701,9 @@ skip_vfb:
     if (!xlu_cfg_get_long (config, "pci_seize", &l, 0))
         pci_seize = l;
 
+    if (!xlu_cfg_get_long (config, "pci_rdmforce", &l, 0))
+        pci_rdmforce = l;
+
     /* To be reworked (automatically enabled) once the auto ballooning
      * after guest starts is done (with PCI devices passed in). */
     if (c_info->type == LIBXL_DOMAIN_TYPE_PV) {

@@ -1719,9 +1724,11 @@ skip_vfb:
             pcidev->power_mgmt = pci_power_mgmt;
             pcidev->permissive = pci_permissive;
             pcidev->seize = pci_seize;
+            pcidev->rdmforce = pci_rdmforce;
             if (!xlu_pci_parse_bdf(config, pcidev, buf))
                 d_config->num_pcidevs++;
         }
+        d_config->b_info.pci_rdmforce = pci_rdmforce;
         if (d_config->num_pcidevs && c_info->type == LIBXL_DOMAIN_TYPE_PV)
             libxl_defbool_set(&b_info->u.pv.e820_host, true);
     }

xen/drivers/passthrough/pci.c +28 -0

index 1eba833..daf259e 100644
@@ -1540,6 +1540,34 @@ int iommu_do_pci_domctl(
         }
         break;
 
+    case XEN_DOMCTL_set_rdm:
+    {
+        struct xen_domctl_set_rdm *xdsr = &domctl->u.set_rdm;
+        struct xen_guest_pcidev_info *pcidevs;
+        int i;
+
+        pcidevs = xmalloc_array(xen_guest_pcidev_info_t,
+                                domctl->u.set_rdm.num_pcidevs);
+        if ( pcidevs == NULL )
+        {
+            return -ENOMEM;
+        }
+
+        for ( i = 0; i < xdsr->num_pcidevs; ++i )
+        {
+            if ( __copy_from_guest_offset(pcidevs, xdsr->pcidevs, i, 1) )
+            {
+                xfree(pcidevs);
+                return -EFAULT;
+            }
+        }
+
+        d->arch.hvm_domain.pci_rdmforce = domctl->u.set_rdm.pci_rdmforce;
+        d->arch.hvm_domain.num_pcidevs = domctl->u.set_rdm.num_pcidevs;
+        d->arch.hvm_domain.pcidevs = pcidevs;
+    }
+        break;
+
     case XEN_DOMCTL_assign_device:
         if ( unlikely(d->is_dying) )
         {

xen/drivers/passthrough/vtd/dmar.c +30 -6

index 546eca9..4b35c04 100644
@@ -28,6 +28,7 @@
 #include <xen/xmalloc.h>
 #include <xen/pci.h>
 #include <xen/pci_regs.h>
+#include <xen/sched.h>
 #include <asm/string.h>
 #include "dmar.h"
 #include "iommu.h"

@@ -925,14 +926,37 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
 {
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;
+    int i, j;
+    u16 bdf, pt_bdf;
+    struct domain *d = current->domain;
 
-    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    for_each_rmrr_device ( rmrr, bdf, i )
     {
-        rc = func(PFN_DOWN(rmrr->base_address),
-                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
-                  ctxt);
-        if ( rc )
-            break;
+        if ( d->arch.hvm_domain.pci_rdmforce )
+        {
+            rc = func(PFN_DOWN(rmrr->base_address),
+                      PFN_UP(rmrr->end_address) -
+                      PFN_DOWN(rmrr->base_address),
+                      ctxt);
+            if ( rc )
+                break;
+        }
+        else
+        {
+            for ( j = 0; j < d->arch.hvm_domain.num_pcidevs; j++ )
+            {
+                pt_bdf = PCI_BDF(d->arch.hvm_domain.pcidevs[j].bus,
+                                 d->arch.hvm_domain.pcidevs[j].dev,
+                                 d->arch.hvm_domain.pcidevs[j].func);
+                if ( pt_bdf == bdf )
+                    rc = func(PFN_DOWN(rmrr->base_address),
+                              PFN_UP(rmrr->end_address) -
+                              PFN_DOWN(rmrr->base_address),
+                              ctxt);
+                if ( rc )
+                    break;
+            }
+        }
     }
 
     return rc;

xen/include/asm-x86/hvm/domain.h +4 -0

index 2757c7f..b18ce40 100644
@@ -90,6 +90,10 @@ struct hvm_domain {
     /* Cached CF8 for guest PCI config cycles */
     uint32_t                pci_cf8;
 
+    uint32_t                pci_rdmforce;
+    uint32_t                num_pcidevs;
+    struct xen_guest_pcidev_info      *pcidevs;
+
     struct pl_time         pl_time;
 
     struct hvm_io_handler *io_handler;

xen/include/public/domctl.h +21 -1

index 58b19e7..4e47fac 100644
@@ -484,6 +484,24 @@ struct xen_domctl_assign_device {
 typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
 
+struct xen_guest_pcidev_info {
+    uint8_t func;
+    uint8_t dev;
+    uint8_t bus;
+    int domain;
+    int rdmforce;
+};
+typedef struct xen_guest_pcidev_info xen_guest_pcidev_info_t;
+DEFINE_XEN_GUEST_HANDLE(xen_guest_pcidev_info_t);
+/* Control whether/how we check and reserve device memory. */
+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;
+    uint32_t  num_pcidevs;
+    XEN_GUEST_HANDLE(xen_guest_pcidev_info_t) pcidevs;
+};
+typedef struct xen_domctl_set_rdm xen_domctl_set_rdm_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_rdm_t);
+
 /* Retrieve sibling devices infomation of machine_sbdf */
 /* XEN_DOMCTL_get_device_group */
 struct xen_domctl_get_device_group {

@@ -1056,6 +1074,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_set_vcpu_msrs                 73
 #define XEN_DOMCTL_setvnumainfo                  74
 #define XEN_DOMCTL_psr_cmt_op                    75
+#define XEN_DOMCTL_set_rdm                       76
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002

@@ -1118,7 +1137,8 @@ struct xen_domctl {
         struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
         struct xen_domctl_vnuma             vnuma;
         struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[112];
     } u;
 };
 typedef struct xen_domctl xen_domctl_t;

Thanks
Tiejun

Jan Beulich

2014-11-07 11:08 UTC #81 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 3 lines

Quoted 1 line
Quoted 1 line · On 07.11.14 at 11:27, <[email protected]> wrote:
Are you saying Xen restrict some BDFs specific to emulate some devices? But I don't see these associated codes.

I didn't say so. All I said that some of the SBDFs are being used by them.

Quoted 21 lines

Quoted 11 lines

xen/include/xen/iommu.h +1 -1

@@ -158,14 +158,14 @@ struct iommu_ops {
       void (*crash_shutdown)(void);
       void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
       void (*iotlb_flush_all)(struct domain *d);
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct domain *, void *);
       void (*dump_p2m_table)(struct domain *d);
   };

Quoted 9 lines

xen/include/xen/iommu.h struct iommu_ops { · line 168
Quoted 5 lines
xen/include/xen/iommu.h +1 -1
   void iommu_suspend(void);
   void iommu_resume(void);
   void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain *, void *);
I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).

Quoted 3 lines

I'm not 100% sure if we can call current->domain in all scenarios. If you can help me confirm this I'd really like to remove this change :) Now I assume this should be true as follows:

Which is wrong, and not what I said. Instead you should pass the domain as part of the context that gets made available to the callback function.

Quoted 17 lines

xen/drivers/passthrough/pci.c +11 -0

@@ -1540,6 +1540,34 @@ int iommu_do_pci_domctl(
          }
          break;
 
+    case XEN_DOMCTL_set_rdm:
+    {
+        struct xen_domctl_set_rdm *xdsr = &domctl->u.set_rdm;
+        struct xen_guest_pcidev_info *pcidevs;
+        int i;
+
+        pcidevs = xmalloc_array(xen_guest_pcidev_info_t,
+                                domctl->u.set_rdm.num_pcidevs);
+        if ( pcidevs == NULL )
+        {
+            return -ENOMEM;

xen/drivers/passthrough/pci.c int iommu_do_pci_domctl( · line 1563

Quoted 10 lines

xen/drivers/passthrough/pci.c +10 -0

+        }
+
+        for ( i = 0; i < xdsr->num_pcidevs; ++i )
+        {
+            if ( __copy_from_guest_offset(pcidevs, xdsr->pcidevs, i, 1) )
+            {
+                xfree(pcidevs);
+                return -EFAULT;
+            }
+        }

I don't see the need for a loop here. And you definitely can't use the double-underscore-prefixed variant the way you do.

xen/include/asm-x86/hvm/domain.h struct hvm_domain { · line 93

Quoted 7 lines

xen/include/asm-x86/hvm/domain.h +1 -0

@@ -90,6 +90,10 @@ struct hvm_domain {
      /* Cached CF8 for guest PCI config cycles */
      uint32_t                pci_cf8;
 
+    uint32_t                pci_rdmforce;

I still don't see why this is a uint32_t.

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 493

Quoted 13 lines

xen/include/public/domctl.h +7 -0

@@ -484,6 +484,24 @@ struct xen_domctl_assign_device {
  typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
  DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);
 
+struct xen_guest_pcidev_info {
+    uint8_t func;
+    uint8_t dev;
+    uint8_t bus;
+    int domain;
+    int rdmforce;
+};

Please see struct physdev_pci_device_add for how to properly and space efficiently express what you need. And of course I'd expect the code to actually use all fields you specify here - neither domain (which really ought to be named segment if it is what I think it is meant to be) nor rdmforce are being used anywhere. Plus - again - just "force" would be sufficient as a name, provided the field is needed at all.

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 498

Quoted 2 lines

xen/include/public/domctl.h +2 -0

+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;

Same comment as on the field added to "struct hvm_domain".

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 500

Quoted 2 lines

xen/include/public/domctl.h +2 -0

+    uint32_t  num_pcidevs;
+    XEN_GUEST_HANDLE(xen_guest_pcidev_info_t) pcidevs;

Did you _at all_ look at any of the other domctls when adding this? There's not a single use of XEN_GUEST_HANDLE() in the whole file.

xen/include/public/domctl.h struct xen_domctl { · line 1141

Quoted 7 lines

xen/include/public/domctl.h +2 -1

@@ -1118,7 +1137,8 @@ struct xen_domctl {
          struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
          struct xen_domctl_vnuma             vnuma;
          struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[112];

Why are you altering the padding size here?

Jan

Chen, Tiejun

author

2014-11-11 06:32 UTC #82 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 28 lines · Jan Beulich

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 07.11.14 at 11:27, <[email protected]> wrote:
Are you saying Xen restrict some BDFs specific to emulate some devices? But I don't see these associated codes.
I didn't say so. All I said that some of the SBDFs are being used by them.
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
xen/include/xen/iommu.h +0 -0
@@ -158,14 +158,14 @@ struct iommu_ops {
        void (*crash_shutdown)(void);
        void (*iotlb_flush)(struct domain *d, unsigned long gfn, unsigned int page_count);
        void (*iotlb_flush_all)(struct domain *d);
xen/include/xen/iommu.h struct iommu_ops { · line 168
Quoted 10 lines
xen/include/xen/iommu.h +2 -2
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct domain *, void *);
        void (*dump_p2m_table)(struct domain *d);
    };
 
    void iommu_suspend(void);
    void iommu_resume(void);
    void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain *, void *);
I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).

Quoted 7 lines

Quoted 3 lines
I'm not 100% sure if we can call current->domain in all scenarios. If you can help me confirm this I'd really like to remove this change :) Now I assume this should be true as follows:
Which is wrong, and not what I said. Instead you should pass the domain as part of the context that gets made available to the callback function.

Okay I will try to go there.

Quoted 23 lines

xen/drivers/passthrough/pci.c +16 -0

@@ -1540,6 +1540,34 @@ int iommu_do_pci_domctl(
           }
           break;
 
+    case XEN_DOMCTL_set_rdm:
+    {
+        struct xen_domctl_set_rdm *xdsr = &domctl->u.set_rdm;
+        struct xen_guest_pcidev_info *pcidevs;
+        int i;
+
+        pcidevs = xmalloc_array(xen_guest_pcidev_info_t,
+                                domctl->u.set_rdm.num_pcidevs);
+        if ( pcidevs == NULL )
+        {
+            return -ENOMEM;
+        }
+
+        for ( i = 0; i < xdsr->num_pcidevs; ++i )
+        {
+            if ( __copy_from_guest_offset(pcidevs, xdsr->pcidevs, i, 1) )

Quoted 8 lines

xen/drivers/passthrough/pci.c int iommu_do_pci_domctl( · line 1563
Quoted 5 lines
xen/drivers/passthrough/pci.c +5 -0
+            {
+                xfree(pcidevs);
+                return -EFAULT;
+            }
+        }
I don't see the need for a loop here. And you definitely can't use the double-underscore-prefixed variant the way you do.

Do you mean this line?

copy_from_guest_offset(pcidevs, xdsr->pcidevs, 0, xdsr->num_pcidevs*sizeof(xen_guest_pcidev_info_t))

Quoted 10 lines

xen/include/asm-x86/hvm/domain.h struct hvm_domain { · line 93
Quoted 8 lines
xen/include/asm-x86/hvm/domain.h +1 -0
@@ -90,6 +90,10 @@ struct hvm_domain {
       /* Cached CF8 for guest PCI config cycles */
       uint32_t                pci_cf8;
 
+    uint32_t                pci_rdmforce;
I still don't see why this is a uint32_t.

How about bool_t?

Quoted 7 lines

xen/include/public/domctl.h +0 -0

@@ -484,6 +484,24 @@ struct xen_domctl_assign_device {
   typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
   DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);

Quoted 10 lines

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 493
Quoted 7 lines
xen/include/public/domctl.h +7 -0
+struct xen_guest_pcidev_info {
+    uint8_t func;
+    uint8_t dev;
+    uint8_t bus;
+    int domain;
+    int rdmforce;
+};
Please see struct physdev_pci_device_add for how to properly and space efficiently express what you need. And of course I'd expect

Yes. Actually I ever considered this point but I just think we may need to keep a complete set of fields.

You're right and anywhere what we should do is focusing on on-demand.

Quoted 5 lines

the code to actually use all fields you specify here - neither domain (which really ought to be named segment if it is what I think it is meant to be) nor rdmforce are being used anywhere. Plus - again - just "force" would be sufficient as a name, provided the field is needed at all.

Okay I can use 'force' directly.

In Xen side we have 'bool_t', but we have 'bool' in tools side. So how to define this in xen/include/public/domctl.h?

Quoted 5 lines

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 498
Quoted 3 lines
xen/include/public/domctl.h +2 -0
+struct xen_domctl_set_rdm {
+    uint32_t  pci_rdmforce;
Same comment as on the field added to "struct hvm_domain".

Ditto.

Quoted 6 lines

xen/include/public/domctl.h struct xen_domctl_assign_device { · line 500
Quoted 3 lines
xen/include/public/domctl.h +2 -0
+    uint32_t  num_pcidevs;
+    XEN_GUEST_HANDLE(xen_guest_pcidev_info_t) pcidevs;
Did you _at all_ look at any of the other domctls when adding this? There's not a single use of XEN_GUEST_HANDLE() in the whole file.

Looks I should do this,

XEN_GUEST_HANDLE_64(xen_guest_pcidev_info_t) pcidevs;

Quoted 10 lines

xen/include/public/domctl.h struct xen_domctl { · line 1141

Quoted 8 lines

xen/include/public/domctl.h +2 -1

@@ -1118,7 +1137,8 @@ struct xen_domctl {
           struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
           struct xen_domctl_vnuma             vnuma;
           struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[112];

Why are you altering the padding size here?

As I understand we should shrink this pad when we introduce new filed, shouldn't we?

Thanks
Tiejun

Chen, Tiejun

author

2014-11-11 07:49 UTC #83 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 28 lines

Quoted 28 lines
Quoted 27 lines
Quoted 13 lines
Quoted 13 lines
xen/include/xen/iommu.h +1 -1
@@ -158,14 +158,14 @@ struct iommu_ops {
        void (*crash_shutdown)(void);
        void (*iotlb_flush)(struct domain *d, unsigned long gfn,
 unsigned int page_count);
        void (*iotlb_flush_all)(struct domain *d);
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct
 domain *, void *);
        void (*dump_p2m_table)(struct domain *d);
    };
 
Quoted 10 lines
xen/include/xen/iommu.h struct iommu_ops { · line 171
Quoted 6 lines
xen/include/xen/iommu.h +1 -1
    void iommu_suspend(void);
    void iommu_resume(void);
    void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain
 *, void *);
I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).
I'm not 100% sure if we can call current->domain in all scenarios. If you can help me confirm this I'd really like to remove this change :) Now I assume this should be true as follows:

Quoted 6 lines

Quoted 3 lines
Which is wrong, and not what I said. Instead you should pass the domain as part of the context that gets made available to the callback function.
Okay I will try to go there.

Are you saying this change?

@@ -898,14 +899,36 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)

 {
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;
+    int i, j;
+    u16 bdf, pt_bdf;
+    struct domain *d = ctxt->domain;

- list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+ for_each_rmrr_device ( rmrr, bdf, i )

- rc = func(PFN_DOWN(rmrr->base_address),
- PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
- ctxt);
- if ( rc )
- break;
+ if ( d->arch.hvm_domain.pci_force )
+ {
+ rc = func(PFN_DOWN(rmrr->base_address),

+                      PFN_UP(rmrr->end_address) -
+                      PFN_DOWN(rmrr->base_address),
+                      ctxt);
+            if ( rc )
+                break;
+        }
+        else
+        {

+ for ( j = 0; j < d->arch.hvm_domain.num_pcidevs; j++ )
+ {

But,

dmar.c: In function 'intel_iommu_get_reserved_device_memory'" dmar.c:904:28: error: dereferencing 'void *' pointer [-Werror]

     struct domain *d = ctxt->domain;
                            ^

dmar.c:904:28: error: request for member 'domain' in something not a structure or union
cc1: all warnings being treated as errors
make[6]: *** [dmar.o] Error 1
make[6]: *** Waiting for unfinished jobs.

Unless we move all check inside each callback functions.

Thanks
Tiejun

Jan Beulich

2014-11-11 09:03 UTC #84 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 75 lines

Quoted 29 lines

Quoted 29 lines · On 11.11.14 at 08:49, <[email protected]> wrote:
Quoted 27 lines
Quoted 13 lines
Quoted 13 lines
xen/include/xen/iommu.h +1 -1
@@ -158,14 +158,14 @@ struct iommu_ops {
        void (*crash_shutdown)(void);
        void (*iotlb_flush)(struct domain *d, unsigned long gfn,
 unsigned int page_count);
        void (*iotlb_flush_all)(struct domain *d);
-    int (*get_reserved_device_memory)(iommu_grdm_t *, void *);
+    int (*get_reserved_device_memory)(iommu_grdm_t *, struct
 domain *, void *);
        void (*dump_p2m_table)(struct domain *d);
    };
 
Quoted 10 lines
xen/include/xen/iommu.h struct iommu_ops { · line 171
Quoted 6 lines
xen/include/xen/iommu.h +1 -1
    void iommu_suspend(void);
    void iommu_resume(void);
    void iommu_crash_shutdown(void);
-int iommu_get_reserved_device_memory(iommu_grdm_t *, void *);
+int iommu_get_reserved_device_memory(iommu_grdm_t *, struct domain
 *, void *);
I don't see why these generic interfaces would need to change; the only thing that would seem to need changing is the callback function (and of course the context passed to it).
I'm not 100% sure if we can call current->domain in all scenarios. If you can help me confirm this I'd really like to remove this change :) Now I assume this should be true as follows:

Quoted 6 lines

Quoted 3 lines
Which is wrong, and not what I said. Instead you should pass the domain as part of the context that gets made available to the callback function.
Okay I will try to go there.

Are you saying this change?

xen/drivers/passthrough/vtd/dmar.c +17 -6

@@ -898,14 +899,36 @@ int 
 intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
  {
      struct acpi_rmrr_unit *rmrr;
      int rc = 0;
+    int i, j;
+    u16 bdf, pt_bdf;
+    struct domain *d = ctxt->domain;
 
-    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    for_each_rmrr_device ( rmrr, bdf, i )
      {
-        rc = func(PFN_DOWN(rmrr->base_address),
-                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
-                  ctxt);
-        if ( rc )
-            break;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            rc = func(PFN_DOWN(rmrr->base_address),
+                      PFN_UP(rmrr->end_address) -
+                      PFN_DOWN(rmrr->base_address),
+                      ctxt);
+            if ( rc )
+                break;
+        }
+        else
+        {
+            for ( j = 0; j < d->arch.hvm_domain.num_pcidevs; j++ )
+            {
 
 But,

dmar.c: In function 'intel_iommu_get_reserved_device_memory'" dmar.c:904:28: error: dereferencing 'void *' pointer [-Werror]

      struct domain *d = ctxt->domain;
                             ^

Quoted 7 lines

dmar.c:904:28: error: request for member 'domain' in something not a structure or union
cc1: all warnings being treated as errors
make[6]: *** [dmar.o] Error 1
make[6]: *** Waiting for unfinished jobs.
Unless we move all check inside each callback functions.

That's what you should do imo, albeit I realize that the comparing of the specific SBDFs will need additional consideration.

Jan

Jan Beulich

2014-11-11 09:06 UTC #85 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines

Quoted 3 lines
Quoted 2 lines · On 11.11.14 at 10:03, <[email protected]> wrote:
Quoted 1 line · On 11.11.14 at 08:49, <[email protected]> wrote:
Unless we move all check inside each callback functions.
That's what you should do imo, albeit I realize that the comparing of the specific SBDFs will need additional consideration.

Part of which would involve re-considering whether device assignment shouldn't be done before memory population in the tool stack.

Jan

Chen, Tiejun

author

2014-11-11 09:42 UTC #86 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 5 lines · Jan Beulich

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines · On 11.11.14 at 10:03, <[email protected]> wrote:
Quoted 1 line · On 11.11.14 at 08:49, <[email protected]> wrote:
Unless we move all check inside each callback functions.
That's what you should do imo, albeit I realize that the comparing

Yes, I can do this in all existing callback functions but I'm somewhat afraid when other guys want to introduce new callback function, who can guarantee this should be done as well?

Quoted 6 lines

Quoted 1 line
of the specific SBDFs will need additional consideration.
Part of which would involve re-considering whether device assignment shouldn't be done before memory population in the tool stack.

Yes, after we introduce this new domctl, we just make sure this domctl should be called before memory population no matter when we do assign a device as passthrough.

Thanks
Tiejun

Jan Beulich

2014-11-11 10:07 UTC #87 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 10 lines

Quoted 1 line
Quoted 1 line · On 11.11.14 at 10:42, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Part of which would involve re-considering whether device assignment shouldn't be done before memory population in the tool stack.
Yes, after we introduce this new domctl, we just make sure this domctl should be called before memory population no matter when we do assign a device as passthrough.

You didn't think through the implications then: If device assignment happens early enough, there's no need to report SBDF tuples via a new domctl (or only if we want to still allow for post-boot assignment of affected devices without using the global enforcement flag, in which case assigning devices early at boot time is pointless).

Jan

Chen, Tiejun

author

2014-11-12 01:36 UTC #88 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 7 lines · Jan Beulich

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 11.11.14 at 10:42, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Part of which would involve re-considering whether device assignment shouldn't be done before memory population in the tool stack.

Quoted 6 lines

Quoted 3 lines
Yes, after we introduce this new domctl, we just make sure this domctl should be called before memory population no matter when we do assign a device as passthrough.
You didn't think through the implications then: If device assignment happens early enough, there's no need to report SBDF tuples via a

In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk.

Quoted 3 lines

new domctl (or only if we want to still allow for post-boot assignment of affected devices without using the global enforcement flag, in which case assigning devices early at boot time is pointless).

The global enforcement flag is mainly used to provide such an approach the user know exactly he/she may need a hotplug later, we'd better check to reserve all RMRR ranges in advance.

So I guess you mean we need to separate this interface from our original device assignment like this,

#1 pci_force as a global variable would control whether we force to check/reserve __all__ RMRR ranges.

#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately. This means we should introduce new parameters just like current 'pci' construction, right? Or extend current device assignment to be compatible with this case, for instance, new field to indicate if we really want to do device assignment in boot time.

Thanks
Tiejun

Jan Beulich

2014-11-12 08:37 UTC #89 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 12 lines

Quoted 1 line
Quoted 1 line · On 12.11.14 at 02:36, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 11.11.14 at 10:42, <[email protected]> wrote:
Quoted 3 lines · Jan Beulich
Part of which would involve re-considering whether device assignment shouldn't be done before memory population in the tool stack.
Yes, after we introduce this new domctl, we just make sure this domctl should be called before memory population no matter when we do assign a device as passthrough.

Quoted 5 lines

Quoted 2 lines
You didn't think through the implications then: If device assignment happens early enough, there's no need to report SBDF tuples via a
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk.

And I didn't question that; instead I suggested to re-consider whether that should be changed.

Quoted 8 lines

Quoted 3 lines
new domctl (or only if we want to still allow for post-boot assignment of affected devices without using the global enforcement flag, in which case assigning devices early at boot time is pointless).
The global enforcement flag is mainly used to provide such an approach the user know exactly he/she may need a hotplug later, we'd better check to reserve all RMRR ranges in advance.

Quoted 5 lines

So I guess you mean we need to separate this interface from our original device assignment like this,
#1 pci_force as a global variable would control whether we force to check/reserve __all__ RMRR ranges.

Yes.

Quoted 4 lines

#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.

And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.

Jan

Chen, Tiejun

author

2014-11-12 08:45 UTC #90 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 8 lines

Quoted 4 lines
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.

Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.

Thanks
Tiejun

Jan Beulich

2014-11-12 09:02 UTC #91 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.

Quoted 6 lines

Quoted 3 lines
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.

I can't connect your question to what I said. What I tried to tell you was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's no need for a new domctl to communicate SBDFs, as devices needing holes will be known to the hypervisor already).

Jan

Chen, Tiejun

author

2014-11-12 09:13 UTC #92 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines · Jan Beulich

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.
I can't connect your question to what I said. What I tried to tell you

Something is misunderstanding to me.

Quoted 6 lines

was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's

Here what do you mean, 'if we decide to require device assignment to happen before memory getting populated'?

Because -quote-
"
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk. "

Or you already plan or deciede to change this sequence?

Thanks
Tiejun

Quoted 7 lines

no need for a new domctl to communicate SBDFs, as devices needing holes will be known to the hypervisor already).
Jan

Jan Beulich

2014-11-12 09:56 UTC #93 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 29 lines

Quoted 1 line
Quoted 1 line · On 12.11.14 at 10:13, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.
I can't connect your question to what I said. What I tried to tell you
Something is misunderstanding to me.
Quoted 6 lines
was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's
Here what do you mean, 'if we decide to require device assignment to happen before memory getting populated'?

Quoted 7 lines

Because -quote-
"
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk. "
Or you already plan or deciede to change this sequence?

So it is now the 3rd time that I'm telling you that part of your decision making as to which route to follow should be to re-consider whether the current sequence of operations shouldn't be changed. Please also consult with the VT-d maintainers (hint to
them: participating in this discussion publicly would be really nice) on _all_ decisions to be made here.

Jan

Chen, Tiejun

author

2014-11-12 10:18 UTC #94 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 35 lines · Jan Beulich

Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 10:13, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can use them to control which devices need to work as hotplug later, separately.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.
I can't connect your question to what I said. What I tried to tell you
Something is misunderstanding to me.
Quoted 6 lines
was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's
Here what do you mean, 'if we decide to require device assignment to happen before memory getting populated'?
Because -quote-
"
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk. "

Quoted 5 lines

Quoted 1 line
Or you already plan or deciede to change this sequence?
So it is now the 3rd time that I'm telling you that part of your decision making as to which route to follow should be to re-consider whether the current sequence of operations shouldn't

As I said previously it may corrupt some existing frameworks to move device assignment codes.

Anyway, who can determine formally this approach? Let me ping actively them.

Quoted 1 line

be changed. Please also consult with the VT-d maintainers (hint to

Especially, Yang and Kevin are always included in this thread.

Quoted 1 line

them: participating in this discussion publicly would be really nice)

Also in public.

Thanks
Tiejun

Quoted 6 lines

on _all_ decisions to be made here.
Jan

Tian, Kevin

2014-11-19 08:17 UTC #95 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Jan Beulich

Quoted 39 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, November 12, 2014 5:57 PM
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 10:13, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But its not dependent on current device assignment domctl, so the user can
use
Quoted 29 lines
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
them to control which devices need to work as hotplug later, separately.
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.
I can't connect your question to what I said. What I tried to tell you
Something is misunderstanding to me.
Quoted 6 lines
was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's
Here what do you mean, 'if we decide to require device assignment to happen before memory getting populated'?
Because -quote-
"
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk. "

Quoted 9 lines

Quoted 1 line
Or you already plan or deciede to change this sequence?
So it is now the 3rd time that I'm telling you that part of your decision making as to which route to follow should be to re-consider whether the current sequence of operations shouldn't be changed. Please also consult with the VT-d maintainers (hint to
them: participating in this discussion publicly would be really nice) on _all_ decisions to be made here.

there's no decision made privately. we hope all the discussions publicly. will get back w/ our thoughts soon.

Thanks
Kevin

Tian, Kevin

2014-11-20 07:45 UTC #96 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Jan Beulich

Quoted 8 lines

From: Tian, Kevin
Sent: Wednesday, November 19, 2014 4:18 PM
Quoted 5 lines
From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, November 12, 2014 5:57 PM
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 10:13, <[email protected]> wrote:
On 2014/11/12 17:02, Jan Beulich wrote:

Quoted 46 lines

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · On 12.11.14 at 09:45, <[email protected]> wrote:
#2 flags field in each specific device of new domctl would control whether this device need to check/reserve its own RMRR range. But
its
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
not dependent on current device assignment domctl, so the user can
use
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
them to control which devices need to work as hotplug later,
separately.
Quoted 37 lines
Quoted 22 lines
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
And this could be left as a second step, in order for what needs to be done now to not get more complicated that necessary.
Do you mean currently we still rely on the device assignment domctl to provide SBDF? So looks nothing should be changed in our policy.
I can't connect your question to what I said. What I tried to tell you
Something is misunderstanding to me.
Quoted 6 lines
was that I don't currently see a need to make this overly complicated:
Having the option to punch holes for all devices and (by default) dealing with just the devices assigned at boot may be sufficient as a first step. Yet (repeating just to avoid any misunderstanding) that makes things easier only if we decide to require device assignment to happen before memory getting populated (since in that case there's
Here what do you mean, 'if we decide to require device assignment to happen before memory getting populated'?
Quoted 7 lines
Because -quote-
"
In the present the device assignment is always after memory population. And I also mentioned previously I double checked this sequence with printk. "
Or you already plan or deciede to change this sequence?
So it is now the 3rd time that I'm telling you that part of your decision making as to which route to follow should be to re-consider whether the current sequence of operations shouldn't be changed. Please also consult with the VT-d maintainers (hint to
them: participating in this discussion publicly would be really nice) on _all_ decisions to be made here.

Yang and I did some discussion here. We understand your point to avoid introducing new interface if we can leverage existing code. However it's not a trivial effort to move device assignment before populating p2m, and there is no other benefit of doing so except for this purpose. So we'd not suggest this way.

Current option sounds a reasonable one, i.e. passing a list of BDFs assigned to this VM before populating p2m, and then having hypervisor to filter out reserved regions associated with those BDFs. This way libxc teaches Xen to create reserved regions once, and then later the filtered info is returned upon query.

The limitation of wasted memory due to confliction can be mitigated, and we considered further enhancement can be made later in libxc that when populating p2m, the reserved regions can be skipped explicitly at initial p2m creation phase and then there would be no waste at all. But this optimization takes some time and can be built incrementally on current patch and interface, post 4.5 release. For now let's focus on the very correctness first.

If you agree, Tiejun will move forward to send another series for 4.5. So far lots of opens have been closed with your help, but it also means original v7 needs a serious update then (latest code is in deep discussion list)

Thanks
Kevin

Jan Beulich

2014-11-20 08:04 UTC #97 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines

Quoted 1 line
Quoted 1 line · On 20.11.14 at 08:45, <[email protected]> wrote:
Yang and I did some discussion here. We understand your point to avoid introducing new interface if we can leverage existing code. However it's not a trivial effort to move device assignment before populating p2m, and there is no other benefit of doing so except for this purpose. So we'd not suggest this way.

"It's not a trivial effort" is pretty vague: What specifically is it that makes this difficult? I wouldn't expect there to be any strong dependencies on the ordering of these two operations...

Quoted 5 lines

Current option sounds a reasonable one, i.e. passing a list of BDFs assigned to this VM before populating p2m, and then having hypervisor to filter out reserved regions associated with those BDFs. This way libxc teaches Xen to create reserved regions once, and then later the filtered info is returned upon query.

Reasonable, but partly redundant. The positive aspect being that it permits this list and the list of actually being assigned devices to be different, i.e. allowing holes to be set up for devices that only _may_ get assigned at some point.

Quoted 7 lines

The limitation of wasted memory due to confliction can be mitigated, and we considered further enhancement can be made later in libxc that when populating p2m, the reserved regions can be skipped explicitly at initial p2m creation phase and then there would be no waste at all. But this optimization takes some time and can be built incrementally on current patch and interface, post 4.5 release. For now let's focus on the very correctness first.

I agree, as long as the optimization part doesn't get dropped after the correctness part went in.

Jan

Tian, Kevin

2014-11-20 08:51 UTC #98 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 10 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, November 20, 2014 4:04 PM
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 20.11.14 at 08:45, <[email protected]> wrote:
Yang and I did some discussion here. We understand your point to avoid introducing new interface if we can leverage existing code. However it's not a trivial effort to move device assignment before populating p2m, and there is no other benefit of doing so except for this purpose. So we'd not suggest this way.

Quoted 3 lines

"It's not a trivial effort" is pretty vague: What specifically is it that makes this difficult? I wouldn't expect there to be any strong dependencies on the ordering of these two operations...

I'll leave to Yang to answer this part, who did a detail investigation on that, e.g. on IOMMU page table setup, etc. But what really matters here is not only about complexity, but also flexibility. Doing so will tie the policy to assigned device only, which removes the option to support hotpluggable device.

Quoted 11 lines

Quoted 6 lines
Current option sounds a reasonable one, i.e. passing a list of BDFs assigned to this VM before populating p2m, and then having hypervisor to filter out reserved regions associated with those BDFs. This way libxc teaches Xen to create reserved regions once, and then later the filtered info is returned upon query.
Reasonable, but partly redundant. The positive aspect being that it permits this list and the list of actually being assigned devices to be different, i.e. allowing holes to be set up for devices that only _may_ get assigned at some point.

redundant if we think the list only exactly matching the statically assigned devices. but that's just current point.

reasonable if we think there may be other policies impacting the list (e.g. if hotplugable device may have a config option and then we have a potential list larger than static assigned devices). From this angle I think the new interface actually makes sense for this very purpose.

Quoted 11 lines

Quoted 8 lines
The limitation of wasted memory due to confliction can be mitigated, and we considered further enhancement can be made later in libxc that when populating p2m, the reserved regions can be skipped explicitly at initial p2m creation phase and then there would be no waste at all. But this optimization takes some time and can be built incrementally on current patch and interface, post 4.5 release. For now let's focus on the very correctness first.
I agree, as long as the optimization part doesn't get dropped after the correctness part went in.

definitely. after putting so much effort in last months from Tiejun, you can see our willingness to make it correct and continuously improved.

Thanks
Kevin

Tian, Kevin

2014-11-20 14:40 UTC #99 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Jan Beulich

Quoted 35 lines

From: Tian, Kevin
Sent: Thursday, November 20, 2014 4:51 PM
Quoted 10 lines
From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, November 20, 2014 4:04 PM
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 20.11.14 at 08:45, <[email protected]> wrote:
Yang and I did some discussion here. We understand your point to avoid introducing new interface if we can leverage existing code. However it's not a trivial effort to move device assignment before populating p2m, and there is no other benefit of doing so except for this purpose. So we'd not suggest this way.
Quoted 3 lines
"It's not a trivial effort" is pretty vague: What specifically is it that makes this difficult? I wouldn't expect there to be any strong dependencies on the ordering of these two operations...
I'll leave to Yang to answer this part, who did a detail investigation on that, e.g. on IOMMU page table setup, etc. But what really matters here is not only about complexity, but also flexibility. Doing so will tie the policy to assigned device only, which removes the option to support hotpluggable device.
Quoted 11 lines
Quoted 6 lines
Current option sounds a reasonable one, i.e. passing a list of BDFs assigned to this VM before populating p2m, and then having hypervisor to filter out reserved regions associated with those BDFs. This way libxc teaches Xen to create reserved regions once, and then later the filtered info is returned upon query.
Reasonable, but partly redundant. The positive aspect being that it permits this list and the list of actually being assigned devices to be different, i.e. allowing holes to be set up for devices that only _may_ get assigned at some point.

Quoted 8 lines

redundant if we think the list only exactly matching the statically assigned devices. but that's just current point.
reasonable if we think there may be other policies impacting the list (e.g. if hotplugable device may have a config option and then we have a potential list larger than static assigned devices). From this angle I think the new interface actually makes sense for this very purpose.

Jan are you OK with this? In previous approach we reserved all the RMRR regions so hotplug scenario is covered automatically. Now since we want to do BDF specific filtering, this new interface is actually necessary for hotplug support. If OK, Tiejun will send out a new series to see whether it's ready for 4.5 check-in.

Thanks
Kevin

Jan Beulich

2014-11-20 14:46 UTC #100 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 20 lines

Quoted 20 lines
Quoted 1 line · On 20.11.14 at 15:40, <[email protected]> wrote:
From: Tian, Kevin
Sent: Thursday, November 20, 2014 4:51 PM
Quoted 9 lines
From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, November 20, 2014 4:04 PM
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 20.11.14 at 08:45, <[email protected]> wrote:
Current option sounds a reasonable one, i.e. passing a list of BDFs assigned to this VM before populating p2m, and then having hypervisor to filter out reserved regions associated with those BDFs. This way libxc teaches Xen to create reserved regions once, and then later the filtered info is returned upon query.
Quoted 4 lines
Reasonable, but partly redundant. The positive aspect being that it permits this list and the list of actually being assigned devices to be different, i.e. allowing holes to be set up for devices that only _may_ get assigned at some point.
redundant if we think the list only exactly matching the statically assigned devices. but that's just current point.

Quoted 6 lines

Quoted 4 lines
reasonable if we think there may be other policies impacting the list (e.g. if hotplugable device may have a config option and then we have a potential list larger than static assigned devices). From this angle I think the new interface actually makes sense for this very purpose.
Jan are you OK with this?

I can live with it, yes.

Jan

Konrad Rzeszutek Wilk

2014-11-20 20:11 UTC #101 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Tian, Kevin

Quoted 5 lines

Jan are you OK with this? In previous approach we reserved all the RMRR regions so hotplug scenario is covered automatically. Now since we want to do BDF specific filtering, this new interface is actually necessary for hotplug support. If OK, Tiejun will send out a new series to see whether it's ready for 4.5 check-in.

Could you also drop the 'RFC' part please? And please do mention in your cover letter how to reproduce the initial bug (perhaps a pointer to the hardware and software that this affects?)

Thank you.

Tian, Kevin

2014-11-21 00:32 UTC #102 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 9 lines

From: Konrad Rzeszutek Wilk [mailto:[email protected]]
Sent: Friday, November 21, 2014 4:11 AM
Quoted 5 lines
Jan are you OK with this? In previous approach we reserved all the RMRR regions so hotplug scenario is covered automatically. Now since we want to do BDF specific filtering, this new interface is actually necessary for hotplug support. If OK, Tiejun will send out a new series to see whether it's ready for 4.5 check-in.

Quoted 4 lines

Could you also drop the 'RFC' part please? And please do mention in your cover letter how to reproduce the initial bug (perhaps a pointer to the hardware and software that this affects?)

good suggestion. will do.

Thanks
Kevin

Chen, Tiejun

author

2014-11-12 03:05 UTC #103 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Chen, Tiejun

Quoted 11 lines · Tiejun

Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines · On 11.11.14 at 10:03, <[email protected]> wrote:
Quoted 1 line · On 11.11.14 at 08:49, <[email protected]> wrote:
Unless we move all check inside each callback functions.
That's what you should do imo, albeit I realize that the comparing
Yes, I can do this in all existing callback functions but I'm somewhat afraid when other guys want to introduce new callback function, who can guarantee this should be done as well?

I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.

I tried to address this but obviously we have to pass each 'pdf' to callback functions,

xen/common/compat/memory.c +25 -5

index af613b9..db4b90f 100644
@@ -20,12 +20,16 @@ CHECK_mem_access_op;
 struct get_reserved_device_memory {
     struct compat_reserved_device_memory_map map;
     unsigned int used_entries;
+    struct domain *domain;
 };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr,
+                                      u16 bdf, void *ctxt)
 {
     struct get_reserved_device_memory *grdm = ctxt;
+    u16 pt_bdf;
+    int i;
+    struct domain *d = grdm->domain;
 
     if ( grdm->used_entries < grdm->map.nr_entries )
     {

@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
         if ( rdm.start_pfn != start || rdm.nr_pages != nr )
             return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                                 &rdm, 1) )
+                        return -EFAULT;
+            }
+        }
     }
 
     ++grdm->used_entries;

@@ -314,6 +333,7 @@ int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
                 return -EFAULT;
 
             grdm.used_entries = 0;
+            grdm.domain = current->domain;
             rc =

iommu_get_reserved_device_memory(get_reserved_device_memory,

                                                   &grdm);

xen/common/memory.c +25 -5

index 2177c56..f5e9c1f 100644
@@ -696,12 +696,16 @@ out:
 struct get_reserved_device_memory {
     struct xen_reserved_device_memory_map map;
     unsigned int used_entries;
+    struct domain *domain;
 };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr,
+                                      u16 bdf, void *ctxt)
 {
     struct get_reserved_device_memory *grdm = ctxt;
+    u16 pt_bdf;
+    int i;
+    struct domain *d = grdm->domain;
 
     if ( grdm->used_entries < grdm->map.nr_entries )
     {

@@ -709,9 +713,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
             .start_pfn = start, .nr_pages = nr
         };
 
-        if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
-                                    &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
+                                        &rdm, 1) )
+                return -EFAULT;
+        }
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
+                                                &rdm, 1) )
+                        return -EFAULT;
+            }
+        }
     }
 
     ++grdm->used_entries;

@@ -1139,6 +1158,7 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
             return -EFAULT;
 
         grdm.used_entries = 0;
+        grdm.domain = current->domain;
         rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
                                               &grdm);

xen/drivers/passthrough/vtd/dmar.c +3 -1

index 141e735..68da9d0 100644
@@ -898,11 +898,14 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
 {
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;
+    int i;
+    u16 bdf;
 
-    list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+    for_each_rmrr_device ( rmrr, bdf, i )
     {
         rc = func(PFN_DOWN(rmrr->base_address),
                   PFN_UP(rmrr->end_address) -

PFN_DOWN(rmrr->base_address),
+ bdf,

                   ctxt);
         if ( rc )
             break;

xen/include/xen/iommu.h +1 -1

index 409f6f8..ddea0be 100644
@@ -120,7 +120,7 @@ void iommu_dt_domain_destroy(struct domain *d);
 
 struct page_info;
 
-typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, void *ctxt);
+typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, u16 bdf, void *ctxt);
 
 struct iommu_ops {
     int (*init)(struct domain *d);

Thanks
Tiejun

Jan Beulich

2014-11-12 08:55 UTC #104 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 3 lines

Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.

As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).

Quoted 2 lines

I tried to address this but obviously we have to pass each 'pdf' to callback functions,

Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the segment there.

Quoted 13 lines

xen/common/compat/memory.c +6 -3

@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
          if ( rdm.start_pfn != start || rdm.nr_pages != nr )
              return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }

xen/common/compat/memory.c static int get_reserved_device_memory(xen_pfn_t start, · line 58

Quoted 10 lines

xen/common/compat/memory.c +10 -0

+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                                 &rdm, 1) )
+                        return -EFAULT;

I think d->arch.hvm_domain.pcidevs[] shouldn't contain duplicates, and hence there's no point continuing the loop if you found a match.

xen/common/compat/memory.c static int get_reserved_device_memory(xen_pfn_t start, · line 63

Quoted 5 lines

xen/common/compat/memory.c +2 -0

+            }
+        }
      }
 
      ++grdm->used_entries;

This should no longer get incremented unconditionally.

xen/common/memory.c int compat_memory_op(unsigned int cmd, · line 339

Quoted 9 lines

xen/common/memory.c +1 -0

@@ -314,6 +333,7 @@ int compat_memory_op(unsigned int cmd, 
 XEN_GUEST_HANDLE_PARAM(void) compat)
                  return -EFAULT;
 
              grdm.used_entries = 0;
+            grdm.domain = current->domain;
              rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
                                                    &grdm);

Maybe I misguided you with an earlier response, or maybe the earlier reply was in a different context: There's no point communicating current->domain to the callback function; there would be a point communicating the domain if it was _not_ always current->domain.

Jan

Chen, Tiejun

author

2014-11-12 10:18 UTC #105 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 10 lines · Jan Beulich

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).

Quoted 5 lines

Quoted 2 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the

Okay.

Quoted 1 line

segment there.

I think we don't need segment since when we passthrough a device, that domain doesn't matter with the real segment in phydev.

Quoted 19 lines

xen/common/compat/memory.c +11 -3

@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
           if ( rdm.start_pfn != start || rdm.nr_pages != nr )
               return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,

Quoted 8 lines

xen/common/compat/memory.c static int get_reserved_device_memory(xen_pfn_t start, · line 58
Quoted 5 lines
xen/common/compat/memory.c +5 -0
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                                 &rdm, 1) )
+                        return -EFAULT;
I think d->arch.hvm_domain.pcidevs[] shouldn't contain duplicates, and hence there's no point continuing the loop if you found a match.

You're right.

Quoted 8 lines

xen/common/compat/memory.c static int get_reserved_device_memory(xen_pfn_t start, · line 63
Quoted 6 lines
xen/common/compat/memory.c +2 -0
+            }
+        }
       }
 
       ++grdm->used_entries;
This should no longer get incremented unconditionally.

Yes.

Quoted 5 lines

xen/common/compat/memory.c +0 -0

@@ -314,6 +333,7 @@ int compat_memory_op(unsigned int cmd,
 XEN_GUEST_HANDLE_PARAM(void) compat)
                   return -EFAULT;

Quoted 12 lines

xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1163
Quoted 5 lines
xen/common/memory.c +1 -0
               grdm.used_entries = 0;
+            grdm.domain = current->domain;
               rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
                                                     &grdm);
Maybe I misguided you with an earlier response, or maybe the earlier reply was in a different context: There's no point communicating current->domain to the callback function; there would be a point communicating the domain if it was _not_ always current->domain.

I will look into this.

Thanks
Tiejun

Jan Beulich

2014-11-12 10:24 UTC #106 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 18 lines

Quoted 1 line
Quoted 1 line · On 12.11.14 at 11:18, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).
Quoted 5 lines
Quoted 2 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the

Quoted 6 lines

Okay.
Quoted 1 line
segment there.
I think we don't need segment since when we passthrough a device, that domain doesn't matter with the real segment in phydev.

How can this not matter? If 0001:bb:dd.f is associated with an RMRR but 0000:bb:dd.f isn't, it's quite relevant which one is being handed to a guest.

Jan

Chen, Tiejun

author

2014-11-12 10:32 UTC #107 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 20 lines · Jan Beulich

Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 11:18, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).
Quoted 5 lines
Quoted 2 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the
Okay.

Quoted 9 lines

Quoted 4 lines
Quoted 1 line
segment there.
I think we don't need segment since when we passthrough a device, that domain doesn't matter with the real segment in phydev.
How can this not matter? If 0001:bb:dd.f is associated with an RMRR but 0000:bb:dd.f isn't, it's quite relevant which one is being handed to a guest.

In passthrough case this is needed so I will add this.

Thanks
Tiejun

Chen, Tiejun

author

2014-11-13 03:09 UTC #108 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Jan Beulich

Quoted 35 lines · Jan Beulich

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).
Quoted 2 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the segment there.
Quoted 18 lines
xen/common/compat/memory.c +11 -3
@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
           if ( rdm.start_pfn != start || rdm.nr_pages != nr )
               return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,

Quoted 9 lines

xen/common/compat/memory.c static int get_reserved_device_memory(xen_pfn_t start, · line 58
Quoted 5 lines
xen/common/compat/memory.c +5 -0
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                                 &rdm, 1) )
+                        return -EFAULT;
I think d->arch.hvm_domain.pcidevs[] shouldn't contain duplicates, and hence there's no point continuing the loop if you found a match.

I take a look at this again. Seems we shouldn't just check bdf since as you know its possible to occupy one entry by multiple different BDFs, so we have to filter-to-keep one entry. Instead, I really hope we'd check to expose before we do the hypercall.

BTW, I already ping Yang in local to look that possibility to reorder the sequence of the device assignment and the memory population in iommu side.

Thanks
Tiejun

Chen, Tiejun

author

2014-11-14 02:21 UTC #109 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 50 lines · Tiejun

Quoted 39 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).
Quoted 2 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the segment there.
Quoted 22 lines
xen/common/compat/memory.c +12 -3
@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t
 start,
           if ( rdm.start_pfn != start || rdm.nr_pages != nr )
               return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer,
 grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer,
 grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
Quoted 9 lines
Review
Quoted 5 lines
(quoted diff) +2 -0
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer,
grdm->used_entries,
(quoted diff) +2 -0
+                                                 &rdm, 1) )
+                        return -EFAULT;
I think d->arch.hvm_domain.pcidevs[] shouldn't contain duplicates, and hence there's no point continuing the loop if you found a match.

Quoted 4 lines

I take a look at this again. Seems we shouldn't just check bdf since as you know its possible to occupy one entry by multiple different BDFs, so we have to filter-to-keep one entry. Instead, I really hope we'd check to expose before we do the hypercall.

Even if eventually we'll reorder that sequence, this just change that approach to get BDF. Are you fine to this subsequent change?

@@ -894,18 +894,55 @@ int platform_supports_x2apic(void)

     return cpu_has_x2apic && ((dmar_flags & mask) ==

ACPI_DMAR_INTR_REMAP);
}

-int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt) +int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, struct domain *d,
+ void *ctxt)

 {
     struct acpi_rmrr_unit *rmrr;
-    int rc = 0;
+    int rc = 0, i, j, seg_check = 1;
+    u16 id, bdf;

- list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+ if ( d->arch.hvm_domain.pci_force )

- rc = func(PFN_DOWN(rmrr->base_address),
- PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
- ctxt);
- if ( rc )
- break;
+ list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+ {
+ rc = func(PFN_DOWN(rmrr->base_address),
+ PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
+ ctxt);
+ if ( rc )
+ break;
+ }

+    }
+    else
+    {

+ list_for_each_entry(rmrr, &acpi_rmrr_units, list)
+ {
+ for ( i = 0; i < d->arch.hvm_domain.num_pcidevs &&
+ seg_check; i++ )
+ {
+ if ( rmrr->segment == d->arch.hvm_domain.pcidevs[i].seg )
+ {
+ bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[j].bus,
+ d->arch.hvm_domain.pcidevs[j].devfn);
+ for (j = 0; (id = rmrr->scope.devices[j]) &&
+ j < rmrr->scope.devices_cnt && seg_check; j++)
+ {
+ if ( bdf == id )
+ {
+ rc = func(PFN_DOWN(rmrr->base_address),
+ PFN_UP(rmrr->end_address) -
+ PFN_DOWN(rmrr->base_address),
+ ctxt);
+ if ( rc )
+ return;
+ /* Hit this seg entry. */
+ seg_check = 0;
+ }
+ }
+ }

+            }
+            /* goto next seg entry. */
+            seg_check = 1;
+        }
     }

     return rc;

Quoted 4 lines

BTW, I already ping Yang in local to look that possibility to reorder the sequence of the device assignment and the memory population in iommu side.

Yang and Kevin,

Any comments to this requirement?

Thanks
Tiejun

Jan Beulich

2014-11-14 08:21 UTC #110 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 3 lines

Quoted 1 line
Quoted 1 line · On 14.11.14 at 03:21, <[email protected]> wrote:
Even if eventually we'll reorder that sequence, this just change that approach to get BDF. Are you fine to this subsequent change?

You again pass a struct domain pointer to the IOMMU-specific function. I already told you not to do so - the domain specific aspect should be taken care of by the callback function, i.e. you need to make SBDF available to it (just like you already properly did in the previous round for BDF). I suppose that'll at once make the ugly open coding of for_each_rmrr_device() unnecessary - you can just use that construct as replacement for what right now is list_for_each_entry().

Jan

Chen, Tiejun

author

2014-11-17 07:31 UTC #111 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines · Jan Beulich

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 14.11.14 at 03:21, <[email protected]> wrote:
Even if eventually we'll reorder that sequence, this just change that approach to get BDF. Are you fine to this subsequent change?
You again pass a struct domain pointer to the IOMMU-specific function. I already told you not to do so - the domain specific

I remembered this comment but I want to show this may introduce many duplicated codes. As I understand this kind of check should be a common thing dependent on one given platform.

Quoted 7 lines

aspect should be taken care of by the callback function, i.e. you need to make SBDF available to it (just like you already properly did in the previous round for BDF). I suppose that'll at once make the ugly open coding of for_each_rmrr_device() unnecessary - you can just use that construct as replacement for what right now is list_for_each_entry().

Okay, I will try to go there.

Thanks
Tiejun

Chen, Tiejun

author

2014-11-17 07:57 UTC #112 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Jan Beulich

Quoted 9 lines · Jan Beulich

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 12.11.14 at 04:05, <[email protected]> wrote:
I don't see any feedback to this point, so I think you still prefer we should do all check in the callback function.
As a draft this looks reasonable, but there are various bugs to be dealt with along with cosmetic issues (I'll point out the former, but I'm tired of pointing out the latter once again - please go back to earlier reviews of patches to refresh e.g. what types to use for loop variables).

'earlier reviews' means this subject email? I go back to check this but just see this comment related to our loop codes:

Quoted 9 lines

Quoted 8 lines

(quoted diff) +8 -0

+        for ( i = 0; i < xdsr->num_pcidevs; ++i )
+        {
+            if ( __copy_from_guest_offset(pcidevs, xdsr->pcidevs, i, 1) )
+            {
+                xfree(pcidevs);
+                return -EFAULT;
+            }
+        }

Quoted 7 lines

Quoted 2 lines
I don't see the need for a loop here. And you definitely can't use the double-underscore-prefixed variant the way you do.
Do you mean this line?
copy_from_guest_offset(pcidevs, xdsr->pcidevs, 0, xdsr->num_pcidevs*sizeof(xen_guest_pcidev_info_t))

Almost:

    copy_from_guest(pcidevs, xdsr->pcidevs, xdsr->num_pcidevs *

sizeof(*pcidevs))
"

Or I need to set as 'unsigned int' here?

Anyway I did this in the following codes firstly. If I'm still wrong I will correct that.

Quoted 47 lines

Quoted 3 lines
I tried to address this but obviously we have to pass each 'pdf' to callback functions,
Yes, but at the generic IOMMU layer this shouldn't be named "bdf", but something more neutral (maybe "id"). And you again lost the segment there.
Quoted 13 lines
(quoted diff) +6 -3
@@ -36,9 +40,24 @@ static int get_reserved_device_memory(xen_pfn_t start,
           if ( rdm.start_pfn != start || rdm.nr_pages != nr )
               return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+        }
Review static int get_reserved_device_memory(xen_pfn_t start, · line 58
Quoted 10 lines
(quoted diff) +10 -0
+        else
+        {
+            for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+            {
+                pt_bdf = PCI_BDF2(d->arch.hvm_domain.pcidevs[i].bus,
+                                  d->arch.hvm_domain.pcidevs[i].devfn);
+                if ( pt_bdf == bdf )
+                    if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                                 &rdm, 1) )
+                        return -EFAULT;
I think d->arch.hvm_domain.pcidevs[] shouldn't contain duplicates, and hence there's no point continuing the loop if you found a match.
Review static int get_reserved_device_memory(xen_pfn_t start, · line 63
Quoted 5 lines
(quoted diff) +2 -0
+            }
+        }
       }
 
       ++grdm->used_entries;
This should no longer get incremented unconditionally.
Quoted 4 lines
(quoted diff) +0 -0
@@ -314,6 +333,7 @@ int compat_memory_op(unsigned int cmd,
 XEN_GUEST_HANDLE_PARAM(void) compat)
                   return -EFAULT;
 

Quoted 12 lines

Review int compat_memory_op(unsigned int cmd, · line 339
Quoted 5 lines
(quoted diff) +1 -0
               grdm.used_entries = 0;
+            grdm.domain = current->domain;
               rc = iommu_get_reserved_device_memory(get_reserved_device_memory,
                                                     &grdm);
Maybe I misguided you with an earlier response, or maybe the earlier reply was in a different context: There's no point communicating current->domain to the callback function; there would be a point communicating the domain if it was _not_ always current->domain.

So we need the caller to pass domain ID, right?

xen/common/compat/memory.c +34 -7

index af613b9..314d7e6 100644
@@ -22,10 +22,13 @@ struct get_reserved_device_memory {
     unsigned int used_entries;
 };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16 seg,
+                                      u16 *ids, int cnt, void *ctxt)
 {
     struct get_reserved_device_memory *grdm = ctxt;
+    struct domain *d = get_domain_by_id(grdm->map.domid);
+    unsigned int i, j;
+    u32 sbdf, pt_sbdf;
 
     if ( grdm->used_entries < grdm->map.nr_entries )
     {

@@ -36,13 +39,37 @@ static int get_reserved_device_memory(xen_pfn_t start,
         if ( rdm.start_pfn != start || rdm.nr_pages != nr )
             return -ERANGE;
 
-        if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
-                                     &rdm, 1) )
-            return -EFAULT;
+        if ( d->arch.hvm_domain.pci_force )
+        {
+            if ( __copy_to_compat_offset(grdm->map.buffer, grdm->used_entries,
+                                         &rdm, 1) )
+                return -EFAULT;
+            ++grdm->used_entries;
+        }
+        else
+        {
+            for ( i = 0; i < cnt; i++ )
+            {
+                sbdf = PCI_SBDF(seg, ids[i]);
+                for ( j = 0; j < d->arch.hvm_domain.num_pcidevs; j++ )
+                {
+                    pt_sbdf = PCI_SBDF2(d->arch.hvm_domain.pcidevs[j].seg,
+                                        d->arch.hvm_domain.pcidevs[j].bus,
+ d->arch.hvm_domain.pcidevs[j].devfn);
+                    if ( pt_sbdf == sbdf )
+                    {
+                        if ( __copy_to_compat_offset(grdm->map.buffer,
+                                                     grdm->used_entries,
+                                                     &rdm, 1) )
+                            return -EFAULT;
+                        ++grdm->used_entries;
+                        break;
+                    }
+                }
+            }
+        }
     }
 
-    ++grdm->used_entries;
-
     return 0;
 }
 #endif

xen/common/memory.c +34 -7

57 lines not on this page · raw message

xen/drivers/passthrough/vtd/dmar.c +0 -0

index 141e735..fa813c5 100644
@@ -903,6 +903,9 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
     {
         rc = func(PFN_DOWN(rmrr->base_address),
                   PFN_UP(rmrr->end_address) -

PFN_DOWN(rmrr->base_address),

+                  rmrr->segment,
+                  rmrr->scope.devices,
+                  rmrr->scope.devices_cnt,
                   ctxt);
         if ( rc )
             break;

xen/include/public/memory.h +5 -0

index f1b6a48..f8964e1 100644
@@ -588,6 +588,11 @@ typedef struct xen_reserved_device_memory xen_reserved_device_memory_t;
 DEFINE_XEN_GUEST_HANDLE(xen_reserved_device_memory_t);
 
 struct xen_reserved_device_memory_map {
+    /*
+     * Domain whose reservation is being changed.
+     * Unprivileged domains can specify only DOMID_SELF.
+     */
+    domid_t        domid;
     /* IN/OUT */
     unsigned int nr_entries;
     /* OUT */

xen/include/xen/iommu.h +2 -1

index 409f6f8..75c6759 100644
@@ -120,7 +120,8 @@ void iommu_dt_domain_destroy(struct domain *d);
 
 struct page_info;
 
-typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, void *ctxt);
+typedef int iommu_grdm_t(xen_pfn_t start, xen_ulong_t nr, u16 seg, u16 *ids,
+                         int cnt, void *ctxt);
 
 struct iommu_ops {
     int (*init)(struct domain *d);

xen/include/xen/pci.h +2 -0

index 91520bc..ba881ef 100644
@@ -31,6 +31,8 @@
 #define PCI_DEVFN2(bdf) ((bdf) & 0xff)
 #define PCI_BDF(b,d,f)  ((((b) & 0xff) << 8) | PCI_DEVFN(d,f))
 #define PCI_BDF2(b,df)  ((((b) & 0xff) << 8) | ((df) & 0xff))
+#define PCI_SBDF(s,bdf) (((s & 0xffff) << 16) | (bdf & 0xffff))
+#define PCI_SBDF2(s,b,df) (((s & 0xffff) << 16) | PCI_BDF2(b,df))
 
 struct pci_dev_info {
     bool_t is_extfn;

Thanks
Tiejun

Jan Beulich

2014-11-17 10:05 UTC #113 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

xen/common/memory.c struct get_reserved_device_memory { · line 702

Quoted 11 lines

Quoted 1 line

Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:

xen/common/memory.c +2 -2

@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
      unsigned int used_entries;
  };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16 seg,
+                                      u16 *ids, int cnt, void *ctxt)

While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid introducing a hidden dependency on how the VT-d code manages its internal data.

Jan

Chen, Tiejun

author

2014-11-17 11:08 UTC #114 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 7 lines · Jan Beulich

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:
xen/common/memory.c +0 -0
@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
       unsigned int used_entries;
   };
 

Quoted 8 lines

xen/common/memory.c struct get_reserved_device_memory { · line 702
Quoted 4 lines
xen/common/memory.c +2 -2
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16 seg,
+                                      u16 *ids, int cnt, void *ctxt)
While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid

Do you mean I should do this?

for_each_rmrr_device ( rmrr, bdf, i )

{
	 sbdf = PCI_SBDF(seg, rmrr->scope.devices[i]);
         rc = func(PFN_DOWN(rmrr->base_address),
                   PFN_UP(rmrr->end_address) - 
PFN_DOWN(rmrr->base_address),
		   sbdf,	
                   ctxt);

But each different sbdf may occupy one same rmrr entry as I said previously, so we have to introduce more codes to filter them as one identified entry in the callback.

Thanks
Tiejun

Quoted 7 lines

introducing a hidden dependency on how the VT-d code manages its internal data.
Jan

Jan Beulich

2014-11-17 11:17 UTC #115 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 31 lines

Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:08, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
xen/common/memory.c struct get_reserved_device_memory { · line 701
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:
xen/common/memory.c +1 -2
@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
       unsigned int used_entries;
   };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16 
seg,
Quoted 5 lines
xen/common/memory.c struct get_reserved_device_memory { · line 702
Quoted 1 line
xen/common/memory.c +1 -0
+                                      u16 *ids, int cnt, void *ctxt)
While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid
Do you mean I should do this?
for_each_rmrr_device ( rmrr, bdf, i )
{
	 sbdf = PCI_SBDF(seg, rmrr->scope.devices[i]);
          rc = func(PFN_DOWN(rmrr->base_address),
                    PFN_UP(rmrr->end_address) - 
PFN_DOWN(rmrr->base_address),
		   sbdf,	
                    ctxt);

Quoted 3 lines

But each different sbdf may occupy one same rmrr entry as I said previously, so we have to introduce more codes to filter them as one identified entry in the callback.

Not really - remember that part of what needs to be done is to make sure all devices associated with a given RMRR get assigned to the same guest? Or the callback function could use a special return value (e.g. 1) to signal that the iteration for the current RMRR can be terminated (or further entries skipped).

Jan

Chen, Tiejun

author

2014-11-17 11:32 UTC #116 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 31 lines · Jan Beulich

Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:08, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
xen/common/memory.c struct get_reserved_device_memory { · line 701
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:
xen/common/memory.c +1 -2
@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
        unsigned int used_entries;
    };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16
seg,
Quoted 5 lines
xen/common/memory.c struct get_reserved_device_memory { · line 702
Quoted 1 line
xen/common/memory.c +1 -0
+                                      u16 *ids, int cnt, void *ctxt)
While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid
Do you mean I should do this?
for_each_rmrr_device ( rmrr, bdf, i )
{
	 sbdf = PCI_SBDF(seg, rmrr->scope.devices[i]);
           rc = func(PFN_DOWN(rmrr->base_address),
                     PFN_UP(rmrr->end_address) -
PFN_DOWN(rmrr->base_address),
		   sbdf,	
                     ctxt);

Quoted 7 lines

Quoted 3 lines
But each different sbdf may occupy one same rmrr entry as I said previously, so we have to introduce more codes to filter them as one identified entry in the callback.
Not really - remember that part of what needs to be done is to make sure all devices associated with a given RMRR get assigned to the same guest? Or the callback function could use a special return value

Yes, but I means in the callback,

get_reserved_device_memory()

{
	...
	for(each assigned pci devs:pt_sbdf)
		if (sbdf == pt_sbdf)
			__copy_to_guest_offset(buffer, ...)

Buffer may be copied to include multiple same entries if we have two or more assigned devices associated to one give RMRR entry.

Thanks
Tiejun

Quoted 7 lines

(e.g. 1) to signal that the iteration for the current RMRR can be terminated (or further entries skipped).
Jan

Jan Beulich

2014-11-17 11:51 UTC #117 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 44 lines

Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:32, <[email protected]> wrote:
Quoted 31 lines · Jan Beulich
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:08, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
xen/common/memory.c struct get_reserved_device_memory { · line 701
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:
xen/common/memory.c +1 -2
@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
        unsigned int used_entries;
    };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16
seg,
Quoted 5 lines
xen/common/memory.c struct get_reserved_device_memory { · line 702
Quoted 1 line
xen/common/memory.c +1 -0
+                                      u16 *ids, int cnt, void *ctxt)
While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid
Do you mean I should do this?
for_each_rmrr_device ( rmrr, bdf, i )
{
	 sbdf = PCI_SBDF(seg, rmrr->scope.devices[i]);
           rc = func(PFN_DOWN(rmrr->base_address),
                     PFN_UP(rmrr->end_address) -
PFN_DOWN(rmrr->base_address),
		   sbdf,	
                     ctxt);
Quoted 7 lines
Quoted 3 lines
But each different sbdf may occupy one same rmrr entry as I said previously, so we have to introduce more codes to filter them as one identified entry in the callback.
Not really - remember that part of what needs to be done is to make sure all devices associated with a given RMRR get assigned to the same guest? Or the callback function could use a special return value
Yes, but I means in the callback,
get_reserved_device_memory()

Quoted 8 lines

{
	...
	for(each assigned pci devs:pt_sbdf)
		if (sbdf == pt_sbdf)
			__copy_to_guest_offset(buffer, ...)
Buffer may be copied to include multiple same entries if we have two or more assigned devices associated to one give RMRR entry.

Which would be easily avoided by ...

Quoted 2 lines

Quoted 2 lines
(e.g. 1) to signal that the iteration for the current RMRR can be terminated (or further entries skipped).

... the approach I outlined.

Jan

Chen, Tiejun

author

2014-11-18 03:08 UTC #118 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 53 lines · Jan Beulich

Quoted 44 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:32, <[email protected]> wrote:
Quoted 31 lines · Jan Beulich
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 12:08, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
xen/common/memory.c struct get_reserved_device_memory { · line 701
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 17.11.14 at 08:57, <[email protected]> wrote:
xen/common/memory.c +1 -2
@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
         unsigned int used_entries;
     };
 
-static int get_reserved_device_memory(xen_pfn_t start,
-                                      xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr, u16
seg,
Quoted 5 lines
xen/common/memory.c struct get_reserved_device_memory { · line 702
Quoted 1 line
xen/common/memory.c +1 -0
+                                      u16 *ids, int cnt, void *ctxt)
While the approach is a lot better than what you did previously, I still don't like you adding 3 new parameters when one would do (calling the callback for each SBDF individually): That way you avoid
Do you mean I should do this?
for_each_rmrr_device ( rmrr, bdf, i )
{
	 sbdf = PCI_SBDF(seg, rmrr->scope.devices[i]);
            rc = func(PFN_DOWN(rmrr->base_address),
                      PFN_UP(rmrr->end_address) -
PFN_DOWN(rmrr->base_address),
		   sbdf,	
                      ctxt);
Quoted 7 lines
Quoted 3 lines
But each different sbdf may occupy one same rmrr entry as I said previously, so we have to introduce more codes to filter them as one identified entry in the callback.
Not really - remember that part of what needs to be done is to make sure all devices associated with a given RMRR get assigned to the same guest? Or the callback function could use a special return value
Yes, but I means in the callback,
get_reserved_device_memory()
Quoted 8 lines
{
	...
	for(each assigned pci devs:pt_sbdf)
		if (sbdf == pt_sbdf)
			__copy_to_guest_offset(buffer, ...)
Buffer may be copied to include multiple same entries if we have two or more assigned devices associated to one give RMRR entry.

Quoted 7 lines

Which would be easily avoided by ...
Quoted 2 lines
Quoted 2 lines
(e.g. 1) to signal that the iteration for the current RMRR can be terminated (or further entries skipped).
... the approach I outlined.

Here I tried to implement what you want. Note just pick two key fragments since others have no big deal.

#1:

@@ -898,14 +898,25 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)

 {
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;
+    unsigned int i;
+    u32 id;
+    u16 bdf;

list_for_each_entry(rmrr, &acpi_rmrr_units, list) {
- rc = func(PFN_DOWN(rmrr->base_address),
- PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
- ctxt);
- if ( rc )
- break;
+ for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+ i < rmrr->scope.devices_cnt && !rc; i++)
+ {
+ id = PCI_SBDF(rmrr->segment, bdf);
+ rc = func(PFN_DOWN(rmrr->base_address),
+ PFN_UP(rmrr->end_address) -
+ PFN_DOWN(rmrr->base_address),
+ id,
+ ctxt);
+ if ( rc < 0 )
+ return rc;
+ }

+        rc = 0;
     }

     return rc;

and #2,

@@ -698,10 +698,13 @@ struct get_reserved_device_memory {
     unsigned int used_entries;

};

-static int get_reserved_device_memory(xen_pfn_t start,
- xen_ulong_t nr, void *ctxt)
+static int get_reserved_device_memory(xen_pfn_t start, xen_ulong_t nr,
+ u32 id, void *ctxt)

 {
     struct get_reserved_device_memory *grdm = ctxt;
+    struct domain *d = get_domain_by_id(grdm->map.domid);
+    unsigned int i;
+    u32 sbdf;

     if ( grdm->used_entries < grdm->map.nr_entries )
     {

@@ -709,13 +712,34 @@ static int get_reserved_device_memory(xen_pfn_t start,
.start_pfn = start, .nr_pages = nr
};

- if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
- &rdm, 1) )
- return -EFAULT;
+ if ( d->arch.hvm_domain.pci_force )
+ {
+ if ( __copy_to_guest_offset(grdm->map.buffer, grdm->used_entries,
+ &rdm, 1) )
+ return -EFAULT;
+ ++grdm->used_entries;
+ return 1;

+        }
+        else
+        {

+ for ( i = 0; i < d->arch.hvm_domain.num_pcidevs; i++ )
+ {
+ sbdf = PCI_SBDF2(d->arch.hvm_domain.pcidevs[i].seg,
+ d->arch.hvm_domain.pcidevs[i].bus,
+ d->arch.hvm_domain.pcidevs[i].devfn);
+ if ( sbdf == id )
+ {
+ if ( __copy_to_guest_offset(grdm->map.buffer,
+ grdm->used_entries,
+ &rdm, 1) )
+ return -EFAULT;
+ ++grdm->used_entries;
+ return 1;
+ }
+ }
+ }

- ++grdm->used_entries;
-

     return 0;

}
#endif

Thanks
Tiejun

Jan Beulich

2014-11-18 08:01 UTC #119 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 25 lines

Quoted 1 line

Quoted 1 line · On 18.11.14 at 04:08, <[email protected]> wrote:

Here I tried to implement what you want. Note just pick two key fragments since others have no big deal.

#1:

(quoted diff) +6 -5

@@ -898,14 +898,25 @@ int 
 intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
  {
      struct acpi_rmrr_unit *rmrr;
      int rc = 0;
+    unsigned int i;
+    u32 id;
+    u16 bdf;
 
      list_for_each_entry(rmrr, &acpi_rmrr_units, list)
      {
-        rc = func(PFN_DOWN(rmrr->base_address),
-                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
-                  ctxt);
-        if ( rc )
-            break;
+        for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+                    i < rmrr->scope.devices_cnt && !rc; i++)
+        {

Review int · line 920

Quoted 10 lines

(quoted diff) +10 -0

+            id = PCI_SBDF(rmrr->segment, bdf);
+            rc = func(PFN_DOWN(rmrr->base_address),
+                               PFN_UP(rmrr->end_address) -
+                                PFN_DOWN(rmrr->base_address),
+                               id,
+                               ctxt);
+            if ( rc < 0 )
+                return rc;
+        }
+        rc = 0;

Getting close - the main issue is that (as previously mentioned) you should avoid open-coding for_each_rmrr_device(). It also doesn't look like you really need the local variable 'id'.

Jan

Chen, Tiejun

author

2014-11-18 08:16 UTC #120 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 30 lines · Jan Beulich

Quoted 30 lines

Quoted 1 line

Quoted 1 line · On 18.11.14 at 04:08, <[email protected]> wrote:

Here I tried to implement what you want. Note just pick two key fragments since others have no big deal.

#1:

(quoted diff) +11 -5

@@ -898,14 +898,25 @@ int
 intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
   {
       struct acpi_rmrr_unit *rmrr;
       int rc = 0;
+    unsigned int i;
+    u32 id;
+    u16 bdf;
 
       list_for_each_entry(rmrr, &acpi_rmrr_units, list)
       {
-        rc = func(PFN_DOWN(rmrr->base_address),
-                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
-                  ctxt);
-        if ( rc )
-            break;
+        for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+                    i < rmrr->scope.devices_cnt && !rc; i++)
+        {
+            id = PCI_SBDF(rmrr->segment, bdf);
+            rc = func(PFN_DOWN(rmrr->base_address),
+                               PFN_UP(rmrr->end_address) -
+                                PFN_DOWN(rmrr->base_address),
+                               id,

Quoted 8 lines

Review int · line 920
Quoted 5 lines
(quoted diff) +5 -0
+                               ctxt);
+            if ( rc < 0 )
+                return rc;
+        }
+        rc = 0;
Getting close - the main issue is that (as previously mentioned) you should avoid open-coding for_each_rmrr_device(). It also doesn't

Sorry, are you saying these lines?

Quoted 2 lines

Review int · line 922

Quoted 2 lines

(quoted diff) +2 -0

+        for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+                    i < rmrr->scope.devices_cnt && !rc; i++)

So without lookuping devices[i], how can we call func() for each sbdf as you mentioned?

Quoted 1 line

look like you really need the local variable 'id'.

Okay, I can pass PCI_SBDF(rmrr->segment, bdf) directly.

Thanks
Tiejun

Jan Beulich

2014-11-18 09:33 UTC #121 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 41 lines

Quoted 1 line
Quoted 1 line · On 18.11.14 at 09:16, <[email protected]> wrote:
Quoted 30 lines · Jan Beulich
Quoted 30 lines
Quoted 1 line
Quoted 1 line · On 18.11.14 at 04:08, <[email protected]> wrote:
Here I tried to implement what you want. Note just pick two key fragments since others have no big deal.
#1:
(quoted diff) +11 -5
@@ -898,14 +898,25 @@ int
 intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
   {
       struct acpi_rmrr_unit *rmrr;
       int rc = 0;
+    unsigned int i;
+    u32 id;
+    u16 bdf;
 
       list_for_each_entry(rmrr, &acpi_rmrr_units, list)
       {
-        rc = func(PFN_DOWN(rmrr->base_address),
-                  PFN_UP(rmrr->end_address) - PFN_DOWN(rmrr->base_address),
-                  ctxt);
-        if ( rc )
-            break;
+        for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+                    i < rmrr->scope.devices_cnt && !rc; i++)
+        {
+            id = PCI_SBDF(rmrr->segment, bdf);
+            rc = func(PFN_DOWN(rmrr->base_address),
+                               PFN_UP(rmrr->end_address) -
+                                PFN_DOWN(rmrr->base_address),
+                               id,
Quoted 8 lines
Review int · line 920
Quoted 5 lines
(quoted diff) +5 -0
+                               ctxt);
+            if ( rc < 0 )
+                return rc;
+        }
+        rc = 0;
Getting close - the main issue is that (as previously mentioned) you should avoid open-coding for_each_rmrr_device(). It also doesn't

Quoted 7 lines

Sorry, are you saying these lines?
Quoted 2 lines
Review int · line 922
Quoted 2 lines
(quoted diff) +2 -0
+        for (i = 0; (bdf = rmrr->scope.devices[i]) &&
+                    i < rmrr->scope.devices_cnt && !rc; i++)
So without lookuping devices[i], how can we call func() for each sbdf as you mentioned?

You've got both rmrr and bdf in the body of for_each_rmrr_device(). After all - as I said - you just open-coded it.

Jan

Chen, Tiejun

author

2014-11-19 01:26 UTC #122 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 6 lines

Quoted 2 lines
So without lookuping devices[i], how can we call func() for each sbdf as you mentioned?
You've got both rmrr and bdf in the body of for_each_rmrr_device(). After all - as I said - you just open-coded it.

Yeah, so change this again,

int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)

{
    struct acpi_rmrr_unit *rmrr;
    int rc = 0;
    unsigned int i;
    u16 bdf;

    for_each_rmrr_device ( rmrr, bdf, i )
    {
        rc = func(PFN_DOWN(rmrr->base_address),
                           PFN_UP(rmrr->end_address) -
                            PFN_DOWN(rmrr->base_address),
                           PCI_SBDF(rmrr->segment, bdf),
                          ctxt);
        /* Hit this entry so just go next. */
        if ( rc == 1 )
            i = rmrr->scope.devices_cnt;
        else if ( rc < 0 )
            return rc;
    }

    return rc;
}

Thanks
Tiejun

Jan Beulich

2014-11-20 07:31 UTC #123 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 25 lines

Quoted 7 lines
Quoted 3 lines · On 19.11.14 at 02:26, <[email protected]> wrote:
So without lookuping devices[i], how can we call func() for each sbdf as you mentioned?
You've got both rmrr and bdf in the body of for_each_rmrr_device(). After all - as I said - you just open-coded it.
Yeah, so change this again,
int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
{
     struct acpi_rmrr_unit *rmrr;
     int rc = 0;
     unsigned int i;
     u16 bdf;

     for_each_rmrr_device ( rmrr, bdf, i )
     {
         rc = func(PFN_DOWN(rmrr->base_address),
                            PFN_UP(rmrr->end_address) -
                             PFN_DOWN(rmrr->base_address),
                            PCI_SBDF(rmrr->segment, bdf),
                           ctxt);
         /* Hit this entry so just go next. */

Quoted 8 lines

         if ( rc == 1 )
             i = rmrr->scope.devices_cnt;
         else if ( rc < 0 )
             return rc;
     }

     return rc;
}

Better. Another improvement would be make it not depend on the internal workings of for_each_rmrr_device()... And in any case you should not special case 1 - just return when rc is negative and skip the rest of the current RMRR when it's positive. And of course make the function's final return value predictable.

Jan

Chen, Tiejun

author

2014-11-20 08:12 UTC #124 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 31 lines · Jan Beulich

Quoted 31 lines
Quoted 7 lines
Quoted 3 lines · On 19.11.14 at 02:26, <[email protected]> wrote:
So without lookuping devices[i], how can we call func() for each sbdf as you mentioned?
You've got both rmrr and bdf in the body of for_each_rmrr_device(). After all - as I said - you just open-coded it.
Yeah, so change this again,
int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
{
      struct acpi_rmrr_unit *rmrr;
      int rc = 0;
      unsigned int i;
      u16 bdf;

      for_each_rmrr_device ( rmrr, bdf, i )
      {
          rc = func(PFN_DOWN(rmrr->base_address),
                             PFN_UP(rmrr->end_address) -
                              PFN_DOWN(rmrr->base_address),
                             PCI_SBDF(rmrr->segment, bdf),
                            ctxt);
          /* Hit this entry so just go next. */
          if ( rc == 1 )
              i = rmrr->scope.devices_cnt;
          else if ( rc < 0 )
              return rc;
      }

Quoted 5 lines

Quoted 2 lines
      return rc;
}
Better. Another improvement would be make it not depend on the internal workings of for_each_rmrr_device()... And in any case you

Yes but as you see,

#define for_each_rmrr_device(rmrr, bdf, idx)            \
    list_for_each_entry(rmrr, &acpi_rmrr_units, list)   \
        /* assume there never is a bdf == 0 */          \
        for (idx = 0; (bdf = rmrr->scope.devices[idx]) && \
                 idx < rmrr->scope.devices_cnt; idx++)

So,

    for_each_rmrr_device ( rmrr, bdf, i )
    {
        rc = func(...);
        /* Hit this entry so just go next. */
        if ( rc > 0 )
            i = rmrr->scope.devices_cnt;

If you don't intend to reset something of this internal working, its hard to go next rmrr entry. Maybe you already have idea, so could you give me some hints?

Quoted 4 lines

should not special case 1 - just return when rc is negative and skip the rest of the current RMRR when it's positive. And of course make the function's final return value predictable.

Like this,

        /* Hit this entry so just go next. */
        if ( rc > 0 )
            xxxx;
        else if ( rc < 0 )
            return rc;
    }

    return 0;

Thanks
Tiejun

Jan Beulich

2014-11-20 08:59 UTC #125 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 46 lines

Quoted 1 line

Quoted 1 line · On 20.11.14 at 09:12, <[email protected]> wrote:

Quoted 22 lines · Jan Beulich

Quoted 22 lines

Quoted 1 line

Quoted 1 line · On 19.11.14 at 02:26, <[email protected]> wrote:

int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)

{
      struct acpi_rmrr_unit *rmrr;
      int rc = 0;
      unsigned int i;
      u16 bdf;

      for_each_rmrr_device ( rmrr, bdf, i )
      {
          rc = func(PFN_DOWN(rmrr->base_address),
                             PFN_UP(rmrr->end_address) -
                              PFN_DOWN(rmrr->base_address),
                             PCI_SBDF(rmrr->segment, bdf),
                            ctxt);
          /* Hit this entry so just go next. */
          if ( rc == 1 )
              i = rmrr->scope.devices_cnt;
          else if ( rc < 0 )
              return rc;
      }

Quoted 5 lines

Quoted 2 lines
      return rc;
}
Better. Another improvement would be make it not depend on the internal workings of for_each_rmrr_device()... And in any case you

Yes but as you see,

#define for_each_rmrr_device(rmrr, bdf, idx)            \
     list_for_each_entry(rmrr, &acpi_rmrr_units, list)   \
         /* assume there never is a bdf == 0 */          \
         for (idx = 0; (bdf = rmrr->scope.devices[idx]) && \
                  idx < rmrr->scope.devices_cnt; idx++)

So,

     for_each_rmrr_device ( rmrr, bdf, i )
     {
         rc = func(...);
         /* Hit this entry so just go next. */
         if ( rc > 0 )
             i = rmrr->scope.devices_cnt;

Quoted 3 lines

If you don't intend to reset something of this internal working, its hard to go next rmrr entry. Maybe you already have idea, so could you give me some hints?

Have a second struct acpi_rmrr_unit pointer, starting out as NULL and getting set to the current one when the callback returns a positive value. Skip further iterations as long as both pointers match.

Quoted 5 lines

Quoted 4 lines
should not special case 1 - just return when rc is negative and skip the rest of the current RMRR when it's positive. And of course make the function's final return value predictable.

Quoted 8 lines

Like this,

         /* Hit this entry so just go next. */
         if ( rc > 0 )
             xxxx;
         else if ( rc < 0 )
             return rc;
     }

Yes, albeit swapping the order (and dropping the "else" along with adding unlikely() to the error case) would be preferred.

Jan

Chen, Tiejun

author

2014-11-20 10:28 UTC #126 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 4 lines

Have a second struct acpi_rmrr_unit pointer, starting out as NULL and getting set to the current one when the callback returns a positive value. Skip further iterations as long as both pointers match.

Great!

int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)

{
    struct acpi_rmrr_unit *rmrr, *rmrr_cur = NULL;
    int rc = 0;
    unsigned int i;
    u16 bdf;

    for_each_rmrr_device ( rmrr, bdf, i )
    {
        if ( rmrr != rmrr_cur )
        {
            rc = func(PFN_DOWN(rmrr->base_address),
                      PFN_UP(rmrr->end_address) -
                        PFN_DOWN(rmrr->base_address),
                      PCI_SBDF(rmrr->segment, bdf),
                      ctxt);

            if ( unlikely(rc < 0) )
                return rc;
            /* Hit this entry so just go next. */
            if ( rc > 0 )
                rmrr_cur = rmrr;
        }
    }

    return 0;
}

Thanks
Tiejun

Jan Beulich

2014-11-11 08:59 UTC #127 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

In reply to Chen, Tiejun

Quoted 34 lines

Quoted 1 line
Quoted 1 line · On 11.11.14 at 07:32, <[email protected]> wrote:
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.11.14 at 11:27, <[email protected]> wrote:
xen/drivers/passthrough/pci.c +16 -0
@@ -1540,6 +1540,34 @@ int iommu_do_pci_domctl(
           }
           break;
 
+    case XEN_DOMCTL_set_rdm:
+    {
+        struct xen_domctl_set_rdm *xdsr = &domctl->u.set_rdm;
+        struct xen_guest_pcidev_info *pcidevs;
+        int i;
+
+        pcidevs = xmalloc_array(xen_guest_pcidev_info_t,
+                                domctl->u.set_rdm.num_pcidevs);
+        if ( pcidevs == NULL )
+        {
+            return -ENOMEM;
+        }
+
+        for ( i = 0; i < xdsr->num_pcidevs; ++i )
+        {
+            if ( __copy_from_guest_offset(pcidevs, xdsr->pcidevs, i, 1) )
Quoted 8 lines
xen/drivers/passthrough/pci.c int iommu_do_pci_domctl( · line 1563
Quoted 5 lines
xen/drivers/passthrough/pci.c +5 -0
+            {
+                xfree(pcidevs);
+                return -EFAULT;
+            }
+        }
I don't see the need for a loop here. And you definitely can't use the double-underscore-prefixed variant the way you do.

Quoted 4 lines

Do you mean this line?
copy_from_guest_offset(pcidevs, xdsr->pcidevs, 0, xdsr->num_pcidevs*sizeof(xen_guest_pcidev_info_t))

Almost:

    copy_from_guest(pcidevs, xdsr->pcidevs, xdsr->num_pcidevs * sizeof(*pcidevs))

Quoted 11 lines

Quoted 9 lines
xen/include/asm-x86/hvm/domain.h struct hvm_domain { · line 93
Quoted 7 lines
xen/include/asm-x86/hvm/domain.h +1 -0
@@ -90,6 +90,10 @@ struct hvm_domain {
       /* Cached CF8 for guest PCI config cycles */
       uint32_t                pci_cf8;
 
+    uint32_t                pci_rdmforce;
I still don't see why this is a uint32_t.
How about bool_t?

Exactly.

Quoted 2 lines

In Xen side we have 'bool_t', but we have 'bool' in tools side. So how to define this in xen/include/public/domctl.h?

Have a structure field named e.g. "flags" and a #define consuming exactly one bit of it. Just like it's being done everywhere else.

Quoted 10 lines

Quoted 9 lines

xen/include/asm-x86/hvm/domain.h struct xen_domctl { · line 1141

Quoted 7 lines

xen/include/asm-x86/hvm/domain.h +2 -1

@@ -1118,7 +1137,8 @@ struct xen_domctl {
           struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
           struct xen_domctl_vnuma             vnuma;
           struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[112];

Why are you altering the padding size here?

Quoted 2 lines

As I understand we should shrink this pad when we introduce new filed, shouldn't we?

You realize that this is inside a union?

Jan

Chen, Tiejun

author

2014-11-11 09:35 UTC #128 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 8 lines

Quoted 4 lines
Do you mean this line?
copy_from_guest_offset(pcidevs, xdsr->pcidevs, 0, xdsr->num_pcidevs*sizeof(xen_guest_pcidev_info_t))
Almost:
     copy_from_guest(pcidevs, xdsr->pcidevs, xdsr->num_pcidevs * sizeof(*pcidevs))

Fixed.

Quoted 13 lines

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/include/asm-x86/hvm/domain.h +0 -0
@@ -90,6 +90,10 @@ struct hvm_domain {
        /* Cached CF8 for guest PCI config cycles */
        uint32_t                pci_cf8;
 
Quoted 5 lines
Quoted 3 lines
xen/include/asm-x86/hvm/domain.h struct hvm_domain { · line 93
Quoted 1 line
xen/include/asm-x86/hvm/domain.h +1 -0
+    uint32_t                pci_rdmforce;
I still don't see why this is a uint32_t.
How about bool_t?

Quoted 7 lines

Exactly.
Quoted 2 lines
In Xen side we have 'bool_t', but we have 'bool' in tools side. So how to define this in xen/include/public/domctl.h?
Have a structure field named e.g. "flags" and a #define consuming exactly one bit of it. Just like it's being done everywhere else.

Like this?

typedef struct xen_domctl_assign_device xen_domctl_assign_device_t; DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);

+struct xen_guest_pcidev_info {

+    uint8_t bus;
+    uint8_t devfn;
+    struct {
+        uint32_t    force           : 1,
+                    reserved        : 31;

+ }flags;
+};
+typedef struct xen_guest_pcidev_info xen_guest_pcidev_info_t; +DEFINE_XEN_GUEST_HANDLE(xen_guest_pcidev_info_t); +/* Control whether/how we check and reserve device memory. */ +struct xen_domctl_set_rdm {
+ struct {

+        uint32_t    force           : 1,
+                    reserved        : 31;
+    }flags;
+    uint32_t    num_pcidevs;
+    XEN_GUEST_HANDLE_64(xen_guest_pcidev_info_t) pcidevs;

+};
+typedef struct xen_domctl_set_rdm xen_domctl_set_rdm_t; +DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_rdm_t);
+
/* Retrieve sibling devices infomation of machine_sbdf */ /* XEN_DOMCTL_get_device_group */
struct xen_domctl_get_device_group {

Quoted 11 lines

Quoted 10 lines

xen/include/asm-x86/hvm/domain.h struct xen_domctl { · line 1141

Quoted 8 lines

xen/include/asm-x86/hvm/domain.h +2 -1

@@ -1118,7 +1137,8 @@ struct xen_domctl {
            struct xen_domctl_gdbsx_domstatus   gdbsx_domstatus;
            struct xen_domctl_vnuma             vnuma;
            struct xen_domctl_psr_cmt_op        psr_cmt_op;
-        uint8_t                             pad[128];
+        struct xen_domctl_set_rdm           set_rdm;
+        uint8_t                             pad[112];

Why are you altering the padding size here?

Quoted 4 lines

Quoted 2 lines
As I understand we should shrink this pad when we introduce new filed, shouldn't we?
You realize that this is inside a union?

Sorry I didn't realize this point.

Thanks
Tiejun

Jan Beulich

2014-11-11 09:42 UTC #129 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 9 lines

Quoted 3 lines
Quoted 1 line · On 11.11.14 at 10:35, <[email protected]> wrote:
Have a structure field named e.g. "flags" and a #define consuming exactly one bit of it. Just like it's being done everywhere else.
Like this?
  typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
  DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);

Review

Quoted 8 lines

(quoted diff) +8 -0

+struct xen_guest_pcidev_info {
+    uint8_t bus;
+    uint8_t devfn;
+    struct {
+        uint32_t    force           : 1,
+                    reserved        : 31;
+    }flags;
+};

I said #define for a reason - with a few exceptions we try to avoid using bit fields in the public interface.

Jan

Chen, Tiejun

author

2014-11-11 09:51 UTC #130 lore raw

Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

Quoted 9 lines · Jan Beulich

Quoted 9 lines
Quoted 3 lines
Quoted 1 line · On 11.11.14 at 10:35, <[email protected]> wrote:
Have a structure field named e.g. "flags" and a #define consuming exactly one bit of it. Just like it's being done everywhere else.
Like this?
   typedef struct xen_domctl_assign_device xen_domctl_assign_device_t;
   DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);

Quoted 12 lines

Review
Quoted 8 lines
(quoted diff) +8 -0
+struct xen_guest_pcidev_info {
+    uint8_t bus;
+    uint8_t devfn;
+    struct {
+        uint32_t    force           : 1,
+                    reserved        : 31;
+    }flags;
+};
I said #define for a reason - with a few exceptions we try to avoid using bit fields in the public interface.

Ok,

typedef struct xen_domctl_assign_device xen_domctl_assign_device_t; DEFINE_XEN_GUEST_HANDLE(xen_domctl_assign_device_t);

+/* Currently just one bit to indicate force to check Reserved Device Memory. */
+#define PCI_DEV_RDM_CHECK 0x1
+struct xen_guest_pcidev_info {

+    uint8_t bus;
+    uint8_t devfn;
+    uint32_t    flags;

+};
+typedef struct xen_guest_pcidev_info xen_guest_pcidev_info_t; +DEFINE_XEN_GUEST_HANDLE(xen_guest_pcidev_info_t); +/* Control whether/how we check and reserve device memory. */ +struct xen_domctl_set_rdm {

+    uint32_t    flags;
+    uint32_t    num_pcidevs;
+    XEN_GUEST_HANDLE_64(xen_guest_pcidev_info_t) pcidevs;

Thanks
Tiejun

Tiejun Chen

author

2014-10-24 07:34 UTC #131 lore raw

RFC PATCH v7 7/13 xen/x86/p2m: introduce p2m_check_reserved_device_memory

In reply to Tiejun Chen

This can be used conveniently in many cases later.

Signed-off-by Tiejun Chen <[email protected]>

---
 xen/include/asm-x86/p2m.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)

xen/include/asm-x86/p2m.h +13 -0

index 90ddd15..934324e 100644
@@ -713,6 +713,19 @@ extern int arch_grant_map_page_identity(struct domain *d, unsigned long frame,
                                  bool_t writeable);
 extern int arch_grant_unmap_page_identity(struct domain *d, unsigned long frame);
 
+/* Check if we are accessing rdm. */
+static inline int p2m_check_reserved_device_memory(xen_pfn_t start,
+                                                   xen_ulong_t nr, void *d)
+{
+    unsigned long *gfn = d;
+    xen_pfn_t end = start + nr;
+
+    if ( *gfn >= start && *gfn <= end )
+        return 1;
+
+    return 0;
+}
+
 #endif /* _XEN_P2M_H */
 
 /*

Signature · 1 line

1.9.1

Jan Beulich

2014-10-24 15:02 UTC #132 lore raw

Re: [v7][RFC][PATCH 07/13] xen/x86/p2m: introduce p2m_check_reserved_device_memory

Quoted 9 lines

Quoted 1 line

Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:

xen/include/asm-x86/p2m.h +2 -0

@@ -713,6 +713,19 @@ extern int arch_grant_map_page_identity(struct domain *d, unsigned long frame,
                                  bool_t writeable);
 extern int arch_grant_unmap_page_identity(struct domain *d, unsigned long frame);
 
+/* Check if we are accessing rdm. */
+static inline int p2m_check_reserved_device_memory(xen_pfn_t start,

xen/include/asm-x86/p2m.h extern int arch_grant_map_page_identity(struct domain *d, unsigned long frame, · line 727

Quoted 10 lines

xen/include/asm-x86/p2m.h +10 -0

+                                                   xen_ulong_t nr, void *d)
+{
+    unsigned long *gfn = d;
+    xen_pfn_t end = start + nr;
+
+    if ( *gfn >= start && *gfn <= end )
+        return 1;
+
+    return 0;
+}

There is absolutely nothing in the function connecting it to its name. And the last parameter being void looks pretty bogus too. Both signs of it (again) being wrong to add such a function without at least one first caller.

Also the way you return from the function suggests the return type really ought to be bool_t. And instead of the last four lines in the function body you could again use a single simple return statement.

Jan

Chen, Tiejun

author

2014-10-27 08:50 UTC #133 lore raw

Re: [v7][RFC][PATCH 07/13] xen/x86/p2m: introduce p2m_check_reserved_device_memory

Quoted 16 lines · Jan Beulich

Quoted 16 lines

Quoted 1 line

Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:

xen/include/asm-x86/p2m.h +9 -0

@@ -713,6 +713,19 @@ extern int arch_grant_map_page_identity(struct domain *d, unsigned long frame,
                                   bool_t writeable);
  extern int arch_grant_unmap_page_identity(struct domain *d, unsigned long frame);
 
+/* Check if we are accessing rdm. */
+static inline int p2m_check_reserved_device_memory(xen_pfn_t start,
+                                                   xen_ulong_t nr, void *d)
+{
+    unsigned long *gfn = d;
+    xen_pfn_t end = start + nr;
+
+    if ( *gfn >= start && *gfn <= end )
+        return 1;

Quoted 8 lines

xen/include/asm-x86/p2m.h extern int arch_grant_map_page_identity(struct domain *d, unsigned long frame, · line 727
Quoted 3 lines
xen/include/asm-x86/p2m.h +3 -0
+
+    return 0;
+}
There is absolutely nothing in the function connecting it to its name. And the last parameter being void looks pretty bogus too. Both signs of it (again) being wrong to add such a function without at least one first caller.

Looks you hope I fold this into sequent patch, lets address this in that sequent patch as well.

Thanks
Tiejun

Quoted 9 lines

Also the way you return from the function suggests the return type really ought to be bool_t. And instead of the last four lines in the function body you could again use a single simple return statement.
Jan

Tiejun Chen

author

2014-10-24 07:34 UTC #134 lore raw

RFC PATCH v7 8/13 xen/x86/p2m: set p2m_access_n for reserved device memory mapping

In reply to Tiejun Chen

In case of shared-ept or non-shared-ept but 1:1 mapping, we need to set p2m_access_n to make sure all reserved device memory can't be accessed by any !iommu approach.

Signed-off-by Tiejun Chen <[email protected]>

---
 xen/arch/x86/mm/p2m.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

xen/arch/x86/mm/p2m.c +24 -0

index efa49dd..97eb6fd 100644
@@ -686,6 +686,30 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
     /* Now, actually do the two-way mapping */
     if ( mfn_valid(_mfn(mfn)) ) 
     {
+
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            if ( rc )
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);
+                    goto out; /* Failed to update rdm p2m. */
+                }
+            }
+        }
+
         rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
                            p2m->default_access);
         if ( rc )

Signature · 1 line

1.9.1

Jan Beulich

2014-10-24 15:11 UTC #135 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 693

Quoted 12 lines

Quoted 1 line

Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:

xen/arch/x86/mm/p2m.c +5 -0

@@ -686,6 +686,30 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
     /* Now, actually do the two-way mapping */
     if ( mfn_valid(_mfn(mfn)) ) 
     {
+
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);

Okay, no I see what that function is needed for. It being an inline function is of course very questionable looking at this use site.

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 694

Quoted 1 line

xen/arch/x86/mm/p2m.c +1 -0
+            if ( rc )

And the return value from the called function is of type int - non-zero may not just mean "true" but (when negative) also "error". You need to distinguish these cases.

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 701

Quoted 7 lines

xen/arch/x86/mm/p2m.c +7 -0

+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )

How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 707

Quoted 6 lines

xen/arch/x86/mm/p2m.c +6 -0

+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);

Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.

Jan

Chen, Tiejun

author

2014-10-27 09:05 UTC #136 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 15 lines · Jan Beulich

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 693
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
xen/arch/x86/mm/p2m.c +5 -0
@@ -686,6 +686,30 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
      /* Now, actually do the two-way mapping */
      if ( mfn_valid(_mfn(mfn)) )
      {
+
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
Okay, no I see what that function is needed for. It being an inline function is of course very questionable looking at this use site.

xen/arch/x86/mm/p2m.c +11 -0

@@ -556,6 +556,17 @@ guest_physmap_remove_page(struct domain *d, unsigned long gfn,
     gfn_unlock(p2m, gfn, page_order);
 }
 
+/* Check if we are accessing rdm. */
+int p2m_check_reserved_device_memory(xen_pfn_t start,
+                                     xen_ulong_t nr,
+                                     void *d)
+{
+    unsigned long *gfn = d;
+    xen_pfn_t end = start + nr;
+
+   return ( *gfn >= start && *gfn <= end ) ? 1 : 0;
+}
+
 int
 guest_physmap_add_entry(struct domain *d, unsigned long gfn,
                         unsigned long mfn, unsigned int page_order,

Quoted 6 lines

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 694
Quoted 2 lines
xen/arch/x86/mm/p2m.c +1 -0
+            if ( rc )
And the return value from the called function is of type int - non-zero may not just mean "true" but (when negative) also "error". You need to distinguish these cases.

But in our case its impossible to get a negative value.

Quoted 11 lines

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 701
Quoted 8 lines
xen/arch/x86/mm/p2m.c +7 -0
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.

There are two scenarios we should concern:

#1 in case of shared-ept.

We always need to check so iommu_use_hap_pt(d) is good.

#2 in case of non-sharepd-ept

If mfn != gfn I think guest don't access RMRR range, so its allowed.

Quoted 8 lines

xen/arch/x86/mm/p2m.c guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 707

Quoted 7 lines

xen/arch/x86/mm/p2m.c +6 -0

+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);

Quoted 5 lines

Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.

Can P2M_DEBUG work here?

P2M_DEBUG("set rdm p2m failed: %#lx\n", gfn);

Thanks
Tiejun

Jan Beulich

2014-10-27 10:33 UTC #137 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 8 lines

Quoted 1 line
Quoted 1 line · On 27.10.14 at 10:05, <[email protected]> wrote:
Quoted 1 line · Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
int
guest_physmap_add_entry(struct domain *d, unsigned long gfn,
unsigned long mfn, unsigned int page_order,
Quoted 1 line

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
+ if ( rc )
And the return value from the called function is of type int - non-zero may not just mean "true" but (when negative) also "error". You need to distinguish these cases.
But in our case its impossible to get a negative value.

Being guaranteed by what? Please don't simply take the _current implementation_ of iommu_get_reserved_device_memory() as reference - it could be changed at any time, and it allowing for an error return status already would make it perfectly fine for someone adding an actual case thereof not to go through all existing callers to check whether they can cope. This is a general code quality requirement to assure things remain maintainable.

Quoted 13 lines

Quoted 10 lines
Review
Quoted 7 lines
(quoted diff) +7 -0
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.
There are two scenarios we should concern:

Quoted 7 lines

#1 in case of shared-ept.
We always need to check so iommu_use_hap_pt(d) is good.
#2 in case of non-sharepd-ept
If mfn != gfn I think guest don't access RMRR range, so its allowed.

And what if subsequently a device needing a 1:1 mapping at this GFN gets assigned? (I simply don't see why shared vs non-shared would matter here.)

Quoted 13 lines

Quoted 7 lines
Review
Quoted 6 lines
(quoted diff) +6 -0
+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);
Quoted 5 lines
Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.

Quoted 3 lines

Can P2M_DEBUG work here?
P2M_DEBUG("set rdm p2m failed: %#lx\n", gfn);

I don't think this would magically add the missing information. Plus it would limit output to the !NDEBUG case, putting the practical usefulness of this under question even more.

But anyway, looking at the existing code again, I think you'd be better off falling through to the p2m_set_entry() that's already there, just altering the access permission value you pass. Less code, better readable.

Jan

Chen, Tiejun

author

2014-10-28 08:26 UTC #138 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 14 lines · Jan Beulich

Quoted 14 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 10:05, <[email protected]> wrote:
Quoted 1 line · Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
int
guest_physmap_add_entry(struct domain *d, unsigned long gfn,
unsigned long mfn, unsigned int page_order,
Quoted 6 lines
Quoted 2 lines
+ if ( rc )
And the return value from the called function is of type int - non-zero may not just mean "true" but (when negative) also "error". You need to distinguish these cases.

Quoted 5 lines

Quoted 1 line
But in our case its impossible to get a negative value.
Being guaranteed by what? Please don't simply take the _current implementation_ of iommu_get_reserved_device_memory() as reference - it could be changed at any time, and it allowing for an

Okay.

	if ( rc == 1 )

This means we hit a RMRR page. Other failed cases just post a warning message.

Quoted 22 lines

error return status already would make it perfectly fine for someone adding an actual case thereof not to go through all existing callers to check whether they can cope. This is a general code quality requirement to assure things remain maintainable.
Quoted 17 lines
Quoted 10 lines
Review
Quoted 7 lines
(quoted diff) +7 -0
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.
There are two scenarios we should concern:
#1 in case of shared-ept.
We always need to check so iommu_use_hap_pt(d) is good.

Quoted 7 lines

Quoted 3 lines
#2 in case of non-sharepd-ept
If mfn != gfn I think guest don't access RMRR range, so its allowed.
And what if subsequently a device needing a 1:1 mapping at this GFN gets assigned? (I simply don't see why shared vs non-shared would matter here.)

In case of non-shared ept we just create VT-d table, if we assign a device with 1:1 RMRR mapping. So as long as mfn != gfn, its not necessary to set p2m_access_n.

In case of shared ept, we have to set p2m_access_n in any scenarios.

Quoted 14 lines

Quoted 14 lines
Quoted 13 lines
Review
Quoted 7 lines
(quoted diff) +6 -0
+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);
Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.

Quoted 5 lines

Quoted 3 lines
Can P2M_DEBUG work here?
P2M_DEBUG("set rdm p2m failed: %#lx\n", gfn);
I don't think this would magically add the missing information. Plus it

Sorry, is it okay?

gdprintk(XENLOG_WARNING, "Domain %hu set rdm p2m failed: %#lx\n",

                                 d->domain_id, gfn);

And I think I don't understand what you said properly, so I will ask other guys.

Quoted 5 lines

would limit output to the !NDEBUG case, putting the practical usefulness of this under question even more.
But anyway, looking at the existing code again, I think you'd be better off falling through to the p2m_set_entry() that's already there, just

Are you saying I should do this in p2m_set_entry()?

Quoted 1 line

altering the access permission value you pass. Less code, better

But here, guest_physmap_add_entry() just initiate to set p2m_access_n. We will alter the access permission until if we really assign device to create a 1:1 mapping.

Thanks
Tiejun

Quoted 5 lines

readable.
Jan

Jan Beulich

2014-10-28 10:12 UTC #139 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 26 lines

Quoted 1 line
Quoted 1 line · On 28.10.14 at 09:26, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 10:05, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Review
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
(quoted diff) +7 -0
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.
There are two scenarios we should concern:
#1 in case of shared-ept.
We always need to check so iommu_use_hap_pt(d) is good.
#2 in case of non-sharepd-ept
If mfn != gfn I think guest don't access RMRR range, so its allowed.

Quoted 7 lines

Quoted 3 lines
And what if subsequently a device needing a 1:1 mapping at this GFN gets assigned? (I simply don't see why shared vs non-shared would matter here.)
In case of non-shared ept we just create VT-d table, if we assign a device with 1:1 RMRR mapping. So as long as mfn != gfn, its not necessary to set p2m_access_n.

I think it was mentioned before (and not just by me) that it's at least risky to allow the two page tables to get out of sync wrt the translations they do (as opposed to permissions they set).

Quoted 15 lines

Quoted 15 lines
Quoted 15 lines
Quoted 7 lines
Review
Quoted 6 lines
(quoted diff) +6 -0
+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);
Quoted 5 lines
Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.
Can P2M_DEBUG work here?

Quoted 8 lines

Quoted 3 lines
Quoted 1 line
P2M_DEBUG("set rdm p2m failed: %#lx\n", gfn);
I don't think this would magically add the missing information. Plus it
Sorry, is it okay?
gdprintk(XENLOG_WARNING, "Domain %hu set rdm p2m failed: %#lx\n",
                                  d->domain_id, gfn);

Almost. Use printk() instead of gdprintk(), and %d instead of %hu.

Quoted 7 lines

Quoted 5 lines
would limit output to the !NDEBUG case, putting the practical usefulness of this under question even more.
But anyway, looking at the existing code again, I think you'd be better off falling through to the p2m_set_entry() that's already there, just
Are you saying I should do this in p2m_set_entry()?

No. I said that I'd prefer you to use the _existing call to_ p2m_set_entry().

Quoted 5 lines

Quoted 1 line
altering the access permission value you pass. Less code, better
But here, guest_physmap_add_entry() just initiate to set p2m_access_n. We will alter the access permission until if we really assign device to create a 1:1 mapping.

And I didn't question that. All I said is to use the existing call by simply replacing the unconditional use of the default access value with one conditionally set to p2m_access_n.

Jan

Chen, Tiejun

author

2014-10-29 08:20 UTC #140 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 30 lines · Jan Beulich

Quoted 30 lines
Quoted 1 line
Quoted 1 line · On 28.10.14 at 09:26, <[email protected]> wrote:
Quoted 27 lines · Jan Beulich
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 27.10.14 at 10:05, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Review
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 24.10.14 at 09:34, <[email protected]> wrote:
(quoted diff) +7 -0
+            {
+                /*
+                 * Just set p2m_access_n in case of shared-ept
+                 * or non-shared ept but 1:1 mapping.
+                 */
+                if ( iommu_use_hap_pt(d) ||
+                     (!iommu_use_hap_pt(d) && mfn == gfn) )
How would, other than by chance, mfn equal gfn here? Also the double use of iommu_use_hap_pt(d) is pointless here.
There are two scenarios we should concern:
Quoted 7 lines
#1 in case of shared-ept.
We always need to check so iommu_use_hap_pt(d) is good.
#2 in case of non-sharepd-ept
If mfn != gfn I think guest don't access RMRR range, so its allowed.
And what if subsequently a device needing a 1:1 mapping at this GFN gets assigned? (I simply don't see why shared vs non-shared would matter here.)

Quoted 7 lines

Quoted 3 lines
In case of non-shared ept we just create VT-d table, if we assign a device with 1:1 RMRR mapping. So as long as mfn != gfn, its not necessary to set p2m_access_n.
I think it was mentioned before (and not just by me) that it's at least risky to allow the two page tables to get out of sync wrt the translations they do (as opposed to permissions they set).

Okay I will remove all condition check here.

Quoted 40 lines

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Review
Quoted 7 lines
(quoted diff) +6 -0
+                {
+                    rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
+                                       p2m_access_n);
+                    if ( rc )
+                        gdprintk(XENLOG_WARNING, "set rdm p2m failed: (%#lx)\n",
+                                 gfn);
Quoted 5 lines
Such messages are (due to acting on a foreign domain) relatively useless without also logging the domain that is affected. Conversely, logging the current domain and vCPU (due to using gdprintk()) is rather pointless. Also please drop either the colon or the parentheses in the message.
Can P2M_DEBUG work here?
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
P2M_DEBUG("set rdm p2m failed: %#lx\n", gfn);
I don't think this would magically add the missing information. Plus it
Sorry, is it okay?
gdprintk(XENLOG_WARNING, "Domain %hu set rdm p2m failed: %#lx\n",
                                   d->domain_id, gfn);
Almost. Use printk() instead of gdprintk(), and %d instead of %hu.
Quoted 7 lines
Quoted 5 lines
would limit output to the !NDEBUG case, putting the practical usefulness of this under question even more.
But anyway, looking at the existing code again, I think you'd be better off falling through to the p2m_set_entry() that's already there, just
Are you saying I should do this in p2m_set_entry()?
No. I said that I'd prefer you to use the _existing call to_ p2m_set_entry().
Quoted 2 lines
Quoted 1 line
altering the access permission value you pass. Less code, better

Quoted 7 lines

Quoted 3 lines
But here, guest_physmap_add_entry() just initiate to set p2m_access_n. We will alter the access permission until if we really assign device to create a 1:1 mapping.
And I didn't question that. All I said is to use the existing call by simply replacing the unconditional use of the default access value with one conditionally set to p2m_access_n.

Okay,

@@ -686,8 +686,19 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,

     /* Now, actually do the two-way mapping */
     if ( mfn_valid(_mfn(mfn)) )
     {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        if ( !is_hardware_domain(d) )
+        {

+ rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+ &gfn);
+ if ( rc < 0 )
+ printk("Domain %d can't can't check reserved device memory.\n",
+ d->domain_id);
+ }
+

+        /* We need to set reserved device memory as p2m_access_n. */
+        a =  ( rc == 1 ) ? p2m_access_n : p2m->default_access;
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);

if ( rc )
goto out; /* Failed to update p2m, bail without updating
m2p. */

Thanks
Tiejun

Quoted 4 lines

Jan

Jan Beulich

2014-10-29 09:20 UTC #141 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 12 lines

Quoted 1 line

Quoted 1 line · On 29.10.14 at 09:20, <[email protected]> wrote:

(quoted diff) +5 -2

@@ -686,8 +686,19 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
      /* Now, actually do the two-way mapping */
      if ( mfn_valid(_mfn(mfn)) )
      {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);

Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 703

Quoted 10 lines

(quoted diff) +8 -0

+            if ( rc < 0 )
+                printk("Domain %d can't can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        /* We need to set reserved device memory as p2m_access_n. */
+        a =  ( rc == 1 ) ? p2m_access_n : p2m->default_access;
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
          if ( rc )
              goto out; /* Failed to update p2m, bail without updating m2p. */

Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).

But then again this code may change altogether if you avoid populating the reserved regions in the first place.

Jan

Chen, Tiejun

author

2014-10-30 07:39 UTC #142 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 19 lines · Jan Beulich

Quoted 19 lines

Quoted 1 line

Quoted 1 line · On 29.10.14 at 09:20, <[email protected]> wrote:

(quoted diff) +12 -2

@@ -686,8 +686,19 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
       /* Now, actually do the two-way mapping */
       if ( mfn_valid(_mfn(mfn)) )
       {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            if ( rc < 0 )
+                printk("Domain %d can't can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        /* We need to set reserved device memory as p2m_access_n. */
+        a =  ( rc == 1 ) ? p2m_access_n : p2m->default_access;

Quoted 8 lines

Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 703
Quoted 3 lines
(quoted diff) +1 -0
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
           if ( rc )
               goto out; /* Failed to update p2m, bail without updating m2p. */
Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).

Please check the follows:

@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,

     /* Now, actually do the two-way mapping */
     if ( mfn_valid(_mfn(mfn)) )
     {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {

+ rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+ &gfn);
+ /* We need to set reserved device memory as p2m_access_n. */
+ if ( rc == 1 )
+ a = p2m_access_n;
+ else if ( rc < 0 )
+ printk(XENLOG_WARNING
+ "Domain %d can't check reserved device memory.\n",
+ d->domain_id);
+ }
+
+ rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
if ( rc )
goto out; /* Failed to update p2m, bail without updating
m2p. */

Quoted 3 lines

But then again this code may change altogether if you avoid populating the reserved regions in the first place.

Are you saying this scenario?

#1 Here we first set these ranges as p2m_access_n
#2 We reset them as 1:1 RMRR mapping with p2m_access_rw somewhere #3 Someone may try to populate these ranges again

Thanks
Tiejun

Quoted 5 lines

Jan

Jan Beulich

2014-10-30 09:24 UTC #143 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 23 lines

Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).
Please check the follows:
(quoted diff) +8 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
      /* Now, actually do the two-way mapping */
      if ( mfn_valid(_mfn(mfn)) )
      {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )

Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706

Quoted 10 lines

(quoted diff) +8 -0

+                a = p2m_access_n;
+            else if ( rc < 0 )
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
          if ( rc )
              goto out; /* Failed to update p2m, bail without updating m2p. */

The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.

Quoted 8 lines

Quoted 2 lines
But then again this code may change altogether if you avoid populating the reserved regions in the first place.
Are you saying this scenario?
#1 Here we first set these ranges as p2m_access_n
#2 We reset them as 1:1 RMRR mapping with p2m_access_rw somewhere #3 Someone may try to populate these ranges again

No. I pointed at the fact that if you avoid populating the holes, there's no need to force them to p2m_access_n. Any attempts to map other than the 1:1 thing there could then simply be rejected.

Jan

Chen, Tiejun

author

2014-10-31 02:50 UTC #144 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 28 lines · Jan Beulich

Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).
Please check the follows:
(quoted diff) +13 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
       /* Now, actually do the two-way mapping */
       if ( mfn_valid(_mfn(mfn)) )
       {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )
+                a = p2m_access_n;
+            else if ( rc < 0 )
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);

Quoted 8 lines

Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706
Quoted 5 lines
(quoted diff) +3 -0
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
           if ( rc )
               goto out; /* Failed to update p2m, bail without updating m2p. */
The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.

Do you mean I'm missing some necessary info? Like gfn and mfn, so domain id, gfn and mfn can show enough message.

Sorry I'm poor to understand what you expect.

Quoted 6 lines

Quoted 6 lines
Quoted 3 lines
But then again this code may change altogether if you avoid populating the reserved regions in the first place.
Are you saying this scenario?

Quoted 8 lines

Quoted 3 lines
#1 Here we first set these ranges as p2m_access_n
#2 We reset them as 1:1 RMRR mapping with p2m_access_rw somewhere #3 Someone may try to populate these ranges again
No. I pointed at the fact that if you avoid populating the holes, there's no need to force them to p2m_access_n. Any attempts to map other than the 1:1 thing there could then simply be rejected.

I think any population should be rejected totally, because 1:1 mapping means guest can access these RMRR ranges in case of no any device assignment with RMRR, right? Any access to these range corrupt the real device usage.

Thanks
Tiejun

Jan Beulich

2014-10-31 08:25 UTC #145 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 37 lines

Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:50, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).
Please check the follows:
(quoted diff) +10 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
       /* Now, actually do the two-way mapping */
       if ( mfn_valid(_mfn(mfn)) )
       {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )
+                a = p2m_access_n;
+            else if ( rc < 0 )
Quoted 8 lines
Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706
Quoted 8 lines
(quoted diff) +6 -0
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
           if ( rc )
               goto out; /* Failed to update p2m, bail without updating m2p. 
*/
Quoted 1 line

Quoted 7 lines

Quoted 2 lines
The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.
Do you mean I'm missing some necessary info? Like gfn and mfn, so domain id, gfn and mfn can show enough message.
Sorry I'm poor to understand what you expect.

But I explained it already, and that explanation is still visible in the quotes above. But to avoid any doubt, I'll repeat: "And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me)."

Quoted 14 lines

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
But then again this code may change altogether if you avoid populating the reserved regions in the first place.
Are you saying this scenario?
Quoted 8 lines
Quoted 3 lines
#1 Here we first set these ranges as p2m_access_n
#2 We reset them as 1:1 RMRR mapping with p2m_access_rw somewhere #3 Someone may try to populate these ranges again
No. I pointed at the fact that if you avoid populating the holes, there's no need to force them to p2m_access_n. Any attempts to map other than the 1:1 thing there could then simply be rejected.

Quoted 4 lines

I think any population should be rejected totally, because 1:1 mapping means guest can access these RMRR ranges in case of no any device assignment with RMRR, right? Any access to these range corrupt the real device usage.

Oh yes, of course I implied that the 1:1 mapping would be permitted only for those ranges where the RMRR corresponds to a device the guest got assigned.

Jan

Chen, Tiejun

author

2014-11-03 06:20 UTC #146 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 40 lines · Jan Beulich

Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:50, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Getting closer. Just set a to p2m->default_access before the if(), and overwrite it when rc == 1 inside the if(). And properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me).
Please check the follows:
(quoted diff) +10 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
        /* Now, actually do the two-way mapping */
        if ( mfn_valid(_mfn(mfn)) )
        {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )
+                a = p2m_access_n;
+            else if ( rc < 0 )
Quoted 8 lines
Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706
Quoted 8 lines
(quoted diff) +6 -0
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
            if ( rc )
                goto out; /* Failed to update p2m, bail without updating m2p.
*/
Quoted 3 lines
The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.

Quoted 7 lines

Quoted 4 lines
Do you mean I'm missing some necessary info? Like gfn and mfn, so domain id, gfn and mfn can show enough message.
Sorry I'm poor to understand what you expect.
But I explained it already, and that explanation is still visible in the quotes above. But to avoid any doubt, I'll repeat: "And

I tried to understand what you said but felt a confusion so ask if you show me directly.

Quoted 3 lines

properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me)."

Looks there are two problems:

#1: the error message

If current line is not fine,
printk(XENLOG_G_WARNING "Domain %d can't check reserved device
memory.\n", d->domain_id);

I mean could you change this directly.

#2 the error handling

In an error case what should I do? Currently we still create these mapping as normal. This means these mfns will be valid so later we can't set them again then device can't be assigned as passthrough. I think this makes sense. Or we should just stop them from setting 1:1 mapping?

Quoted 15 lines

Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
But then again this code may change altogether if you avoid populating the reserved regions in the first place.
Are you saying this scenario?
#1 Here we first set these ranges as p2m_access_n
#2 We reset them as 1:1 RMRR mapping with p2m_access_rw somewhere #3 Someone may try to populate these ranges again
No. I pointed at the fact that if you avoid populating the holes, there's no need to force them to p2m_access_n. Any attempts to map other than the 1:1 thing there could then simply be rejected.

Quoted 9 lines

Quoted 4 lines
I think any population should be rejected totally, because 1:1 mapping means guest can access these RMRR ranges in case of no any device assignment with RMRR, right? Any access to these range corrupt the real device usage.
Oh yes, of course I implied that the 1:1 mapping would be permitted only for those ranges where the RMRR corresponds to a device the guest got assigned.

Yeah.

Tim,

Please take a look at this, and I hope this can make sure we'll be the same page finally :)

Thanks
Tiejun

Jan Beulich

2014-11-03 09:00 UTC #147 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 51 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 07:20, <[email protected]> wrote:
Quoted 32 lines · Jan Beulich
Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:50, <[email protected]> wrote:
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
(quoted diff) +10 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
        /* Now, actually do the two-way mapping */
        if ( mfn_valid(_mfn(mfn)) )
        {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )
+                a = p2m_access_n;
+            else if ( rc < 0 )
Quoted 8 lines
Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706
Quoted 8 lines
(quoted diff) +6 -0
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
            if ( rc )
                goto out; /* Failed to update p2m, bail without updating m2p.
*/
Quoted 3 lines
The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.
Quoted 7 lines
Quoted 4 lines
Do you mean I'm missing some necessary info? Like gfn and mfn, so domain id, gfn and mfn can show enough message.
Sorry I'm poor to understand what you expect.
But I explained it already, and that explanation is still visible in the quotes above. But to avoid any doubt, I'll repeat: "And
I tried to understand what you said but felt a confusion so ask if you show me directly.
Quoted 3 lines
properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me)."
Looks there are two problems:

Quoted 7 lines

#1: the error message
If current line is not fine,
printk(XENLOG_G_WARNING "Domain %d can't check reserved device
memory.\n", d->domain_id);
I mean could you change this directly.

This looks reasonable, albeit we generally prefer Dom%d or dom%d so that messages are somewhat grep-able.

Quoted 6 lines

#2 the error handling
In an error case what should I do? Currently we still create these mapping as normal. This means these mfns will be valid so later we can't set them again then device can't be assigned as passthrough. I think this makes sense. Or we should just stop them from setting 1:1 mapping?

You should, with very few exceptions, not ignore errors (which includes "handling" them by just logging a message. Instead, you should propagate the error back up the call chain.

Jan

Chen, Tiejun

author

2014-11-03 09:51 UTC #148 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 53 lines · Jan Beulich

Quoted 53 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 07:20, <[email protected]> wrote:
Quoted 32 lines · Jan Beulich
Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 31.10.14 at 03:50, <[email protected]> wrote:
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 30.10.14 at 08:39, <[email protected]> wrote:
(quoted diff) +10 -2
@@ -686,8 +686,22 @@ guest_physmap_add_entry(struct domain *d, unsigned long gfn,
         /* Now, actually do the two-way mapping */
         if ( mfn_valid(_mfn(mfn)) )
         {
-        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t,
-                           p2m->default_access);
+        rc = 0;
+        a =  p2m->default_access;
+        if ( !is_hardware_domain(d) )
+        {
+            rc = iommu_get_reserved_device_memory(p2m_check_reserved_device_memory,
+                                                  &gfn);
+            /* We need to set reserved device memory as p2m_access_n. */
+            if ( rc == 1 )
+                a = p2m_access_n;
+            else if ( rc < 0 )
Quoted 8 lines
Review guest_physmap_add_entry(struct domain *d, unsigned long gfn, · line 706
Quoted 8 lines
(quoted diff) +6 -0
+                printk(XENLOG_WARNING
+                       "Domain %d can't check reserved device memory.\n",
+                       d->domain_id);
+        }
+
+        rc = p2m_set_entry(p2m, gfn, _mfn(mfn), page_order, t, a);
             if ( rc )
                 goto out; /* Failed to update p2m, bail without updating m2p.
*/
Quoted 3 lines
The handling of "a" looks good now, but the error handling and logging is still as broken as it was before.
Quoted 7 lines
Quoted 4 lines
Do you mean I'm missing some necessary info? Like gfn and mfn, so domain id, gfn and mfn can show enough message.
Sorry I'm poor to understand what you expect.
But I explained it already, and that explanation is still visible in the quotes above. But to avoid any doubt, I'll repeat: "And
I tried to understand what you said but felt a confusion so ask if you show me directly.
Quoted 3 lines
properly handle the error case (just logging a message - which btw lacks a proper XENLOG_G_* prefix - doesn't seem enough to me)."
Looks there are two problems:
#1: the error message

Quoted 8 lines

Quoted 5 lines
If current line is not fine,
printk(XENLOG_G_WARNING "Domain %d can't check reserved device
memory.\n", d->domain_id);
I mean could you change this directly.
This looks reasonable, albeit we generally prefer Dom%d or dom%d so that messages are somewhat grep-able.

Fixed.

Quoted 12 lines

Quoted 7 lines
#2 the error handling
In an error case what should I do? Currently we still create these mapping as normal. This means these mfns will be valid so later we can't set them again then device can't be assigned as passthrough. I think this makes sense. Or we should just stop them from setting 1:1 mapping?
You should, with very few exceptions, not ignore errors (which includes "handling" them by just logging a message. Instead, you should propagate the error back up the call chain.

Do you mean in your patch,

+int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt) +{
+ const struct iommu_ops *ops = iommu_get_ops();
+
+ if ( !iommu_enabled || !ops->get_reserved_device_memory )
+ return 0;
+
+ return ops->get_reserved_device_memory(func, ctxt);
+}
+

I shouldn't return that directly. Then instead, we should handle all error scenarios here?

Thanks
Tiejun

Jan Beulich

2014-11-03 10:03 UTC #149 lore raw

Re: [v7][RFC][PATCH 08/13] xen/x86/p2m: set p2m_access_n for reserved device memory mapping

Quoted 28 lines

Quoted 1 line
Quoted 1 line · On 03.11.14 at 10:51, <[email protected]> wrote:
Quoted 12 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 03.11.14 at 07:20, <[email protected]> wrote:
#2 the error handling
In an error case what should I do? Currently we still create these mapping as normal. This means these mfns will be valid so later we can't set them again then device can't be assigned as passthrough. I think this makes sense. Or we should just stop them from setting 1:1 mapping?
You should, with very few exceptions, not ignore errors (which includes "handling" them by just logging a message. Instead, you should propagate the error back up the call chain.
Do you mean in your patch,
(quoted diff) +10 -0
+int iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt)
+{
+    const struct iommu_ops *ops = iommu_get_ops();
+
+    if ( !iommu_enabled || !ops->get_reserved_device_memory )
+        return 0;
+
+    return ops->get_reserved_device_memory(func, ctxt);
+}
+

Quoted 2 lines

I shouldn't return that directly. Then instead, we should handle all error scenarios here?

No. All error scenarios are already being handled here (by propagating the error code to the caller).

Jan