Skip to contentSkip to conversation

Multiple fixes to XENMEM_acquire_resource

PATCH Superseded Andrew Cooper 5 participants 37 messages Jul 28, 2020 – Sep 22, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#9 lore raw

PATCH 2/5 xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper

The existing logic doesn't function in the general case for mapping a guests grant table, due to arbitrary 32 frame limit, and the default grant table limit being 64.

In order to start addressing this, rework the existing grant table logic by implementing a single gnttab_acquire_resource(). This is far more efficient than the previous acquire_grant_table() in memory.c because it doesn't take the grant table write lock, and attempt to grow the table, for every single frame.

The new gnttab_acquire_resource() function subsumes the previous two gnttab_get_{shared,status}_frame() helpers.

No functional change.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/common/grant_table.c      | 93 ++++++++++++++++++++++++++++++-------------
 xen/common/memory.c           | 42 +------------------
 xen/include/xen/grant_table.h | 19 ++++-----
 3 files changed, 75 insertions(+), 79 deletions(-)
xen/common/grant_table.c +66 -27
index 9f0cae52c0..122d1e7596 100644
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;
+
+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;
+    if ( tot_frames != frame + nr_frames )
+        return -EINVAL;
+
+    /* Grow table if necessary. */
+    grant_write_lock(gt);
+    switch ( id )
+    {
+        mfn_t tmp;
+
+    case XENMEM_resource_grant_table_id_shared:
+        rc = gnttab_get_shared_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        if ( gt->gt_version != 2 )
+        {
+    default:
+            rc = -EINVAL;
+            break;
+        }
+        rc = gnttab_get_status_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+    }
+
+    /* Any errors from growing the table? */
+    if ( rc )
+        goto out;
+
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        vaddrs = gt->shared_raw;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        vaddrs = (void **)gt->status;
+        break;
+    }
+
+    for ( i = 0; i < nr_frames; ++i )
+        mfn_list[i] = virt_to_mfn(vaddrs[frame + i]);
+
+ out:
+    grant_write_unlock(gt);
+
+    return rc;
+}
+
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn, mfn_t *mfn)
 {
     int rc = 0;
@@ -4047,33 +4113,6 @@ int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn, mfn_t *mfn)
     return rc;
 }
 
-int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn)
-{
-    struct grant_table *gt = d->grant_table;
-    int rc;
-
-    grant_write_lock(gt);
-    rc = gnttab_get_shared_frame_mfn(d, idx, mfn);
-    grant_write_unlock(gt);
-
-    return rc;
-}
-
-int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn)
-{
-    struct grant_table *gt = d->grant_table;
-    int rc;
-
-    grant_write_lock(gt);
-    rc = (gt->gt_version == 2) ?
-        gnttab_get_status_frame_mfn(d, idx, mfn) : -EINVAL;
-    grant_write_unlock(gt);
-
-    return rc;
-}
-
 static void gnttab_usage_print(struct domain *rd)
 {
     int first = 1;
xen/common/memory.c +2 -40

54 lines not on this page · raw message

xen/include/xen/grant_table.h +7 -12
index 98603604b8..5a2c75b880 100644
@@ -56,10 +56,10 @@ int mem_sharing_gref_to_gfn(struct grant_table *gt, grant_ref_t ref,
 
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn,
                      mfn_t *mfn);
-int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn);
-int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn);
+
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[]);
 
 #else
 
@@ -93,14 +93,9 @@ static inline int gnttab_map_frame(struct domain *d, unsigned long idx,
     return -EINVAL;
 }
 
-static inline int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                                          mfn_t *mfn)
-{
-    return -EINVAL;
-}
-
-static inline int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                                          mfn_t *mfn)
+static inline int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
 {
     return -EINVAL;
 }
Signature · 1 line
2.11.0
#24 lore raw

PATCH 4/5 xen/memory: Fix acquire_resource size semantics

In reply to Andrew Cooper

Calling XENMEM_acquire_resource with a NULL frame_list is a request for the size of the resource, but the returned 32 is bogus.

If someone tries to follow it for XENMEM_resource_ioreq_server, the acquire call will fail as IOREQ servers currently top out at 2 frames, and it is only half the size of the default grant table limit for guests.

Also, no users actually request a resource size, because it was never wired up in the sole implemenation of resource aquisition in Linux.

Introduce a new resource_max_frames() to calculate the size of a resource, and implement it the IOREQ and grant subsystems.

It is impossible to guarentee that a mapping call following a successful size call will succedd (e.g. The target IOREQ server gets destroyed, or the domain switches from grant v2 to v1). Document the restriction, and use the flexibility to simplify the paths to be lockless.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/mm.c             | 20 ++++++++++++++++
 xen/common/grant_table.c      | 19 +++++++++++++++
 xen/common/memory.c           | 55 +++++++++++++++++++++++++++++++++----------
 xen/include/asm-x86/mm.h      |  3 +++
 xen/include/public/memory.h   | 16 +++++++++----
 xen/include/xen/grant_table.h |  8 +++++++
 xen/include/xen/mm.h          |  6 +++++
 7 files changed, 110 insertions(+), 17 deletions(-)
xen/arch/x86/mm.c +20 -0
index 82bc676553..f73a90a2ab 100644
@@ -4600,6 +4600,26 @@ int xenmem_add_to_physmap_one(
     return rc;
 }
 
+unsigned int arch_resource_max_frames(
+    struct domain *d, unsigned int type, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    switch ( type )
+    {
+#ifdef CONFIG_HVM
+    case XENMEM_resource_ioreq_server:
+        if ( !is_hvm_domain(d) )
+            break;
+        /* One frame for the buf-ioreq ring, and one frame per 128 vcpus. */
+        nr = 1 + DIV_ROUND_UP(d->max_vcpus * sizeof(struct ioreq), PAGE_SIZE);
+        break;
+#endif
+    }
+
+    return nr;
+}
+
 int arch_acquire_resource(struct domain *d, unsigned int type,
                           unsigned int id, unsigned long frame,
                           unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/common/grant_table.c +19 -0
index 122d1e7596..0962fc7169 100644
@@ -4013,6 +4013,25 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    /* Don't need the grant lock.  This limit is fixed at domain create time. */
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        nr = d->grant_table->max_grant_frames;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        nr = grant_to_status_frames(d->grant_table->max_grant_frames);
+        break;
+    }
+
+    return nr;
+}
+
 int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/common/memory.c +42 -13

79 lines not on this page · raw message

xen/include/asm-x86/mm.h +3 -0
index 7e74996053..b0caf372a8 100644
@@ -649,6 +649,9 @@ static inline bool arch_mfn_in_directmap(unsigned long mfn)
     return mfn <= (virt_to_mfn(eva - 1) + 1);
 }
 
+unsigned int arch_resource_max_frames(struct domain *d,
+                                      unsigned int type, unsigned int id);
+
 int arch_acquire_resource(struct domain *d, unsigned int type,
                           unsigned int id, unsigned long frame,
                           unsigned int nr_frames, xen_pfn_t mfn_list[]);
xen/include/public/memory.h +12 -4
index 21057ed78e..cea88cf40c 100644
@@ -639,10 +639,18 @@ struct xen_mem_acquire_resource {
 #define XENMEM_resource_grant_table_id_status 1
 
     /*
-     * IN/OUT - As an IN parameter number of frames of the resource
-     *          to be mapped. However, if the specified value is 0 and
-     *          frame_list is NULL then this field will be set to the
-     *          maximum value supported by the implementation on return.
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the resource to be mapped.
+     *
+     * When frame_list is NULL and nr_frames is 0, this is interpreted as a
+     * request for the size of the resource, which shall be returned in the
+     * nr_frames field.
+     *
+     * The size of a resource will never be zero, but a nonzero result doesn't
+     * guarentee that a subsequent mapping request will be successful.  There
+     * are further type/id specific constraints which may change between the
+     * two calls.
      */
     uint32_t nr_frames;
     uint32_t pad;
xen/include/xen/grant_table.h +8 -0
index 5a2c75b880..bae4d79623 100644
@@ -57,6 +57,8 @@ int mem_sharing_gref_to_gfn(struct grant_table *gt, grant_ref_t ref,
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn,
                      mfn_t *mfn);
 
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id);
+
 int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[]);
@@ -93,6 +95,12 @@ static inline int gnttab_map_frame(struct domain *d, unsigned long idx,
     return -EINVAL;
 }
 
+static inline unsigned int gnttab_resource_max_frames(
+    struct domain *d, unsigned int id)
+{
+    return 0;
+}
+
 static inline int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/include/xen/mm.h +6 -0
index 1b2c1f6b32..c184dc1db1 100644
@@ -686,6 +686,12 @@ static inline void put_page_alloc_ref(struct page_info *page)
 }
 
 #ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline unsigned int arch_resource_max_frames(
+    struct domain *d, unsigned int type, unsigned int id)
+{
+    return 0;
+}
+
 static inline int arch_acquire_resource(
     struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
Signature · 1 line
2.11.0
#33 lore raw

PATCH 5/5 tools/foreignmem: Support querying the size of a resource

In reply to Andrew Cooper

With the Xen side of this interface fixed to return real sizes, userspace needs to be able to make the query.

Introduce xenforeignmemory_resource_size() for the purpose, bumping the library minor version and providing compatiblity for the non-Linux builds.

Its not possible to reuse the IOCTL_PRIVCMD_MMAP_RESOURCE infrastructure, because it depends on having already mmap()'d a suitably sized region before it will make an XENMEM_acquire_resource hypercall to Xen.

Instead, open a xencall handle and make an XENMEM_acquire_resource hypercall directly.

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 tools/libs/foreignmemory/Makefile                  |  2 +-
 tools/libs/foreignmemory/core.c                    | 14 +++++++++
 .../libs/foreignmemory/include/xenforeignmemory.h  | 15 ++++++++++
 tools/libs/foreignmemory/libxenforeignmemory.map   |  4 +++
 tools/libs/foreignmemory/linux.c                   | 35 ++++++++++++++++++++++
 tools/libs/foreignmemory/private.h                 | 14 +++++++++
 6 files changed, 83 insertions(+), 1 deletion(-)
tools/libs/foreignmemory/Makefile +1 -1
index 28f1bddc96..8e07f92c59 100644
@@ -2,7 +2,7 @@ XEN_ROOT = $(CURDIR)/../../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 MAJOR    = 1
-MINOR    = 3
+MINOR    = 4
 LIBNAME  := foreignmemory
 USELIBS  := toollog toolcore
 
tools/libs/foreignmemory/core.c +14 -0
index 63f12e2450..5d95c59c48 100644
@@ -53,6 +53,10 @@ xenforeignmemory_handle *xenforeignmemory_open(xentoollog_logger *logger,
         if (!fmem->logger) goto err;
     }
 
+    fmem->xcall = xencall_open(fmem->logger, 0);
+    if ( !fmem->xcall )
+        goto err;
+
     rc = osdep_xenforeignmemory_open(fmem);
     if ( rc  < 0 ) goto err;
 
@@ -61,6 +65,7 @@ xenforeignmemory_handle *xenforeignmemory_open(xentoollog_logger *logger,
 err:
     xentoolcore__deregister_active_handle(&fmem->tc_ah);
     osdep_xenforeignmemory_close(fmem);
+    xencall_close(fmem->xcall);
     xtl_logger_destroy(fmem->logger_tofree);
     free(fmem);
     return NULL;
@@ -75,6 +80,7 @@ int xenforeignmemory_close(xenforeignmemory_handle *fmem)
 
     xentoolcore__deregister_active_handle(&fmem->tc_ah);
     rc = osdep_xenforeignmemory_close(fmem);
+    xencall_close(fmem->xcall);
     xtl_logger_destroy(fmem->logger_tofree);
     free(fmem);
     return rc;
@@ -188,6 +194,14 @@ int xenforeignmemory_unmap_resource(
     return rc;
 }
 
+int xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    return osdep_xenforeignmemory_resource_size(fmem, domid, type,
+                                                id, nr_frames);
+}
+
 /*
  * Local variables:
  * mode: C
tools/libs/foreignmemory/include/xenforeignmemory.h +15 -0
index d594be8df0..1ba2f5316b 100644
@@ -179,6 +179,21 @@ xenforeignmemory_resource_handle *xenforeignmemory_map_resource(
 int xenforeignmemory_unmap_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
 
+/**
+ * Determine the maximum size of a specific resource.
+ *
+ * @parm fmem handle to the open foreignmemory interface
+ * @parm domid the domain id
+ * @parm type the resource type
+ * @parm id the type-specific resource identifier
+ *
+ * Return 0 on success and fills in *nr_frames.  Sets errno and return -1 on
+ * error.
+ */
+int xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames);
+
 #endif
 
 /*
tools/libs/foreignmemory/libxenforeignmemory.map +4 -0
index d5323c87d9..8aca341b99 100644
@@ -19,3 +19,7 @@ VERS_1.3 {
 		xenforeignmemory_map_resource;
 		xenforeignmemory_unmap_resource;
 } VERS_1.2;
+VERS_1.4 {
+	global:
+		xenforeignmemory_resource_size;
+} VERS_1.3;
tools/libs/foreignmemory/linux.c +35 -0
index 8daa5828e3..67e0ca1e83 100644
@@ -28,6 +28,8 @@
 
 #include "private.h"
 
+#include <xen/memory.h>
+
 #define ROUNDUP(_x,_w) (((unsigned long)(_x)+(1UL<<(_w))-1) & ~((1UL<<(_w))-1))
 
 #ifndef O_CLOEXEC
@@ -340,6 +342,39 @@ int osdep_xenforeignmemory_map_resource(
     return 0;
 }
 
+int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    int rc;
+    struct xen_mem_acquire_resource *xmar =
+        xencall_alloc_buffer(fmem->xcall, sizeof(*xmar));
+
+    if ( !xmar )
+    {
+        PERROR("Could not bounce memory for acquire_resource hypercall");
+        return -1;
+    }
+
+    *xmar = (struct xen_mem_acquire_resource){
+        .domid = domid,
+        .type = type,
+        .id = id,
+    };
+
+    rc = xencall2(fmem->xcall, __HYPERVISOR_memory_op,
+                  XENMEM_acquire_resource, (uintptr_t)xmar);
+    if ( rc )
+        goto out;
+
+    *nr_frames = xmar->nr_frames;
+
+ out:
+    xencall_free_buffer(fmem->xcall, xmar);
+
+    return rc;
+}
+
 /*
  * Local variables:
  * mode: C
tools/libs/foreignmemory/private.h +14 -0
index 8f1bf081ed..1a6b685f45 100644
@@ -4,6 +4,7 @@
 #include <xentoollog.h>
 
 #include <xenforeignmemory.h>
+#include <xencall.h>
 
 #include <xentoolcore_internal.h>
 
@@ -20,6 +21,7 @@
 
 struct xenforeignmemory_handle {
     xentoollog_logger *logger, *logger_tofree;
+    xencall_handle *xcall;
     unsigned flags;
     int fd;
     Xentoolcore__Active_Handle tc_ah;
@@ -74,6 +76,15 @@ static inline int osdep_xenforeignmemory_unmap_resource(
 {
     return 0;
 }
+
+static inline int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    errno = EOPNOTSUPP;
+    return -1;
+}
+
 #else
 int osdep_xenforeignmemory_restrict(xenforeignmemory_handle *fmem,
                                     domid_t domid);
@@ -81,6 +92,9 @@ int osdep_xenforeignmemory_map_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
 int osdep_xenforeignmemory_unmap_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
+int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames);
 #endif
 
 #define PERROR(_f...) \
Signature · 1 line
2.11.0
#19 lore raw

PATCH 3/5 xen/memory: Fix compat XENMEM_acquire_resource for size requests

In reply to Andrew Cooper

Copy the nr_frames from the correct structure, so the caller doesn't unconditionally receive 0.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/common/compat/memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/compat/memory.c +1 -1
index 3851f756c7..ed92e05b08 100644
@@ -599,7 +599,7 @@ int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
                 if ( __copy_field_to_guest(
                          guest_handle_cast(compat,
                                            compat_mem_acquire_resource_t),
-                         &cmp.mar, nr_frames) )
+                         nat.mar, nr_frames) )
                     return -EFAULT;
             }
             else
Signature · 1 line
2.11.0
#0 lore raw

cover Multiple fixes to XENMEM_acquire_resource

I thought this was going to be a very simple small bugfix for Michał's Processor Trace series. Serves me right for expecting it not to be full of bear traps...

The sole implementation of acquire_resource never asks for size, so its little surprise that Xen is broken for compat callers, and returns incorrect information for regular callers.

Patches 1-2 are cleanup with no net functional change. Patches 3-5 are fixes to the size side of this interface, and allow userspace to obtain the actual size of resources.

I'm still working on fixing the batch mapping support, which has further sharp corners, especially for compat callers.

This is sufficenet of a series so far to post for comments. A branch can be obtained from:

https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=shortlog;h=refs/heads/xen-acquire-size

Andrew Cooper (5):
xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE xen/gnttab: Rework resource acquisition
xen/memory: Fix compat XENMEM_acquire_resource for size requests xen/memory: Fix acquire_resource size semantics tools/foreignmem: Support querying the size of a resource

 tools/libs/foreignmemory/Makefile                  |   2 +-
 tools/libs/foreignmemory/core.c                    |  14 +++
 .../libs/foreignmemory/include/xenforeignmemory.h  |  15 +++
 tools/libs/foreignmemory/libxenforeignmemory.map   |   4 +
 tools/libs/foreignmemory/linux.c                   |  35 +++++++
 tools/libs/foreignmemory/private.h                 |  14 +++
 xen/arch/x86/Kconfig                               |   1 +
 xen/arch/x86/mm.c                                  |  20 ++++
 xen/common/Kconfig                                 |   3 +
 xen/common/compat/memory.c                         |   2 +-
 xen/common/grant_table.c                           | 112 ++++++++++++++++-----
 xen/common/memory.c                                |  85 +++++++---------
 xen/include/asm-arm/mm.h                           |   8 --
 xen/include/asm-x86/mm.h                           |   3 +
 xen/include/public/memory.h                        |  16 ++-
 xen/include/xen/grant_table.h                      |  21 ++--
 xen/include/xen/mm.h                               |  15 +++
 17 files changed, 273 insertions(+), 97 deletions(-)
Signature · 1 line
2.11.0
#1 lore raw

PATCH 1/5 xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

New architectures shouldn't be forced to implement no-op stubs for unused functionality.

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide compatibility logic in xen/mm.h

No functional change.

---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/Kconfig     | 1 +
 xen/common/Kconfig       | 3 +++
 xen/include/asm-arm/mm.h | 8 --------
 xen/include/xen/mm.h     | 9 +++++++++
 4 files changed, 13 insertions(+), 8 deletions(-)
xen/arch/x86/Kconfig +1 -0
index a636a4bb1e..e7644a0a9d 100644
@@ -6,6 +6,7 @@ config X86
 	select ACPI
 	select ACPI_LEGACY_TABLES_LOOKUP
 	select ARCH_SUPPORTS_INT128
+	select ARCH_ACQUIRE_RESOURCE
 	select COMPAT
 	select CORE_PARKING
 	select HAS_ALTERNATIVE
xen/common/Kconfig +3 -0
index 15e3b79ff5..593459ea6e 100644
@@ -22,6 +22,9 @@ config GRANT_TABLE
 
 	  If unsure, say Y.
 
+config ARCH_ACQUIRE_RESOURCE
+	bool
+
 config HAS_ALTERNATIVE
 	bool
 
xen/include/asm-arm/mm.h +0 -8
index f8ba49b118..0b7de3102e 100644
@@ -358,14 +358,6 @@ static inline void put_page_and_type(struct page_info *page)
 
 void clear_and_clean_page(struct page_info *page);
 
-static inline
-int arch_acquire_resource(struct domain *d, unsigned int type, unsigned int id,
-                          unsigned long frame, unsigned int nr_frames,
-                          xen_pfn_t mfn_list[])
-{
-    return -EOPNOTSUPP;
-}
-
 unsigned int arch_get_dma_bitsize(void);
 
 #endif /*  __ARCH_ARM_MM__ */
xen/include/xen/mm.h +9 -0
index 1061765bcd..1b2c1f6b32 100644
@@ -685,4 +685,13 @@ static inline void put_page_alloc_ref(struct page_info *page)
     }
 }
 
+#ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline int arch_acquire_resource(
+    struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    return -EOPNOTSUPP;
+}
+#endif /* !CONFIG_ARCH_ACQUIRE_RESOURCE */
+
 #endif /* __XEN_MM_H__ */
Signature · 1 line
2.11.0
#10 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper
Quoted 5 lines · Andrew Cooper
xen/common/grant_table.c +0 -0
index 9f0cae52c0..122d1e7596 100644

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4022

Quoted 10 lines
xen/common/grant_table.c +7 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;

I've folded an "= NULL" here to fix the CentOS 6 build.

~Andrew

#34 lore raw

Re: [PATCH 5/5] tools/foreignmem: Support querying the size of a resource

In reply to Andrew Cooper
Quoted 13 lines · Andrew Cooper

With the Xen side of this interface fixed to return real sizes, userspace needs to be able to make the query.

Introduce xenforeignmemory_resource_size() for the purpose, bumping the library minor version and providing compatiblity for the non-Linux builds.

Its not possible to reuse the IOCTL_PRIVCMD_MMAP_RESOURCE infrastructure, because it depends on having already mmap()'d a suitably sized region before it will make an XENMEM_acquire_resource hypercall to Xen.

Instead, open a xencall handle and make an XENMEM_acquire_resource hypercall directly.

Quoted 7 lines

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

I've folded:

tools/Rules.mk +0 -0
index 5ed5664bf7..b8ccf03ea9 100644
@@ -123,7 +123,7 @@ LDLIBS_libxencall = $(SHDEPS_libxencall)

$(XEN_LIBXENCALL)/libxencall$(libextens
 SHLIB_libxencall  = $(SHDEPS_libxencall) -Wl,-rpath-link=$(XEN_LIBXENCALL)
 CFLAGS_libxenforeignmemory = -I$(XEN_LIBXENFOREIGNMEMORY)/include $(CFLAGS_xeninclude)
-SHDEPS_libxenforeignmemory = $(SHLIB_libxentoolcore) +SHDEPS_libxenforeignmemory = $(SHLIB_libxentoolcore) $(SHDEPS_libxencall) LDLIBS_libxenforeignmemory = $(SHDEPS_libxenforeignmemory) $(XEN_LIBXENFOREIGNMEMORY)/libxenforeignmemory$(libextension) SHLIB_libxenforeignmemory  = $(SHDEPS_libxenforeignmemory) -Wl,-rpath-link=$(XEN_LIBXENFOREIGNMEMORY)

(quoted diff) +0 -0

b/tools/libs/foreignmemory/Makefile
index 8e07f92c59..f3a61e27c7 100644

tools/libs/foreignmemory/Makefile +0 -0
@@ -4,7 +4,7 @@ include $(XEN_ROOT)/tools/Rules.mk

 MAJOR    = 1
 MINOR    = 4
 LIBNAME  := foreignmemory
-USELIBS  := toollog toolcore
+USELIBS  := toollog toolcore call

 SRCS-y                 += core.c
 SRCS-$(CONFIG_Linux)   += linux.c

to fix the build in certain containers.

~Andrew

#11 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper
Quoted 9 lines · Andrew Cooper

The existing logic doesn't function in the general case for mapping a guests grant table, due to arbitrary 32 frame limit, and the default grant table limit being 64.

In order to start addressing this, rework the existing grant table logic by implementing a single gnttab_acquire_resource(). This is far more efficient than the previous acquire_grant_table() in memory.c because it doesn't take the grant table write lock, and attempt to grow the table, for every single frame.

Among the code you replace there is a comment "Iterate backwards in case table needs to grow" explaining why what you say about growing the grant table didn't actually happen.

Quoted 8 lines
xen/common/grant_table.c +2 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
      return 0;
  }
 
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4027

Quoted 10 lines
xen/common/grant_table.c +10 -0
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

I can't seem to be able to find an equivalent of this in the old logic, and hence this looks like an unwarranted change in behavior to me. We have quite a few hypercall ops where some count being zero is simply a no-op, i.e. yielding success without doing anything.

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4035

Quoted 7 lines
xen/common/grant_table.c +7 -0
+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;
+    if ( tot_frames != frame + nr_frames )
+        return -EINVAL;

I find the naming here quite confusing. I realize part of this stems from the code you replace, but anyway: "unsigned long frame" typically represents a memory frame number of some sort, making the calculation look as if it was wrong. (Initially I merely meant to ask whether this check isn't redundant with the prior one, or vice versa.)

Quoted 23 lines
xen/common/grant_table.c +23 -0
+    /* Grow table if necessary. */
+    grant_write_lock(gt);
+    switch ( id )
+    {
+        mfn_t tmp;
+
+    case XENMEM_resource_grant_table_id_shared:
+        rc = gnttab_get_shared_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        if ( gt->gt_version != 2 )
+        {
+    default:
+            rc = -EINVAL;
+            break;
+        }
+        rc = gnttab_get_status_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+    }
+
+    /* Any errors from growing the table? */
+    if ( rc )

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4069

Quoted 10 lines
xen/common/grant_table.c +10 -0
+        goto out;
+
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        vaddrs = gt->shared_raw;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        vaddrs = (void **)gt->status;

Now this is the kind of cast that I consider really dangerous, and hence worth trying hard to avoid. With the code structure as is, I don't see an immediate solution though.

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4069

Quoted 2 lines
xen/common/grant_table.c +2 -0
+        break;
+    }

Worth having an ASSERT_UNREACHABLE() default case here?

Jan

Reviewed-by on v1 3/5
#20 lore raw

Re: [PATCH 3/5] xen/memory: Fix compat XENMEM_acquire_resource for size requests

In reply to Andrew Cooper
Quoted 2 lines · Andrew Cooper

Copy the nr_frames from the correct structure, so the caller doesn't unconditionally receive 0.

Well, no - it does get copied from the correct structure. It's just that the field doesn't get set properly up front. Otherwise you'll
(a) build in an unchecked assumption that the native and compat fields match in type and (b) set a bad example for people looking here and then cloning this code in perhaps a case where (a) is not even true. If you agree, the alternative change of setting cmp.mar.nr_frames from nat.mar->nr_frames before the call is

Jan

Reviewed-by on v1 1/5
#3 lore raw

RE: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

In reply to Andrew Cooper
Quoted 10 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Volodymyr Babchuk <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

Quoted 7 lines

New architectures shouldn't be forced to implement no-op stubs for unused functionality.

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide compatibility logic in xen/mm.h

No functional change.

Code-wise, it looks fine, so...

...but ...

Quoted 24 lines

---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/Kconfig     | 1 +
 xen/common/Kconfig       | 3 +++
 xen/include/asm-arm/mm.h | 8 --------
 xen/include/xen/mm.h     | 9 +++++++++
 4 files changed, 13 insertions(+), 8 deletions(-)
xen/arch/x86/Kconfig +0 -0
index a636a4bb1e..e7644a0a9d 100644

xen/arch/x86/Kconfig config X86 · line 9

Quoted 4 lines
xen/arch/x86/Kconfig +1 -0
@@ -6,6 +6,7 @@ config X86
 	select ACPI
 	select ACPI_LEGACY_TABLES_LOOKUP
 	select ARCH_SUPPORTS_INT128
+	select ARCH_ACQUIRE_RESOURCE

... I do wonder whether 'HAS_ACQUIRE_RESOURCE' is a better and more descriptive name.

Quoted 56 lines
xen/arch/x86/Kconfig +0 -0
 	select COMPAT
 	select CORE_PARKING
 	select HAS_ALTERNATIVE
xen/common/Kconfig +3 -0
index 15e3b79ff5..593459ea6e 100644
@@ -22,6 +22,9 @@ config GRANT_TABLE
 
 	  If unsure, say Y.
 
+config ARCH_ACQUIRE_RESOURCE
+	bool
+
 config HAS_ALTERNATIVE
 	bool
 
xen/include/asm-arm/mm.h +0 -8
index f8ba49b118..0b7de3102e 100644
@@ -358,14 +358,6 @@ static inline void put_page_and_type(struct page_info *page)
 
 void clear_and_clean_page(struct page_info *page);
 
-static inline
-int arch_acquire_resource(struct domain *d, unsigned int type, unsigned int id,
-                          unsigned long frame, unsigned int nr_frames,
-                          xen_pfn_t mfn_list[])
-{
-    return -EOPNOTSUPP;
-}
-
 unsigned int arch_get_dma_bitsize(void);
 
 #endif /*  __ARCH_ARM_MM__ */
xen/include/xen/mm.h +9 -1
index 1061765bcd..1b2c1f6b32 100644
@@ -685,4 +685,13 @@ static inline void put_page_alloc_ref(struct page_info *page)
     }
 }
 
+#ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline int arch_acquire_resource(
+    struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    return -EOPNOTSUPP;
+}
+#endif /* !CONFIG_ARCH_ACQUIRE_RESOURCE */
+
 #endif /* __XEN_MM_H__ */
--

2.11.0

#16 lore raw

RE: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper
Quoted 15 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Julien Grall <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 2/5] xen/gnttab: Rework resource acquisition

The existing logic doesn't function in the general case for mapping a guests grant table, due to arbitrary 32 frame limit, and the default grant table limit being 64.

Quoted 6 lines

In order to start addressing this, rework the existing grant table logic by implementing a single gnttab_acquire_resource(). This is far more efficient than the previous acquire_grant_table() in memory.c because it doesn't take the grant table write lock, and attempt to grow the table, for every single frame.

But that should not have happened before because the code deliberately iterates backwards, thereby starting with the last frame, thereby growing the table at most once. (I agree that dropping and re-acquiring the lock every time was sub-optimal).

Quoted 33 lines

The new gnttab_acquire_resource() function subsumes the previous two gnttab_get_{shared,status}_frame() helpers.

No functional change.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/common/grant_table.c      | 93 ++++++++++++++++++++++++++++++-------------
 xen/common/memory.c           | 42 +------------------
 xen/include/xen/grant_table.h | 19 ++++-----
 3 files changed, 75 insertions(+), 79 deletions(-)
xen/common/grant_table.c +2 -0
index 9f0cae52c0..122d1e7596 100644
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4027

Quoted 10 lines
xen/common/grant_table.c +10 -0
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

This was not an error before. Does mapping 0 frames really need to be a failure?

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4033

Quoted 6 lines
xen/common/grant_table.c +6 -0
+
+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;

That name is confusing. 'last_frame' perhaps (and then make the -1 implicit)?

Quoted 26 lines
xen/common/grant_table.c +26 -0
+    if ( tot_frames != frame + nr_frames )
+        return -EINVAL;
+
+    /* Grow table if necessary. */
+    grant_write_lock(gt);
+    switch ( id )
+    {
+        mfn_t tmp;
+
+    case XENMEM_resource_grant_table_id_shared:
+        rc = gnttab_get_shared_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        if ( gt->gt_version != 2 )
+        {
+    default:
+            rc = -EINVAL;
+            break;
+        }
+        rc = gnttab_get_status_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+    }
+
+    /* Any errors from growing the table? */
+    if ( rc )

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4069

Quoted 10 lines
xen/common/grant_table.c +10 -0
+        goto out;
+
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        vaddrs = gt->shared_raw;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        vaddrs = (void **)gt->status;

Erk. Could we avoid this casting nastiness by putting a loop in each switch case. I know it could be considered code duplication but this is a bit icky.

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4077

Quoted 8 lines
xen/common/grant_table.c +8 -0
+        break;
+    }
+
+    for ( i = 0; i < nr_frames; ++i )
+        mfn_list[i] = virt_to_mfn(vaddrs[frame + i]);
+
+ out:
+    grant_write_unlock(gt);

Since you deliberately grew the table first, could you not drop the write lock and acquire it a read lock before looping over the frames?

Paul

Quoted 141 lines
xen/common/grant_table.c +4 -27
+
+    return rc;
+}
+
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn, mfn_t *mfn)
 {
     int rc = 0;
@@ -4047,33 +4113,6 @@ int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn, mfn_t
 *mfn)
     return rc;
 }
 
-int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn)
-{
-    struct grant_table *gt = d->grant_table;
-    int rc;
-
-    grant_write_lock(gt);
-    rc = gnttab_get_shared_frame_mfn(d, idx, mfn);
-    grant_write_unlock(gt);
-
-    return rc;
-}
-
-int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn)
-{
-    struct grant_table *gt = d->grant_table;
-    int rc;
-
-    grant_write_lock(gt);
-    rc = (gt->gt_version == 2) ?
-        gnttab_get_status_frame_mfn(d, idx, mfn) : -EINVAL;
-    grant_write_unlock(gt);
-
-    return rc;
-}
-
 static void gnttab_usage_print(struct domain *rd)
 {
     int first = 1;
xen/common/memory.c +2 -40
index 714077c1e5..dc3a7248e3 100644
@@ -1007,44 +1007,6 @@ static long xatp_permission_check(struct domain *d, unsigned int space)
     return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
 }
 
-static int acquire_grant_table(struct domain *d, unsigned int id,
-                               unsigned long frame,
-                               unsigned int nr_frames,
-                               xen_pfn_t mfn_list[])
-{
-    unsigned int i = nr_frames;
-
-    /* Iterate backwards in case table needs to grow */
-    while ( i-- != 0 )
-    {
-        mfn_t mfn = INVALID_MFN;
-        int rc;
-
-        switch ( id )
-        {
-        case XENMEM_resource_grant_table_id_shared:
-            rc = gnttab_get_shared_frame(d, frame + i, &mfn);
-            break;
-
-        case XENMEM_resource_grant_table_id_status:
-            rc = gnttab_get_status_frame(d, frame + i, &mfn);
-            break;
-
-        default:
-            rc = -EINVAL;
-            break;
-        }
-
-        if ( rc )
-            return rc;
-
-        ASSERT(!mfn_eq(mfn, INVALID_MFN));
-        mfn_list[i] = mfn_x(mfn);
-    }
-
-    return 0;
-}
-
 static int acquire_resource(
     XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
 {
@@ -1091,8 +1053,8 @@ static int acquire_resource(
     switch ( xmar.type )
     {
     case XENMEM_resource_grant_table:
-        rc = acquire_grant_table(d, xmar.id, xmar.frame, xmar.nr_frames,
-                                 mfn_list);
+        rc = gnttab_acquire_resource(d, xmar.id, xmar.frame, xmar.nr_frames,
+                                     mfn_list);
         break;
 
     default:
xen/include/xen/grant_table.h +7 -13
index 98603604b8..5a2c75b880 100644
@@ -56,10 +56,10 @@ int mem_sharing_gref_to_gfn(struct grant_table *gt, grant_ref_t ref,
 
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn,
                      mfn_t *mfn);
-int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn);
-int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                            mfn_t *mfn);
+
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[]);
 
 #else
 
@@ -93,14 +93,9 @@ static inline int gnttab_map_frame(struct domain *d, unsigned long idx,
     return -EINVAL;
 }
 
-static inline int gnttab_get_shared_frame(struct domain *d, unsigned long idx,
-                                          mfn_t *mfn)
-{
-    return -EINVAL;
-}
-
-static inline int gnttab_get_status_frame(struct domain *d, unsigned long idx,
-                                          mfn_t *mfn)
+static inline int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
 {
     return -EINVAL;
 }
--

2.11.0

Reviewed-by on v1 3/5
#23 lore raw

RE: [PATCH 3/5] xen/memory: Fix compat XENMEM_acquire_resource for size requests

In reply to Andrew Cooper
Quoted 11 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Julien Grall <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 3/5] xen/memory: Fix compat XENMEM_acquire_resource for size requests

Quoted 4 lines

Copy the nr_frames from the correct structure, so the caller doesn't unconditionally receive 0.

Quoted 30 lines

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/common/compat/memory.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
xen/common/compat/memory.c +1 -2
index 3851f756c7..ed92e05b08 100644
@@ -599,7 +599,7 @@ int compat_memory_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) compat)
                 if ( __copy_field_to_guest(
                          guest_handle_cast(compat,
                                            compat_mem_acquire_resource_t),
-                         &cmp.mar, nr_frames) )
+                         nat.mar, nr_frames) )
                     return -EFAULT;
             }
             else
--

2.11.0

#25 lore raw

RE: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Andrew Cooper
Quoted 18 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Hubert Jasudowicz <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Michał Leszczyński <[email protected]>; Ian Jackson <[email protected]>
Subject: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

Calling XENMEM_acquire_resource with a NULL frame_list is a request for the size of the resource, but the returned 32 is bogus.

If someone tries to follow it for XENMEM_resource_ioreq_server, the acquire call will fail as IOREQ servers currently top out at 2 frames, and it is only half the size of the default grant table limit for guests.

Quoted 7 lines

Also, no users actually request a resource size, because it was never wired up in the sole implemenation of resource aquisition in Linux.

Introduce a new resource_max_frames() to calculate the size of a resource, and implement it the IOREQ and grant subsystems.

It is impossible to guarentee that a mapping call following a successful size

s/guarantee/guarantee

Quoted 1 line

call will succedd (e.g. The target IOREQ server gets destroyed, or the domain

s/succedd/succeed

Quoted 67 lines

switches from grant v2 to v1). Document the restriction, and use the flexibility to simplify the paths to be lockless.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/mm.c             | 20 ++++++++++++++++
 xen/common/grant_table.c      | 19 +++++++++++++++
 xen/common/memory.c           | 55 +++++++++++++++++++++++++++++++++----------
 xen/include/asm-x86/mm.h      |  3 +++
 xen/include/public/memory.h   | 16 +++++++++----
 xen/include/xen/grant_table.h |  8 +++++++
 xen/include/xen/mm.h          |  6 +++++
 7 files changed, 110 insertions(+), 17 deletions(-)
xen/arch/x86/mm.c +20 -0
index 82bc676553..f73a90a2ab 100644
@@ -4600,6 +4600,26 @@ int xenmem_add_to_physmap_one(
     return rc;
 }
 
+unsigned int arch_resource_max_frames(
+    struct domain *d, unsigned int type, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    switch ( type )
+    {
+#ifdef CONFIG_HVM
+    case XENMEM_resource_ioreq_server:
+        if ( !is_hvm_domain(d) )
+            break;
+        /* One frame for the buf-ioreq ring, and one frame per 128 vcpus. */
+        nr = 1 + DIV_ROUND_UP(d->max_vcpus * sizeof(struct ioreq), PAGE_SIZE);
+        break;
+#endif
+    }
+
+    return nr;
+}
+
 int arch_acquire_resource(struct domain *d, unsigned int type,
                           unsigned int id, unsigned long frame,
                           unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/common/grant_table.c +3 -0
index 122d1e7596..0962fc7169 100644
@@ -4013,6 +4013,25 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id)
+{
+    unsigned int nr = 0;

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4028

Quoted 10 lines
xen/common/grant_table.c +10 -0
+
+    /* Don't need the grant lock.  This limit is fixed at domain create time. */
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        nr = d->grant_table->max_grant_frames;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        nr = grant_to_status_frames(d->grant_table->max_grant_frames);

Two uses of d->grant_table, so perhaps define a stack variable for it? Also, should you not make sure 0 is returned in the case of a v1 table?

Quoted 14 lines
xen/common/grant_table.c +6 -0
+        break;
+    }
+
+    return nr;
+}
+
 int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/common/memory.c +0 -0
index dc3a7248e3..21edabf9cc 100644

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1014

Quoted 8 lines
xen/common/memory.c +5 -0
@@ -1007,6 +1007,26 @@ static long xatp_permission_check(struct domain *d, unsigned int space)
     return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
 }
 
+/*
+ * Return 0 on any kind of error.  Caller converts to -EINVAL.
+ *
+ * All nonzero values should be repeatable (i.e. derived from some fixed
+ * proerty of the domain), and describe the full resource (i.e. mapping the

s/property/property

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1015

Quoted 1 line
xen/common/memory.c +1 -0
+ * result of this call will be the entire resource).

This precludes dynamically adding a resource to a running domain. Do we really want to bake in that restriction?

Quoted 99 lines
xen/common/memory.c +36 -13
+ */
+static unsigned int resource_max_frames(struct domain *d,
+                                        unsigned int type, unsigned int id)
+{
+    switch ( type )
+    {
+    case XENMEM_resource_grant_table:
+        return gnttab_resource_max_frames(d, id);
+
+    default:
+        return arch_resource_max_frames(d, type, id);
+    }
+}
+
 static int acquire_resource(
     XEN_GUEST_HANDLE_PARAM(xen_mem_acquire_resource_t) arg)
 {
@@ -1018,6 +1038,7 @@ static int acquire_resource(
      * use-cases then per-CPU arrays or heap allocations may be required.
      */
     xen_pfn_t mfn_list[32];
+    unsigned int max_frames;
     int rc;
 
     if ( copy_from_guest(&xmar, arg, 1) )
@@ -1026,19 +1047,6 @@ static int acquire_resource(
     if ( xmar.pad != 0 )
         return -EINVAL;
 
-    if ( guest_handle_is_null(xmar.frame_list) )
-    {
-        if ( xmar.nr_frames )
-            return -EINVAL;
-
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
-
-        if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
-            return -EFAULT;
-
-        return 0;
-    }
-
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
         return -E2BIG;
 
@@ -1050,6 +1058,27 @@ static int acquire_resource(
     if ( rc )
         goto out;
 
+    max_frames = resource_max_frames(d, xmar.type, xmar.id);
+
+    rc = -EINVAL;
+    if ( !max_frames )
+        goto out;
+
+    if ( guest_handle_is_null(xmar.frame_list) )
+    {
+        if ( xmar.nr_frames )
+            goto out;
+
+        xmar.nr_frames = max_frames;
+
+        rc = -EFAULT;
+        if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
+            goto out;
+
+        rc = 0;
+        goto out;
+    }
+
     switch ( xmar.type )
     {
     case XENMEM_resource_grant_table:
xen/include/asm-x86/mm.h +3 -0
index 7e74996053..b0caf372a8 100644
@@ -649,6 +649,9 @@ static inline bool arch_mfn_in_directmap(unsigned long mfn)
     return mfn <= (virt_to_mfn(eva - 1) + 1);
 }
 
+unsigned int arch_resource_max_frames(struct domain *d,
+                                      unsigned int type, unsigned int id);
+
 int arch_acquire_resource(struct domain *d, unsigned int type,
                           unsigned int id, unsigned long frame,
                           unsigned int nr_frames, xen_pfn_t mfn_list[]);
xen/include/public/memory.h +0 -4
index 21057ed78e..cea88cf40c 100644
@@ -639,10 +639,18 @@ struct xen_mem_acquire_resource {
 #define XENMEM_resource_grant_table_id_status 1
 
     /*
-     * IN/OUT - As an IN parameter number of frames of the resource
-     *          to be mapped. However, if the specified value is 0 and
-     *          frame_list is NULL then this field will be set to the
-     *          maximum value supported by the implementation on return.

xen/include/public/memory.h struct xen_mem_acquire_resource { · line 651

Quoted 10 lines
xen/include/public/memory.h +10 -0
+     * IN/OUT
+     *
+     * As an IN parameter number of frames of the resource to be mapped.
+     *
+     * When frame_list is NULL and nr_frames is 0, this is interpreted as a
+     * request for the size of the resource, which shall be returned in the
+     * nr_frames field.
+     *
+     * The size of a resource will never be zero, but a nonzero result doesn't
+     * guarentee that a subsequent mapping request will be successful.  There

s/guarantee/guarantee

Paul

Quoted 51 lines
xen/include/public/memory.h +2 -0
+     * are further type/id specific constraints which may change between the
+     * two calls.
      */
     uint32_t nr_frames;
     uint32_t pad;
xen/include/xen/grant_table.h +8 -0
index 5a2c75b880..bae4d79623 100644
@@ -57,6 +57,8 @@ int mem_sharing_gref_to_gfn(struct grant_table *gt, grant_ref_t ref,
 int gnttab_map_frame(struct domain *d, unsigned long idx, gfn_t gfn,
                      mfn_t *mfn);
 
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id);
+
 int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[]);
@@ -93,6 +95,12 @@ static inline int gnttab_map_frame(struct domain *d, unsigned long idx,
     return -EINVAL;
 }
 
+static inline unsigned int gnttab_resource_max_frames(
+    struct domain *d, unsigned int id)
+{
+    return 0;
+}
+
 static inline int gnttab_acquire_resource(
     struct domain *d, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/include/xen/mm.h +6 -1
index 1b2c1f6b32..c184dc1db1 100644
@@ -686,6 +686,12 @@ static inline void put_page_alloc_ref(struct page_info *page)
 }
 
 #ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline unsigned int arch_resource_max_frames(
+    struct domain *d, unsigned int type, unsigned int id)
+{
+    return 0;
+}
+
 static inline int arch_acquire_resource(
     struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
     unsigned int nr_frames, xen_pfn_t mfn_list[])
--

2.11.0

Reviewed-by on v1 5/5
#36 lore raw

RE: [PATCH 5/5] tools/foreignmem: Support querying the size of a resource

In reply to Andrew Cooper
Quoted 15 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 5/5] tools/foreignmem: Support querying the size of a resource

With the Xen side of this interface fixed to return real sizes, userspace needs to be able to make the query.

Introduce xenforeignmemory_resource_size() for the purpose, bumping the library minor version and providing compatiblity for the non-Linux builds.

Quoted 6 lines

Its not possible to reuse the IOCTL_PRIVCMD_MMAP_RESOURCE infrastructure, because it depends on having already mmap()'d a suitably sized region before it will make an XENMEM_acquire_resource hypercall to Xen.

Instead, open a xencall handle and make an XENMEM_acquire_resource hypercall directly.

Shame we have to do that but, as you say, it's the only option.

Quoted 2 lines
Quoted 215 lines

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 tools/libs/foreignmemory/Makefile                  |  2 +-
 tools/libs/foreignmemory/core.c                    | 14 +++++++++
 .../libs/foreignmemory/include/xenforeignmemory.h  | 15 ++++++++++
 tools/libs/foreignmemory/libxenforeignmemory.map   |  4 +++
 tools/libs/foreignmemory/linux.c                   | 35 ++++++++++++++++++++++
 tools/libs/foreignmemory/private.h                 | 14 +++++++++
 6 files changed, 83 insertions(+), 1 deletion(-)
tools/libs/foreignmemory/Makefile +1 -1
index 28f1bddc96..8e07f92c59 100644
@@ -2,7 +2,7 @@ XEN_ROOT = $(CURDIR)/../../..
 include $(XEN_ROOT)/tools/Rules.mk
 
 MAJOR    = 1
-MINOR    = 3
+MINOR    = 4
 LIBNAME  := foreignmemory
 USELIBS  := toollog toolcore
 
tools/libs/foreignmemory/core.c +14 -0
index 63f12e2450..5d95c59c48 100644
@@ -53,6 +53,10 @@ xenforeignmemory_handle *xenforeignmemory_open(xentoollog_logger *logger,
         if (!fmem->logger) goto err;
     }
 
+    fmem->xcall = xencall_open(fmem->logger, 0);
+    if ( !fmem->xcall )
+        goto err;
+
     rc = osdep_xenforeignmemory_open(fmem);
     if ( rc  < 0 ) goto err;
 
@@ -61,6 +65,7 @@ xenforeignmemory_handle *xenforeignmemory_open(xentoollog_logger *logger,
 err:
     xentoolcore__deregister_active_handle(&fmem->tc_ah);
     osdep_xenforeignmemory_close(fmem);
+    xencall_close(fmem->xcall);
     xtl_logger_destroy(fmem->logger_tofree);
     free(fmem);
     return NULL;
@@ -75,6 +80,7 @@ int xenforeignmemory_close(xenforeignmemory_handle *fmem)
 
     xentoolcore__deregister_active_handle(&fmem->tc_ah);
     rc = osdep_xenforeignmemory_close(fmem);
+    xencall_close(fmem->xcall);
     xtl_logger_destroy(fmem->logger_tofree);
     free(fmem);
     return rc;
@@ -188,6 +194,14 @@ int xenforeignmemory_unmap_resource(
     return rc;
 }
 
+int xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    return osdep_xenforeignmemory_resource_size(fmem, domid, type,
+                                                id, nr_frames);
+}
+
 /*
  * Local variables:
  * mode: C
tools/libs/foreignmemory/include/xenforeignmemory.h +15 -0
index d594be8df0..1ba2f5316b 100644
@@ -179,6 +179,21 @@ xenforeignmemory_resource_handle *xenforeignmemory_map_resource(
 int xenforeignmemory_unmap_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
 
+/**
+ * Determine the maximum size of a specific resource.
+ *
+ * @parm fmem handle to the open foreignmemory interface
+ * @parm domid the domain id
+ * @parm type the resource type
+ * @parm id the type-specific resource identifier
+ *
+ * Return 0 on success and fills in *nr_frames.  Sets errno and return -1 on
+ * error.
+ */
+int xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames);
+
 #endif
 
 /*
tools/libs/foreignmemory/libxenforeignmemory.map +4 -0
index d5323c87d9..8aca341b99 100644
@@ -19,3 +19,7 @@ VERS_1.3 {
 		xenforeignmemory_map_resource;
 		xenforeignmemory_unmap_resource;
 } VERS_1.2;
+VERS_1.4 {
+	global:
+		xenforeignmemory_resource_size;
+} VERS_1.3;
tools/libs/foreignmemory/linux.c +35 -0
index 8daa5828e3..67e0ca1e83 100644
@@ -28,6 +28,8 @@
 
 #include "private.h"
 
+#include <xen/memory.h>
+
 #define ROUNDUP(_x,_w) (((unsigned long)(_x)+(1UL<<(_w))-1) & ~((1UL<<(_w))-1))
 
 #ifndef O_CLOEXEC
@@ -340,6 +342,39 @@ int osdep_xenforeignmemory_map_resource(
     return 0;
 }
 
+int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    int rc;
+    struct xen_mem_acquire_resource *xmar =
+        xencall_alloc_buffer(fmem->xcall, sizeof(*xmar));
+
+    if ( !xmar )
+    {
+        PERROR("Could not bounce memory for acquire_resource hypercall");
+        return -1;
+    }
+
+    *xmar = (struct xen_mem_acquire_resource){
+        .domid = domid,
+        .type = type,
+        .id = id,
+    };
+
+    rc = xencall2(fmem->xcall, __HYPERVISOR_memory_op,
+                  XENMEM_acquire_resource, (uintptr_t)xmar);
+    if ( rc )
+        goto out;
+
+    *nr_frames = xmar->nr_frames;
+
+ out:
+    xencall_free_buffer(fmem->xcall, xmar);
+
+    return rc;
+}
+
 /*
  * Local variables:
  * mode: C
tools/libs/foreignmemory/private.h +14 -1
index 8f1bf081ed..1a6b685f45 100644
@@ -4,6 +4,7 @@
 #include <xentoollog.h>
 
 #include <xenforeignmemory.h>
+#include <xencall.h>
 
 #include <xentoolcore_internal.h>
 
@@ -20,6 +21,7 @@
 
 struct xenforeignmemory_handle {
     xentoollog_logger *logger, *logger_tofree;
+    xencall_handle *xcall;
     unsigned flags;
     int fd;
     Xentoolcore__Active_Handle tc_ah;
@@ -74,6 +76,15 @@ static inline int osdep_xenforeignmemory_unmap_resource(
 {
     return 0;
 }
+
+static inline int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames)
+{
+    errno = EOPNOTSUPP;
+    return -1;
+}
+
 #else
 int osdep_xenforeignmemory_restrict(xenforeignmemory_handle *fmem,
                                     domid_t domid);
@@ -81,6 +92,9 @@ int osdep_xenforeignmemory_map_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
 int osdep_xenforeignmemory_unmap_resource(
     xenforeignmemory_handle *fmem, xenforeignmemory_resource_handle *fres);
+int osdep_xenforeignmemory_resource_size(
+    xenforeignmemory_handle *fmem, domid_t domid, unsigned int type,
+    unsigned int id, unsigned long *nr_frames);
 #endif
 
 #define PERROR(_f...) \
--

2.11.0

Acked-by on v1 1/5
#6 lore raw

Re: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

In reply to Andrew Cooper

Hi Andrew,

Quoted 9 lines · Andrew Cooper

New architectures shouldn't be forced to implement no-op stubs for unused functionality.

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide compatibility logic in xen/mm.h

No functional change.

With one question below:

xen/include/xen/mm.h static inline void put_page_alloc_ref(struct page_info *page) · line 691

Quoted 12 lines
xen/include/xen/mm.h +4 -0
index 1061765bcd..1b2c1f6b32 100644
@@ -685,4 +685,13 @@ static inline void put_page_alloc_ref(struct page_info *page)
      }
  }
 
+#ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline int arch_acquire_resource(
+    struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])

Any reason to change the way we indent the arguments?

xen/include/xen/mm.h static inline void put_page_alloc_ref(struct page_info *page) · line 697

Quoted 7 lines
xen/include/xen/mm.h +5 -0
+{
+    return -EOPNOTSUPP;
+}
+#endif /* !CONFIG_ARCH_ACQUIRE_RESOURCE */
+
  #endif /* __XEN_MM_H__ */

Cheers,

Signature · 1 line
Julien Grall
#18 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper

Hi Andrew,

Quoted 38 lines · Andrew Cooper

The existing logic doesn't function in the general case for mapping a guests grant table, due to arbitrary 32 frame limit, and the default grant table limit being 64.

In order to start addressing this, rework the existing grant table logic by implementing a single gnttab_acquire_resource(). This is far more efficient than the previous acquire_grant_table() in memory.c because it doesn't take the grant table write lock, and attempt to grow the table, for every single frame.

The new gnttab_acquire_resource() function subsumes the previous two gnttab_get_{shared,status}_frame() helpers.

No functional change.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
  xen/common/grant_table.c      | 93 ++++++++++++++++++++++++++++++-------------
  xen/common/memory.c           | 42 +------------------
  xen/include/xen/grant_table.h | 19 ++++-----
  3 files changed, 75 insertions(+), 79 deletions(-)
xen/common/grant_table.c +0 -0
index 9f0cae52c0..122d1e7596 100644

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4018

Quoted 6 lines
xen/common/grant_table.c +3 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
      return 0;
  }
 
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])

Any reasons for changing the way we usually indent parameters?

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4023

Quoted 5 lines
xen/common/grant_table.c +5 -0
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;

AFAICT rc will always be initialized when used. So is it necessary?

Quoted 19 lines
xen/common/grant_table.c +19 -0
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;
+
+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;
+    if ( tot_frames != frame + nr_frames )
+        return -EINVAL;
+
+    /* Grow table if necessary. */
+    grant_write_lock(gt);
+    switch ( id )
+    {
+        mfn_t tmp;
+

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4052

Quoted 10 lines
xen/common/grant_table.c +10 -0
+    case XENMEM_resource_grant_table_id_shared:
+        rc = gnttab_get_shared_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        if ( gt->gt_version != 2 )
+        {
+    default:
+            rc = -EINVAL;
+            break;

I am aware this is valid C code, but I find this construct confusing. Could you just duplicate the 2 lines and have a separate default case?

Quoted 10 lines
xen/common/grant_table.c +10 -0
+        }
+        rc = gnttab_get_status_frame_mfn(d, tot_frames - 1, &tmp);
+        break;
+    }
+
+    /* Any errors from growing the table? */
+    if ( rc )
+        goto out;
+
+    switch ( id )

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4072

Quoted 10 lines
xen/common/grant_table.c +10 -0
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        vaddrs = gt->shared_raw;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        vaddrs = (void **)gt->status;
+        break;
+    }
+

NIT: Would it make sense to check vaddrs? This would avoid NULL dereference pointers if something went wrong.

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4081

Quoted 9 lines
xen/common/grant_table.c +9 -0
+    for ( i = 0; i < nr_frames; ++i )
+        mfn_list[i] = virt_to_mfn(vaddrs[frame + i]);
+
+ out:
+    grant_write_unlock(gt);
+
+    return rc;
+}
+

Cheers,

Signature · 1 line
Julien Grall
#26 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Paul Durrant

Hi Paul,

Quoted 16 lines · Paul Durrant

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1014

Quoted 13 lines
xen/common/memory.c +5 -0
index dc3a7248e3..21edabf9cc 100644
@@ -1007,6 +1007,26 @@ static long xatp_permission_check(struct domain *d, unsigned int space)
      return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
  }
 
+/*
+ * Return 0 on any kind of error.  Caller converts to -EINVAL.
+ *
+ * All nonzero values should be repeatable (i.e. derived from some fixed
+ * proerty of the domain), and describe the full resource (i.e. mapping the

s/property/property

Quoted 3 lines

xen/common/memory.c static long xatp_permission_check(struct domain *d, unsigned int space) · line 1015

Quoted 1 line
xen/common/memory.c +1 -0
+ * result of this call will be the entire resource).

This precludes dynamically adding a resource to a running domain. Do we really want to bake in that restriction?

AFAICT, this restriction is not documented in the ABI. In particular, it is written:

"
The size of a resource will never be zero, but a nonzero result doesn't guarentee that a subsequent mapping request will be successful. There are further type/id specific constraints which may change between the two calls.
"

So I think a domain couldn't rely on this behavior. Although, it might be good to clarify in the comment on top of resource_max_frames that this an implementation decision and not part of the ABI.

Cheers,

Signature · 1 line
Julien Grall
#7 lore raw

Re: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

In reply to Julien Grall
Quoted 11 lines · Julien Grall

Hi Andrew,

Quoted 8 lines · Andrew Cooper

New architectures shouldn't be forced to implement no-op stubs for unused
functionality.

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide
compatibility logic in xen/mm.h

Quoted 7 lines
Quoted 3 lines

No functional change.

With one question below:

Thanks,

Quoted 17 lines
Quoted 10 lines
xen/include/xen/mm.h +0 -0
index 1061765bcd..1b2c1f6b32 100644
@@ -685,4 +685,13 @@ static inline void put_page_alloc_ref(struct
 page_info *page)
       }
   }
Quoted 5 lines

  +#ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline int arch_acquire_resource(
+    struct domain *d, unsigned int type, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])

Any reason to change the way we indent the arguments?

So its not all squashed on the right hand side.

~Andrew

#4 lore raw

Re: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

In reply to Paul Durrant
Quoted 13 lines · Paul Durrant
Quoted 13 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; Jan Beulich <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Volodymyr Babchuk <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

New architectures shouldn't be forced to implement no-op stubs for unused functionality.

Quoted 7 lines
Quoted 4 lines

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide compatibility logic in xen/mm.h

No functional change.

Code-wise, it looks fine, so...

Thanks,

Quoted 21 lines

...but ...

Quoted 18 lines

---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/Kconfig     | 1 +
 xen/common/Kconfig       | 3 +++
 xen/include/asm-arm/mm.h | 8 --------
 xen/include/xen/mm.h     | 9 +++++++++
 4 files changed, 13 insertions(+), 8 deletions(-)
Quoted 10 lines

xen/arch/x86/Kconfig config X86 · line 9

Quoted 9 lines
xen/arch/x86/Kconfig +1 -0
index a636a4bb1e..e7644a0a9d 100644
@@ -6,6 +6,7 @@ config X86
 	select ACPI
 	select ACPI_LEGACY_TABLES_LOOKUP
 	select ARCH_SUPPORTS_INT128
+	select ARCH_ACQUIRE_RESOURCE

... I do wonder whether 'HAS_ACQUIRE_RESOURCE' is a better and more descriptive name.

We don't have a coherent policy for how to categorise these things.  I can change the name if you insist, but I'm not sure it makes a useful difference.

~Andrew

#5 lore raw

RE: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

Quoted 68 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 30 July 2020 18:34
To: [email protected]; 'Xen-devel' <[email protected]>
Cc: 'Jan Beulich' <[email protected]>; 'Wei Liu' <[email protected]>; 'Roger Pau Monné' <[email protected]>; 'Stefano Stabellini' <[email protected]>; 'Julien Grall' <[email protected]>; 'Volodymyr Babchuk' <[email protected]>; 'Michał Leszczyński' <[email protected]>; 'Hubert Jasudowicz' <[email protected]>
Subject: Re: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; Jan Beulich <[email protected]>; Wei Liu

<[email protected]>;

Quoted 2 lines
Quoted 2 lines

Roger Pau Monné <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Volodymyr Babchuk <[email protected]>; Paul Durrant <[email protected]>;

Michał

Quoted 6 lines
Quoted 6 lines

Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

New architectures shouldn't be forced to implement no-op stubs for unused functionality.

Quoted 7 lines
Quoted 4 lines

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide compatibility logic in xen/mm.h

No functional change.

Code-wise, it looks fine, so...

Thanks,

Quoted 21 lines

...but ...

Quoted 18 lines

---
CC: Jan Beulich <[email protected]>
CC: Wei Liu <[email protected]>
CC: Roger Pau Monné <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Julien Grall <[email protected]>
CC: Volodymyr Babchuk <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/Kconfig     | 1 +
 xen/common/Kconfig       | 3 +++
 xen/include/asm-arm/mm.h | 8 --------
 xen/include/xen/mm.h     | 9 +++++++++
 4 files changed, 13 insertions(+), 8 deletions(-)
Quoted 10 lines

xen/arch/x86/Kconfig config X86 · line 9

Quoted 9 lines
xen/arch/x86/Kconfig +1 -0
index a636a4bb1e..e7644a0a9d 100644
@@ -6,6 +6,7 @@ config X86
 	select ACPI
 	select ACPI_LEGACY_TABLES_LOOKUP
 	select ARCH_SUPPORTS_INT128
+	select ARCH_ACQUIRE_RESOURCE

... I do wonder whether 'HAS_ACQUIRE_RESOURCE' is a better and more descriptive name.

Quoted 4 lines

We don't have a coherent policy for how to categorise these things. I can change the name if you insist, but I'm not sure it makes a useful difference.

Ok, it's fine. My R-b stands.

Paul

Quoted 1 line

~Andrew

#8 lore raw

Re: [PATCH 1/5] xen/memory: Introduce CONFIG_ARCH_ACQUIRE_RESOURCE

In reply to Andrew Cooper
Quoted 40 lines · Andrew Cooper
Quoted 11 lines · Julien Grall

Hi Andrew,

Quoted 8 lines · Andrew Cooper

New architectures shouldn't be forced to implement no-op stubs for unused
functionality.

Introduce CONFIG_ARCH_ACQUIRE_RESOURCE which can be opted in to, and provide
compatibility logic in xen/mm.h

Quoted 7 lines
Quoted 3 lines

No functional change.

With one question below:

Thanks,

Quoted 17 lines
Quoted 13 lines
xen/include/xen/mm.h +2 -0
index 1061765bcd..1b2c1f6b32 100644
@@ -685,4 +685,13 @@ static inline void put_page_alloc_ref(struct
 page_info *page)
       }
   }
   +#ifndef CONFIG_ARCH_ACQUIRE_RESOURCE
+static inline int arch_acquire_resource(
+    struct domain *d, unsigned int type, unsigned int id, unsigned
Quoted 2 lines

long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])

Any reason to change the way we indent the arguments?

Quoted 1 line

So its not all squashed on the right hand side.

Fair enough. I have asked the same question on a follow-up patch. Feel free to ignore it :).

Cheers,

Signature · 1 line
Julien Grall
#21 lore raw

Re: [PATCH 3/5] xen/memory: Fix compat XENMEM_acquire_resource for size requests

In reply to Jan Beulich
Quoted 6 lines · Jan Beulich
Quoted 2 lines · Andrew Cooper

Copy the nr_frames from the correct structure, so the caller doesn't unconditionally receive 0.

Well, no - it does get copied from the correct structure. It's just that the field doesn't get set properly up front.

You appear to be objecting to my use of the term "correct".

There are two structures.  One contains the correct value, and one contains the wrong value, which happens to always be 0.

I stand by sentence as currently written.

Quoted 3 lines

Otherwise you'll
(a) build in an unchecked assumption that the native and compat fields match in type

Did you actually check?  Because I did before embarking on this course of action.

In file included from /local/xen.git/xen/include/xen/guest_access.h:10:0, from compat/memory.c:5:
/local/xen.git/xen/include/asm/guest_access.h:152:28: error: comparison of distinct pointer types lacks a cast [-Werror] (void)(&(hnd).p->field == _s);                      \
                            ^
compat/memory.c:628:22: note: in expansion of macro ‘__copy_field_to_guest’ if ( __copy_field_to_guest( ^~~~~~~~~~~~~~~~~~~~~

This is what the compiler thinks of the code, when nr_frames is changed from uint32_t to unsigned long.

Quoted 2 lines

and (b) set a bad example for people looking
here

This entire function is a massive set of bad examples; the worst IMO being the fact that there isn't a single useful comment anywhere in it concerning how the higher level loop structure works.

I'm constantly annoyed that I need to reverse engineer it from scratch every time I look at it, despite having a better-than-most understanding of what it is trying to achieve, and how it is supposed to work.

I realise this is noones fault in particular, but it is not fair/reasonable to claim that this change is the thing setting a bad example in this file.

Quoted 3 lines

and then cloning this code in perhaps a case where (a) is not even true. If you agree, the alternative change of setting cmp.mar.nr_frames from nat.mar->nr_frames before the call is

Is there more to this sentence?

While this example could be implemented (at even higher overhead) by copying nat back to cmp before passing it back to the guest, the same is not true for the changes required to fix batching (which is another series the same size as this).

~Andrew

#28 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Paul Durrant
Quoted 94 lines · Paul Durrant
Quoted 18 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Hubert Jasudowicz <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Michał Leszczyński <[email protected]>; Ian Jackson <[email protected]>
Subject: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

Calling XENMEM_acquire_resource with a NULL frame_list is a request for the size of the resource, but the returned 32 is bogus.

If someone tries to follow it for XENMEM_resource_ioreq_server, the acquire call will fail as IOREQ servers currently top out at 2 frames, and it is only half the size of the default grant table limit for guests.

Quoted 7 lines

Also, no users actually request a resource size, because it was never wired up in the sole implemenation of resource aquisition in Linux.

Introduce a new resource_max_frames() to calculate the size of a resource, and implement it the IOREQ and grant subsystems.

It is impossible to guarentee that a mapping call following a successful size

s/guarantee/guarantee

Quoted 1 line

call will succedd (e.g. The target IOREQ server gets destroyed, or the domain

s/succedd/succeed

Quoted 64 lines

switches from grant v2 to v1). Document the restriction, and use the flexibility to simplify the paths to be lockless.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/arch/x86/mm.c             | 20 ++++++++++++++++
 xen/common/grant_table.c      | 19 +++++++++++++++
 xen/common/memory.c           | 55 +++++++++++++++++++++++++++++++++----------
 xen/include/asm-x86/mm.h      |  3 +++
 xen/include/public/memory.h   | 16 +++++++++----
 xen/include/xen/grant_table.h |  8 +++++++
 xen/include/xen/mm.h          |  6 +++++
 7 files changed, 110 insertions(+), 17 deletions(-)
xen/arch/x86/mm.c +20 -0
index 82bc676553..f73a90a2ab 100644
@@ -4600,6 +4600,26 @@ int xenmem_add_to_physmap_one(
     return rc;
 }
 
+unsigned int arch_resource_max_frames(
+    struct domain *d, unsigned int type, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    switch ( type )
+    {
+#ifdef CONFIG_HVM
+    case XENMEM_resource_ioreq_server:
+        if ( !is_hvm_domain(d) )
+            break;
+        /* One frame for the buf-ioreq ring, and one frame per 128 vcpus. */
+        nr = 1 + DIV_ROUND_UP(d->max_vcpus * sizeof(struct ioreq), PAGE_SIZE);
+        break;
+#endif
+    }
+
+    return nr;
+}
+
 int arch_acquire_resource(struct domain *d, unsigned int type,
                           unsigned int id, unsigned long frame,
                           unsigned int nr_frames, xen_pfn_t mfn_list[])
xen/common/grant_table.c +0 -0
index 122d1e7596..0962fc7169 100644
@@ -4013,6 +4013,25 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
Quoted 14 lines

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4028

Quoted 13 lines
xen/common/grant_table.c +13 -0
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    /* Don't need the grant lock.  This limit is fixed at domain create time. */
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        nr = d->grant_table->max_grant_frames;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        nr = grant_to_status_frames(d->grant_table->max_grant_frames);

Two uses of d->grant_table, so perhaps define a stack variable for it?

Can do.

Quoted 1 line

Also, should you not make sure 0 is returned in the case of a v1 table?

This was the case specifically discussed in the commit message, but perhaps it needs expanding.

Doing so would be buggy.

Some utility is going to query the resource size, and then try to map it (if it doesn't blindly know the size and/or subset it cares about already).

In between these two hypercalls from the utility, the guest can do a v1=>v2 or v2=>v1 switch and make the resource spontaneously appear or disappear.

The only case where we can know for certain whether the resource is available is when we're in the map hypercall.  Therefore, userspace has to be able to get to the map call if there is potentially a resource available.

The semantics of the size call are really "this resource might exist, and if it does, this is how large it is".

As for the grant status frames specifically, I think making them a mappable resource might have been a poor choice in hind sight.

Only the guest can switch between grant versions.  GNTTABOP_set_version strictly operates on current, unlike most of the other grant hypercalls which take a domid and let dom0 specify something other than DOMID_SELF.

There is GNTTABOP_get_version, but it is racy to use in the same way as described above, and if some utility does successfully map the status frames, what will happen in practice is that a guest attempting to switch from v2 back to v1 will have the set_version hypercall fail due to outstanding refs on the frames.

~Andrew

#27 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Julien Grall
Quoted 37 lines · Julien Grall

Hi Paul,

Quoted 19 lines · Paul Durrant
Quoted 13 lines
xen/common/memory.c +4 -0
index dc3a7248e3..21edabf9cc 100644
@@ -1007,6 +1007,26 @@ static long xatp_permission_check(struct
 domain *d, unsigned int space)
       return xsm_add_to_physmap(XSM_TARGET, current->domain, d);
   }
 
+/*
+ * Return 0 on any kind of error.  Caller converts to -EINVAL.
+ *
+ * All nonzero values should be repeatable (i.e. derived from some
Quoted 3 lines

fixed
+ * proerty of the domain), and describe the full resource (i.e. mapping the

s/property/property

Quoted 4 lines
Quoted 1 line

+ * result of this call will be the entire resource).

This precludes dynamically adding a resource to a running domain. Do we really want to bake in that restriction?

AFAICT, this restriction is not documented in the ABI. In particular, it is written:

"
The size of a resource will never be zero, but a nonzero result doesn't guarentee that a subsequent mapping request will be successful.  There are further type/id specific constraints which may change between the two calls.
"

Quoted 3 lines

So I think a domain couldn't rely on this behavior. Although, it might be good to clarify in the comment on top of resource_max_frames that this an implementation decision and not part of the ABI.

There are two aspects here.

First, yes - I deliberately didn't state it in the ABI, just in case we might want to use it in the future.  I could theoretically foresee using -EBUSY for the purpose.

That said however, we are currently deliberately taking dynamic resources out of Xen, because they've proved to be unnecessary in practice and a fertile source of complexity and security bugs.

I don't foresee accepting new dynamic resources, but that's not to say that someone can't theoretically come up with a sufficiently compelling counterexample.

~Andrew

#29 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Andrew Cooper

Hi Andrew,

Quoted 4 lines · Andrew Cooper

On 30/07/2020 09:31, Paul Durrant wrote:
In between these two hypercalls from the utility, the guest can do a v1=>v2 or v2=>v1 switch and make the resource spontaneously appear or disappear.

This can only happen on platform where grant-table v2 is enabled. Where this is not enabled (e.g Arm), then I think we want to return 0 as there is nothing to map.

Cheers,

Signature · 1 line
Julien Grall
#22 lore raw

Re: [PATCH 3/5] xen/memory: Fix compat XENMEM_acquire_resource for size requests

In reply to Andrew Cooper
Quoted 8 lines · Andrew Cooper
Quoted 6 lines · Jan Beulich
Quoted 2 lines · Andrew Cooper

Copy the nr_frames from the correct structure, so the caller doesn't unconditionally receive 0.

Well, no - it does get copied from the correct structure. It's just that the field doesn't get set properly up front.

Quoted 6 lines

You appear to be objecting to my use of the term "correct".

There are two structures.  One contains the correct value, and one contains the wrong value, which happens to always be 0.

I stand by sentence as currently written.

At the risk of splitting hair, what you copy from is a field holding the correct value, but not the correct field. This only works correctly because of the way __copy_field_{from,to}_guest() happen to be implemented; there are possible alternative implementations where this would break, despite ...

Quoted 17 lines
Quoted 3 lines

Otherwise you'll
(a) build in an unchecked assumption that the native and compat fields match in type

Did you actually check?  Because I did before embarking on this course of action.

In file included from /local/xen.git/xen/include/xen/guest_access.h:10:0, from compat/memory.c:5:
/local/xen.git/xen/include/asm/guest_access.h:152:28: error: comparison of distinct pointer types lacks a cast [-Werror] (void)(&(hnd).p->field == _s);                      \
                            ^
compat/memory.c:628:22: note: in expansion of macro ‘__copy_field_to_guest’ if ( __copy_field_to_guest( ^~~~~~~~~~~~~~~~~~~~~

Quoted 2 lines

This is what the compiler thinks of the code, when nr_frames is changed from uint32_t to unsigned long.

... this type safety check (which, I admit, I didn't consider when writing my reply). I continue to think that handle and struct should match up not just for {,__}copy_{from,to}_guest() but also for {,__}copy_field_{from,to}_guest().

Quoted 7 lines
Quoted 2 lines

and (b) set a bad example for people looking
here

This entire function is a massive set of bad examples; the worst IMO being the fact that there isn't a single useful comment anywhere in it concerning how the higher level loop structure works.

Quoted 7 lines

I'm constantly annoyed that I need to reverse engineer it from scratch every time I look at it, despite having a better-than-most understanding of what it is trying to achieve, and how it is supposed to work.

I realise this is noones fault in particular, but it is not fair/reasonable to claim that this change is the thing setting a bad example in this file.

I'd be happy to see "bad examples" be corrected. As stated at various occasions, at the time I first implemented the compat layer this seemed like the most reasonable approach to me. If you see room for improvement, then I'm all for it.

Quoted 5 lines
Quoted 3 lines

and then cloning this code in perhaps a case where (a) is not even true. If you agree, the alternative change of setting cmp.mar.nr_frames from nat.mar->nr_frames before the call is

Is there more to this sentence?

I guess I can't figure what you mean here.

Quoted 4 lines

While this example could be implemented (at even higher overhead) by copying nat back to cmp before passing it back to the guest, the same is not true for the changes required to fix batching (which is another series the same size as this).

I'll see when you post this, but I think we will want the principle outlined above to continue to hold.

Jan

#30 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Andrew Cooper
Quoted 41 lines · Andrew Cooper
Quoted 9 lines · Paul Durrant
Quoted 9 lines

From: Xen-devel <[email protected]> On Behalf Of Andrew Cooper
Sent: 28 July 2020 12:37

xen/common/grant_table.c +0 -0
@@ -4013,6 +4013,25 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
Quoted 14 lines

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4028

Quoted 13 lines
xen/common/grant_table.c +13 -0
+unsigned int gnttab_resource_max_frames(struct domain *d, unsigned int id)
+{
+    unsigned int nr = 0;
+
+    /* Don't need the grant lock.  This limit is fixed at domain create time. */
+    switch ( id )
+    {
+    case XENMEM_resource_grant_table_id_shared:
+        nr = d->grant_table->max_grant_frames;
+        break;
+
+    case XENMEM_resource_grant_table_id_status:
+        nr = grant_to_status_frames(d->grant_table->max_grant_frames);

Two uses of d->grant_table, so perhaps define a stack variable for it?

Can do.

Quoted 1 line

Also, should you not make sure 0 is returned in the case of a v1 table?

This was the case specifically discussed in the commit message, but perhaps it needs expanding.

Doing so would be buggy.

Some utility is going to query the resource size, and then try to map it (if it doesn't blindly know the size and/or subset it cares about already).

In between these two hypercalls from the utility, the guest can do a v1=>v2 or v2=>v1 switch and make the resource spontaneously appear or disappear.

Quoted 7 lines

The only case where we can know for certain whether the resource is available is when we're in the map hypercall.  Therefore, userspace has to be able to get to the map call if there is potentially a resource available.

The semantics of the size call are really "this resource might exist, and if it does, this is how large it is".

With you deriving from d->grant_table->max_grant_frames, this approach would imply that by obtaining a mapping the grant tables will get grown to their permitted maximum, no matter whether as much is actually needed by the guest. If this is indeed the intention, then we could as well set up maximum grant structures right at domain creation. Not something I would favor, but anyway...

Jan

#31 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

In reply to Andrew Cooper
Quoted 9 lines · Andrew Cooper
xen/common/memory.c +0 -5
@@ -1026,19 +1047,6 @@ static int acquire_resource(
     if ( xmar.pad != 0 )
         return -EINVAL;
 
-    if ( guest_handle_is_null(xmar.frame_list) )
-    {
-        if ( xmar.nr_frames )
-            return -EINVAL;
-

xen/common/memory.c static int acquire_resource( · line 1051

Quoted 10 lines
xen/common/memory.c +0 -8
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
-
-        if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
-            return -EFAULT;
-
-        return 0;
-    }
-
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
         return -E2BIG;

While arguably minor, the error code in the null-handle case would imo better be the same, no matter how big xmar.nr_frames is.

Jan

#32 lore raw

Re: [PATCH 4/5] xen/memory: Fix acquire_resource size semantics

Quoted 15 lines · Jan Beulich
Quoted 14 lines · Andrew Cooper
xen/common/memory.c +0 -10
@@ -1026,19 +1047,6 @@ static int acquire_resource(
     if ( xmar.pad != 0 )
         return -EINVAL;
 
-    if ( guest_handle_is_null(xmar.frame_list) )
-    {
-        if ( xmar.nr_frames )
-            return -EINVAL;
-
-        xmar.nr_frames = ARRAY_SIZE(mfn_list);
-
-        if ( __copy_field_to_guest(arg, &xmar, nr_frames) )
-            return -EFAULT;
-
Quoted 8 lines

xen/common/memory.c static int acquire_resource( · line 1051

Quoted 5 lines
xen/common/memory.c +0 -3
-        return 0;
-    }
-
     if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
         return -E2BIG;

While arguably minor, the error code in the null-handle case would imo better be the same, no matter how big xmar.nr_frames is.

This clause doesn't survive the fixes to batching.

Given how broken this infrastructure is, I'm not concerned with transient differences in error codes for users which will ultimately fail anyway.

~Andrew

Acked-by
#35 lore raw

Re: [PATCH 5/5] tools/foreignmem: Support querying the size of a resource

In reply to Andrew Cooper
Quoted 56 lines · Andrew Cooper
Quoted 13 lines · Andrew Cooper

With the Xen side of this interface fixed to return real sizes, userspace needs to be able to make the query.

Introduce xenforeignmemory_resource_size() for the purpose, bumping the library minor version and providing compatiblity for the non-Linux builds.

Its not possible to reuse the IOCTL_PRIVCMD_MMAP_RESOURCE infrastructure, because it depends on having already mmap()'d a suitably sized region before it will make an XENMEM_acquire_resource hypercall to Xen.

Instead, open a xencall handle and make an XENMEM_acquire_resource hypercall directly.

Quoted 7 lines

---
CC: Ian Jackson <[email protected]>
CC: Wei Liu <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

I've folded:

tools/Rules.mk +0 -0
index 5ed5664bf7..b8ccf03ea9 100644
@@ -123,7 +123,7 @@ LDLIBS_libxencall = $(SHDEPS_libxencall)

$(XEN_LIBXENCALL)/libxencall$(libextens
 SHLIB_libxencall  = $(SHDEPS_libxencall) -Wl,-rpath-link=$(XEN_LIBXENCALL)
 CFLAGS_libxenforeignmemory = -I$(XEN_LIBXENFOREIGNMEMORY)/include $(CFLAGS_xeninclude)

tools/Rules.mk +1 -1
-SHDEPS_libxenforeignmemory = $(SHLIB_libxentoolcore)
+SHDEPS_libxenforeignmemory = $(SHLIB_libxentoolcore) $(SHDEPS_libxencall)

 LDLIBS_libxenforeignmemory = $(SHDEPS_libxenforeignmemory) $(XEN_LIBXENFOREIGNMEMORY)/libxenforeignmemory$(libextension) SHLIB_libxenforeignmemory  = $(SHDEPS_libxenforeignmemory) -Wl,-rpath-link=$(XEN_LIBXENFOREIGNMEMORY)

tools/libs/foreignmemory/Makefile +0 -0
index 8e07f92c59..f3a61e27c7 100644
@@ -4,7 +4,7 @@ include $(XEN_ROOT)/tools/Rules.mk

 MAJOR    = 1
 MINOR    = 4
 LIBNAME  := foreignmemory

tools/libs/foreignmemory/Makefile +1 -1
-USELIBS  := toollog toolcore
+USELIBS  := toollog toolcore call


 SRCS-y                 += core.c
 SRCS-$(CONFIG_Linux)   += linux.c

Quoted 1 line

to fix the build in certain containers.

#12 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Jan Beulich
Quoted 17 lines · Jan Beulich
Quoted 6 lines · Andrew Cooper

The existing logic doesn't function in the general case for mapping a guests
grant table, due to arbitrary 32 frame limit, and the default grant table
limit being 64.

Quoted 9 lines

In order to start addressing this, rework the existing grant table logic by
implementing a single gnttab_acquire_resource().  This is far more efficient
than the previous acquire_grant_table() in memory.c because it doesn't take
the grant table write lock, and attempt to grow the table, for every single
frame.

Quoted 3 lines

Among the code you replace there is a comment "Iterate backwards in case table needs to grow" explaining why what you say about growing the grant table didn't actually happen.

What I have said is accurate.

Iterating backwards caused it to actually grow once, but every iteration still takes the lock and attempts to grow the table.

Quoted 16 lines
Quoted 16 lines
xen/common/grant_table.c +0 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct
 domain *d,
       return 0;
   }

  +int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame, +    unsigned int nr_frames, xen_pfn_t mfn_list[])

xen/common/grant_table.c +6 -0
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
Quoted 8 lines
Quoted 3 lines

+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

I can't seem to be able to find an equivalent of this in the old logic, and hence this looks like an unwarranted change in behavior to me. We have quite a few hypercall ops where some count being zero is simply a no-op, i.e. yielding success without doing anything.

There is no possible circumstance when getting here requesting 0 is legitimate.

Tolerating a zero input for a mapping request is a very expensive way of hiding caller bugs.

Most importantly however, the correctness of the logic does depends on nr_frames being nonzero.

Quoted 9 lines
Quoted 8 lines

+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;
+    if ( tot_frames != frame + nr_frames )
+        return -EINVAL;

Quoted 5 lines

I find the naming here quite confusing. I realize part of this stems from the code you replace, but anyway: "unsigned long frame" typically represents a memory frame number of some sort, making the calculation look as if it was wrong. (Initially I merely meant to ask whether this check isn't redundant with the prior one, or vice versa.)

Both checks are strictly necessary.  The top one checks 64bit wrapping, while the second is checking 32bit truncation.

This code is horrible, both in terms of the acquire_resource API, and the grant API being off by one compared to normal, and needing the max_frame index, not the number of frames.

~Andrew

#17 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Paul Durrant
Quoted 15 lines · Paul Durrant
Quoted 15 lines

-----Original Message-----
From: Andrew Cooper <[email protected]>
Sent: 28 July 2020 12:37
To: Xen-devel <[email protected]>
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Wei Liu <[email protected]>; Julien Grall <[email protected]>; Paul Durrant <[email protected]>; Michał Leszczyński <[email protected]>; Hubert Jasudowicz <[email protected]>
Subject: [PATCH 2/5] xen/gnttab: Rework resource acquisition

The existing logic doesn't function in the general case for mapping a guests grant table, due to arbitrary 32 frame limit, and the default grant table limit being 64.

Quoted 7 lines
Quoted 6 lines

In order to start addressing this, rework the existing grant table logic by implementing a single gnttab_acquire_resource(). This is far more efficient than the previous acquire_grant_table() in memory.c because it doesn't take the grant table write lock, and attempt to grow the table, for every single frame.

But that should not have happened before because the code deliberately iterates backwards, thereby starting with the last frame, thereby growing the table at most once. (I agree that dropping and re-acquiring the lock every time was sub-optimal).

It still attempts to grow on every iteration.  Its just growing to a smaller size than already succeeded.

Quoted 32 lines
Quoted 32 lines

The new gnttab_acquire_resource() function subsumes the previous two gnttab_get_{shared,status}_frame() helpers.

No functional change.

---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Paul Durrant <[email protected]>
CC: Michał Leszczyński <[email protected]>
CC: Hubert Jasudowicz <[email protected]>

---
 xen/common/grant_table.c      | 93 ++++++++++++++++++++++++++++++-------------
 xen/common/memory.c           | 42 +------------------
 xen/include/xen/grant_table.h | 19 ++++-----
 3 files changed, 75 insertions(+), 79 deletions(-)
xen/common/grant_table.c +0 -0
index 9f0cae52c0..122d1e7596 100644
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct domain *d,
     return 0;
 }
 
Quoted 13 lines

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4027

Quoted 12 lines
xen/common/grant_table.c +12 -0
+int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame,
+    unsigned int nr_frames, xen_pfn_t mfn_list[])
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

This was not an error before. Does mapping 0 frames really need to be a failure?

Yes.

You have spotted the -1 which depends on nr_frames being nonzero to function correctly.

Quoted 7 lines

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4033

Quoted 6 lines
xen/common/grant_table.c +6 -0
+
+    /* Overflow checks */
+    if ( frame + nr_frames < frame )
+        return -EINVAL;
+
+    tot_frames = frame + nr_frames;

That name is confusing. 'last_frame' perhaps (and then make the -1 implicit)?

How is that naming any less confusing?

Quoted 9 lines

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4077

Quoted 8 lines
xen/common/grant_table.c +8 -0
+        break;
+    }
+
+    for ( i = 0; i < nr_frames; ++i )
+        mfn_list[i] = virt_to_mfn(vaddrs[frame + i]);
+
+ out:
+    grant_write_unlock(gt);

Since you deliberately grew the table first, could you not drop the write lock and acquire it a read lock before looping over the frames?

I tried originally.  That's not an operation supported by percpu read/write locks, and this isn't a fastpath.

~Andrew

#13 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

In reply to Andrew Cooper
Quoted 26 lines · Andrew Cooper
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Andrew Cooper
xen/common/grant_table.c +0 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct
 domain *d,
       return 0;
   }

  +int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame, +    unsigned int nr_frames, xen_pfn_t mfn_list[])

xen/common/grant_table.c +6 -0
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
Quoted 8 lines
Quoted 3 lines

+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

I can't seem to be able to find an equivalent of this in the old logic, and hence this looks like an unwarranted change in behavior to me. We have quite a few hypercall ops where some count being zero is simply a no-op, i.e. yielding success without doing anything.

Quoted 5 lines

There is no possible circumstance when getting here requesting 0 is legitimate.

Tolerating a zero input for a mapping request is a very expensive way of hiding caller bugs.

That's just one possible view. There are people thinking that some extra runtime overhead doesn't outweigh the "clutter" of extra conditionals in their code, and hence would rather avoid checking if some calculation of theirs yielded zero before making a hypercall. I think we should try to consistently regard counts of zero as "nothing to do" in all our hypercalls.

Quoted 2 lines

Most importantly however, the correctness of the logic does depends on nr_frames being nonzero.

I didn't mean to ask to drop the conditional if it's needed; all I'm opposed to is raising an error in this case.

Jan

#14 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

Quoted 25 lines · Jan Beulich
Quoted 24 lines · Andrew Cooper
Quoted 23 lines · Jan Beulich
Quoted 9 lines · Andrew Cooper
xen/common/grant_table.c +0 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct
 domain *d,
       return 0;
   }

  +int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame, +    unsigned int nr_frames, xen_pfn_t mfn_list[])

xen/common/grant_table.c static int gnttab_get_shared_frame_mfn(struct domain *d, · line 4027

Quoted 9 lines
xen/common/grant_table.c +9 -0
+{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;
+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

I can't seem to be able to find an equivalent of this in the old logic, and hence this looks like an unwarranted change in behavior to me. We have quite a few hypercall ops where some count being zero is simply a no-op, i.e. yielding success without doing anything.

Quoted 6 lines
Quoted 5 lines

There is no possible circumstance when getting here requesting 0 is legitimate.

Tolerating a zero input for a mapping request is a very expensive way of hiding caller bugs.

That's just one possible view.

... that is fully enforced in the ecosystem we work in.

You can't get a 0 to here in the first place.

However, when it comes to the XLAT and the chunking fixes, getting 0 here is a hard error, and I'm not interested in breaking the that logic for a non-existent corner case.

~Andrew

#15 lore raw

Re: [PATCH 2/5] xen/gnttab: Rework resource acquisition

Quoted 29 lines · Andrew Cooper
Quoted 28 lines · Jan Beulich
Quoted 27 lines · Andrew Cooper
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Andrew Cooper
xen/common/grant_table.c +0 -0
@@ -4013,6 +4013,72 @@ static int gnttab_get_shared_frame_mfn(struct
 domain *d,
       return 0;
   }

  +int gnttab_acquire_resource(
+    struct domain *d, unsigned int id, unsigned long frame, +    unsigned int nr_frames, xen_pfn_t mfn_list[]) +{
+    struct grant_table *gt = d->grant_table;
+    unsigned int i = nr_frames, tot_frames;
+    void **vaddrs;
+    int rc = 0;

Quoted 8 lines
Quoted 4 lines

+
+    /* Input sanity. */
+    if ( !nr_frames )
+        return -EINVAL;

I can't seem to be able to find an equivalent of this in the old logic, and hence this looks like an unwarranted change in behavior to me. We have quite a few hypercall ops where some count being zero is simply a no-op, i.e. yielding success without doing anything.

There is no possible circumstance when getting here requesting 0 is legitimate.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Tolerating a zero input for a mapping request is a very expensive way of hiding caller bugs.

That's just one possible view.

... that is fully enforced in the ecosystem we work in.

You can't get a 0 to here in the first place.

Would you mind pointing me at why this is? All I can see is

if ( xmar.nr_frames > ARRAY_SIZE(mfn_list) )
return -E2BIG;

as far as the caller's argument checking goes here. And acquire_grant_table(), which is what you're replacing, also is dealing with zero quite fine afaics. As is arch_acquire_resource().

Quoted 3 lines

However, when it comes to the XLAT and the chunking fixes, getting 0 here is a hard error, and I'm not interested in breaking the that logic for a non-existent corner case.

I don't share this view. It's also interesting that you had to address similar questions by Paul, isn't it?

Jan