Skip to contentSkip to conversation

ioreq: add support for internal servers

PATCH Superseded Roger Pau Monne 2 participants 20 messages Aug 21, 2019 – Aug 22, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover ioreq: add support for internal servers

Such internal servers are implemented by a single function that handles ioreqs inside the hypervisor.

The motivation behind this change is to switch vPCI to become an internal ioreq server, so that accesses to the PCI config space can be multiplexed between devices handled by vPCI and devices handled by other ioreq servers.

The implementation is fairly simple and limited to what's needed by vPCI, but can be expanded in the future if other more complex users appear.

The series can also be found at:

git://xenbits.xen.org/people/royger/xen.git ioreq_vpci_v1

Thanks, Roger.

Roger Pau Monne (7):
ioreq: add fields to allow internal ioreq servers
ioreq: add internal ioreq initialization support
ioreq: allow dispatching ioreqs to internal servers
ioreq: allow registering internal ioreq server handler
ioreq: allow decoding accesses to MMCFG regions
vpci: register as an internal ioreq server
ioreq: provide support for long-running operations...

 xen/arch/x86/hvm/dm.c               |   9 +-
 xen/arch/x86/hvm/dom0_build.c       |   9 +-
 xen/arch/x86/hvm/hvm.c              |   7 +-
 xen/arch/x86/hvm/io.c               | 288 +----------------------
 xen/arch/x86/hvm/ioreq.c            | 349 +++++++++++++++++++++++-----
 xen/arch/x86/physdev.c              |   7 +-
 xen/drivers/passthrough/x86/iommu.c |   2 +-
 xen/drivers/vpci/vpci.c             |  57 +++++
 xen/include/asm-x86/hvm/domain.h    |  31 ++-
 xen/include/asm-x86/hvm/io.h        |  14 +-
 xen/include/asm-x86/hvm/ioreq.h     |  19 +-
 xen/include/xen/vpci.h              |   3 +
 12 files changed, 415 insertions(+), 380 deletions(-)
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/7 ioreq: add fields to allow internal ioreq servers

Internal ioreq servers are plain function handlers implemented inside of the hypervisor. Note that most fields used by current (external) ioreq servers are not needed for internal ones, and hence have been placed inside of a struct and packed in an union together with the only internal specific field, a function pointer to a handler.

This is required in order to have PCI config accesses forwarded to external ioreq servers or to internal ones (ie: QEMU emulated devices vs vPCI passthrough), and is the first step in order to allow unprivileged domains to use vPCI.

---
 xen/include/asm-x86/hvm/domain.h | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)
xen/include/asm-x86/hvm/domain.h +19 -11
index 6c7c4f5aa6..f0be303517 100644
@@ -52,21 +52,29 @@ struct hvm_ioreq_vcpu {
 #define MAX_NR_IO_RANGES  256
 
 struct hvm_ioreq_server {
-    struct domain          *target, *emulator;
-
+    struct domain          *target;
     /* Lock to serialize toolstack modifications */
     spinlock_t             lock;
-
-    struct hvm_ioreq_page  ioreq;
-    struct list_head       ioreq_vcpu_list;
-    struct hvm_ioreq_page  bufioreq;
-
-    /* Lock to serialize access to buffered ioreq ring */
-    spinlock_t             bufioreq_lock;
-    evtchn_port_t          bufioreq_evtchn;
     struct rangeset        *range[NR_IO_RANGE_TYPES];
     bool                   enabled;
-    uint8_t                bufioreq_handling;
+    bool                   internal;
+
+    union {
+        struct {
+            struct domain          *emulator;
+            struct hvm_ioreq_page  ioreq;
+            struct list_head       ioreq_vcpu_list;
+            struct hvm_ioreq_page  bufioreq;
+
+            /* Lock to serialize access to buffered ioreq ring */
+            spinlock_t             bufioreq_lock;
+            evtchn_port_t          bufioreq_evtchn;
+            uint8_t                bufioreq_handling;
+        };
+        struct {
+            int (*handler)(struct vcpu *v, ioreq_t *);
+        };
+    };
 };
 
 /*
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 2/7 ioreq: add internal ioreq initialization support

In reply to Roger Pau Monne

Add support for internal ioreq servers to initialization and deinitialization routines, prevent some functions from being executed against internal ioreq servers and add guards to only allow internal callers to modify internal ioreq servers. External callers (ie: from hypercalls) are only allowed to deal with external ioreq servers.

---
 xen/arch/x86/hvm/dm.c           |   9 +-
 xen/arch/x86/hvm/ioreq.c        | 150 +++++++++++++++++++++-----------
 xen/include/asm-x86/hvm/ioreq.h |   8 +-
 3 files changed, 108 insertions(+), 59 deletions(-)
xen/arch/x86/hvm/dm.c +5 -4
index d6d0e8be89..5ca8b66d67 100644
@@ -417,7 +417,7 @@ static int dm_op(const struct dmop_args *op_args)
             break;
 
         rc = hvm_create_ioreq_server(d, data->handle_bufioreq,
-                                     &data->id);
+                                     &data->id, false);
         break;
     }
 
@@ -452,7 +452,7 @@ static int dm_op(const struct dmop_args *op_args)
             break;
 
         rc = hvm_map_io_range_to_ioreq_server(d, data->id, data->type,
-                                              data->start, data->end);
+                                              data->start, data->end, false);
         break;
     }
 
@@ -466,7 +466,8 @@ static int dm_op(const struct dmop_args *op_args)
             break;
 
         rc = hvm_unmap_io_range_from_ioreq_server(d, data->id, data->type,
-                                                  data->start, data->end);
+                                                  data->start, data->end,
+                                                  false);
         break;
     }
 
@@ -529,7 +530,7 @@ static int dm_op(const struct dmop_args *op_args)
         if ( data->pad )
             break;
 
-        rc = hvm_set_ioreq_server_state(d, data->id, !!data->enabled);
+        rc = hvm_set_ioreq_server_state(d, data->id, !!data->enabled, false);
         break;
     }
 
xen/arch/x86/hvm/ioreq.c +99 -51

297 lines not on this page · raw message

xen/include/asm-x86/hvm/ioreq.h +4 -4
index e2588e912f..e8119b26a6 100644
@@ -24,7 +24,7 @@ bool handle_hvm_io_completion(struct vcpu *v);
 bool is_ioreq_server_page(struct domain *d, const struct page_info *page);
 
 int hvm_create_ioreq_server(struct domain *d, int bufioreq_handling,
-                            ioservid_t *id);
+                            ioservid_t *id, bool internal);
 int hvm_destroy_ioreq_server(struct domain *d, ioservid_t id);
 int hvm_get_ioreq_server_info(struct domain *d, ioservid_t id,
                               unsigned long *ioreq_gfn,
@@ -34,14 +34,14 @@ int hvm_get_ioreq_server_frame(struct domain *d, ioservid_t id,
                                unsigned long idx, mfn_t *mfn);
 int hvm_map_io_range_to_ioreq_server(struct domain *d, ioservid_t id,
                                      uint32_t type, uint64_t start,
-                                     uint64_t end);
+                                     uint64_t end, bool internal);
 int hvm_unmap_io_range_from_ioreq_server(struct domain *d, ioservid_t id,
                                          uint32_t type, uint64_t start,
-                                         uint64_t end);
+                                         uint64_t end, bool internal);
 int hvm_map_mem_type_to_ioreq_server(struct domain *d, ioservid_t id,
                                      uint32_t type, uint32_t flags);
 int hvm_set_ioreq_server_state(struct domain *d, ioservid_t id,
-                               bool enabled);
+                               bool enabled, bool internal);
 
 int hvm_all_ioreq_servers_add_vcpu(struct domain *d, struct vcpu *v);
 void hvm_all_ioreq_servers_remove_vcpu(struct domain *d, struct vcpu *v);
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw

Re: [Xen-devel] [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 5 lines

Add support for internal ioreq servers to initialization and deinitialization routines, prevent some functions from being executed against internal ioreq servers and add guards to only allow internal callers to modify internal ioreq servers. External callers (ie: from hypercalls) are only allowed to deal with external ioreq servers.

It's kind of ugly to have the extra 'internal' argument passed to anything other than the create function so I wonder whether it would be neater to encode it in the ioreq server id. I.e. we have distinct id ranges for internal and external servers. What do you think?

Paul
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: [Xen-devel] [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 8 lines · Paul Durrant
Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 7 lines
Quoted 5 lines

Add support for internal ioreq servers to initialization and deinitialization routines, prevent some functions from being executed against internal ioreq servers and add guards to only allow internal callers to modify internal ioreq servers. External callers (ie: from hypercalls) are only allowed to deal with external ioreq servers.

It's kind of ugly to have the extra 'internal' argument passed to anything other than the create function so I wonder whether it would be neater to encode it in the ioreq server id. I.e. we have distinct id ranges for internal and external servers. What do you think?

That would be fine, I guess I could use the most significant bit in the id to signal whether the server is internal or external, and reject dmop calls that target internal servers based on the provided id. Does that sound sensible?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 23 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 22 August 2019 08:24
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH 2/7] ioreq: add internal ioreq initialization support

Quoted 14 lines · Paul Durrant
Quoted 13 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 2/7] ioreq: add internal ioreq initialization support

Add support for internal ioreq servers to initialization and deinitialization routines, prevent some functions from being executed against internal ioreq servers and add guards to only allow internal callers to modify internal ioreq servers. External callers (ie: from hypercalls) are only allowed to deal with external ioreq servers.

Quoted 9 lines
Quoted 1 line

It's kind of ugly to have the extra 'internal' argument passed to anything other than the create

function so I wonder whether it would be neater to encode it in the ioreq server id. I.e. we have distinct id ranges for internal and external servers. What do you think?

That would be fine, I guess I could use the most significant bit in the id to signal whether the server is internal or external, and reject dmop calls that target internal servers based on the provided id. Does that sound sensible?

Yes, that's basically what I was thinking initially although, as you observe, in the thread for patch #3 having two smaller consecutive ranges would be more convenient.

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

PATCH 3/7 ioreq: allow dispatching ioreqs to internal servers

In reply to Roger Pau Monne

Internal ioreq servers are always processed first, and ioreqs are dispatched by calling the handler function. If no internal servers have registered for an ioreq it's then forwarded to external callers.

---
 xen/arch/x86/hvm/ioreq.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/ioreq.c +18 -1
index 23ef9b0c02..3fb6fe9585 100644
@@ -1305,6 +1305,7 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
     uint8_t type;
     uint64_t addr;
     unsigned int id;
+    bool internal = true;
 
     if ( p->type != IOREQ_TYPE_COPY && p->type != IOREQ_TYPE_PIO )
         return NULL;
@@ -1345,11 +1346,12 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
         addr = p->addr;
     }
 
+ retry:
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
         struct rangeset *r;
 
-        if ( !s->enabled )
+        if ( !s->enabled || s->internal != internal )
             continue;
 
         r = s->range[type];
@@ -1387,6 +1389,12 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
         }
     }
 
+    if ( internal )
+    {
+        internal = false;
+        goto retry;
+    }
+
     return NULL;
 }
 
@@ -1492,9 +1500,18 @@ int hvm_send_ioreq(struct hvm_ioreq_server *s, ioreq_t *proto_p,
 
     ASSERT(s);
 
+    if ( s->internal && buffered )
+    {
+        ASSERT_UNREACHABLE();
+        return X86EMUL_UNHANDLEABLE;
+    }
+
     if ( buffered )
         return hvm_send_buffered_ioreq(s, proto_p);
 
+    if ( s->internal )
+        return s->handler(curr, proto_p);
+
     if ( unlikely(!vcpu_start_shutdown_deferral(curr)) )
         return X86EMUL_RETRY;
 
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [Xen-devel] [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 11 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Internal ioreq servers are always processed first, and ioreqs are dispatched by calling the handler function. If no internal servers have registered for an ioreq it's then forwarded to external callers.

Distinct id ranges would help here... Internal ids could be walked first, then external. If there's no possibility of interleaving then you don't need the retry.

Paul

Quoted 65 lines
---
 xen/arch/x86/hvm/ioreq.c | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
xen/arch/x86/hvm/ioreq.c +18 -2
index 23ef9b0c02..3fb6fe9585 100644
@@ -1305,6 +1305,7 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
     uint8_t type;
     uint64_t addr;
     unsigned int id;
+    bool internal = true;
 
     if ( p->type != IOREQ_TYPE_COPY && p->type != IOREQ_TYPE_PIO )
         return NULL;
@@ -1345,11 +1346,12 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
         addr = p->addr;
     }
 
+ retry:
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
         struct rangeset *r;
 
-        if ( !s->enabled )
+        if ( !s->enabled || s->internal != internal )
             continue;
 
         r = s->range[type];
@@ -1387,6 +1389,12 @@ struct hvm_ioreq_server *hvm_select_ioreq_server(struct domain *d,
         }
     }
 
+    if ( internal )
+    {
+        internal = false;
+        goto retry;
+    }
+
     return NULL;
 }
 
@@ -1492,9 +1500,18 @@ int hvm_send_ioreq(struct hvm_ioreq_server *s, ioreq_t *proto_p,
 
     ASSERT(s);
 
+    if ( s->internal && buffered )
+    {
+        ASSERT_UNREACHABLE();
+        return X86EMUL_UNHANDLEABLE;
+    }
+
     if ( buffered )
         return hvm_send_buffered_ioreq(s, proto_p);
 
+    if ( s->internal )
+        return s->handler(curr, proto_p);
+
     if ( unlikely(!vcpu_start_shutdown_deferral(curr)) )
         return X86EMUL_RETRY;
 
--
Quoted 1 line

2.22.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [Xen-devel] [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 8 lines · Paul Durrant
Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 5 lines
Quoted 3 lines

Internal ioreq servers are always processed first, and ioreqs are dispatched by calling the handler function. If no internal servers have registered for an ioreq it's then forwarded to external callers.

Distinct id ranges would help here... Internal ids could be walked first, then external. If there's no possibility of interleaving then you don't need the retry.

So if internal vs external is keyed on the ID then we would end up with two different arrays in hvm_domain, one for internal and one for external ioreq servers.

Maybe instead of my previous suggestion it would be better to just define consecutive ranges for external and internal servers, like:

#define MAX_NR_EXTERNAL_IOREQ_SERVERS 8
#define MAX_NR_INTERNAL_IOREQ_SERVERS 1
#define MAX_NR_IOREQ_SERVERS \
(MAX_NR_EXTERNAL_IOREQ_SERVERS + MAX_NR_INTERNAL_IOREQ_SERVERS)

#define FOR_EACH_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_IOREQ_SERVERS * 2; (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else
#define FOR_EACH_INTERNAL_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_IOREQ_SERVERS; (id) > MAX_NR_INTERNAL_IOREQ_SERVERS && (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else
#define FOR_EACH_EXTERNAL_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_INTERNAL_IOREQ_SERVERS; (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else

That would also force FOR_EACH_IOREQ_SERVER to always process internal ioreq servers first.

We could even have something like:

union {
    struct {
        struct hvm_ioreq_server *external_server[MAX_NR_EXTERNAL_IOREQ_SERVERS];
        struct hvm_ioreq_server *internal_server[MAX_NR_INTERNAL_IOREQ_SERVERS];
    }
    struct hvm_ioreq_server *server[MAX_NR_IOREQ_SERVERS];
}

In order to split the arrays if required.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [Xen-devel] [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 48 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 22 August 2019 08:41
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 8 lines · Paul Durrant
Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 3/7] ioreq: allow dispatching ioreqs to internal servers

Quoted 5 lines
Quoted 3 lines

Internal ioreq servers are always processed first, and ioreqs are dispatched by calling the handler function. If no internal servers have registered for an ioreq it's then forwarded to external callers.

Distinct id ranges would help here... Internal ids could be walked first, then external. If there's

no possibility of interleaving then you don't need the retry.

So if internal vs external is keyed on the ID then we would end up with two different arrays in hvm_domain, one for internal and one for external ioreq servers.

Maybe instead of my previous suggestion it would be better to just define consecutive ranges for external and internal servers, like:

#define MAX_NR_EXTERNAL_IOREQ_SERVERS 8
#define MAX_NR_INTERNAL_IOREQ_SERVERS 1
#define MAX_NR_IOREQ_SERVERS \
(MAX_NR_EXTERNAL_IOREQ_SERVERS + MAX_NR_INTERNAL_IOREQ_SERVERS)

#define FOR_EACH_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_IOREQ_SERVERS * 2; (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else
#define FOR_EACH_INTERNAL_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_IOREQ_SERVERS; (id) > MAX_NR_INTERNAL_IOREQ_SERVERS && (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else
Quoted 8 lines
#define FOR_EACH_EXTERNAL_IOREQ_SERVER(d, id, s) \
    for ( (id) = MAX_NR_INTERNAL_IOREQ_SERVERS; (id) != 0; ) \
        if ( !(s = GET_IOREQ_SERVER(d, --(id))) ) \
            continue; \
        else

That would also force FOR_EACH_IOREQ_SERVER to always process internal ioreq servers first.

Exactly what I was thinking.

Quoted 11 lines

We could even have something like:

union {
    struct {
        struct hvm_ioreq_server *external_server[MAX_NR_EXTERNAL_IOREQ_SERVERS];
        struct hvm_ioreq_server *internal_server[MAX_NR_INTERNAL_IOREQ_SERVERS];
    }
    struct hvm_ioreq_server *server[MAX_NR_IOREQ_SERVERS];
}
Quoted 2 lines

In order to split the arrays if required.

I'd not considered a union, but it makes sense :-)

Paul

Quoted 1 line

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

PATCH 4/7 ioreq: allow registering internal ioreq server handler

In reply to Roger Pau Monne

Provide a routine to register the handler for an internal ioreq server. Note the handler can only be set once.

---
 xen/arch/x86/hvm/ioreq.c        | 32 ++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/ioreq.h |  3 +++
 2 files changed, 35 insertions(+)
xen/arch/x86/hvm/ioreq.c +32 -0
index 3fb6fe9585..d8fea191aa 100644
@@ -486,6 +486,38 @@ static int hvm_add_ioreq_gfn(struct hvm_ioreq_server *s, bool buf)
     return rc;
 }
 
+int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
+                          int (*handler)(struct vcpu *v, ioreq_t *))
+{
+    struct hvm_ioreq_server *s;
+    int rc = 0;
+
+    spin_lock_recursive(&d->arch.hvm.ioreq_server.lock);
+    s = get_ioreq_server(d, id);
+    if ( !s )
+    {
+        rc = -ENOENT;
+        goto out;
+    }
+    if ( !s->internal )
+    {
+        rc = -EINVAL;
+        goto out;
+    }
+    if ( s->handler != NULL )
+    {
+        rc = -EBUSY;
+        goto out;
+    }
+
+    s->handler = handler;
+
+ out:
+    spin_unlock_recursive(&d->arch.hvm.ioreq_server.lock);
+
+    return rc;
+}
+
 static void hvm_update_ioreq_evtchn(struct hvm_ioreq_server *s,
                                     struct hvm_ioreq_vcpu *sv)
 {
xen/include/asm-x86/hvm/ioreq.h +3 -0
index e8119b26a6..2131c944d4 100644
@@ -55,6 +55,9 @@ unsigned int hvm_broadcast_ioreq(ioreq_t *p, bool buffered);
 
 void hvm_ioreq_init(struct domain *d);
 
+int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
+                          int (*handler)(struct vcpu *v, ioreq_t *));
+
 #endif /* __ASM_X86_HVM_IOREQ_H__ */
 
 /*
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#11 lore raw

Re: [Xen-devel] [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 10 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Provide a routine to register the handler for an internal ioreq server. Note the handler can only be set once.

I'd prefer hvm_set_ioreq_handler() and some sort of guard to prevent enabling of an internal server with no handler (probably in the previous patch) would be prudent, I think. Also, why the set-once semantic?

Paul

Quoted 65 lines
---
 xen/arch/x86/hvm/ioreq.c        | 32 ++++++++++++++++++++++++++++++++
 xen/include/asm-x86/hvm/ioreq.h |  3 +++
 2 files changed, 35 insertions(+)
xen/arch/x86/hvm/ioreq.c +32 -0
index 3fb6fe9585..d8fea191aa 100644
@@ -486,6 +486,38 @@ static int hvm_add_ioreq_gfn(struct hvm_ioreq_server *s, bool buf)
     return rc;
 }
 
+int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
+                          int (*handler)(struct vcpu *v, ioreq_t *))
+{
+    struct hvm_ioreq_server *s;
+    int rc = 0;
+
+    spin_lock_recursive(&d->arch.hvm.ioreq_server.lock);
+    s = get_ioreq_server(d, id);
+    if ( !s )
+    {
+        rc = -ENOENT;
+        goto out;
+    }
+    if ( !s->internal )
+    {
+        rc = -EINVAL;
+        goto out;
+    }
+    if ( s->handler != NULL )
+    {
+        rc = -EBUSY;
+        goto out;
+    }
+
+    s->handler = handler;
+
+ out:
+    spin_unlock_recursive(&d->arch.hvm.ioreq_server.lock);
+
+    return rc;
+}
+
 static void hvm_update_ioreq_evtchn(struct hvm_ioreq_server *s,
                                     struct hvm_ioreq_vcpu *sv)
 {
xen/include/asm-x86/hvm/ioreq.h +3 -1
index e8119b26a6..2131c944d4 100644
@@ -55,6 +55,9 @@ unsigned int hvm_broadcast_ioreq(ioreq_t *p, bool buffered);
 
 void hvm_ioreq_init(struct domain *d);
 
+int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
+                          int (*handler)(struct vcpu *v, ioreq_t *));
+
 #endif /* __ASM_X86_HVM_IOREQ_H__ */
 
 /*
--
Quoted 1 line

2.22.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 8 lines · Paul Durrant
Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 4 lines
Quoted 2 lines

Provide a routine to register the handler for an internal ioreq server. Note the handler can only be set once.

I'd prefer hvm_set_ioreq_handler() and some sort of guard to prevent enabling of an internal server with no handler (probably in the previous patch) would be prudent, I think.

Right, I will add it.

Quoted 1 line

Also, why the set-once semantic?

Well, I didn't have the need to change the handler of internal ioreq servers (vPCI) so I've coded it that way. If you think it's better to allow run time changes of the handler that's fine, I just didn't have the need for it given the current use-case and I thought it would be safer.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [Xen-devel] [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 25 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 22 August 2019 08:44
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: Re: [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 8 lines · Paul Durrant
Quoted 8 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>
Subject: [PATCH 4/7] ioreq: allow registering internal ioreq server handler

Quoted 4 lines
Quoted 2 lines

Provide a routine to register the handler for an internal ioreq server. Note the handler can only be set once.

I'd prefer hvm_set_ioreq_handler() and some sort of guard to prevent enabling of an internal server

with no handler (probably in the previous patch) would be prudent, I think.

Right, I will add it.

Quoted 8 lines
Quoted 1 line

Also, why the set-once semantic?

Well, I didn't have the need to change the handler of internal ioreq servers (vPCI) so I've coded it that way. If you think it's better to allow run time changes of the handler that's fine, I just didn't have the need for it given the current use-case and I thought it would be safer.

I think a more relaxed semantic of only being able to change the handler when the ioreq server is disabled would be fine. Also, I wonder whether you ought to allow handler registration to set some opaque caller context too, rather than assuming that the vcpu is the appropriate context to pass to all handlers?

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

PATCH 5/7 ioreq: allow decoding accesses to MMCFG regions

In reply to Roger Pau Monne

Pick up on the infrastructure already added for vPCI and allow ioreq to decode accesses to MMCFG regions registered for a domain. This infrastructure is still only accessible from internal callers, so MMCFG regions can only be registered from the internal domain builder used by PVH dom0.

Note that the vPCI infrastructure to decode and handle accesses to MMCFG regions will be removed in following patches when vPCI is switched to become an internal ioreq server.

---
 xen/arch/x86/hvm/hvm.c          |  2 +-
 xen/arch/x86/hvm/io.c           | 36 +++++---------
 xen/arch/x86/hvm/ioreq.c        | 88 +++++++++++++++++++++++++++++++--
 xen/include/asm-x86/hvm/io.h    | 12 ++++-
 xen/include/asm-x86/hvm/ioreq.h |  6 +++
 5 files changed, 113 insertions(+), 31 deletions(-)
xen/arch/x86/hvm/hvm.c +1 -1
index 029eea3b85..b7a53377a5 100644
@@ -741,7 +741,7 @@ void hvm_domain_destroy(struct domain *d)
         xfree(ioport);
     }
 
-    destroy_vpci_mmcfg(d);
+    hvm_ioreq_free_mmcfg(d);
 }
 
 static int hvm_save_tsc_adjust(struct vcpu *v, hvm_domain_context_t *h)
xen/arch/x86/hvm/io.c +12 -24
index a5b0a23f06..6585767c03 100644
@@ -279,6 +279,18 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
     return CF8_ADDR_LO(cf8) | (addr & 3);
 }
 
+unsigned int hvm_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
+                                   paddr_t addr, pci_sbdf_t *sbdf)
+{
+    addr -= mmcfg->addr;
+    sbdf->bdf = MMCFG_BDF(addr);
+    sbdf->bus += mmcfg->start_bus;
+    sbdf->seg = mmcfg->segment;
+
+    return addr & (PCI_CFG_SPACE_EXP_SIZE - 1);
+}
+
+
 /* Do some sanity checks. */
 static bool vpci_access_allowed(unsigned int reg, unsigned int len)
 {
@@ -383,14 +395,6 @@ void register_vpci_portio_handler(struct domain *d)
     handler->ops = &vpci_portio_ops;
 }
 
-struct hvm_mmcfg {
-    struct list_head next;
-    paddr_t addr;
-    unsigned int size;
-    uint16_t segment;
-    uint8_t start_bus;
-};
-
 /* Handlers to trap PCI MMCFG config accesses. */
 static const struct hvm_mmcfg *vpci_mmcfg_find(const struct domain *d,
                                                paddr_t addr)
@@ -558,22 +562,6 @@ int register_vpci_mmcfg_handler(struct domain *d, paddr_t addr,
     return 0;
 }
 
-void destroy_vpci_mmcfg(struct domain *d)
-{
-    struct list_head *mmcfg_regions = &d->arch.hvm.mmcfg_regions;
-
-    write_lock(&d->arch.hvm.mmcfg_lock);
-    while ( !list_empty(mmcfg_regions) )
-    {
-        struct hvm_mmcfg *mmcfg = list_first_entry(mmcfg_regions,
-                                                   struct hvm_mmcfg, next);
-
-        list_del(&mmcfg->next);
-        xfree(mmcfg);
-    }
-    write_unlock(&d->arch.hvm.mmcfg_lock);
-}
-
 /*
  * Local variables:
  * mode: C
xen/arch/x86/hvm/ioreq.c +83 -5

134 lines not on this page · raw message

xen/include/asm-x86/hvm/io.h +11 -1
index 7ceb119b64..26f0489171 100644
@@ -165,9 +165,19 @@ void stdvga_deinit(struct domain *d);
 
 extern void hvm_dpci_msi_eoi(struct domain *d, int vector);
 
-/* Decode a PCI port IO access into a bus/slot/func/reg. */
+struct hvm_mmcfg {
+    struct list_head next;
+    paddr_t addr;
+    unsigned int size;
+    uint16_t segment;
+    uint8_t start_bus;
+};
+
+/* Decode a PCI port IO or MMCFG access into a bus/slot/func/reg. */
 unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr,
                                  pci_sbdf_t *sbdf);
+unsigned int hvm_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg,
+                                   paddr_t addr, pci_sbdf_t *sbdf);
 
 /*
  * HVM port IO handler that performs forwarding of guest IO ports into machine
xen/include/asm-x86/hvm/ioreq.h +6 -0
index 2131c944d4..10b9586885 100644
@@ -58,6 +58,12 @@ void hvm_ioreq_init(struct domain *d);
 int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
                           int (*handler)(struct vcpu *v, ioreq_t *));
 
+int hvm_ioreq_register_mmcfg(struct domain *d, paddr_t addr,
+                             unsigned int start_bus, unsigned int end_bus,
+                             unsigned int seg);
+
+void hvm_ioreq_free_mmcfg(struct domain *d);
+
 #endif /* __ASM_X86_HVM_IOREQ_H__ */
 
 /*
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#15 lore raw

PATCH 6/7 vpci: register as an internal ioreq server

In reply to Roger Pau Monne

Switch vPCI to become an internal ioreq server, and hence drop all the vPCI specific decoding and trapping to PCI IO ports and MMCFG regions.

This allows to unify the vPCI code with the ioreq infrastructure, opening the door for domains having PCI accesses handled by vPCI and other ioreq servers at the same time.

---
 xen/arch/x86/hvm/dom0_build.c       |   9 +-
 xen/arch/x86/hvm/hvm.c              |   5 +-
 xen/arch/x86/hvm/io.c               | 272 ----------------------------
 xen/arch/x86/hvm/ioreq.c            |   5 +
 xen/arch/x86/physdev.c              |   7 +-
 xen/drivers/passthrough/x86/iommu.c |   2 +-
 xen/drivers/vpci/vpci.c             |  54 ++++++
 xen/include/asm-x86/hvm/io.h        |   2 +-
 xen/include/xen/vpci.h              |   3 +
 9 files changed, 77 insertions(+), 282 deletions(-)
xen/arch/x86/hvm/dom0_build.c +5 -4
index 8845399ae9..7925189fed 100644
@@ -29,6 +29,7 @@
 
 #include <asm/bzimage.h>
 #include <asm/dom0_build.h>
+#include <asm/hvm/ioreq.h>
 #include <asm/hvm/support.h>
 #include <asm/io_apic.h>
 #include <asm/p2m.h>
@@ -1117,10 +1118,10 @@ static void __hwdom_init pvh_setup_mmcfg(struct domain *d)
 
     for ( i = 0; i < pci_mmcfg_config_num; i++ )
     {
-        rc = register_vpci_mmcfg_handler(d, pci_mmcfg_config[i].address,
-                                         pci_mmcfg_config[i].start_bus_number,
-                                         pci_mmcfg_config[i].end_bus_number,
-                                         pci_mmcfg_config[i].pci_segment);
+        rc = hvm_ioreq_register_mmcfg(d, pci_mmcfg_config[i].address,
+                                      pci_mmcfg_config[i].start_bus_number,
+                                      pci_mmcfg_config[i].end_bus_number,
+                                      pci_mmcfg_config[i].pci_segment);
         if ( rc )
             printk("Unable to setup MMCFG handler at %#lx for segment %u\n",
                    pci_mmcfg_config[i].address,
xen/arch/x86/hvm/hvm.c +4 -1
index b7a53377a5..3fcf46779b 100644
@@ -644,10 +644,13 @@ int hvm_domain_initialise(struct domain *d)
         d->arch.hvm.io_bitmap = hvm_io_bitmap;
 
     register_g2m_portio_handler(d);
-    register_vpci_portio_handler(d);
 
     hvm_ioreq_init(d);
 
+    rc = vpci_register_ioreq(d);
+    if ( rc )
+        goto fail1;
+
     hvm_init_guest_time(d);
 
     d->arch.hvm.params[HVM_PARAM_TRIPLE_FAULT_REASON] = SHUTDOWN_reboot;
xen/arch/x86/hvm/io.c +0 -272

278 lines not on this page · raw message

xen/arch/x86/hvm/ioreq.c +5 -0
index 10c0f7a574..b2582bd3a0 100644
@@ -1707,6 +1707,11 @@ int hvm_ioreq_register_mmcfg(struct domain *d, paddr_t addr,
     return 0;
 }
 
+bool hvm_is_mmcfg_address(const struct domain *d, paddr_t addr)
+{
+    return mmcfg_find(d, addr);
+}
+
 /*
  * Local variables:
  * mode: C
xen/arch/x86/physdev.c +4 -3
index 3a3c15890b..a48b220fc3 100644
@@ -11,6 +11,7 @@
 #include <asm/current.h>
 #include <asm/io_apic.h>
 #include <asm/msi.h>
+#include <asm/hvm/ioreq.h>
 #include <asm/hvm/irq.h>
 #include <asm/hypercall.h>
 #include <public/xen.h>
@@ -562,9 +563,9 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
              * For HVM (PVH) domains try to add the newly found MMCFG to the
              * domain.
              */
-            ret = register_vpci_mmcfg_handler(currd, info.address,
-                                              info.start_bus, info.end_bus,
-                                              info.segment);
+            ret = hvm_ioreq_register_mmcfg(currd, info.address,
+                                           info.start_bus, info.end_bus,
+                                           info.segment);
         }
 
         break;
xen/drivers/passthrough/x86/iommu.c +1 -1
index fd05075bb5..e0f3da91ce 100644
@@ -244,7 +244,7 @@ static bool __hwdom_init hwdom_iommu_map(const struct domain *d,
      * TODO: runtime added MMCFG regions are not checked to make sure they
      * don't overlap with already mapped regions, thus preventing trapping.
      */
-    if ( has_vpci(d) && vpci_is_mmcfg_address(d, pfn_to_paddr(pfn)) )
+    if ( hvm_is_mmcfg_address(d, pfn_to_paddr(pfn)) )
         return false;
 
     return true;
xen/drivers/vpci/vpci.c +54 -0

66 lines not on this page · raw message

xen/include/asm-x86/hvm/io.h +1 -1
index 26f0489171..75a24f33bc 100644
@@ -196,7 +196,7 @@ int register_vpci_mmcfg_handler(struct domain *d, paddr_t addr,
 void destroy_vpci_mmcfg(struct domain *d);
 
 /* Check if an address is between a MMCFG region for a domain. */
-bool vpci_is_mmcfg_address(const struct domain *d, paddr_t addr);
+bool hvm_is_mmcfg_address(const struct domain *d, paddr_t addr);
 
 #endif /* __ASM_X86_HVM_IO_H__ */
 
xen/include/xen/vpci.h +3 -0
index 4cf233c779..666dd1ca68 100644
@@ -23,6 +23,9 @@ typedef int vpci_register_init_t(struct pci_dev *dev);
   static vpci_register_init_t *const x##_entry  \
                __used_section(".data.vpci." p) = x
 
+/* Register vPCI handler wit ioreq. */
+int vpci_register_ioreq(struct domain *d);
+
 /* Add vPCI handlers to device. */
 int __must_check vpci_add_handlers(struct pci_dev *dev);
 
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#16 lore raw

PATCH 7/7 ioreq: provide support for long-running operations...

In reply to Roger Pau Monne

...and switch vPCI to use this infrastructure for long running physmap modification operations.

This allows to get rid of the vPCI specific modifications done to handle_hvm_io_completion and allows generalizing the support for long-running operations to other internal ioreq servers. Such support is implemented as a specific handler that can be registers by internal ioreq servers and that will be called to check for pending work. Returning true from this handler will prevent the vcpu from running until the handler returns false.

---
 xen/arch/x86/hvm/ioreq.c         | 55 ++++++++++++++++++++++++++++----
 xen/drivers/vpci/vpci.c          |  3 ++
 xen/include/asm-x86/hvm/domain.h |  1 +
 xen/include/asm-x86/hvm/ioreq.h  |  2 ++
 4 files changed, 55 insertions(+), 6 deletions(-)
xen/arch/x86/hvm/ioreq.c +49 -6
index b2582bd3a0..8e160a0a14 100644
@@ -186,18 +186,29 @@ bool handle_hvm_io_completion(struct vcpu *v)
     enum hvm_io_completion io_completion;
     unsigned int id;
 
-    if ( has_vpci(d) && vpci_process_pending(v) )
-    {
-        raise_softirq(SCHEDULE_SOFTIRQ);
-        return false;
-    }
-
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
         struct hvm_ioreq_vcpu *sv;
 
         if ( s->internal )
+        {
+            if ( s->pending && s->pending(v) )
+            {
+                /*
+                 * Need to raise a scheduler irq in order to prevent the guest
+                 * vcpu from resuming execution.
+                 *
+                 * Note this is not required for external ioreq operations
+                 * because in that case the vcpu is marked as blocked, but this
+                 * cannot be done for long-running internal operations, since
+                 * it would prevent the vcpu from being scheduled and thus the
+                 * long running operation from finishing.
+                 */
+                raise_softirq(SCHEDULE_SOFTIRQ);
+                return false;
+            }
             continue;
+        }
 
         list_for_each_entry ( sv,
                               &s->ioreq_vcpu_list,
@@ -518,6 +529,38 @@ int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
     return rc;
 }
 
+int hvm_add_ioreq_pending_handler(struct domain *d, ioservid_t id,
+                                  bool (*pending)(struct vcpu *v))
+{
+    struct hvm_ioreq_server *s;
+    int rc = 0;
+
+    spin_lock_recursive(&d->arch.hvm.ioreq_server.lock);
+    s = get_ioreq_server(d, id);
+    if ( !s )
+    {
+        rc = -ENOENT;
+        goto out;
+    }
+    if ( !s->internal )
+    {
+        rc = -EINVAL;
+        goto out;
+    }
+    if ( s->pending != NULL )
+    {
+        rc = -EBUSY;
+        goto out;
+    }
+
+    s->pending = pending;
+
+ out:
+    spin_unlock_recursive(&d->arch.hvm.ioreq_server.lock);
+
+    return rc;
+}
+
 static void hvm_update_ioreq_evtchn(struct hvm_ioreq_server *s,
                                     struct hvm_ioreq_vcpu *sv)
 {
xen/drivers/vpci/vpci.c +3 -0
index 510e3ee771..54b0f31612 100644
@@ -508,6 +508,9 @@ int vpci_register_ioreq(struct domain *d)
         return rc;
 
     rc = hvm_add_ioreq_handler(d, id, ioreq_handler);
+    if ( rc )
+        return rc;
+    rc = hvm_add_ioreq_pending_handler(d, id, vpci_process_pending);
     if ( rc )
         return rc;
 
xen/include/asm-x86/hvm/domain.h +1 -0
index f0be303517..80a38ffe48 100644
@@ -73,6 +73,7 @@ struct hvm_ioreq_server {
         };
         struct {
             int (*handler)(struct vcpu *v, ioreq_t *);
+            bool (*pending)(struct vcpu *v);
         };
     };
 };
xen/include/asm-x86/hvm/ioreq.h +2 -0
index 10b9586885..cc3e27d059 100644
@@ -57,6 +57,8 @@ void hvm_ioreq_init(struct domain *d);
 
 int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
                           int (*handler)(struct vcpu *v, ioreq_t *));
+int hvm_add_ioreq_pending_handler(struct domain *d, ioservid_t id,
+                                  bool (*pending)(struct vcpu *v));
 
 int hvm_ioreq_register_mmcfg(struct domain *d, paddr_t addr,
                              unsigned int start_bus, unsigned int end_bus,
Signature · 7 lines
2.22.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#17 lore raw

Re: [Xen-devel] [PATCH 7/7] ioreq: provide support for long-running operations...

Quoted 14 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>
Subject: [PATCH 7/7] ioreq: provide support for long-running operations...

...and switch vPCI to use this infrastructure for long running physmap modification operations.

Quoted 7 lines

This allows to get rid of the vPCI specific modifications done to handle_hvm_io_completion and allows generalizing the support for long-running operations to other internal ioreq servers. Such support is implemented as a specific handler that can be registers by internal ioreq servers and that will be called to check for pending work. Returning true from this handler will prevent the vcpu from running until the handler returns false.

Rather than having another callback can the handler not be re-called with same ioreq? It could return different values depending on whether there is more work to do (requiring another call) or whether it is done and the vcpu can be resumed. Would that work?

Paul

Quoted 128 lines
---
 xen/arch/x86/hvm/ioreq.c         | 55 ++++++++++++++++++++++++++++----
 xen/drivers/vpci/vpci.c          |  3 ++
 xen/include/asm-x86/hvm/domain.h |  1 +
 xen/include/asm-x86/hvm/ioreq.h  |  2 ++
 4 files changed, 55 insertions(+), 6 deletions(-)
xen/arch/x86/hvm/ioreq.c +49 -6
index b2582bd3a0..8e160a0a14 100644
@@ -186,18 +186,29 @@ bool handle_hvm_io_completion(struct vcpu *v)
     enum hvm_io_completion io_completion;
     unsigned int id;
 
-    if ( has_vpci(d) && vpci_process_pending(v) )
-    {
-        raise_softirq(SCHEDULE_SOFTIRQ);
-        return false;
-    }
-
     FOR_EACH_IOREQ_SERVER(d, id, s)
     {
         struct hvm_ioreq_vcpu *sv;
 
         if ( s->internal )
+        {
+            if ( s->pending && s->pending(v) )
+            {
+                /*
+                 * Need to raise a scheduler irq in order to prevent the guest
+                 * vcpu from resuming execution.
+                 *
+                 * Note this is not required for external ioreq operations
+                 * because in that case the vcpu is marked as blocked, but this
+                 * cannot be done for long-running internal operations, since
+                 * it would prevent the vcpu from being scheduled and thus the
+                 * long running operation from finishing.
+                 */
+                raise_softirq(SCHEDULE_SOFTIRQ);
+                return false;
+            }
             continue;
+        }
 
         list_for_each_entry ( sv,
                               &s->ioreq_vcpu_list,
@@ -518,6 +529,38 @@ int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
     return rc;
 }
 
+int hvm_add_ioreq_pending_handler(struct domain *d, ioservid_t id,
+                                  bool (*pending)(struct vcpu *v))
+{
+    struct hvm_ioreq_server *s;
+    int rc = 0;
+
+    spin_lock_recursive(&d->arch.hvm.ioreq_server.lock);
+    s = get_ioreq_server(d, id);
+    if ( !s )
+    {
+        rc = -ENOENT;
+        goto out;
+    }
+    if ( !s->internal )
+    {
+        rc = -EINVAL;
+        goto out;
+    }
+    if ( s->pending != NULL )
+    {
+        rc = -EBUSY;
+        goto out;
+    }
+
+    s->pending = pending;
+
+ out:
+    spin_unlock_recursive(&d->arch.hvm.ioreq_server.lock);
+
+    return rc;
+}
+
 static void hvm_update_ioreq_evtchn(struct hvm_ioreq_server *s,
                                     struct hvm_ioreq_vcpu *sv)
 {
xen/drivers/vpci/vpci.c +3 -0
index 510e3ee771..54b0f31612 100644
@@ -508,6 +508,9 @@ int vpci_register_ioreq(struct domain *d)
         return rc;
 
     rc = hvm_add_ioreq_handler(d, id, ioreq_handler);
+    if ( rc )
+        return rc;
+    rc = hvm_add_ioreq_pending_handler(d, id, vpci_process_pending);
     if ( rc )
         return rc;
 
xen/include/asm-x86/hvm/domain.h +1 -0
index f0be303517..80a38ffe48 100644
@@ -73,6 +73,7 @@ struct hvm_ioreq_server {
         };
         struct {
             int (*handler)(struct vcpu *v, ioreq_t *);
+            bool (*pending)(struct vcpu *v);
         };
     };
 };
xen/include/asm-x86/hvm/ioreq.h +2 -1
index 10b9586885..cc3e27d059 100644
@@ -57,6 +57,8 @@ void hvm_ioreq_init(struct domain *d);
 
 int hvm_add_ioreq_handler(struct domain *d, ioservid_t id,
                           int (*handler)(struct vcpu *v, ioreq_t *));
+int hvm_add_ioreq_pending_handler(struct domain *d, ioservid_t id,
+                                  bool (*pending)(struct vcpu *v));
 
 int hvm_ioreq_register_mmcfg(struct domain *d, paddr_t addr,
                              unsigned int start_bus, unsigned int end_bus,
--
Quoted 1 line

2.22.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw

Re: [Xen-devel] [PATCH 7/7] ioreq: provide support for long-running operations...

Quoted 22 lines · Paul Durrant
Quoted 14 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>
Subject: [PATCH 7/7] ioreq: provide support for long-running operations...

...and switch vPCI to use this infrastructure for long running physmap modification operations.

Quoted 7 lines

This allows to get rid of the vPCI specific modifications done to handle_hvm_io_completion and allows generalizing the support for long-running operations to other internal ioreq servers. Such support is implemented as a specific handler that can be registers by internal ioreq servers and that will be called to check for pending work. Returning true from this handler will prevent the vcpu from running until the handler returns false.

Quoted 1 line

Rather than having another callback can the handler not be re-called with same ioreq? It could return different values depending on whether there is more work to do (requiring another call) or whether it is done and the vcpu can be resumed. Would that work?

I guess this would work also. The issue with this approach is that I would have to find somewhere to store the ioreq while the operation is being processed, which is not required with the proposed two handler approach.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw

Re: [Xen-devel] [PATCH 7/7] ioreq: provide support for long-running operations...

Quoted 37 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 22 August 2019 13:55
To: Paul Durrant <[email protected]>
Cc: [email protected]; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>
Subject: Re: [PATCH 7/7] ioreq: provide support for long-running operations...

Quoted 6 lines · Paul Durrant
Quoted 6 lines

-----Original Message-----
From: Roger Pau Monne <[email protected]>
Sent: 21 August 2019 15:59
To: [email protected]
Cc: Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; George

Dunlap

Quoted 1 line
Quoted 1 line

<[email protected]>; Ian Jackson <[email protected]>; Julien Grall

<[email protected]>;

Quoted 15 lines
Quoted 14 lines

Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>
Subject: [PATCH 7/7] ioreq: provide support for long-running operations...

...and switch vPCI to use this infrastructure for long running physmap modification operations.

This allows to get rid of the vPCI specific modifications done to handle_hvm_io_completion and allows generalizing the support for long-running operations to other internal ioreq servers. Such support is implemented as a specific handler that can be registers by internal ioreq servers and that will be called to check for pending work. Returning true from this handler will prevent the vcpu from running until the handler returns false.

Quoted 8 lines
Quoted 1 line

Rather than having another callback can the handler not be re-called with same ioreq? It could

return different values depending on whether there is more work to do (requiring another call) or whether it is done and the vcpu can be resumed. Would that work?

I guess this would work also. The issue with this approach is that I would have to find somewhere to store the ioreq while the operation is being processed, which is not required with the proposed two handler approach.

The ioreq already is stored in v->arch.hvm.hvm_io.io_req anyway, so can't you use that copy?

Paul

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel