Skip to contentSkip to conversation

Xen stubdom support

PATCH In review Jason Andryuk 8 participants 36 messages Mar 11, 2019 – Jan 15, 2020

lore
Share & tools
lore
#0 lore raw

cover Xen stubdom support

Xen supports running QEMU in a dedicated service vm - a stub domain or stubdom. QEMU is then isolated outside of the privileged Domain-0.

When running in a stubdom, there are a few changes needed for QEMU. On older Xen versions, the default ioreq server needs to have the stubdom's domid specified. The stubdom doesn't run PV backends, so that initialization code can be skipped. Stubdom's don't support MSI-X, so that PCI capability must be hidden from passed through devices.

Stubdom mode is enabled by the new -xen-stubdom flag.

Jason Andryuk (5):
xen: Introduce -xen-stubdom option
xen: Move xenstore initialization to common location
xen: Skip backend initialization for stubdom
xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen
xen-pt: Hide MSI-X from xen stubdoms

Simon Gaiser (1):
xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

 hw/i386/xen/xen-hvm.c       | 22 ++++++++++++++++------
 hw/xen/xen-legacy-backend.c |  8 --------
 hw/xen/xen_pt.c             | 10 +++++++---
 hw/xen/xen_pt_config_init.c |  3 +++
 include/hw/xen/xen.h        |  6 ++++++
 include/hw/xen/xen_common.h |  5 +++++
 qemu-options.hx             |  7 +++++++
 vl.c                        |  8 ++++++++
 8 files changed, 52 insertions(+), 17 deletions(-)
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/6 xen: Introduce -xen-stubdom option

With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. There are a few differences when running in a stubdom compared to dom0. Add the -xen-stubdom option to select this mode at runtime. The default is off.

---
 include/hw/xen/xen.h | 6 ++++++
 qemu-options.hx      | 7 +++++++
 vl.c                 | 8 ++++++++
 3 files changed, 21 insertions(+)
include/hw/xen/xen.h +6 -0
index ba039c146d..fed3611623 100644
@@ -21,6 +21,7 @@ enum xen_mode {
 extern uint32_t xen_domid;
 extern enum xen_mode xen_mode;
 extern bool xen_domid_restrict;
+extern bool xen_stubdom;
 
 extern bool xen_allowed;
 
@@ -29,6 +30,11 @@ static inline bool xen_enabled(void)
     return xen_allowed;
 }
 
+static inline bool xen_stubdom_enabled(void)
+{
+    return xen_stubdom;
+}
+
 int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num);
 void xen_piix3_set_irq(void *opaque, int irq_num, int level);
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len);
qemu-options.hx +7 -0
index 1cf9aac1fe..ba56c3dd9a 100644
@@ -3386,6 +3386,10 @@ DEF("xen-domid-restrict", 0, QEMU_OPTION_xen_domid_restrict,
     "                        to specified domain id. (Does not affect\n"
     "                        xenpv machine type).\n",
     QEMU_ARCH_ALL)
+DEF("xen-stubdom", 0, QEMU_OPTION_xen_stubdom,
+    "-xen-stubdom    specify QEMU is running in a stubdom, so certain\n"
+    "                behavior changes. (Does not affect xenpv machine type).\n",
+    QEMU_ARCH_ALL)
 STEXI
 @item -xen-domid @var{id}
 @findex -xen-domid
@@ -3396,6 +3400,9 @@ Attach to existing xen domain.
 libxl will use this when starting QEMU (XEN only).
 @findex -xen-domid-restrict
 Restrict set of available xen operations to specified domain id (XEN only).
+@findex -xen-stubdom
+@item -xen-stubdom
+Run qemu in stubdom-mode (XEN only).
 ETEXI
 
 DEF("no-reboot", 0, QEMU_OPTION_no_reboot, \
vl.c +8 -0
index 4a350de5cd..0d04319d9b 100644
@@ -206,6 +206,7 @@ bool xen_allowed;
 uint32_t xen_domid;
 enum xen_mode xen_mode = XEN_EMULATE;
 bool xen_domid_restrict;
+bool xen_stubdom;
 
 static int has_defaults = 1;
 static int default_serial = 1;
@@ -3796,6 +3797,13 @@ int main(int argc, char **argv, char **envp)
                 }
                 xen_domid_restrict = true;
                 break;
+            case QEMU_OPTION_xen_stubdom:
+                if (!(xen_available())) {
+                    error_report("Option not supported for this target");
+                    exit(1);
+                }
+                xen_stubdom = true;
+                break;
             case QEMU_OPTION_trace:
                 g_free(trace_file);
                 trace_file = trace_opt_parse(optarg);
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

Re: [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option

Quoted 4 lines · Jason Andryuk

With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. There are a few differences when running in a stubdom compared to dom0. Add the -xen-stubdom option to select this mode at runtime. The default is off.

This should be "-accel xen,stubdom=on". You should find examples for tcg that explain how to add a suboption to -accel.

Paolo

Quoted 73 lines
---
 include/hw/xen/xen.h | 6 ++++++
 qemu-options.hx      | 7 +++++++
 vl.c                 | 8 ++++++++
 3 files changed, 21 insertions(+)
include/hw/xen/xen.h +6 -0
index ba039c146d..fed3611623 100644
@@ -21,6 +21,7 @@ enum xen_mode {
 extern uint32_t xen_domid;
 extern enum xen_mode xen_mode;
 extern bool xen_domid_restrict;
+extern bool xen_stubdom;
 
 extern bool xen_allowed;
 
@@ -29,6 +30,11 @@ static inline bool xen_enabled(void)
     return xen_allowed;
 }
 
+static inline bool xen_stubdom_enabled(void)
+{
+    return xen_stubdom;
+}
+
 int xen_pci_slot_get_pirq(PCIDevice *pci_dev, int irq_num);
 void xen_piix3_set_irq(void *opaque, int irq_num, int level);
 void xen_piix_pci_write_config_client(uint32_t address, uint32_t val, int len);
qemu-options.hx +7 -0
index 1cf9aac1fe..ba56c3dd9a 100644
@@ -3386,6 +3386,10 @@ DEF("xen-domid-restrict", 0, QEMU_OPTION_xen_domid_restrict,
     "                        to specified domain id. (Does not affect\n"
     "                        xenpv machine type).\n",
     QEMU_ARCH_ALL)
+DEF("xen-stubdom", 0, QEMU_OPTION_xen_stubdom,
+    "-xen-stubdom    specify QEMU is running in a stubdom, so certain\n"
+    "                behavior changes. (Does not affect xenpv machine type).\n",
+    QEMU_ARCH_ALL)
 STEXI
 @item -xen-domid @var{id}
 @findex -xen-domid
@@ -3396,6 +3400,9 @@ Attach to existing xen domain.
 libxl will use this when starting QEMU (XEN only).
 @findex -xen-domid-restrict
 Restrict set of available xen operations to specified domain id (XEN only).
+@findex -xen-stubdom
+@item -xen-stubdom
+Run qemu in stubdom-mode (XEN only).
 ETEXI
 
 DEF("no-reboot", 0, QEMU_OPTION_no_reboot, \
vl.c +1 -0
index 4a350de5cd..0d04319d9b 100644
@@ -206,6 +206,7 @@ bool xen_allowed;
 uint32_t xen_domid;
 enum xen_mode xen_mode = XEN_EMULATE;
 bool xen_domid_restrict;
+bool xen_stubdom;
 
 static int has_defaults = 1;
 static int default_serial = 1;
@@ -3796,6 +3797,13 @@ int main(int argc, char **argv, char **envp)
                 }
                 xen_domid_restrict = true;
                 break;

vl.c int main(int argc, char **argv, char **envp) · line 3809

Quoted 10 lines
vl.c +7 -0
+            case QEMU_OPTION_xen_stubdom:
+                if (!(xen_available())) {
+                    error_report("Option not supported for this target");
+                    exit(1);
+                }
+                xen_stubdom = true;
+                break;
             case QEMU_OPTION_trace:
                 g_free(trace_file);
                 trace_file = trace_opt_parse(optarg);

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Qemu-devel] [PATCH 1/6] xen: Introduce -xen-stubdom option

Quoted 9 lines · Paolo Bonzini
Quoted 4 lines · Jason Andryuk

With Xen, QEMU can run isolated in a dedicated service VM - a stubdom. There are a few differences when running in a stubdom compared to dom0. Add the -xen-stubdom option to select this mode at runtime. The default is off.

This should be "-accel xen,stubdom=on". You should find examples for tcg that explain how to add a suboption to -accel.

Thanks, Paolo. I'll re-work the option as you suggest.

Regards,
Jason

#4 lore raw

PATCH 2/6 xen: Move xenstore initialization to common location

In reply to Jason Andryuk

For the xen stubdom case, we'll want xenstore initialized, but we'll want to skip the rest of xen_be_init. Move the initialization to xen_hvm_init so we can conditionalize calling xen_be_init.

xs_domain_open() is deprecated for xs_open(0), so make the replacement as well.

---
 hw/i386/xen/xen-hvm.c       | 8 ++++++++
 hw/xen/xen-legacy-backend.c | 8 --------
 2 files changed, 8 insertions(+), 8 deletions(-)
hw/i386/xen/xen-hvm.c +8 -0
index 2939122e7c..c20c4b27f6 100644
@@ -1487,6 +1487,14 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
 
     xen_bus_init();
 
+    xenstore = xs_open(0);
+    if (!xenstore) {
+        error_report("Can't connect to xenstored");
+        goto err;
+    }
+
+    qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
+
     /* Initialize backend core & drivers */
     if (xen_be_init() != 0) {
         error_report("xen backend core setup failed");
hw/xen/xen-legacy-backend.c +0 -8
index 36fd1e9b09..bdf2fa917f 100644
@@ -683,14 +683,6 @@ int xen_be_init(void)
 {
     xengnttab_handle *gnttabdev;
 
-    xenstore = xs_daemon_open();
-    if (!xenstore) {
-        xen_pv_printf(NULL, 0, "can't connect to xenstored\n");
-        return -1;
-    }
-
-    qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
-
     if (xen_xc == NULL || xen_fmem == NULL) {
         /* Check if xen_init() have been called */
         goto err;
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

Re: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>; Paolo Bonzini <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>
Subject: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 6 lines

For the xen stubdom case, we'll want xenstore initialized, but we'll want to skip the rest of xen_be_init. Move the initialization to xen_hvm_init so we can conditionalize calling xen_be_init.

xs_domain_open() is deprecated for xs_open(0), so make the replacement as well.

Can you elaborate as to why you need to do this when the code at the top of xen_hvm_init() already opens xenstore for its own purposes, and AFAICT xenstore_update() is only needed if QEMU is implementing a PV backend?

Paul

Quoted 46 lines
---
 hw/i386/xen/xen-hvm.c       | 8 ++++++++
 hw/xen/xen-legacy-backend.c | 8 --------
 2 files changed, 8 insertions(+), 8 deletions(-)
hw/i386/xen/xen-hvm.c +8 -0
index 2939122e7c..c20c4b27f6 100644
@@ -1487,6 +1487,14 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
 
     xen_bus_init();
 
+    xenstore = xs_open(0);
+    if (!xenstore) {
+        error_report("Can't connect to xenstored");
+        goto err;
+    }
+
+    qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
+
     /* Initialize backend core & drivers */
     if (xen_be_init() != 0) {
         error_report("xen backend core setup failed");
hw/xen/xen-legacy-backend.c +0 -9
index 36fd1e9b09..bdf2fa917f 100644
@@ -683,14 +683,6 @@ int xen_be_init(void)
 {
     xengnttab_handle *gnttabdev;
 
-    xenstore = xs_daemon_open();
-    if (!xenstore) {
-        xen_pv_printf(NULL, 0, "can't connect to xenstored\n");
-        return -1;
-    }
-
-    qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
-
     if (xen_xc == NULL || xen_fmem == NULL) {
         /* Check if xen_init() have been called */
         goto err;
--
Quoted 1 line

2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

Re: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 12 lines · Paul Durrant
Quoted 12 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>; Paolo Bonzini <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>
Subject: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 10 lines
Quoted 6 lines

For the xen stubdom case, we'll want xenstore initialized, but we'll want to skip the rest of xen_be_init. Move the initialization to xen_hvm_init so we can conditionalize calling xen_be_init.

xs_domain_open() is deprecated for xs_open(0), so make the replacement as well.

Can you elaborate as to why you need to do this when the code at the top of xen_hvm_init() already opens xenstore for its own purposes, and AFAICT xenstore_update() is only needed if QEMU is implementing a PV backend?

Hi, Paul. Thanks for reviewing.

I think you are right, that this basically shouldn't be needed if PV backends are disabled. This patch came out of OpenXT, where it is needed for some out-of-tree patches. But that doesn't make it suitable for upstreaming.

However, while reviewing, it looks like the xen accelerator in hw/xen/xen-common.c:xen_init() registers xen_change_state_handler(). xen_change_state_handler() uses the global xenstore handle and will exit(1) if NULL. I'm not sure how to get the XenIOState xenstore handle over to the accelerator's xen_init. Outside of that, I think you are correct that xenstore_update doesn't need to be run when PV backends are disabled.

Thanks,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 43 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 13 March 2019 18:12
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paolo Bonzini <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>
Subject: Re: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 12 lines · Paul Durrant
Quoted 12 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>; Paolo Bonzini <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>
Subject: [PATCH 2/6] xen: Move xenstore initialization to common location

Quoted 8 lines
Quoted 6 lines

For the xen stubdom case, we'll want xenstore initialized, but we'll want to skip the rest of xen_be_init. Move the initialization to xen_hvm_init so we can conditionalize calling xen_be_init.

xs_domain_open() is deprecated for xs_open(0), so make the replacement as well.

Can you elaborate as to why you need to do this when the code at the top of xen_hvm_init() already

opens xenstore for its own purposes, and AFAICT xenstore_update() is only needed if QEMU is implementing a PV backend?

Quoted 2 lines

Hi, Paul. Thanks for reviewing.

I think you are right, that this basically shouldn't be needed if PV backends are disabled. This patch came out of OpenXT, where it is needed for some out-of-tree patches. But that doesn't make it suitable for upstreaming.

Quoted 4 lines

However, while reviewing, it looks like the xen accelerator in hw/xen/xen-common.c:xen_init() registers xen_change_state_handler(). xen_change_state_handler() uses the global xenstore handle and will exit(1) if NULL.

I see it yes. TBH signalling state via xenstore should go away as it is incompatible with deprivileging, and I think Anthony might have some patches for that? In the meantime I suggest just doing a local xs_open(0) in that function.

Quoted 2 lines

I'm not sure how to get the XenIOState xenstore
handle over to the accelerator's xen_init.

That would not be appropriate as the machine type may not be xenfv and hence xen_hvm_init() may not have been called.

Paul

Quoted 6 lines

Outside of that, I think
you are correct that xenstore_update doesn't need to be run when PV backends are disabled.

Thanks,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

PATCH 3/6 xen: Skip backend initialization for stubdom

In reply to Jason Andryuk

When QEMU is running in a stubdom, it does not provide any Paravirtualized backends. Those still run in dom0 or another driver domain. Therefore we skip backend initialization (xen_bus_init and xen_be_init) for the stubdom case.

Original patch by Anthony PERARD <[email protected]>

---
 hw/i386/xen/xen-hvm.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
hw/i386/xen/xen-hvm.c +9 -7
index c20c4b27f6..4b62f070cb 100644
@@ -1485,8 +1485,6 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
     QLIST_INIT(&state->dev_list);
     device_listener_register(&state->device_listener);
 
-    xen_bus_init();
-
     xenstore = xs_open(0);
     if (!xenstore) {
         error_report("Can't connect to xenstored");
@@ -1495,12 +1493,16 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
 
     qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
 
-    /* Initialize backend core & drivers */
-    if (xen_be_init() != 0) {
-        error_report("xen backend core setup failed");
-        goto err;
+    if (!xen_stubdom_enabled()) {
+        xen_bus_init();
+
+        /* Initialize backend core & drivers */
+        if (xen_be_init() != 0) {
+            error_report("xen backend core setup failed");
+            goto err;
+        }
+        xen_be_register_common();
     }
-    xen_be_register_common();
 
     QLIST_INIT(&xen_physmap);
     xen_read_physmap(state);
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 3/6
#9 lore raw

Re: [PATCH 3/6] xen: Skip backend initialization for stubdom

Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>; Paolo Bonzini <[email protected]>; Richard Henderson <[email protected]>; Eduardo Habkost <[email protected]>; Michael S. Tsirkin <[email protected]>; Marcel Apfelbaum <[email protected]>
Subject: [PATCH 3/6] xen: Skip backend initialization for stubdom

Quoted 8 lines

When QEMU is running in a stubdom, it does not provide any Paravirtualized backends. Those still run in dom0 or another driver domain. Therefore we skip backend initialization (xen_bus_init and xen_be_init) for the stubdom case.

Original patch by Anthony PERARD <[email protected]>

Quoted 40 lines
---
 hw/i386/xen/xen-hvm.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
hw/i386/xen/xen-hvm.c +9 -8
index c20c4b27f6..4b62f070cb 100644
@@ -1485,8 +1485,6 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
     QLIST_INIT(&state->dev_list);
     device_listener_register(&state->device_listener);
 
-    xen_bus_init();
-
     xenstore = xs_open(0);
     if (!xenstore) {
         error_report("Can't connect to xenstored");
@@ -1495,12 +1493,16 @@ void xen_hvm_init(PCMachineState *pcms, MemoryRegion **ram_memory)
 
     qemu_set_fd_handler(xs_fileno(xenstore), xenstore_update, NULL, NULL);
 
-    /* Initialize backend core & drivers */
-    if (xen_be_init() != 0) {
-        error_report("xen backend core setup failed");
-        goto err;
+    if (!xen_stubdom_enabled()) {
+        xen_bus_init();
+
+        /* Initialize backend core & drivers */
+        if (xen_be_init() != 0) {
+            error_report("xen backend core setup failed");
+            goto err;
+        }
+        xen_be_register_common();
     }
-    xen_be_register_common();
 
     QLIST_INIT(&xen_physmap);
     xen_read_physmap(state);
--
Quoted 1 line

2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

PATCH 4/6 xen: Set HVM_PARAM_DM_DOMAIN for stubdom on older Xen

In reply to Jason Andryuk

When running in a stubdom, we have to inform the hypervisor that the stubdom and not dom0 is handling the device model. Explicitly created ioreq servers are fine, but a call to HVM_PARAM_DM_DOMAIN is needed for the default ioreq server.

Xen 4.12 removes the default ioreq server. With that, Xen started returning an error when setting HVM_PARAM_DM_DOMAIN. Put the HVM_PARAM_DM_DOMAIN call in the version compatibility header. When we fallback to the default ioreq server, issue the call and don't bother to check the return value.

Original patch by Anthony PERARD <[email protected]>

---
 include/hw/xen/xen_common.h | 5 +++++
 1 file changed, 5 insertions(+)
include/hw/xen/xen_common.h +5 -0
index 9a8155e172..f59f841a43 100644
@@ -616,6 +616,11 @@ static inline void xen_create_ioreq_server(domid_t dom,
 
     *ioservid = 0;
     use_default_ioreq_server = true;
+
+    if (xen_stubdom_enabled()) {
+        xc_hvm_param_set(xen_xc, xen_domid, HVM_PARAM_DM_DOMAIN, DOMID_SELF);
+    }
+
     trace_xen_default_ioreq_server();
 }
 
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#11 lore raw

PATCH 5/6 xen-pt: Hide MSI-X from xen stubdoms

In reply to Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

A compile-time patch was originally written by James McKenzie <[email protected]>

---
 hw/xen/xen_pt_config_init.c | 3 +++
 1 file changed, 3 insertions(+)
hw/xen/xen_pt_config_init.c +3 -0
index 31ec5add1d..b827a493ea 100644
@@ -54,6 +54,9 @@ static int xen_pt_hide_dev_cap(const XenHostPCIDevice *d, uint8_t grp_id)
             return 1;
         }
         break;
+    case PCI_CAP_ID_MSIX:
+        /* stubdoms don't support MSI-X so skip it. */
+        return xen_stubdom_enabled();
     }
     return 0;
 }
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#12 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

Quoted 6 lines · Roger Pau Monné
Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

At least missing "Fix PCI passthrough for HVM with stubdomain" series, but that's mostly on Xen side (+ one change how QEMU enable MSI-X in config space).
Some of it can be worked around by enabling permissive mode. Jason, did you had a chance to test it with any MSI-X device?
I'm not aware of anything thing particular that breaks MSI-X but not MSI. Besides much less devices lying around to test MSI-X...

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

Quoted 9 lines · Marek Marczykowski-Górecki
Quoted 6 lines · Roger Pau Monné
Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

Quoted 7 lines

At least missing "Fix PCI passthrough for HVM with stubdomain" series, but that's mostly on Xen side (+ one change how QEMU enable MSI-X in config space).
Some of it can be worked around by enabling permissive mode. Jason, did you had a chance to test it with any MSI-X device?
I'm not aware of anything thing particular that breaks MSI-X but not MSI. Besides much less devices lying around to test MSI-X...

OpenXT and Qubes have used a compile time patch that disabled MSI-X for a long time. The OpenXT patch description doesn't help:
"""
Currently we do not support MSI-X setup for PCI devices passed through.

Although the specification mentions that PCI-e devices might implement only MSI-X there is not a lot of those and mostly none that we have encountered yet. Considering that, we force devices to use MSI by hiding the MSI-X capability. """

To be honest, I didn't question the reasoning and just made the compile-time disabling into a runtime disabling.

I tested with a NEC uPD720200 XHCI controller supporting MSI-X. There was an error related to setting up MSI-X when I failed to pass the "-xen-stubdom" flag. I can pull that log when I get back to the machine. With this patch, MSI-X was hidden in the guest, but dom0 showed MSI-X present but unused.

Marek, is "Use xc_physdev_msi_set_enable for enabling MSI..." the QEMU patch you are refer to? Do you think permissive mode would allow MSI-X to work without that patch? I could test that out.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

Quoted 29 lines · Jason Andryuk
Quoted 9 lines · Marek Marczykowski-Górecki
Quoted 6 lines · Roger Pau Monné
Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

Quoted 7 lines

At least missing "Fix PCI passthrough for HVM with stubdomain" series, but that's mostly on Xen side (+ one change how QEMU enable MSI-X in config space).
Some of it can be worked around by enabling permissive mode. Jason, did you had a chance to test it with any MSI-X device?
I'm not aware of anything thing particular that breaks MSI-X but not MSI. Besides much less devices lying around to test MSI-X...

OpenXT and Qubes have used a compile time patch that disabled MSI-X for a long time. The OpenXT patch description doesn't help:
"""
Currently we do not support MSI-X setup for PCI devices passed through.

Although the specification mentions that PCI-e devices might implement only MSI-X there is not a lot of those and mostly none that we have encountered yet. Considering that, we force devices to use MSI by hiding the MSI-X capability. """

Quoted 8 lines

To be honest, I didn't question the reasoning and just made the compile-time disabling into a runtime disabling.

I tested with a NEC uPD720200 XHCI controller supporting MSI-X. There was an error related to setting up MSI-X when I failed to pass the "-xen-stubdom" flag. I can pull that log when I get back to the machine. With this patch, MSI-X was hidden in the guest, but dom0 showed MSI-X present but unused.

Given that Marek is working on a series to make both MSI and MSI-X work for pci-passthrough and stubdomains I'm not sure how did you even manage to get plain MSI working. Is there something I'm missing?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

Quoted 40 lines · Roger Pau Monné
Quoted 29 lines · Jason Andryuk
Quoted 9 lines · Marek Marczykowski-Górecki
Quoted 6 lines · Roger Pau Monné
Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

Quoted 7 lines

At least missing "Fix PCI passthrough for HVM with stubdomain" series, but that's mostly on Xen side (+ one change how QEMU enable MSI-X in config space).
Some of it can be worked around by enabling permissive mode. Jason, did you had a chance to test it with any MSI-X device?
I'm not aware of anything thing particular that breaks MSI-X but not MSI. Besides much less devices lying around to test MSI-X...

OpenXT and Qubes have used a compile time patch that disabled MSI-X for a long time. The OpenXT patch description doesn't help:
"""
Currently we do not support MSI-X setup for PCI devices passed through.

Although the specification mentions that PCI-e devices might implement only MSI-X there is not a lot of those and mostly none that we have encountered yet. Considering that, we force devices to use MSI by hiding the MSI-X capability. """

Quoted 8 lines

To be honest, I didn't question the reasoning and just made the compile-time disabling into a runtime disabling.

I tested with a NEC uPD720200 XHCI controller supporting MSI-X. There was an error related to setting up MSI-X when I failed to pass the "-xen-stubdom" flag. I can pull that log when I get back to the machine. With this patch, MSI-X was hidden in the guest, but dom0 showed MSI-X present but unused.

Quoted 3 lines

Given that Marek is working on a series to make both MSI and MSI-X work for pci-passthrough and stubdomains I'm not sure how did you even manage to get plain MSI working. Is there something I'm missing?

Marek's series adds a hypercall to enable MSI/MSI-X since they are not allowed by default in PCI passthrough. PCI passthrough also has a permissive mode to allow access to a device's entire PCI configuration space including enabling MSI.

Qubes 4.0 has an out-of-tree patch that whitelists enabling MSI in non-permissive mode - I've tested these patches on Qubes 4.0 with the MSI-X XHCI device where MSI is enabled but not MSI-X. Other NICs with only MSI also work.

OpenXT linux stubdoms use permissive mode PCI passthrough, so the stubdom can program the PCI config space to enable MSI. I've tested the patches there, but none of my OpenXT test systems have MSI-X. MSI devices work properly.

Marek also tested a "vanilla" linux stubdom version in his osstest suite, but that doesn't test passthrough.

If Marek's pending patch series or "permissive" mode will enable MSI/MSI-X, then maybe this patch should just be dropped in favor of those options. I'll test to verify whether MSI-X works with permissive mode.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

On Tue, Mar 12, 2019 at 11:15 AM Jason Andryuk <[email protected]> wrote:
<snip>

Quoted 2 lines

I'll test to verify whether MSI-X works with
permissive mode.

Dropping this patch and enabling permissive mode allowed MSI-X to work.

{"execute": "device_add", "arguments": {"driver":
"xen-pci-passthrough", "id": "xen-pci-pt_0000-03-00.0", "hostaddr":
"0000:00:01.00", "machine_addr": "0000:03:00.0", "permissive": true}} [00:07.0] xen_pt_realize: Assigning real physical device 00:01.0 to devfn 0x38 [00:07.0] xen_pt_register_regions: IO region 0 registered (size=0x00002000 base_addr=0xe1900000 type: 0x4)
[00:07.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000.
[00:07.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0xe1900004, syncing to 0xe1900004. [00:07.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! Emulated=0x0000, host=0x01c3, syncing to 0x0003.
[00:07.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! Emulated=0x0000, host=0x0086, syncing to 0x0080.
[00:07.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! Emulated=0x0000, host=0x8fc0, syncing to 0x8fc0.
[00:07.0] xen_pt_config_reg_init: Offset 0x00aa mismatch! Emulated=0x0000, host=0x0010, syncing to 0x0010.
[00:07.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! Emulated=0x0000, host=0x1011, syncing to 0x1011.
[00:07.0] xen_pt_msix_init: get MSI-X table BAR base 0xe1900000 [00:07.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 [00:07.0] xen_pt_msix_init: mapping physical MSI-X table to 0x7ad6a82e4000 [00:07.0] xen_pt_config_reg_init: Offset 0x0092 mismatch! Emulated=0x0000, host=0x0007, syncing to 0x0007.
[00:07.0] xen_pt_pci_intx: intx=1
[00:07.0] xen_pt_realize: Real physical device 00:01.0 registered successfully {"return": {}}

hw/xen/xen_pt_msi.c:xen_pt_msix_init() calls open(/dev/mem) and mmaps it, so I had to add CONFIG_DEVMEM to the stubdom linux .config. Without /dev/mem:
[00:07.0] xen_pt_msix_init: get MSI-X table BAR base 0xe1900000 [00:07.0] xen_pt_msix_init: Error: Can't open /dev/mem: No such file or directory
[00:07.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init.
Failed to initialize 12/15, type = 0x1, rc: -2
[00:07.0] xen_pt_msi_set_enable: disabling MSI.
free(): double free detected in tcache 2

[user@sys-usb ~]$ sudo lspci -v -s 00:07.0
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
Subsystem: Lenovo Device 21d2
Physical Slot: 7
Flags: bus master, fast devsel, latency 0, IRQ 44 Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
Capabilities: [50] Power Management version 3
Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
Capabilities: [a0] Express Endpoint, MSI 00 Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

Regards,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw

Re: [PATCH 5/6] xen-pt: Hide MSI-X from xen stubdoms

In reply to Jason Andryuk
Quoted 38 lines · Jason Andryuk
Quoted 9 lines · Marek Marczykowski-Górecki
Quoted 6 lines · Roger Pau Monné
Quoted 2 lines · Jason Andryuk

MSI-X is not supported in Xen stubdoms, so it must be disabled. Use the existing xen_pt_hide_dev_cap to hide when running under -xen-stubdom.

I'm afraid this requires some more context. What's the actual issue that prevents MSI-X from working?

Quoted 7 lines

At least missing "Fix PCI passthrough for HVM with stubdomain" series, but that's mostly on Xen side (+ one change how QEMU enable MSI-X in config space).
Some of it can be worked around by enabling permissive mode. Jason, did you had a chance to test it with any MSI-X device?
I'm not aware of anything thing particular that breaks MSI-X but not MSI. Besides much less devices lying around to test MSI-X...

OpenXT and Qubes have used a compile time patch that disabled MSI-X for a long time. The OpenXT patch description doesn't help:
"""
Currently we do not support MSI-X setup for PCI devices passed through.

Although the specification mentions that PCI-e devices might implement only MSI-X there is not a lot of those and mostly none that we have encountered yet. Considering that, we force devices to use MSI by hiding the MSI-X capability. """

To be honest, I didn't question the reasoning and just made the compile-time disabling into a runtime disabling.

I tested with a NEC uPD720200 XHCI controller supporting MSI-X. There was an error related to setting up MSI-X when I failed to pass the "-xen-stubdom" flag. I can pull that log when I get back to the machine. With this patch, MSI-X was hidden in the guest, but dom0 showed MSI-X present but unused.

Quoted 3 lines

Marek, is "Use xc_physdev_msi_set_enable for enabling MSI..." the QEMU patch you are refer to? Do you think permissive mode would allow MSI-X to work without that patch? I could test that out.

Yes, this one. Permissive mode should work around it. There is also another patch about IRQ permission, but I believe you already have it in OpenXT.

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw

PATCH 6/6 xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

In reply to Jason Andryuk

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned. This breaks the memory mapping via xc_domain_memory_mapping since this function is page based and the "offset" is therefore lost.

Without this patch you will see error like this in the stubdom log:

[00:05.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004

QubesOS/qubes-issues#2849

---
 hw/xen/xen_pt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
hw/xen/xen_pt.c +7 -3
index 5539d56c3a..7f680442ee 100644
@@ -449,9 +449,10 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
     /* Register PIO/MMIO BARs */
     for (i = 0; i < PCI_ROM_SLOT; i++) {
         XenHostPCIIORegion *r = &d->io_regions[i];
+        pcibus_t r_size = r->size;
         uint8_t type;
 
-        if (r->base_addr == 0 || r->size == 0) {
+        if (r->base_addr == 0 || r_size == 0) {
             continue;
         }
 
@@ -469,15 +470,18 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
                 type |= PCI_BASE_ADDRESS_MEM_TYPE_64;
             }
             *cmd |= PCI_COMMAND_MEMORY;
+
+            /* Round up to a full page for the hypercall. */
+            r_size = (r_size + XC_PAGE_SIZE - 1) & XC_PAGE_MASK;
         }
 
         memory_region_init_io(&s->bar[i], OBJECT(s), &ops, &s->dev,
-                              "xen-pci-pt-bar", r->size);
+                              "xen-pci-pt-bar", r_size);
         pci_register_bar(&s->dev, i, type, &s->bar[i]);
 
         XEN_PT_LOG(&s->dev, "IO region %i registered (size=0x%08"PRIx64
                    " base_addr=0x%08"PRIx64" type: %#x)\n",
-                   i, r->size, r->base_addr, type);
+                   i, r_size, r->base_addr, type);
     }
 
     /* Register expansion ROM address */
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#20 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be tolerating BARs smaller than that?

Paul

Quoted 54 lines

This breaks the memory mapping via xc_domain_memory_mapping since this function is page based and the "offset" is therefore lost.

Without this patch you will see error like this in the stubdom log:

[00:05.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004

QubesOS/qubes-issues#2849

---
 hw/xen/xen_pt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
hw/xen/xen_pt.c +7 -4
index 5539d56c3a..7f680442ee 100644
@@ -449,9 +449,10 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
     /* Register PIO/MMIO BARs */
     for (i = 0; i < PCI_ROM_SLOT; i++) {
         XenHostPCIIORegion *r = &d->io_regions[i];
+        pcibus_t r_size = r->size;
         uint8_t type;
 
-        if (r->base_addr == 0 || r->size == 0) {
+        if (r->base_addr == 0 || r_size == 0) {
             continue;
         }
 
@@ -469,15 +470,18 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
                 type |= PCI_BASE_ADDRESS_MEM_TYPE_64;
             }
             *cmd |= PCI_COMMAND_MEMORY;
+
+            /* Round up to a full page for the hypercall. */
+            r_size = (r_size + XC_PAGE_SIZE - 1) & XC_PAGE_MASK;
         }
 
         memory_region_init_io(&s->bar[i], OBJECT(s), &ops, &s->dev,
-                              "xen-pci-pt-bar", r->size);
+                              "xen-pci-pt-bar", r_size);
         pci_register_bar(&s->dev, i, type, &s->bar[i]);
 
         XEN_PT_LOG(&s->dev, "IO region %i registered (size=0x%08"PRIx64
                    " base_addr=0x%08"PRIx64" type: %#x)\n",
-                   i, r->size, r->base_addr, type);
+                   i, r_size, r->base_addr, type);
     }
 
     /* Register expansion ROM address */
--
Quoted 1 line

2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 11 lines · Paul Durrant
Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 9 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be tolerating BARs smaller than that?

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Regards,
Jason

Quoted 52 lines
Quoted 52 lines

This breaks the memory mapping via xc_domain_memory_mapping since this function is page based and the "offset" is therefore lost.

Without this patch you will see error like this in the stubdom log:

[00:05.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004

QubesOS/qubes-issues#2849

---
 hw/xen/xen_pt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
hw/xen/xen_pt.c +7 -3
index 5539d56c3a..7f680442ee 100644
@@ -449,9 +449,10 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
     /* Register PIO/MMIO BARs */
     for (i = 0; i < PCI_ROM_SLOT; i++) {
         XenHostPCIIORegion *r = &d->io_regions[i];
+        pcibus_t r_size = r->size;
         uint8_t type;
 
-        if (r->base_addr == 0 || r->size == 0) {
+        if (r->base_addr == 0 || r_size == 0) {
             continue;
         }
 
@@ -469,15 +470,18 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
                 type |= PCI_BASE_ADDRESS_MEM_TYPE_64;
             }
             *cmd |= PCI_COMMAND_MEMORY;
+
+            /* Round up to a full page for the hypercall. */
+            r_size = (r_size + XC_PAGE_SIZE - 1) & XC_PAGE_MASK;
         }
 
         memory_region_init_io(&s->bar[i], OBJECT(s), &ops, &s->dev,
-                              "xen-pci-pt-bar", r->size);
+                              "xen-pci-pt-bar", r_size);
         pci_register_bar(&s->dev, i, type, &s->bar[i]);
 
         XEN_PT_LOG(&s->dev, "IO region %i registered (size=0x%08"PRIx64
                    " base_addr=0x%08"PRIx64" type: %#x)\n",
-                   i, r->size, r->base_addr, type);
+                   i, r_size, r->base_addr, type);
     }
 
Quoted 4 lines
Quoted 3 lines
hw/xen/xen_pt.c +0 -1
     /* Register expansion ROM address */
--

2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#22 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Jason Andryuk:

Quoted 19 lines · Paul Durrant
Quoted 18 lines
Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be tolerating BARs smaller than that?

Quoted 7 lines
Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

Not sure which one it was. Probably the USB controller or the SD host controller. As your example below shows this is not so uncommon.

Quoted 90 lines

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Regards,
Jason

Quoted 52 lines
Quoted 52 lines

This breaks the memory mapping via xc_domain_memory_mapping since this function is page based and the "offset" is therefore lost.

Without this patch you will see error like this in the stubdom log:

[00:05.0] xen_pt_bar_read: Error: Should not read BAR through QEMU. @0x0000000000000004

QubesOS/qubes-issues#2849

---
 hw/xen/xen_pt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
hw/xen/xen_pt.c +7 -3
index 5539d56c3a..7f680442ee 100644
@@ -449,9 +449,10 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
     /* Register PIO/MMIO BARs */
     for (i = 0; i < PCI_ROM_SLOT; i++) {
         XenHostPCIIORegion *r = &d->io_regions[i];
+        pcibus_t r_size = r->size;
         uint8_t type;
 
-        if (r->base_addr == 0 || r->size == 0) {
+        if (r->base_addr == 0 || r_size == 0) {
             continue;
         }
 
@@ -469,15 +470,18 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s, uint16_t *cmd)
                 type |= PCI_BASE_ADDRESS_MEM_TYPE_64;
             }
             *cmd |= PCI_COMMAND_MEMORY;
+
+            /* Round up to a full page for the hypercall. */
+            r_size = (r_size + XC_PAGE_SIZE - 1) & XC_PAGE_MASK;
         }
 
         memory_region_init_io(&s->bar[i], OBJECT(s), &ops, &s->dev,
-                              "xen-pci-pt-bar", r->size);
+                              "xen-pci-pt-bar", r_size);
         pci_register_bar(&s->dev, i, type, &s->bar[i]);
 
         XEN_PT_LOG(&s->dev, "IO region %i registered (size=0x%08"PRIx64
                    " base_addr=0x%08"PRIx64" type: %#x)\n",
-                   i, r->size, r->base_addr, type);
+                   i, r_size, r->base_addr, type);
     }
 
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
hw/xen/xen_pt.c +0 -1
     /* Register expansion ROM address */
--

2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#23 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 18 lines · Simon Gaiser

Jason Andryuk:

Quoted 17 lines · Paul Durrant
Quoted 16 lines
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be tolerating BARs smaller than that?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

Not sure which one it was. Probably the USB controller or the SD host controller. As your example below shows this is not so uncommon.

The minimum is 128 bytes, not 4k - I've just checked the PCIe spec.

Xen/Qemu definitely needs to cope with smaller than 4k if we want to be spec compliant.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#24 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 35 lines

-----Original Message-----
From: Andrew Cooper
Sent: 14 March 2019 19:37
To: Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Paul Durrant <[email protected]>
Cc: Anthony Perard <[email protected]>; [email protected]; Stefano Stabellini <[email protected]>; [email protected]; [email protected]
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 17 lines · Simon Gaiser

Jason Andryuk:

Quoted 16 lines · Paul Durrant
Quoted 15 lines
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even

be tolerating BARs smaller than that?

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

Quoted 7 lines
Quoted 2 lines

Not sure which one it was. Probably the USB controller or the SD host controller. As your example below shows this is not so uncommon.

The minimum is 128 bytes, not 4k - I've just checked the PCIe spec.

Xen/Qemu definitely needs to cope with smaller than 4k if we want to be spec compliant.

Well, we have a problem for pass-through if the BAR is smaller than 4k in that page protection is not going to isolate it. I don't see any other way that to trap and emulate such BARs if we want to pass through those devices at all.

Paul

Quoted 2 lines

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#25 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

In reply to Jason Andryuk

Jason Andryuk:

Quoted 45 lines · Paul Durrant
Quoted 11 lines
Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 9 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be tolerating BARs smaller than that?

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

Quoted 1 line

These examples are all 4K aligned, so this is not an issue on this machine.

I wrote this patch quite some time ago, so I might be misremembering something but IIRC the problem was the address qemu allocates with memory_region_init_io(). I.e. the address as seen from inside the VM, so it does not help if the real address is aligned.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

In reply to Jason Andryuk
Quoted 65 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 11 lines · Paul Durrant
Quoted 11 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 6 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 3 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#27 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them.  That way we can safely pass them through even when they are smaller.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#28 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

(*These print statements don't round up the addresses like xen_pt_region_update currently does when calling xc_domain_memory_mapping. I think that rounding is incorrect since address 0xf2028400 in pfn 0xf2028 would actually round to 0xf2029.)

I haven't tracked down all the following, but the guest controls where the BARs get located, right? So rounding up all BARs to at least 1 page prevent the guest locating mutliple BARs in the same page. The other approach would be to make the Guest OS provide a minimum page size alignment to each. Since we're dealing with HVMs that aren't necessarily aware of Xen, making QEMU enforce the minimum size avoids BARs sharing a page. But then is it an issue if QEMU advertises a bar size greater than the device's real size?

Regards,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#29 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 102 lines · Jason Andryuk
Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

Quoted 8 lines

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Thanks, Roger.

---8<---

tools/firmware/hvmloader/pci.c +4 -0
index 0b708bf578..c433b34cd6 100644
@@ -489,6 +489,10 @@ void pci_setup(void)
 
         resource->base = base;
 
+        if ( (bar_data & PCI_BASE_ADDRESS_SPACE) ==
+             PCI_BASE_ADDRESS_SPACE_MEMORY )
+            resource->base = ROUNDUP(resource->base, PAGE_SIZE);
+
         pci_writel(devfn, bar_reg, bar_data);
         if (using_64bar)
             pci_writel(devfn, bar_reg + 4, bar_data_upper);
tools/firmware/hvmloader/util.h +2 -0
index 7bca6418d2..b5554b5844 100644
@@ -51,6 +51,8 @@ void __bug(char *file, int line) __attribute__((noreturn));
 #define MB(mb) (mb##ULL << 20)
 #define GB(gb) (gb##ULL << 30)
 
+#define ROUNDUP(x, a) (((x) + (a) - 1) & ~((a) - 1))
+
 static inline int test_bit(unsigned int b, const void *p)
 {
     return !!(((const uint8_t *)p)[b>>3] & (1u<<(b&7)));

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#30 lore raw

Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 113 lines · Roger Pau Monné
Quoted 102 lines · Jason Andryuk
Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

Quoted 8 lines

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

Quoted 8 lines

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Thanks, Roger.

Hi, Roger.

I've minimally tested this. Yes, this patch seems to place small BARs into separate pages. The linux stubdom and QEMU then use the spacing as provided by hvmloader.

Thanks,
Jason

Quoted 22 lines

---8<---

tools/firmware/hvmloader/pci.c +4 -0
index 0b708bf578..c433b34cd6 100644
@@ -489,6 +489,10 @@ void pci_setup(void)
 
         resource->base = base;
 
+        if ( (bar_data & PCI_BASE_ADDRESS_SPACE) ==
+             PCI_BASE_ADDRESS_SPACE_MEMORY )
+            resource->base = ROUNDUP(resource->base, PAGE_SIZE);
+
         pci_writel(devfn, bar_reg, bar_data);
         if (using_64bar)
             pci_writel(devfn, bar_reg + 4, bar_data_upper);
tools/firmware/hvmloader/util.h +0 -0
index 7bca6418d2..b5554b5844 100644

tools/firmware/hvmloader/util.h void __bug(char *file, int line) __attribute__((noreturn)); · line 58

Quoted 8 lines
tools/firmware/hvmloader/util.h +2 -0
@@ -51,6 +51,8 @@ void __bug(char *file, int line) __attribute__((noreturn));
 #define MB(mb) (mb##ULL << 20)
 #define GB(gb) (gb##ULL << 30)
 
+#define ROUNDUP(x, a) (((x) + (a) - 1) & ~((a) - 1))
+
 static inline int test_bit(unsigned int b, const void *p)
 {
     return !!(((const uint8_t *)p)[b>>3] & (1u<<(b&7)));

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#31 lore raw

Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 122 lines · Jason Andryuk
Quoted 120 lines · Roger Pau Monné
Quoted 102 lines · Jason Andryuk
Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

Quoted 8 lines

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Quoted 7 lines
Quoted 1 line

Thanks, Roger.

Hi, Roger.

I've minimally tested this. Yes, this patch seems to place small BARs into separate pages. The linux stubdom and QEMU then use the spacing as provided by hvmloader.

Roger,

Would you mind submitting this patch to Xen?

Thanks,
Jason

Quoted 26 lines
Quoted 26 lines

---8<---

tools/firmware/hvmloader/pci.c +4 -0
index 0b708bf578..c433b34cd6 100644
@@ -489,6 +489,10 @@ void pci_setup(void)
 
         resource->base = base;
 
+        if ( (bar_data & PCI_BASE_ADDRESS_SPACE) ==
+             PCI_BASE_ADDRESS_SPACE_MEMORY )
+            resource->base = ROUNDUP(resource->base, PAGE_SIZE);
+
         pci_writel(devfn, bar_reg, bar_data);
         if (using_64bar)
             pci_writel(devfn, bar_reg + 4, bar_data_upper);
tools/firmware/hvmloader/util.h +0 -0
index 7bca6418d2..b5554b5844 100644
@@ -51,6 +51,8 @@ void __bug(char *file, int line) __attribute__((noreturn));
 #define MB(mb) (mb##ULL << 20)
 #define GB(gb) (gb##ULL << 30)
 
Quoted 6 lines

tools/firmware/hvmloader/util.h void __bug(char *file, int line) __attribute__((noreturn)); · line 58

Quoted 6 lines
tools/firmware/hvmloader/util.h +2 -0
+#define ROUNDUP(x, a) (((x) + (a) - 1) & ~((a) - 1))
+
 static inline int test_bit(unsigned int b, const void *p)
 {
     return !!(((const uint8_t *)p)[b>>3] & (1u<<(b&7)));

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#32 lore raw

Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 127 lines · Jason Andryuk
Quoted 126 lines · Jason Andryuk
Quoted 113 lines · Roger Pau Monné
Quoted 102 lines · Jason Andryuk
Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

Quoted 8 lines

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

Quoted 8 lines

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Thanks, Roger.

Hi, Roger.

Quoted 7 lines
Quoted 3 lines

I've minimally tested this. Yes, this patch seems to place small BARs into separate pages. The linux stubdom and QEMU then use the spacing as provided by hvmloader.

Roger,

Would you mind submitting this patch to Xen?

Hm, I'm half minded regarding this patch. It feels more like a bandaid than a proper solution. Mapping BARs not multiple of page-sizes is dangerous because AFAIK there's no entity that asserts there isn't any other BAR from a different device on the same page, and hence you might end up mapping some MMIO region from another device inadvertently.

Anyway, I can formally submit the patch since it's no worse than what's currently done, but I would clearly state this is not safe in it's current state.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#33 lore raw

Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 135 lines · Roger Pau Monné
Quoted 133 lines · Jason Andryuk
Quoted 122 lines · Jason Andryuk
Quoted 120 lines · Roger Pau Monné
Quoted 102 lines · Jason Andryuk
Quoted 71 lines · Andrew Cooper
Quoted 61 lines · Paul Durrant
Quoted 61 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 14 March 2019 18:16
To: Paul Durrant <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Simon Gaiser <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>
Subject: Re: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 10 lines · Paul Durrant
Quoted 10 lines

-----Original Message-----
From: Jason Andryuk [mailto:[email protected]]
Sent: 11 March 2019 18:02
To: [email protected]
Cc: [email protected]; [email protected]; Simon Gaiser <[email protected]>; Jason Andryuk <[email protected]>; Stefano Stabellini <[email protected]>; Anthony Perard <[email protected]>; Paul Durrant <[email protected]>
Subject: [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 5 lines
Quoted 4 lines

From: Simon Gaiser <[email protected]>

If a pci memory region has a size < XEN_PAGE_SIZE it can get located at an address which is not page aligned.

IIRC the PCI spec says that the minimum memory region size should be at least 4k. Should we even be

tolerating BARs smaller than that?

Quoted 2 lines

Paul

Hi, Paul.

Simon found this, so it affects a real device. Simon, do you recall which device was affected?

I think BARs only need to be power-of-two size and aligned, and 4k is not a minimum. 16bytes may be a minimum, but I don't know what the spec says.

On an Ivy Bridge system, here are some of the devices with BARs smaller than 4K:
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
Memory at d0735000 (64-bit, non-prefetchable) [disabled] [size=16]
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) (prog-if 20 [EHCI])
Memory at d0739000 (32-bit, non-prefetchable) [disabled] [size=1K]
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
Memory at d0734000 (64-bit, non-prefetchable) [disabled] [size=256]
02:00.0 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
Memory at d0503000 (32-bit, non-prefetchable) [disabled] [size=256]

These examples are all 4K aligned, so this is not an issue on this machine.

Reviewing the code, I'm now wondering if the following in
hw/xen/xen_pt.c:xen_pt_region_update is wrong: rc =
xc_domain_memory_mapping(xen_xc, xen_domid,

                                     XEN_PFN(guest_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(machine_addr + XC_PAGE_SIZE - 1),
                                     XEN_PFN(size + XC_PAGE_SIZE - 1),
                                     op);
Quoted 7 lines
Quoted 6 lines

If a bar of size 0x100 is at 0xd0500800, then the machine_addr passed in would be 0xd0501000 which is past the actual location. Should the call arguments just be XEN_PFN(guest_addr) & XEN_PFN(machine_addr)?

BARs smaller than a page would also be a problem if BARs for different devices shared the same page.

Exactly. We cannot pass them through with any degree of safety (not that passthrough of an arbitrary device is a particularly safe thing to do anyway). The xen-pt code would instead need to trap those BARs and perform the accesses to the real BAR itself. Ultimately though I think we should be retiring the xen-pt code in favour of a standalone emulator.

Quoted 8 lines

It doesn't matter if the BAR is smaller than 4k, if there are holes next to it.

Do we know what the case is in practice for these USB controllers?

If the worst comes to the worst, we can re-enumerate the PCI bus to ensure that all bars smaller than 4k still have 4k alignment between them. That way we can safely pass them through even when they are smaller.

Andrew, thanks for checking the spec on the minimum BAR size.

Dropping the Round PCI region patch from QMEU, the guest HVM will have:

00:06.0 SD Host controller: Ricoh Co Ltd PCIe SDXC/MMC Host Controller (rev 07)

    Memory at f2028800 (32-bit, non-prefetchable) [size=256]
00:07.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host
Controller (rev 04) (prog-if 30 [XHCI])
    Memory at f2024000 (64-bit, non-prefetchable) [size=8K]
00:08.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028000 (32-bit, non-prefetchable) [size=1K]
00:09.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
    Memory at f2028400 (32-bit, non-prefetchable) [size=1K]

00:09.0, 00:08.0 & 00:06.0 all share the same page. Only 00:08.0 is working. With some added debugging output, you'll see that the same page* is used for three of the BARs.

Quoted 8 lines

[00:06.0] mapping guest_addr 0xf2028800 gfn 0xf2028 to maddr 0xe1a30000 mfn 0xe1a30
[00:07.0] mapping guest_addr 0xf2024000 gfn 0xf2024 to maddr 0xe0800000 mfn 0xe0800
[00:09.0] mapping guest_addr 0xf2028400 gfn 0xf2028 to maddr 0xe1900000 mfn 0xe1900
[00:08.0] mapping guest_addr 0xf2028000 gfn 0xf2028 to maddr 0xe1a2f000 mfn 0xe1a2f

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Quoted 7 lines
Quoted 1 line

Thanks, Roger.

Hi, Roger.

I've minimally tested this. Yes, this patch seems to place small BARs into separate pages. The linux stubdom and QEMU then use the spacing as provided by hvmloader.

Roger,

Quoted 8 lines
Quoted 1 line

Would you mind submitting this patch to Xen?

Hm, I'm half minded regarding this patch. It feels more like a bandaid than a proper solution. Mapping BARs not multiple of page-sizes is dangerous because AFAIK there's no entity that asserts there isn't any other BAR from a different device on the same page, and hence you might end up mapping some MMIO region from another device inadvertently.

We have the guest, linux stubdom with qemu, & dom0. Are you concerned that all of them need a minimum of page alignment?

Linux PCI subsytem has an option resource_alignment that can be applied to either a single device or all devices. Booting with pci=resource_aligment=4096 will align each device to a page. Do you think pciback should force resource_alignment=4096 for dom0? Are there other MMIO ranges to be concerned about adjacent to BARs?

On my one test machine with a BAR smaller than 4096, the firmware already sets an alignment of 4096. Linux dom0 seems to keep the firmware BAR alignment by default.

Quoted 3 lines

Anyway, I can formally submit the patch since it's no worse than what's currently done, but I would clearly state this is not safe in it's current state.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#34 lore raw

Re: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 33 lines · Jason Andryuk
Quoted 23 lines · Roger Pau Monné
Quoted 21 lines · Jason Andryuk
Quoted 10 lines · Jason Andryuk
Quoted 8 lines · Roger Pau Monné

The patch below should prevent hvmloader from placing multiple BARs on the same page, could you give it a try?

Note that this is not going to prevent the guest from moving those BARs around and place them in the same page, thus breaking the initial placement done by hvmloader.

Quoted 7 lines
Quoted 1 line

Thanks, Roger.

Hi, Roger.

I've minimally tested this. Yes, this patch seems to place small BARs into separate pages. The linux stubdom and QEMU then use the spacing as provided by hvmloader.

Roger,

Quoted 8 lines
Quoted 1 line

Would you mind submitting this patch to Xen?

Hm, I'm half minded regarding this patch. It feels more like a bandaid than a proper solution. Mapping BARs not multiple of page-sizes is dangerous because AFAIK there's no entity that asserts there isn't any other BAR from a different device on the same page, and hence you might end up mapping some MMIO region from another device inadvertently.

Quoted 2 lines

We have the guest, linux stubdom with qemu, & dom0. Are you concerned that all of them need a minimum of page alignment?

No, not really. The hardware domain (dom0 in normal deployments) should be the one that makes sure there are no BARs sharing physical pages.

Quoted 4 lines

Linux PCI subsytem has an option resource_alignment that can be applied to either a single device or all devices. Booting with pci=resource_aligment=4096 will align each device to a page. Do you think pciback should force resource_alignment=4096 for dom0?

Ideally Xen should keep track of the BARs position and size and refuse to passthrough devices that have BARs sharing a page with other devices BARs.

Quoted 2 lines

Are
there other MMIO ranges to be concerned about adjacent to BARs?

IIRC you can have two BARs of different devices in the same 4K page, BARs are only aligned to it's size, so BARs smaller than 4K are not required to be page aligned.

Quoted 3 lines

On my one test machine with a BAR smaller than 4096, the firmware already sets an alignment of 4096. Linux dom0 seems to keep the firmware BAR alignment by default.

The PCI spec recommend BARs to be sized to a multiple of a page size, but sadly that's not a mandatory requirement.

Will submit the patch now, thanks for the ping, I completely forgot about this TBH.

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#35 lore raw

RE: [Xen-devel] [PATCH 6/6] xen-pt: Round pci regions sizes to XEN_PAGE_SIZE

Quoted 7 lines

-----Original Message-----

Quoted 4 lines

Linux PCI subsytem has an option resource_alignment that can be applied to either a single device or all devices. Booting with pci=resource_aligment=4096 will align each device to a page. Do you think pciback should force resource_alignment=4096 for dom0?

That sounds like a good idea.

Quoted 10 lines

Ideally Xen should keep track of the BARs position and size and refuse to passthrough devices that have BARs sharing a page with other devices BARs.

Quoted 2 lines

Are
there other MMIO ranges to be concerned about adjacent to BARs?

IIRC you can have two BARs of different devices in the same 4K page, BARs are only aligned to it's size, so BARs smaller than 4K are not required to be page aligned.

If we had a notion of assignment groups for this, as well as devices sharing requester id, then Xen would not need to refuse pass-through, it would just require that all devices sharing the page were passed through as a unit.

Paul

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel