Argo: hypervisor-mediated interdomain communication
cover Argo: hypervisor-mediated interdomain communication
This patch series implements the Argo hypervisor-mediated interdomain communication mechanism as an experimental feature for incorporation into the Xen hypervisor.
Relevant to the ARM deadline for inclusion in the Xen 4.12 release, there are very few and only minor ARM-specific changes in this series.
This is derived from the v4v work of XenClient, retained in the OpenXT Project and developed further by Bromium in uxen. It has benefitted from and been improved by previous rounds of review in this Xen community, and is the combined work of a series of Xen engineers that have preceeded the efforts of the current submission.
The motivation for this feature continues to be that a non-networking, non-shared memory, hypervisor-mediated communication mechanism between domains concurrently executing on the same hypervisor has attractive properties for use cases that value strong mechanisms for policy enforcement and isolation.
In this series, Argo is made optional for inclusion via Kconfig. When included, it defaults to disabled and requires a Xen boot parameter to enable it. It has XSM integration for access control over domain-to-domain communication, and a second boot parameter governs the level of permissiveness over shared communication rings when using the non-XSM/Flask default.
Design documentation can be found on the Xen wiki, at:
https://wiki.xenproject.org/wiki/Argo:_Hypervisor-Mediated_Exchange_(HMX)_for_Xen
and it will be updated to correspond to the submission here in the coming days.
Argo has recently been discussed on the Xen x86 Community Call, minutes:
https://docs.google.com/document/d/1VUPdWwd1raDOPhjReVVkmb6YoQB3X5oU12E4ExjO1n0/edit#heading=h.mz1wjb9vekjn
In (very) short, Argo is implemented by a new hypercall with five operations:
* register ring
* unregister ring
* sendv
* notify
* get config
Ring registration is performed by a domain to provide a region of memory for receiving messages from one or many other domains. A domain can issue a send operation to send messages to another domain's ring. The data is transferred synchronously by the hypervisor. There is no shared memory between domains, allowing for increased confidence by the domain that the memory accesses in the registered ring conform to the expected protocol. The hypervisor is able to enforce access control policy over the communication.
== Naming
v4v lives on in the Bromium uxen codebase. It is not the same implementation as this, it doesn't have quite the same properties and I don't expect the two to converge (though I do hope continued cross-pollination will happen). Given that, this feature needs to be describable with a different name.
It's also a complex enough system, with design details that matter and affect important properties of it, that a generic term (eg. "message rings") is not sufficient.
Xen's name originates from Xenia, the ancient Greek concept of hospitality. Argo is the ship from Greek mythology that provided secure transport for the mission to obtain the Golden Fleece. This feature aims to provide secure transport.
With this series, I'm proposing that this work shall use the name: argo. (short, pronouncable, unique within Xen's context so acceptable in code and material artefacts will be discoverable with a search engine.)
Valued feedback was given in review prior to this posting about whether naming aspects of the implementation 'argo' was ok. I took this seriously, and spent significant time looking at how to reduce the level of argo-ness in this implementation. This version does incorporate changes from that effort but in general, my view is that use of the name in the code assists the clarity of it, so much of it has been retained.
The term "Hypervisor-Mediated data eXchange (HMX)" was introduced in a presentation at the Platform Security Summit 2018, to describe the general, hypervisor-agnostic, capability of data transfer between domains performed by the hypervisor. It is viewable at:
https://www.platformsecuritysummit.com/2018/speaker/clark/
Argo conforms to HMX as described, as does Hyper-V's message-sending primitive.
== Future items
The Linux device driver used to test this software is derived from the OpenXT v4v Linux device driver, available at:
https://github.com/OpenXT/v4v
The Argo implementation is not yet ready to publish (focus has been on the hypervisor code to this point). A Linux device driver suitable for inclusion in Xen will be submitted for a future Xen release and incorporation into OpenXT.
This submission does not include a firewall for constraining domain-to-domain communication. The XSM hooks added currently provide granularity of control at domain-to-domain level. We intend to extend this to provide finer-grained access control in a future submission, but the current implementation should be sufficient to provide sufficient isolation for some use cases.
Communication between VMs at different levels of nesting in a multi-hypervisor system is of strong interest and will inform near-term enhancements.
Optimization of notification delivery to VMs is a known area for improvement.
* uxen's v4v uses an edge-triggered interrupt to reduce VMEXIT load.
* delivering extended notification data via a dedicated registered ring will allow a guest to avoid a search to identify notification causes.
Additional items will be noted on the Xen wiki.
== Credits
Contributors to the design and implementation of this software include:
James McKenzie, Jean Guyader, Ross Philipson, Christopher Clark
with the support of the OpenXT Project.
Thanks are due for the helpful reviews of earlier revisions by Tim Deegan, Jan Beulich, Ian Campbell and Eric Chanudet.
Christopher Clark (25):
xen/evtchn: expose evtchn_bind_ipi_vcpu0_domain for use within Xen
argo: Introduce the Kconfig option to govern inclusion of Argo
argo: introduce the argo_message_op hypercall boilerplate
argo: define argo_dprintk for subsystem debugging
argo: Add initial argo_init and argo_destroy
argo: Xen command line parameter 'argo': bool to enable/disable
xen: add errno-returning functions for copy to and from guest
xen: define XEN_GUEST_HANDLE_NULL as null XEN_GUEST_HANDLE
errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
arm: introduce guest_handle_for_field()
xsm, argo: XSM control for argo register operation, argo_mac bootparam xsm, argo: XSM control for argo message send operation
argo: implement the register op
argo: implement the unregister op
argo: implement the sendv op
argo: implement the notify op
xsm, argo: XSM control for any access to argo by a domain
argo: limit the max number of rings that a domain may register.
argo: limit the max number of notify requests in a single operation. argo, xsm: notify: don't describe rings that cannot be sent to
argo: add array_index_nospec to guard the result of the hash func xen/evtchn: expose send_guest_global_virq for use within Xen
argo: signal x86 HVM and ARM via VIRQ
argo: unmap rings on suspend and send signal to ring-owners on resume
argo: implement the get_config op to query notification config
xen/arch/x86/guest/hypercall_page.S | 2 +- xen/arch/x86/hvm/hypercall.c | 3 + xen/arch/x86/hypercall.c | 3 + xen/arch/x86/pv/hypercall.c | 3 + xen/common/Kconfig | 20 + xen/common/Makefile | 1 + xen/common/argo.c | 1960 +++++++++++++++++++++++++++++++++ xen/common/domain.c | 24 + xen/common/event_channel.c | 37 +- xen/include/asm-arm/guest_access.h | 30 + xen/include/asm-x86/guest_access.h | 31 + xen/include/public/argo.h | 280 +++++ xen/include/public/errno.h | 2 + xen/include/public/xen.h | 6 +- xen/include/xen/argo.h | 32 + xen/include/xen/event.h | 10 + xen/include/xen/guest_access.h | 3 + xen/include/xen/hypercall.h | 9 + xen/include/xen/sched.h | 7 + xen/include/xsm/dummy.h | 25 + xen/include/xsm/xsm.h | 29 + xen/xsm/dummy.c | 6 + xen/xsm/flask/hooks.c | 33 + xen/xsm/flask/policy/access_vectors | 16 + xen/xsm/flask/policy/security_classes | 1 + 25 files changed, 2563 insertions(+), 10 deletions(-) create mode 100644 xen/common/argo.c create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.h
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/25 xen/evtchn: expose evtchn_bind_ipi_vcpu0_domain for use within Xen
Allocates an IPI-bound event channel on vcpu0 for specified domain.
Is able to bypass the existence check on vcpu number since vcpu 0 should always exist. Bypass is required at the point of use by Argo.
--- xen/common/event_channel.c | 35 +++++++++++++++++++++++++++++------ xen/include/xen/event.h | 3 +++ 2 files changed, 32 insertions(+), 6 deletions(-)
xen/common/event_channel.c +29 -6
@@ -411,17 +411,12 @@ int evtchn_bind_virq(evtchn_bind_virq_t *bind, evtchn_port_t port) } -static long evtchn_bind_ipi (evtchn_bind_ipi_t *bind) +static long evtchn_bind_ipi_domain (struct domain *d,evtchn_bind_ipi_t *bind) {struct evtchn *chn; -struct domain *d = current->domain;int port, vcpu = bind->vcpu;long rc =0 ; -if ( (vcpu <0 ) || (vcpu >= d->max_vcpus) || - (d->vcpu[vcpu] ==NULL ) ) -return -ENOENT ; -spin_lock (&d->event_lock);if ( (port =get_free_port (d)) <0 )
@@ -446,6 +441,34 @@ static long evtchn_bind_ipi(evtchn_bind_ipi_t *bind) } +static long evtchn_bind_ipi (evtchn_bind_ipi_t *bind) +{ +struct domain *d = current->domain; +int vcpu = bind->vcpu; + +if ( (vcpu <0 ) || (vcpu >= d->max_vcpus) || + (d->vcpu[vcpu] ==NULL ) ) +return -ENOENT ; + +return evtchn_bind_ipi_domain (d, bind); +} + +long evtchn_bind_ipi_vcpu0_domain (struct domain *d,evtchn_port_t *out_port) +{ +evtchn_bind_ipi_t bind_ipi; +long rc; + + bind_ipi.vcpu =0 ; + + rc =evtchn_bind_ipi_domain (d, &bind_ipi); + +if ( !rc ) + *out_port = bind_ipi.port; + +return rc; +} + +static void link_pirq_port (int port,struct evtchn *chn,struct vcpu *v) { chn->u.pirq.prev_port =0 ;
xen/include/xen/event.h +3 -0
@@ -86,6 +86,9 @@ void notify_via_xen_event_channel(struct domain *ld, int lport);/* Inject an event channel notification into the guest */ void arch_evtchn_inject (struct vcpu *v); +/* Allocate an IPI event channel on vcpu0 for the specified domain */ +long evtchn_bind_ipi_vcpu0_domain (struct domain *d,evtchn_port_t *out_port); +/* * Internal event channel object storage. *
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 01/25] xen/evtchn: expose evtchn_bind_ipi_vcpu0_domain for use within Xen
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
Allocates an IPI-bound event channel on vcpu0 for specified domain.
Please can such changes to general code be done at the point where they're needed?
Quoted 2 lines
Is able to bypass the existence check on vcpu number since vcpu 0 should always exist. Bypass is required at the point of use by Argo.
"Should" is not a sufficient criteria. And you leave open why such a bypass may be needed.
As an aside, I question anyway any new interface special casing vCPU 0.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 01/25] xen/evtchn: expose evtchn_bind_ipi_vcpu0_domain for use within Xen
Quoted 7 lines · Jan Beulich
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
Allocates an IPI-bound event channel on vcpu0 for specified domain.
Please can such changes to general code be done at the point where they're needed?
Quoted 8 lines
Quoted 2 lines
Is able to bypass the existence check on vcpu number since vcpu 0 should always exist. Bypass is required at the point of use by Argo.
"Should" is not a sufficient criteria. And you leave open why such a bypass may be needed.
As an aside, I question anyway any new interface special casing vCPU 0.
Ack. I'll take a look at how this initialization interacts with domains with no VCPUs. Agree that the vcpu0 special-case deserves questioning. Thanks.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/25 argo: Introduce the Kconfig option to govern inclusion of Argo
In reply to Christopher Clark--- xen/common/Kconfig | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
xen/common/Kconfig +20 -0
@@ -200,6 +200,26 @@ config LATE_HWDOM If unsure, say N. +config ARGO + bool "Argo: hypervisor-mediated interdomain communication" + default y + ---help--- + Enables a hypercall for domains to ask the hypervisor to perform + data transfer of messages between domains. + + This allows communication channels to be established that do not + require any shared memory between domains; the hypervisor is the + entity that each domain interacts with. The hypervisor is able to + enforce Mandatory Access Control policy over the communication. + + If XSM_FLASK is enabled, XSM policy can govern which domains may + communicate via the Argo system. + + This feature does nothing if the "argo" boot parameter is not present. + Argo is disabled at runtime by default. + + If unsure, say Y. + menu "Schedulers" visible if EXPERT = "y"
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 02/25] argo: Introduce the Kconfig option to govern inclusion of Argo
xen/common/Kconfig config LATE_HWDOM · line 205
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/Kconfig +3
-0@@ -200,6 +200,26 @@ config LATE_HWDOM If unsure, say N. +config ARGO + bool "Argo: hypervisor-mediated interdomain communication" + default y
Until our policy changes as to wider configurability, options not depending on EXPERT should be accompanied by a reason. I also don't think that we want this to default to enabled from the very beginning. Finally please correct indentation.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 02/25] argo: Introduce the Kconfig option to govern inclusion of Argo
Quoted 8 lines · Jan Beulich
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/Kconfig +0
-0@@ -200,6 +200,26 @@ config LATE_HWDOM If unsure, say N.
Quoted 8 lines
xen/common/Kconfig config LATE_HWDOM · line 205
Quoted 3 lines
xen/common/Kconfig +3
-0+config ARGO + bool "Argo: hypervisor-mediated interdomain communication" + default yUntil our policy changes as to wider configurability, options not depending on EXPERT should be accompanied by a reason. I also don't think that we want this to default to enabled from the very beginning. Finally please correct indentation.
ack: will toggle, add dependency, describe and re-indent.
thanks.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 3/25 argo: introduce the argo_message_op hypercall boilerplate
In reply to Christopher ClarkPresence is gated upon CONFIG_ARGO.
Registers the hypercall previously reserved for this. Takes 5 arguments, does nothing and returns -ENOSYS.
Will be avoiding a compat ABI by using fixed-size types in hypercall ops.
--- xen/arch/x86/guest/hypercall_page.S | 2 +- xen/arch/x86/hvm/hypercall.c | 3 +++ xen/arch/x86/hypercall.c | 3 +++ xen/arch/x86/pv/hypercall.c | 3 +++ xen/common/Makefile | 1 + xen/common/argo.c | 28 ++++++++++++++++++++++++++++ xen/include/public/xen.h | 2 +- xen/include/xen/hypercall.h | 9 +++++++++ 8 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 xen/common/argo.c
xen/arch/x86/guest/hypercall_page.S +1 -1
@@ -59,7 +59,7 @@ DECLARE_HYPERCALL(sysctl) DECLARE_HYPERCALL(domctl) DECLARE_HYPERCALL(kexec_op) DECLARE_HYPERCALL(tmem_op) -DECLARE_HYPERCALL(xc_reserved_op) +DECLARE_HYPERCALL(argo_message_op) DECLARE_HYPERCALL(xenpmu_op) DECLARE_HYPERCALL(arch_0)
xen/arch/x86/hvm/hypercall.c +3 -0
@@ -134,6 +134,9 @@ static const hypercall_table_t hvm_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO + HYPERCALL(argo_message_op), +#endif COMPAT_CALL(platform_op),#ifdef CONFIG_PV COMPAT_CALL(mmuext_op),
xen/arch/x86/hypercall.c +3 -0
@@ -64,6 +64,9 @@ const hypercall_args_t hypercall_args_table[NR_hypercalls] =ARGS (domctl,1 ),ARGS (kexec_op,2 ),ARGS (tmem_op,1 ), +#ifdef CONFIG_ARGO +ARGS (argo_message_op,5 ), +#endifARGS (xenpmu_op,2 ), #ifdef CONFIG_HVM ARGS(hvm_op, 2),
xen/arch/x86/pv/hypercall.c +3 -0
@@ -77,6 +77,9 @@ const hypercall_table_t pv_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO + HYPERCALL(argo_message_op), +#endif HYPERCALL(xenpmu_op),#ifdef CONFIG_HVM HYPERCALL(hvm_op),
xen/common/Makefile +1 -0
@@ -1,3 +1,4 @@ +obj-$(CONFIG_ARGO) += argo.o obj-y += bitmap.o obj-y += bsearch.o obj-$(CONFIG_CORE_PARKING) += core_parking.o
xen/common/argo.c new file +28 -0
@@ -0,0 +1,28 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#include <xen/errno.h> +#include <xen/guest_access.h> + +long +do_argo_message_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2, +uint32_t arg3,uint32_t arg4) +{ +return -ENOSYS ; +}
xen/include/public/xen.h +1 -1
@@ -118,7 +118,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);#define __HYPERVISOR_domctl 36#define __HYPERVISOR_kexec_op 37#define __HYPERVISOR_tmem_op 38 -#define __HYPERVISOR_xc_reserved_op 39/* reserved for XenClient */ +#define __HYPERVISOR_argo_message_op 39#define __HYPERVISOR_xenpmu_op 40#define __HYPERVISOR_dm_op 41
xen/include/xen/hypercall.h +9 -0
@@ -136,6 +136,15 @@ do_tmem_op( XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);#endif +#ifdef CONFIG_ARGO +extern long do_argo_message_op ( +int cmd, +XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2, +uint32_t arg3, +uint32_t arg4); +#endif +extern long do_xenoprof_op (int op,XEN_GUEST_HANDLE_PARAM (void ) arg);
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 03/25] argo: introduce the argo_message_op hypercall boilerplate
Quoted 17 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monne <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 03/25] argo: introduce theargo_message_ophypercall boilerplate
Quoted 6 lines
Presence is gated upon
CONFIG_ARGO.Registers the hypercall previously reserved for this. Takes 5 arguments, does nothing and returns
-ENOSYS.Will be avoiding a compat ABI by using fixed-size types in hypercall ops.
You appear to be using handles, so will you not need compat code to deal with those?
Paul
Quoted 151 lines
Signed-off-by Christopher Clark <[email protected]>--- xen/arch/x86/guest/hypercall_page.S | 2 +- xen/arch/x86/hvm/hypercall.c | 3 +++ xen/arch/x86/hypercall.c | 3 +++ xen/arch/x86/pv/hypercall.c | 3 +++ xen/common/Makefile | 1 + xen/common/argo.c | 28 ++++++++++++++++++++++++++++ xen/include/public/xen.h | 2 +- xen/include/xen/hypercall.h | 9 +++++++++ 8 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 xen/common/argo.cxen/arch/x86/guest/hypercall_page.S +1
-1@@ -59,7 +59,7 @@ DECLARE_HYPERCALL(sysctl) DECLARE_HYPERCALL(domctl) DECLARE_HYPERCALL(kexec_op) DECLARE_HYPERCALL(tmem_op) -DECLARE_HYPERCALL(xc_reserved_op) +DECLARE_HYPERCALL(argo_message_op) DECLARE_HYPERCALL(xenpmu_op) DECLARE_HYPERCALL(arch_0)xen/arch/x86/hvm/hypercall.c +3
-0@@ -134,6 +134,9 @@ static const hypercall_table_t hvm_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO +HYPERCALL (argo_message_op), +#endifCOMPAT_CALL (platform_op), #ifdef CONFIG_PV
COMPAT_CALL(mmuext_op),xen/arch/x86/hypercall.c +0
-0@@ -64,6 +64,9 @@ const hypercall_args_thypercall_args_table[NR_hypercalls] =
ARGS(domctl, 1), ARGS(kexec_op, 2), ARGS(tmem_op, 1),xen/arch/x86/hypercall.c +3
-0+#ifdef CONFIG_ARGO + ARGS(argo_message_op, 5), +#endif ARGS(xenpmu_op, 2),#ifdef CONFIG_HVM ARGS(hvm_op, 2),xen/arch/x86/pv/hypercall.c +3
-0@@ -77,6 +77,9 @@ const hypercall_table_t pv_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO + HYPERCALL(argo_message_op), +#endif HYPERCALL(xenpmu_op),#ifdef CONFIG_HVM HYPERCALL(hvm_op),xen/common/Makefile +1
-0@@ -1,3 +1,4 @@ +obj-$(CONFIG_ARGO) += argo.o obj-y += bitmap.o obj-y += bsearch.o obj-$(CONFIG_CORE_PARKING) += core_parking.oxen/common/argo.c new file +1
-0@@ -0,0 +1,28 @@ +/************************************************************************ ******
+ * Argo : Hypervisor-Mediated data eXchange
+ *
+ * Derived from v4v, the version 2 of v2v.
+ *
+ * Copyright (c) 2010, Citrix Systems
+ * Copyright (c) 2018, BAE Systems
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */xen/common/argo.c new file +11
-0+ +#include <xen/errno.h> +#include <xen/guest_access.h> + +long +do_argo_message_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2, +uint32_t arg3,uint32_t arg4) +{ +return -ENOSYS ; +}xen/include/public/xen.h +1
-1@@ -118,7 +118,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);#define __HYPERVISOR_domctl 36#define __HYPERVISOR_kexec_op 37#define __HYPERVISOR_tmem_op 38 -#define __HYPERVISOR_xc_reserved_op 39/* reserved for XenClient */ +#define __HYPERVISOR_argo_message_op 39#define __HYPERVISOR_xenpmu_op 40#define __HYPERVISOR_dm_op 41xen/include/xen/hypercall.h +9
-1@@ -136,6 +136,15 @@ do_tmem_op( XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);#endif +#ifdef CONFIG_ARGO +extern long do_argo_message_op ( +int cmd, +XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2, +uint32_t arg3, +uint32_t arg4); +#endif +extern long do_xenoprof_op (int op,XEN_GUEST_HANDLE_PARAM (void ) arg); --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 03/25] argo: introduce the argo_message_op hypercall boilerplate
Quoted 8 lines · Paul Durrant
Quoted 8 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Subject: [PATCH 03/25] argo: introduce theargo_message_ophypercall boilerplate
Quoted 8 lines
Quoted 6 lines
Presence is gated upon
CONFIG_ARGO.Registers the hypercall previously reserved for this. Takes 5 arguments, does nothing and returns
-ENOSYS.Will be avoiding a compat ABI by using fixed-size types in hypercall ops.
You appear to be using handles, so will you not need compat code to deal with those?
No. The structures that the handles refer to are exactly the same on both 32 and 64 bit, so the memory access operations that work with the handle DTRT.
Communication has been tested and is working fine, eg. between a client in a 32-bit guest and server in a 64-bit VM.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 4/25 argo: define argo_dprintk for subsystem debugging
In reply to Christopher ClarkA convenience for working on development of the argo subsystem:
toggling a local #define variable turns on just the debug messages in this subsystem.
printk("argo: " format, ## args )
--- xen/common/argo.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
xen/common/argo.c +13 -0
@@ -19,6 +19,19 @@#include <xen/errno.h> #include <xen/guest_access.h> +/* +* Debugs +*/ + +#ifdef ARGO_DEBUG +#define argo_dprintk(format, args...) \ + do { \ + printk("argo: " format, ## args ); \ + } while ( 1 == 0 ) +#else +#define argo_dprintk (format, ... ) (void)0 +#endif + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 04/25] argo: define argo_dprintk for subsystem debugging
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
A convenience for working on development of the argo subsystem:
toggling a local #define variable turns on just the debug messages in this subsystem.
I'm afraid I don't see the #define variable to toggle. I assume it's ARGO_DEBUG, but there's no #define line for it anywhere here.
Quoted 1 line
printk("argo: " format, ## args )
Was this line misplaced here? It doesn't look related to the rest of the description.
Quoted 5 lines
xen/common/argo.c +0
-0@@ -19,6 +19,19 @@#include <xen/errno.h> #include <xen/guest_access.h>
xen/common/argo.c line 30
Quoted 10 lines
xen/common/argo.c +9
-0+/* +* Debugs +*/ + +#ifdef ARGO_DEBUG +#define argo_dprintk(format, args...) \ + do { \ + printk("argo: " format, ## args ); \ + } while ( 1 == 0 )
What wrong with
#define argo_dprintk(format, args...) printk("argo: " format, ## args )
?
xen/common/argo.c line 32
Quoted 2 lines
xen/common/argo.c +2
-0+#else +#define argo_dprintk (format, ... ) (void)0
Please fully parenthesize macro expansions.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 5/25 argo: Add initial argo_init and argo_destroy
In reply to Christopher ClarkInitialises basic data structures and performs teardown of argo state for domain shutdown.
Introduces headers:
<public/argo.h> with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor, and <xen/argo.h> to support hooking init and destroy into domain lifecycle.
If CONFIG_ARGO is enabled:
Adds per-domain init of argo data structures to domain_create by calling argo_init, and similarly adds teardown via argo_destroy into domain_destroy and the error exit path of domain_create.
argo_init allocates an event channel for use for signalling to the domain. The event channel is of type IPI since that behaves in the required way; unbound event channels are unsuitable since they silently drop events. The only disadvantage of the IPI type is that the channel cannot be rebound to any other VCPU; that seems to be tolerable and avoids introducing any further changes to add another channel type.
In accordance with recent work on _domain_destroy, argo_destroy is idempotent.
Adds two new fields to struct domain:
rwlock_t argo_lock;
struct argo_domain *argo;--- xen/common/argo.c | 277 +++++++++++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 15 +++ xen/include/public/argo.h | 55 +++++++++ xen/include/xen/argo.h | 30 +++++ xen/include/xen/sched.h | 7 ++ 5 files changed, 383 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.h
xen/common/argo.c +276 -1
@@ -17,7 +17,101 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> + +DEFINE_XEN_GUEST_HANDLE (argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (argo_ring_t); + +struct argo_pending_ent +{ +struct hlist_node node; +domid_t id; +uint32_t len; +}; + +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +argo_ring_id_t id; +/* used to confirm sender id, protected by L2 */ +uint64_t partner_cookie; +/* L3 */ +spinlock_t lock; +/* cached length of the ring (from ring->len), protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct argo_pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* event channel */ +evtchn_port_t evtchn_port; +/* protected by L2 */ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* id cookie, written only at init, so readable with R(L1) */ +uint64_t domain_cookie; +}; + +/* +* locks +*/ + +/* +* locking is organized as follows: +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding this lock +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain lock: d->argo->lock +* Holding a read lock on L2 protects the hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects len, tx_ptr, the guest ring, the guest ring_data and +* the pending list. +* To aquire L3 you must already have R(L2). W(L2) implies L3. +*/ /* * Debugs
@@ -32,10 +126,191 @@#define argo_dprintk (format, ... ) (void)0#endif +/* +* ring buffer +*/ + +/* caller must have L3 or W(L2) */ +static void +argo_ring_unmap (struct argo_ring_info *ring_info) +{ +int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +/* +* pending +*/ +static void +argo_pending_remove_ent (struct argo_pending_ent *ent) +{ +hlist_del (&ent->node); +xfree (ent); +} + +static void +argo_pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *pending_ent; + +hlist_for_each_entry_safe (pending_ent, node, next, + &ring_info->pending, node) + { +argo_pending_remove_ent (pending_ent); + } +} + +static void argo_ring_remove_mfns (const struct domain *d, +struct argo_ring_info *ring_info) +{ +int i; + +ASSERT (rw_is_write_locked (&d->argo->lock)); + +if ( !ring_info->mfns ) +return ; +ASSERT (ring_info->mfn_mapping); + +argo_ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if (mfn_x (ring_info->mfns[i]) !=mfn_x (INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +argo_ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock)); + +/* Holding W(L2) so do not need to acquire L3 */ +argo_pending_remove_all (ring_info); +hlist_del (&ring_info->node); +argo_ring_remove_mfns (d, ring_info); +xfree (ring_info); +} +long do_argo_message_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,uint32_t arg3,uint32_t arg4) { -return -ENOSYS ; +struct domain *d = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_message_op(%d,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +domain_lock (d); + +switch (cmd) + { +default : + rc = -ENOSYS ; +break ; + } + +domain_unlock (d); +argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc); +return rc; +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; +evtchn_port_t port; +int i; +int rc; + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +rwlock_init (&argo->lock); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) +INIT_HLIST_HEAD (&argo->ring_hash[i]); + + rc =evtchn_bind_ipi_vcpu0_domain (d, &port); +if ( rc ) + { +xfree (argo); +return rc; + } + argo->evtchn_port = port; + argo->domain_cookie = (uint64_t )NOW (); + +write_lock (&argo_lock); + d->argo = argo; +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +int i; + +BUG_ON (!d->is_dying); +write_lock (&argo_lock); + +argo_dprintk ("d->v=%p\n" , d->argo); + +if ( d->argo ) + { +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, + next, &d->argo->ring_hash[i], + node) + { +argo_ring_remove_info (d, ring_info); + } + } +/* +* Since this function is only called during domain destruction, +* argo->evtchn_port need not be closed here. ref: evtchn_destroy +*/ + d->argo->domain_cookie =0 ; +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); + +/* +* This (dying) domain's domid may be recorded as the authorized sender +* to rings registered by other domains, and those rings are not +* unregistered here. +* If a later domain is created that has the same domid as this one, the +* domain_cookie will differ, which ensures that the new domain cannot +* use the inherited authorizations to transmit that were issued to this +* domain. +*/ }
xen/common/domain.c +15 -0
@@ -277,6 +277,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif +rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);
@@ -376,6 +380,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;
@@ -445,6 +452,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );
@@ -717,6 +729,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);
xen/include/public/argo.h new file +55 -0
55 lines not on this page · raw message
xen/include/xen/argo.h new file +30 -0
@@ -0,0 +1,30 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +#include <xen/types.h> +#include <public/argo.h> + +struct argo_domain ; + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); + +#endif
xen/include/xen/sched.h +7 -0
@@ -22,6 +22,7 @@#include <asm/atomic.h> #include <xen/vpci.h> #include <xen/wait.h> +#include <xen/argo.h> #include <public/xen.h> #include <public/domctl.h> #include <public/sysctl.h>
@@ -490,6 +491,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_ARGO +/* Argo interdomain communication support */ +rwlock_t argo_lock; +struct argo_domain *argo; +#endif };/* Protect updates/reads (resp.) of domain_list and domain_hash. */
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 05/25] argo: Add initial argo_init and argo_destroy
Quoted 74 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 05/25] argo: Add initialargo_initandargo_destroyInitialises basic data structures and performs teardown of argo state for domain shutdown.
Introduces headers:
<public/argo.h> with definions of addresses and ring structure,
including
indexes for atomic update for communication between domain and
hypervisor,
and <xen/argo.h> to support hooking init and destroy into domain
lifecycle.If
CONFIG_ARGOis enabled:Adds per-domain init of argo data structures to
domain_createby callingargo_init, and similarly adds teardown viaargo_destroyintodomain_destroy
and the error exit path ofdomain_create.
argo_initallocates an event channel for use for signalling to the domain. The event channel is of type IPI since that behaves in the required way; unbound event channels are unsuitable since they silently drop events. The only disadvantage of the IPI type is that the channel cannot be rebound
to any other VCPU; that seems to be tolerable and avoids introducing any further changes to add another channel type.In accordance with recent work on
_domain_destroy,argo_destroyis idempotent.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 277 +++++++++++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 15 +++ xen/include/public/argo.h | 55 +++++++++ xen/include/xen/argo.h | 30 +++++ xen/include/xen/sched.h | 7 ++ 5 files changed, 383 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hxen/common/argo.c +5
-0@@ -17,7 +17,101 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h>
xen/common/argo.c line 34
Quoted 10 lines
xen/common/argo.c +9
-0#include <xen/guest_access.h> +#include <xen/time.h> + +DEFINE_XEN_GUEST_HANDLE (argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (argo_ring_t); + +struct argo_pending_ent +{ +struct hlist_node node; +domid_t id;
I think you want 16 bits of padding here.
Quoted 86 lines
xen/common/argo.c +80
-0+uint32_t len; +}; + +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +argo_ring_id_t id; +/* used to confirm sender id, protected by L2 */ +uint64_t partner_cookie; +/* L3 */ +spinlock_t lock; +/* cached length of the ring (from ring->len), protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct argo_pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* event channel */ +evtchn_port_t evtchn_port; +/* protected by L2 */ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* id cookie, written only at init, so readable with R(L1) */ +uint64_t domain_cookie; +}; + +/* +* locks +*/ + +/* +* locking is organized as follows: +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding this lock +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain lock: d->argo->lock +* Holding a read lock on L2 protects the hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects len, tx_ptr, the guest ring, the guest ring_data and +* the pending list. +* To aquire L3 you must already have R(L2). W(L2) implies L3. +*/ /* * Debugs @@ -32,10 +126,191 @@#define argo_dprintk (format, ... ) (void)0#endif
xen/common/argo.c line 137
Quoted 10 lines
xen/common/argo.c +9
-0+/* +* ring buffer +*/ + +/* caller must have L3 or W(L2) */ +static void +argo_ring_unmap (struct argo_ring_info *ring_info) +{ +int i;
unsigned?
Quoted 27 lines
xen/common/argo.c +26
-0+ +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) + argo_dprintk(XENLOG_ERR "argo: unmapping page %"PRI_mfn" from %p\n", + mfn_x(ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +/* +* pending +*/ +static void +argo_pending_remove_ent (struct argo_pending_ent *ent) +{ +hlist_del (&ent->node); +xfree (ent); +}
xen/common/argo.c line 173
Quoted 10 lines
xen/common/argo.c +10
-0+ +static void +argo_pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *pending_ent; + +hlist_for_each_entry_safe (pending_ent, node, next, + &ring_info->pending, node) + {
Unnecessary braces, I think.
xen/common/argo.c line 181
Quoted 8 lines
xen/common/argo.c +8
-0+argo_pending_remove_ent (pending_ent); + } +} + +static void argo_ring_remove_mfns (const struct domain *d, +struct argo_ring_info *ring_info) +{ +int i;
unsigned?
xen/common/argo.c line 192
Quoted 11 lines
xen/common/argo.c +11
-0+ +ASSERT (rw_is_write_locked (&d->argo->lock)); + +if ( !ring_info->mfns ) +return ; +ASSERT (ring_info->mfn_mapping); + +argo_ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if (mfn_x (ring_info->mfns[i]) !=mfn_x (INVALID_MFN ) )
How about "!mfn_eq(ring_info->mfns[i], INVALID_MFN)"? I think it's a bit neater.
Quoted 19 lines
xen/common/argo.c +19
-0+put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +argo_ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock)); + +/* Holding W(L2) so do not need to acquire L3 */ +argo_pending_remove_all (ring_info); +hlist_del (&ring_info->node); +argo_ring_remove_mfns (d, ring_info);
xen/common/argo.c line 220
Quoted 10 lines
xen/common/argo.c +4
-1+xfree (ring_info); +} +long do_argo_message_op (int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,uint32_t arg3,uint32_t arg4) { -return -ENOSYS ; +struct domain *d = current->domain;
The general preference these days is to use 'currd' to refer to the current domain (and d for an arbitrary one).
Quoted 6 lines
xen/common/argo.c +6
-0+long rc = -EFAULT ; + +argo_dprintk ("->do_argo_message_op(%d,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +domain_lock (d);
xen/common/argo.c line 236
Quoted 10 lines
xen/common/argo.c +10
-0+ +switch (cmd) + { +default : + rc = -ENOSYS ; +break ; + } + +domain_unlock (d); +argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc);
Blank line here.
Quoted 47 lines
xen/common/argo.c +47
-0+return rc; +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; +evtchn_port_t port; +int i; +int rc; + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +rwlock_init (&argo->lock); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) +INIT_HLIST_HEAD (&argo->ring_hash[i]); + + rc =evtchn_bind_ipi_vcpu0_domain (d, &port); +if ( rc ) + { +xfree (argo); +return rc; + } + argo->evtchn_port = port; + argo->domain_cookie = (uint64_t )NOW (); + +write_lock (&argo_lock); + d->argo = argo; +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +int i; + +BUG_ON (!d->is_dying); +write_lock (&argo_lock); + +argo_dprintk ("d->v=%p\n" , d->argo); + +if ( d->argo )
xen/common/argo.c line 293
Quoted 10 lines
xen/common/argo.c +10
-0+ { +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, + next, &d->argo->ring_hash[i], + node) + {
Unnecessary braces I think.
Quoted 91 lines
xen/common/argo.c +15
-0+argo_ring_remove_info (d, ring_info); + } + } +/* +* Since this function is only called during domain destruction, +* argo->evtchn_port need not be closed here. ref: evtchn_destroy +*/ + d->argo->domain_cookie =0 ; +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); + +/* +* This (dying) domain's domid may be recorded as the authorized sender
xen/common/argo.c +3
-0+* to rings registered by other domains, and those rings are not +* unregistered here. +* If a later domain is created that has the same domid as this one, the
+ *domain_cookiewill differ, which ensures that the new domain
cannot
+ * use the inherited authorizations to transmit that were issued to
thisxen/common/argo.c +2
-0+* domain. +*/ }xen/common/domain.c +15
-0@@ -277,6 +277,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif +rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);@@ -376,6 +380,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;@@ -445,6 +452,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );@@ -717,6 +729,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);xen/include/public/argo.h new file +1
-0@@ -0,0 +1,55 @@ +/************************************************************************ ******
+ * Argo : Hypervisor-Mediated data eXchange
+ *
+ * Derived from v4v, the version 2 of v2v.
+ *
+ * Copyright (c) 2010, Citrix Systems
+ * Copyright (c) 2018, BAE Systems
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
Quoted 8 lines
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
Do you really want to license a public header under the GPL?
Paul
Quoted 102 lines
xen/include/public/argo.h new file +38
-0+ +#ifndef __XEN_PUBLIC_ARGO_H__ +#define __XEN_PUBLIC_ARGO_H__ + +#include "xen.h" + +typedef struct argo_addr +{ +uint32_t port; +domid_t domain_id; +uint16_t pad; +}argo_addr_t ; + +typedef struct argo_ring_id +{ +struct argo_addr addr; +domid_t partner; +uint16_t pad; +}argo_ring_id_t ; + +typedef struct argo_ring +{ +uint64_t magic; +argo_ring_id_t id; +uint32_t len; +/* Guests should use atomic operations to access rx_ptr */ +uint32_t rx_ptr; +/* Guests should use atomic operations to access tx_ptr */ +uint32_t tx_ptr; +uint8_t reserved[32 ]; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t ring[]; +#elif defined(__GNUC__) +uint8_t ring[0 ]; +#endif +}argo_ring_t ; + +#endif xen/include/xen/argo.h new file +1
-0@@ -0,0 +1,30 @@ +/************************************************************************ ******
+ * Argo : Hypervisor-Mediated data eXchange
+ *
+ * Derived from v4v, the version 2 of v2v.
+ *
+ * Copyright (c) 2010, Citrix Systems
+ * Copyright (c) 2018, BAE Systems
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */xen/include/xen/argo.h new file +13
-0+ +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +#include <xen/types.h> +#include <public/argo.h> + +struct argo_domain ; + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); + +#endif xen/include/xen/sched.h +7
-1@@ -22,6 +22,7 @@#include <asm/atomic.h> #include <xen/vpci.h> #include <xen/wait.h> +#include <xen/argo.h> #include <public/xen.h> #include <public/domctl.h> #include <public/sysctl.h> @@ -490,6 +491,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_ARGO +/* Argo interdomain communication support */ +rwlock_t argo_lock; +struct argo_domain *argo; +#endif };/* Protect updates/reads (resp.) of domain_list and domain_hash. */ --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 05/25] argo: Add initial argo_init and argo_destroy
In reply to Christopher ClarkQuoted 47 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/include/public/argo.h new file +43
-0@@ -0,0 +1,55 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_PUBLIC_ARGO_H__ +#define __XEN_PUBLIC_ARGO_H__ + +#include "xen.h" + +typedef struct argo_addr +{ + uint32_t port; + domid_t domain_id; + uint16_t pad; +} argo_addr_t; + +typedef struct argo_ring_id +{ + struct argo_addr addr; + domid_t partner; + uint16_t pad; +} argo_ring_id_t; + +typedef struct argo_ring +{ + uint64_t magic; + argo_ring_id_t id; + uint32_t len; + /* Guests should use atomic operations to access rx_ptr */
xen/include/public/argo.h line 53
Quoted 10 lines
xen/include/public/argo.h new file +10
-0+uint32_t rx_ptr; +/* Guests should use atomic operations to access tx_ptr */ +uint32_t tx_ptr; +uint8_t reserved[32 ]; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t ring[]; +#elif defined(__GNUC__) +uint8_t ring[0 ]; +#endif +} argo_ring_t;
Btw, for all structure types you define, and with your desire to avoid compat mode translation, you should add ?-prefixed entries to xen/include/xlat.lst and invoke the produced CHECK_* macros from somewhere. If, for reference, you'd look at existing instances, you'll then also find another reason why all these would better have xen_ prefixes.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 6/25 argo: Xen command line parameter 'argo': bool to enable/disable
In reply to Christopher ClarkDefault to disabled.
--- xen/common/argo.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
xen/common/argo.c +19 -0
@@ -28,6 +28,10 @@DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled =0 ; +boolean_param ("argo" , opt_argo_enabled); +struct argo_pending_ent {struct hlist_node node;
@@ -223,6 +227,13 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,argo_dprintk ("->do_argo_message_op(%d,%p,%p,%d,%d)\n" , cmd, (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); +if (unlikely (!opt_argo_enabled) ) + { + rc = -ENOSYS ; +argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc); +return rc; + } +domain_lock (d); switch (cmd)
@@ -245,6 +256,14 @@ argo_init(struct domain *d)int i;int rc; +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("argo init: domid: %d\n" , d->domain_id); + argo = xmalloc(struct argo_domain);if ( !argo )return -ENOMEM ;
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 06/25] argo: Xen command line parameter 'argo': bool to enable/disable
Quoted 17 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 06/25] argo: Xen command line parameter 'argo': bool to enable/disable
Quoted 1 line
Default to disabled.
Any particular reason not to fold this into patch #5?
Paul
Quoted 52 lines
Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)xen/common/argo.c +11
-0@@ -28,6 +28,10 @@DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled =0 ; +boolean_param ("argo" , opt_argo_enabled); +struct argo_pending_ent {struct hlist_node node;@@ -223,6 +227,13 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, argo_dprintk("->do_argo_message_op(%d,%p,%p,%d,%d)\n", cmd, (void *)arg1.p, (void *)arg2.p, (int) arg3, (int) arg4); +if (unlikely (!opt_argo_enabled) ) + { + rc = -ENOSYS ; +argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc); +return rc; + } +domain_lock (d);switch (cmd)
xen/common/argo.c +8
-1@@ -245,6 +256,14 @@ argo_init(struct domain *d)int i;int rc; +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("argo init: domid: %d\n" , d->domain_id); + argo = xmalloc(struct argo_domain);if ( !argo )return -ENOMEM ; --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 06/25] argo: Xen command line parameter 'argo': bool to enable/disable
In reply to Christopher Clarkxen/common/argo.c line 32
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +2
-0@@ -28,6 +28,10 @@DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled =0 ;
The initializer is pointless here, and if there was one then it should be true or false.
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 233
Quoted 8 lines
xen/common/argo.c +4
-0@@ -223,6 +227,13 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,argo_dprintk ("->do_argo_message_op(%d,%p,%p,%d,%d)\n" , cmd, (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); +if (unlikely (!opt_argo_enabled) ) + { + rc = -ENOSYS ; +argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc);
While I can see debugging printk()s to be useful in certain places, I question the utility of this and the other one. I also question the use of -ENOSYS - I think you mean e.g. -EOPNOTSUPP.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Christopher Clark
AppliedPATCH 7/25 xen (ARM, x86): add errno-returning functions for copy
In reply to Christopher ClarkApplied to both x86 and ARM headers.
--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)
xen/include/asm-arm/guest_access.h +25 -0
@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,#define __raw_copy_from_guest raw_copy_from_guest#define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno (dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0) +/* Remainder copied from x86 -- could be common? */ /* Is the guest handle a NULL reference? */
@@ -113,6 +118,26 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,raw_copy_from_guest (_d, _s,sizeof (*_d)); \ }) +/* errno returning copy functions */ +#define copy_from_guest_offset_errno (ptr, hnd, off, nr) ({ \ + const typeof(*(ptr)) *_s = (hnd).p; \ + typeof(*(ptr)) *_d = (ptr); \ + raw_copy_from_guest_errno(_d, _s + (off), sizeof(*_d) * (nr)); \ + }) + +#define copy_field_to_guest_errno (hnd, ptr, field) ({ \ + const typeof(&(ptr)->field) _s = &(ptr)->field; \ + void *_d = &(hnd).p->field; \ + ((void)(&(hnd).p->field == &(ptr)->field)); \ + raw_copy_to_guest_errno(_d, _s, sizeof(*_s)); \ + }) + +#define copy_field_from_guest_errno (ptr, hnd, field) ({ \ + const typeof(&(ptr)->field) _s = &(hnd).p->field; \ + typeof(&(ptr)->field) _d = &(ptr)->field; \ + raw_copy_from_guest_errno(_d, _s, sizeof(*_d)); \ + }) +/* * Pre-validate a guest handle. * Allows use of faster __copy_* functions.
xen/include/asm-x86/guest_access.h +29 -0
@@ -38,6 +38,15 @@ clear_user_hvm((dst), (len)) : \ clear_user((dst), (len))) +#define raw_copy_from_guest_errno (dst, src, len) \ + (is_hvm_vcpu(current) ? \ + copy_from_user_hvm((dst), (src), (len)) : \ + (copy_from_user((dst), (src), (len)) ? -EFAULT : 0)) +#define raw_copy_to_guest_errno (dst, src, len) \ + (is_hvm_vcpu(current) ? \ + copy_to_user_hvm((dst), (src), (len)) : \ + (copy_to_user((dst), (src), (len)) ? -EFAULT : 0)) +/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL)
@@ -121,6 +130,26 @@raw_copy_from_guest (_d, _s,sizeof (*_d)); \ }) +/* errno returning copy functions */ +#define copy_from_guest_offset_errno (ptr, hnd, off, nr) ({ \ + const typeof(*(ptr)) *_s = (hnd).p; \ + typeof(*(ptr)) *_d = (ptr); \ + raw_copy_from_guest_errno(_d, _s + (off), sizeof(*_d) * (nr)); \ + }) + +#define copy_field_to_guest_errno (hnd, ptr, field) ({ \ + const typeof(&(ptr)->field) _s = &(ptr)->field; \ + void *_d = &(hnd).p->field; \ + ((void)(&(hnd).p->field == &(ptr)->field)); \ + raw_copy_to_guest_errno(_d, _s, sizeof(*_s)); \ + }) + +#define copy_field_from_guest_errno (ptr, hnd, field) ({ \ + const typeof(&(ptr)->field) _s = &(hnd).p->field; \ + typeof(&(ptr)->field) _d = &(ptr)->field; \ + raw_copy_from_guest_errno(_d, _s, sizeof(*_d)); \ + }) +/* * Pre-validate a guest handle. * Allows use of faster __copy_* functions.
xen/include/xen/guest_access.h +3 -0
@@ -26,6 +26,9 @@#define __copy_from_guest (ptr, hnd, nr) \ __copy_from_guest_offset(ptr, hnd, 0, nr) +#define copy_from_guest_errno (ptr, hnd, nr) \ + copy_from_guest_offset_errno(ptr, hnd, 0, nr) +#define __clear_guest (hnd, nr) \ __clear_guest_offset(hnd, 0, nr)
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copy
Quoted 81 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monne <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copyApplied to both x86 and ARM headers.
Signed-off-by Christopher Clark <[email protected]>--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)xen/include/asm- renamed from xen/include/asm-arm/guest_access.h +0
-0
arm/guest_access.h
index 224d2a0..7b6f89c 100644xen/include/asm-arm/guest_access.h +5
-0@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, #define __raw_copy_from_guest raw_copy_from_guest #define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno(dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0) +/* Remainder copied from x86 -- could be common? */ /* Is the guest handle a NULL reference? */
xen/include/asm-arm/guest_access.h +20
-0@@ -113,6 +118,26 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, raw_copy_from_guest(_d, _s, sizeof(*_d)); \ }) +/* errno returning copy functions */ +#define copy_from_guest_offset_errno (ptr, hnd, off, nr) ({ \ + const typeof(*(ptr)) *_s = (hnd).p; \ + typeof(*(ptr)) *_d = (ptr); \ + raw_copy_from_guest_errno(_d, _s + (off), sizeof(*_d) * (nr)); \ + }) + +#define copy_field_to_guest_errno (hnd, ptr, field) ({ \ + const typeof(&(ptr)->field) _s = &(ptr)->field; \ + void *_d = &(hnd).p->field; \ + ((void)(&(hnd).p->field == &(ptr)->field)); \ + raw_copy_to_guest_errno(_d, _s, sizeof(*_s)); \ + }) + +#define copy_field_from_guest_errno (ptr, hnd, field) ({ \ + const typeof(&(ptr)->field) _s = &(hnd).p->field; \ + typeof(&(ptr)->field) _d = &(ptr)->field; \ + raw_copy_from_guest_errno(_d, _s, sizeof(*_d)); \ + }) +/* * Pre-validate a guest handle.
* Allows use of faster __copy_* functions.xen/include/asm- renamed from xen/include/asm-x86/guest_access.h +0
-0
x86/guest_access.h
index ca700c9..9391cd3 100644
xen/include/asm-x86/guest_access.h line 44
Quoted 7 lines
xen/include/asm-x86/guest_access.h +4
-0@@ -38,6 +38,15 @@ clear_user_hvm((dst), (len)) : \ clear_user((dst), (len))) +#define raw_copy_from_guest_errno (dst, src, len) \ + (is_hvm_vcpu(current) ? \ + copy_from_user_hvm((dst), (src), (len)) : \ + (copy_from_user((dst), (src), (len)) ? -EFAULT : 0))
AFAICT copy_from_user_hvm() doesn't return -ve errno (it has comment "/* fake a copy_to_user() return code */" on the return line) so I think your bracketing is wrong here...
xen/include/asm-x86/guest_access.h line 49
Quoted 5 lines
xen/include/asm-x86/guest_access.h +5
-0+#define raw_copy_to_guest_errno (dst, src, len) \ + (is_hvm_vcpu(current) ? \ + copy_to_user_hvm((dst), (src), (len)) : \ + (copy_to_user((dst), (src), (len)) ? -EFAULT : 0)) +
...and similarly here.
Quoted 37 lines
xen/include/asm-x86/guest_access.h +20
-0/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL)@@ -121,6 +130,26 @@raw_copy_from_guest (_d, _s,sizeof (*_d)); \ }) +/* errno returning copy functions */ +#define copy_from_guest_offset_errno (ptr, hnd, off, nr) ({ \ + const typeof(*(ptr)) *_s = (hnd).p; \ + typeof(*(ptr)) *_d = (ptr); \ + raw_copy_from_guest_errno(_d, _s + (off), sizeof(*_d) * (nr)); \ + }) + +#define copy_field_to_guest_errno (hnd, ptr, field) ({ \ + const typeof(&(ptr)->field) _s = &(ptr)->field; \ + void *_d = &(hnd).p->field; \ + ((void)(&(hnd).p->field == &(ptr)->field)); \ + raw_copy_to_guest_errno(_d, _s, sizeof(*_s)); \ + }) + +#define copy_field_from_guest_errno (ptr, hnd, field) ({ \ + const typeof(&(ptr)->field) _s = &(hnd).p->field; \ + typeof(&(ptr)->field) _d = &(ptr)->field; \ + raw_copy_from_guest_errno(_d, _s, sizeof(*_d)); \ + }) +/* * Pre-validate a guest handle. * Allows use of faster __copy_* functions.
xen/include/xen/guest_access.h +0
-0
xen/include/xen/guest_access.h line 34
Quoted 9 lines
xen/include/xen/guest_access.h +3
-0@@ -26,6 +26,9 @@#define __copy_from_guest (ptr, hnd, nr) \ __copy_from_guest_offset(ptr, hnd, 0, nr) +#define copy_from_guest_errno (ptr, hnd, nr) \ + copy_from_guest_offset_errno(ptr, hnd, 0, nr) +#define __clear_guest (hnd, nr) \ __clear_guest_offset(hnd, 0, nr)
Given that the only errno possible seems to be EFAULT, I do have to question why you need these changes?
Paul
Quoted 2 lines
--
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copy
In reply to Christopher ClarkQuoted 14 lines · Christopher Clark
Applied to both x86 and ARM headers.
Signed-off-by Christopher Clark <[email protected]>--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)xen/include/asm-arm/guest_access.h +0
-0
xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 30
Quoted 7 lines
xen/include/asm-arm/guest_access.h +4
-0@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,#define __raw_copy_from_guest raw_copy_from_guest#define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno (dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0)
Since the only error that you return is EFAULT, I don't really see the point in adding all those helpers. You achieve exactly the same by returning a boolean and doing the translation to EFAULT in the caller if required.
It might have been nice to have the copy to/from set of functions return an error value, but adding a new set of helpers that have the same functionality but just differ in the return value look redundant.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copy
Quoted 29 lines · Roger Pau Monné
Quoted 14 lines · Christopher Clark
Applied to both x86 and ARM headers.
Signed-off-by Christopher Clark <[email protected]>--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)xen/include/asm-arm/guest_access.h +0
-0xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 30
Quoted 7 lines
xen/include/asm-arm/guest_access.h +4
-0@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,#define __raw_copy_from_guest raw_copy_from_guest#define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno (dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0)Since the only error that you return is EFAULT, I don't really see the point in adding all those helpers. You achieve exactly the same by returning a boolean and doing the translation to EFAULT in the caller if required.
Quoted 4 lines
It might have been nice to have the copy to/from set of functions return an error value, but adding a new set of helpers that have the same functionality but just differ in the return value look redundant.
It is true that there is redundancy with these -- but I think there are decent arguments in favour of taking these in:
* the errno-providing interface is just a better fit for almost every call site
- which means less source code in total, that is easier to read.
* it is promoting good interface design for error handling:
return of error code.
* since these are in use within the uxen source code, it eases comparison and work across both codebases - relevant for Argo, due to v4v.
I've rewritten the implementation of these for the second version of the patch series -- now much simpler -- and hopefully that will mitigate some of your concern about them.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copy
Quoted 47 lines
Quoted 1 line
Quoted 1 line · On 20.12.18 at 06:16, <[email protected]> wrote:
Quoted 4 lines · Roger Pau Monné
Quoted 2 lines · Christopher Clark
Applied to both x86 and ARM headers.
Quoted 8 lines
Review
Quoted 8 lines
Signed-off-by Christopher Clark <[email protected]>--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)(quoted diff) +0
-0
b/xen/include/asm-arm/guest_access.hQuoted 4 lines
xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t
Quoted 4 lines
index 224d2a0..7b6f89c 100644
xen/include/asm-arm/guest_access.h +0
-0@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_tipa, void *buf,
Quoted 13 lines
xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 30
Quoted 7 lines
xen/include/asm-arm/guest_access.h +4
-0#define __raw_copy_from_guest raw_copy_from_guest#define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno (dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0)Since the only error that you return is EFAULT, I don't really see the point in adding all those helpers. You achieve exactly the same by returning a boolean and doing the translation to EFAULT in the caller if required.
Quoted 4 lines
It might have been nice to have the copy to/from set of functions return an error value, but adding a new set of helpers that have the same functionality but just differ in the return value look redundant.
It is true that there is redundancy with these -- but I think there are decent arguments in favour of taking these in:
* the errno-providing interface is just a better fit for almost every call site
- which means less source code in total, that is easier to read.* it is promoting good interface design for error handling:
return of error code.
Quoted 6 lines
* since these are in use within the uxen source code, it eases comparison and work across both codebases - relevant for Argo, due to v4v.
I've rewritten the implementation of these for the second version of the patch series -- now much simpler -- and hopefully that will mitigate some of your concern about them.
Without having looked at their v2 forms, I continue to fully agree with Roger here: There's no reason to introduce these flavors just for argo to use. We've been doing fine with what we have, and if you want to change things, then you'd want to do so everywhere. That would then also eliminate the need for separate flavors: The ones that are there would then simply behave along the lines of the principles you outline above.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 07/25] xen (ARM, x86): add errno-returning functions for copy
In reply to Christopher ClarkQuoted 35 lines · Christopher Clark
Quoted 29 lines · Roger Pau Monné
Quoted 14 lines · Christopher Clark
Applied to both x86 and ARM headers.
Signed-off-by Christopher Clark <[email protected]>--- xen/include/asm-arm/guest_access.h | 25 +++++++++++++++++++++++++ xen/include/asm-x86/guest_access.h | 29 +++++++++++++++++++++++++++++ xen/include/xen/guest_access.h | 3 +++ 3 files changed, 57 insertions(+)xen/include/asm-arm/guest_access.h +0
-0xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 30
Quoted 7 lines
xen/include/asm-arm/guest_access.h +4
-0@@ -24,6 +24,11 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,#define __raw_copy_from_guest raw_copy_from_guest#define __raw_clear_guest raw_clear_guest +#define raw_copy_from_guest_errno (dst, src, len) \ + (raw_copy_from_guest((dst), (src), (len)) ? -EFAULT : 0) +#define raw_copy_to_guest_errno (dst, src, len) \ + (raw_copy_to_guest((dst), (src), (len)) ? -EFAULT : 0)Since the only error that you return is EFAULT, I don't really see the point in adding all those helpers. You achieve exactly the same by returning a boolean and doing the translation to EFAULT in the caller if required.
Quoted 4 lines
It might have been nice to have the copy to/from set of functions return an error value, but adding a new set of helpers that have the same functionality but just differ in the return value look redundant.
Quoted 8 lines
It is true that there is redundancy with these -- but I think there are decent arguments in favour of taking these in:
* the errno-providing interface is just a better fit for almost every call site
- which means less source code in total, that is easier to read.* it is promoting good interface design for error handling:
return of error code.
Then I'm afraid that you will have to change the current copy to/from helpers to return an error code and fix all the callers. I don't think it's acceptable to have this duplication of functionality in the code base.
IMO having such redundancy creates confusion, specially with new developers, so if returning an error code is much better and provides cleaner code it should be argued for the whole Xen code base, and a global switch should be made.
Quoted 6 lines
* since these are in use within the uxen source code, it eases comparison and work across both codebases - relevant for Argo, due to v4v.
I've rewritten the implementation of these for the second version of the patch series -- now much simpler -- and hopefully that will mitigate some of your concern about them.
My issue is not so much with the implementation, but rather the redundancy.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 8/25 xen: define XEN_GUEST_HANDLE_NULL as null XEN_GUEST_HANDLE
In reply to Christopher Clark--- xen/include/public/xen.h | 2 ++ 1 file changed, 2 insertions(+)
xen/include/public/xen.h +2 -0
@@ -982,6 +982,8 @@ typedef struct {#define XEN_GUEST_HANDLE_64 (name) XEN_GUEST_HANDLE(name)#endif +#define XEN_GUEST_HANDLE_NULL (name) (XEN_GUEST_HANDLE(name)){(name *)0} + #ifndef __ASSEMBLY__ struct xenctl_bitmap { XEN_GUEST_HANDLE_64(uint8) bitmap;
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 08/25] xen: define XEN_GUEST_HANDLE_NULL as null XEN_GUEST_HANDLE
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
Signed-off-by Christopher Clark <[email protected]>
Here and elsewhere: Such additions, which we've found no need for till now, should not be submitted without giving a reason for why they become necessary or at least desirable.
xen/include/public/xen.h typedef struct { · line 985
Quoted 7 lines
xen/include/public/xen.h +1
-0@@ -982,6 +982,8 @@ typedef struct {#define XEN_GUEST_HANDLE_64 (name) XEN_GUEST_HANDLE(name)#endif +#define XEN_GUEST_HANDLE_NULL (name) (XEN_GUEST_HANDLE(name)){(name *)0}
Public headers are intended to be usable in C89 mode. While this won't cause compilation to fail when not used, it still is a violation of this principle. Furthermore the construct is incompletely parenthesized.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 9/25 errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
In reply to Christopher Clarkhttp://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
describes these codes thus:
EMSGSIZE : "Message too large"
ECONNREFUSED : "Connection refused".
--- xen/include/public/errno.h | 2 ++ 1 file changed, 2 insertions(+)
xen/include/public/errno.h +2 -0
@@ -102,6 +102,7 @@ XEN_ERRNO(EILSEQ, 84) /* Illegal byte sequence */ XEN_ERRNO(ERESTART, 85) /* Interrupted system call should be restarted */#endif XEN_ERRNO(ENOTSOCK, 88) /* Socket operation on non-socket */ +XEN_ERRNO(EMSGSIZE, 90) /* Message too large. */ XEN_ERRNO(EOPNOTSUPP, 95) /* Operation not supported on transport endpoint */ XEN_ERRNO(EADDRINUSE, 98) /* Address already in use */XEN_ERRNO (EADDRNOTAVAIL ,99 )/* Cannot assign requested address */
@@ -109,6 +110,7 @@ XEN_ERRNO(ENOBUFS, 105) /* No buffer space available */ XEN_ERRNO(EISCONN, 106) /* Transport endpoint is already connected */ XEN_ERRNO(ENOTCONN, 107) /* Transport endpoint is not connected */ XEN_ERRNO(ETIMEDOUT, 110) /* Connection timed out */ +XEN_ERRNO(ECONNREFUSED, 111) /* Connection refused */#undef XEN_ERRNO#endif /* XEN_ERRNO */
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 09/25] errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
describes these codes thus:
EMSGSIZE : "Message too large"
ECONNREFUSED : "Connection refused".
If you were to go solely by what POSIX mandates to have, more additions would be necessary afaict. We had limited ourselves to some basic set, so selective additions need further rationale put here. The more that for both added error codes the use case in the hypervisor isn't immediately obvious.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 09/25] errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
Quoted 7 lines · Jan Beulich
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
describes these codes thus:
EMSGSIZE : "Message too large"
ECONNREFUSED : "Connection refused".
Quoted 5 lines
If you were to go solely by what POSIX mandates to have, more additions would be necessary afaict. We had limited ourselves to some basic set, so selective additions need further rationale put here. The more that for both added error codes the use case in the hypervisor isn't immediately obvious.
Thanks for reviewing the series and the previous iterations of this work.
I note your other message indicating a preference for including these changes at point of first use and I will do so in the next revision.
An aside before the rationales below: part of the motivation for selection of these error codes is to continue alignment with the modern v4v implementation in uxen where possible.
EMSGSIZE:
This series proposes to return EMSGSIZE for a sendv operation (patch #15) where an excess amount of data, across all iovs, has been supplied, exceeding either the statically configured maximum size of a transmittable message, or the (variable) size of the ring registered by another domain.
If the new code EMSGSIZE is not wanted, an alternative error code could be EINVAL, though that is returned for other errors in the same operation, such as supplying incorrectly sized individual iovs.
ECONNREFUSED:
This series proposes to return ECONNREFUSED whenever a remote domain is specified that either does not exist or is not argo-enabled. This affects both the ring registration and sending data operations. (register op, patch #13; sendv op, patch #15)
ECONNREFUSED seems plausible for this use as it is determined by the remote domain state within the hypervisor.
Elsewhere, ENODEV is already used to indicate that the local sending domain cannot perform the operation due to its own state. ENOENT is used in the unregister operation to indicate that the domain's own ring that it is attempting to unregister is not present.
ENXIO could work; ECONNREFUSED just seems more descriptive.
Are you OK with these new codes if they are used as described here, and melded into the patches which first use them?
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 09/25] errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 04.12.18 at 10:10, <[email protected]> wrote:
Quoted 7 lines · Jan Beulich
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
describes these codes thus:
EMSGSIZE : "Message too large"
ECONNREFUSED : "Connection refused".Quoted 5 lines
If you were to go solely by what POSIX mandates to have, more additions would be necessary afaict. We had limited ourselves to some basic set, so selective additions need further rationale put here. The more that for both added error codes the use case in the hypervisor isn't immediately obvious.
Quoted 4 lines
Thanks for reviewing the series and the previous iterations of this work.
I note your other message indicating a preference for including these changes at point of first use and I will do so in the next revision.
Actually, for the error code additions here I wouldn't insist on them getting folded into patches using them, as long as it is explained well here why the additions are desired.
Quoted 11 lines
An aside before the rationales below: part of the motivation for selection of these error codes is to continue alignment with the modern v4v implementation in uxen where possible.
EMSGSIZE:
This series proposes to return EMSGSIZE for a sendv operation (patch #15) where an excess amount of data, across all iovs, has been supplied, exceeding either the statically configured maximum size of a transmittable message, or the (variable) size of the ring registered by another domain.
Ah yes, for a send-like operation I can see its use.
Quoted 9 lines
ECONNREFUSED:
This series proposes to return ECONNREFUSED whenever a remote domain is specified that either does not exist or is not argo-enabled. This affects both the ring registration and sending data operations. (register op, patch #13; sendv op, patch #15)
ECONNREFUSED seems plausible for this use as it is determined by the remote domain state within the hypervisor.
Makes sense for the not argo-enabled case. The domain not existing case, however, is nothing argo-specific, and we use a pretty consistent -ESRCH in such cases, I think.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 10/25 arm: introduce guest_handle_for_field()
In reply to Christopher Clarkarm port of commit bb544585137259545d4adc9afe6eed8dc7c7376d
This helper turns a field of a GUEST_HANDLE into a GUEST_HANDLE.
--- xen/include/asm-arm/guest_access.h | 3 +++ 1 file changed, 3 insertions(+)
xen/include/asm-arm/guest_access.h +3 -0
@@ -68,6 +68,9 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, _y; \ }) +#define guest_handle_for_field (hnd, type, fld) \ + ((XEN_GUEST_HANDLE(type)) { &(hnd).p->fld }) +#define guest_handle_from_ptr (ptr, type) \ ((XEN_GUEST_HANDLE_PARAM(type)) { (type *)ptr })#define const_guest_handle_from_ptr (ptr, type) \
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 10/25] arm: introduce guest_handle_for_field()
Quoted 12 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Paul Durrant <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 10/25] arm: introduceguest_handle_for_field()
Quoted 1 line
arm port of commit
bb544585137259545d4adc9afe6eed8dc7c7376d
Capitalize 'arm'? Also maybe shorten the quoted hash, and you should include the commit title here.
Quoted 4 lines
This helper turns a field of a
GUEST_HANDLEinto aGUEST_HANDLE.Signed-off-by Christopher Clark <[email protected]>
The code looks fine, so with a revised comment...
Quoted 19 lines
--- xen/include/asm-arm/guest_access.h | 3 +++ 1 file changed, 3 insertions(+)xen/include/asm- renamed from xen/include/asm-arm/guest_access.h +0
-0
arm/guest_access.h
index 7b6f89c..1137c54 100644xen/include/asm-arm/guest_access.h +3
-0@@ -68,6 +68,9 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, _y; \ }) +#define guest_handle_for_field (hnd, type, fld) \ + ((XEN_GUEST_HANDLE(type)) { &(hnd).p->fld }) +#define guest_handle_from_ptr (ptr, type) \ ((XEN_GUEST_HANDLE_PARAM(type)) { (type *)ptr })
Quoted 3 lines
#define
const_guest_handle_from_ptr(ptr, type)\
--
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 11/25 xsm, argo: XSM control for argo register operation, argo_mac bootparam
In reply to Christopher ClarkXSM hooks implement distinct permissions for these two distinct cases of Argo ring registration:
* Single source: registering a ring for communication to receive messages from a specified single other domain. Default policy: allow.
* Any source: registering a ring for communication to receive messages
from any, or all, other domains (ie. wildcard).
Default policy: deny, with runtime policy configuration via new bootparam.
The reason why the default for wildcard rings is 'deny' is that there is currently no means other than XSM to protect the ring from DoS by a noisy domain spamming the ring, reducing the ability of other domains to send to it. Using XSM at least allows per-domain control over access to the send permission, to limit communication to domains that can be trusted.
Since denying access to any-sender rings unless a flask XSM policy is active will prevent many users from using a key Argo feature, also introduce a bootparam that can override this constraint:
"argo_mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order to make it obvious to an administrator that this has potential security impact.
--- xen/common/argo.c | 15 +++++++++++++++ xen/include/xsm/dummy.h | 15 +++++++++++++++ xen/include/xsm/xsm.h | 17 +++++++++++++++++ xen/xsm/dummy.c | 4 ++++ xen/xsm/flask/hooks.c | 19 +++++++++++++++++++ xen/xsm/flask/policy/access_vectors | 11 +++++++++++ xen/xsm/flask/policy/security_classes | 1 + 7 files changed, 82 insertions(+)
xen/common/argo.c +15 -0
@@ -32,6 +32,21 @@ DEFINE_XEN_GUEST_HANDLE(argo_ring_t);static bool __read_mostly opt_argo_enabled =0 ;boolean_param ("argo" , opt_argo_enabled); +/* Xen command line option for conservative or relaxed access control */ +bool __read_mostly argo_mac_bootparam_enforcing = true; + +static int __init parse_argo_mac_param (const char *s) +{ +if ( !strncmp (s,"enforcing" ,10 ) ) + argo_mac_bootparam_enforcing = true; +else if ( !strncmp (s,"permissive" ,11 ) ) + argo_mac_bootparam_enforcing = false; +else +return -EINVAL ; +return 0 ; +} +custom_param ("argo_mac" , parse_argo_mac_param); +struct argo_pending_ent {struct hlist_node node;
xen/include/xsm/dummy.h +15 -0
@@ -720,6 +720,21 @@ static XSM_INLINE int xsm_dm_op(XSM_DEFAULT_ARG struct domain *d)#endif /* CONFIG_X86 */ +#ifdef CONFIG_ARGO +static XSM_INLINE int xsm_argo_register_single_source(struct domain *d, +struct domain *t) +{ +return 0 ; +} + +static XSM_INLINE int xsm_argo_register_any_source(struct domain *d, +bool strict) +{ +return strict ? -EPERM :0 ; +} + +#endif /* CONFIG_ARGO */ +#include <public/version.h> static XSM_INLINE int xsm_xen_version (XSM_DEFAULT_ARG uint32_t op) {
xen/include/xsm/xsm.h +17 -0
@@ -181,6 +181,10 @@ struct xsm_operations {#endif int (*xen_version) (uint32_t cmd);int (*domain_resource_map) (struct domain *d); +#ifdef CONFIG_ARGO +int (*argo_register_single_source) (struct domain *d,struct domain *t); +int (*argo_register_any_source) (struct domain *d); +#endif };#ifdef CONFIG_XSM
@@ -698,6 +702,19 @@ static inline int xsm_domain_resource_map(xsm_default_t def, struct domain *d)return xsm_ops->domain_resource_map (d); } +#ifdef CONFIG_ARGO +static inline xsm_argo_register_single_source(struct domain *d, struct domain *t) +{ + return xsm_ops->argo_register_single_source(d, t); +} + +static inline xsm_argo_register_any_source(struct domain *d, bool strict) +{ +return xsm_ops->argo_register_any_source (d); +} + +#endif /* CONFIG_ARGO */ +#endif /* XSM_NO_WRAPPERS */ #ifdef CONFIG_MULTIBOOT
xen/xsm/dummy.c +4 -0
@@ -152,4 +152,8 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)#endif set_to_dummy_if_null (ops, xen_version);set_to_dummy_if_null (ops, domain_resource_map); +#ifdef CONFIG_ARGO +set_to_dummy_if_null (ops, argo_register_single_source); +set_to_dummy_if_null (ops, argo_register_any_source); +#endif }
xen/xsm/flask/hooks.c +19 -0
@@ -1717,6 +1717,21 @@ static int flask_domain_resource_map(struct domain *d)return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__RESOURCE_MAP ); } +#ifdef CONFIG_ARGO +static int flask_argo_register_single_source (struct domain *d, +struct domain *t) +{ +return domain_has_perm (d, t,SECCLASS_ARGO , +ARGO__REGISTER_SINGLE_SOURCE ); +} + +static int flask_argo_register_any_source (struct domain *d) +{ +return avc_has_perm (domain_sid (d),SECINITSID_XEN ,SECCLASS_ARGO , +ARGO__REGISTER_ANY_SOURCE ,NULL ); +} +#endif +long do_flask_op (XEN_GUEST_HANDLE_PARAM (xsm_op_t) u_flask_op);int compat_flask_op (XEN_GUEST_HANDLE_PARAM (xsm_op_t) u_flask_op);
@@ -1851,6 +1866,10 @@ static struct xsm_operations flask_ops = {#endif .xen_version = flask_xen_version, .domain_resource_map = flask_domain_resource_map, +#ifdef CONFIG_ARGO + .argo_register_single_source = flask_argo_register_single_source, + .argo_register_any_source = flask_argo_register_any_source, +#endif };void __init flask_init (const void *policy_buffer,size_t policy_size)
xen/xsm/flask/policy/access_vectors +11 -0
@@ -531,3 +531,14 @@ class version # Xen build id xen_build_id } + +# Class argo is used to describe the Argo interdomain communication system. +class argo +{ + # Domain requesting registration of a communication ring + # to receive messages from a specific other domain. + register_single_source + # Domain requesting registration of a communication ring + # to receive messages from any other domain. + register_any_source +}
xen/xsm/flask/policy/security_classes +1 -0
@@ -19,5 +19,6 @@ class event class grant class security class version +class argo # FLASK
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 11/25] xsm, argo: XSM control for argo register operation, argo_mac bootparam
Quoted 66 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Daniel De Graaf <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 11/25] xsm, argo: XSM control for argo register operation,argo_macbootparamXSM hooks implement distinct permissions for these two distinct cases of Argo ring registration:
* Single source: registering a ring for communication to receive messages from a specified single other domain. Default policy: allow.
* Any source: registering a ring for communication to receive messages
from any, or all, other domains (ie. wildcard).
Default policy: deny, with runtime policy configuration via new
bootparam.The reason why the default for wildcard rings is 'deny' is that there is currently no means other than XSM to protect the ring from DoS by a noisy domain spamming the ring, reducing the ability of other domains to send to it.
Using XSM at least allows per-domain control over access to the send permission, to limit communication to domains that can be trusted.Since denying access to any-sender rings unless a flask XSM policy is active
will prevent many users from using a key Argo feature, also introduce a bootparam
that can override this constraint:
"argo_mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order
to make it obvious to an administrator that this has potential security impact.Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 15 +++++++++++++++ xen/include/xsm/dummy.h | 15 +++++++++++++++ xen/include/xsm/xsm.h | 17 +++++++++++++++++ xen/xsm/dummy.c | 4 ++++ xen/xsm/flask/hooks.c | 19 +++++++++++++++++++ xen/xsm/flask/policy/access_vectors | 11 +++++++++++ xen/xsm/flask/policy/security_classes | 1 + 7 files changed, 82 insertions(+)xen/common/argo.c +0
-0@@ -32,6 +32,21 @@ DEFINE_XEN_GUEST_HANDLE(argo_ring_t);static bool __read_mostly opt_argo_enabled =0 ;boolean_param ("argo" , opt_argo_enabled);
xen/common/argo.c DEFINE_XEN_GUEST_HANDLE(argo_ring_t); · line 44
Quoted 10 lines
xen/common/argo.c +10
-0+/* Xen command line option for conservative or relaxed access control */ +bool __read_mostly argo_mac_bootparam_enforcing = true; + +static int __init parse_argo_mac_param (const char *s) +{ +if ( !strncmp (s,"enforcing" ,10 ) ) + argo_mac_bootparam_enforcing = true; +else if ( !strncmp (s,"permissive" ,11 ) ) + argo_mac_bootparam_enforcing = false; +else
Do you really want to parse e.g. 'enforcingfoobar' as 'enforcing'?
Paul
Quoted 158 lines
xen/common/argo.c +5
-0+return -EINVAL ; +return 0 ; +} +custom_param ("argo_mac" , parse_argo_mac_param); +struct argo_pending_ent {struct hlist_node node;xen/include/xsm/dummy.h +15
-0@@ -720,6 +720,21 @@ static XSM_INLINE int xsm_dm_op(XSM_DEFAULT_ARG struct domain *d)#endif /* CONFIG_X86 */ +#ifdef CONFIG_ARGO +static XSM_INLINE int xsm_argo_register_single_source(struct domain *d, +struct domain *t) +{ +return 0 ; +} + +static XSM_INLINE int xsm_argo_register_any_source(struct domain *d, +bool strict) +{ +return strict ? -EPERM :0 ; +} + +#endif /* CONFIG_ARGO */ +#include <public/version.h> static XSM_INLINE int xsm_xen_version (XSM_DEFAULT_ARG uint32_t op){
xen/include/xsm/xsm.h +4
-0@@ -181,6 +181,10 @@ struct xsm_operations {#endif int (*xen_version) (uint32_t cmd);int (*domain_resource_map) (struct domain *d); +#ifdef CONFIG_ARGO +int (*argo_register_single_source) (struct domain *d,struct domain *t); +int (*argo_register_any_source) (struct domain *d); +#endif };#ifdef
CONFIG_XSMxen/include/xsm/xsm.h +13
-0@@ -698,6 +702,19 @@ static inline int xsm_domain_resource_map(xsm_default_t def, struct domain *d) return xsm_ops->domain_resource_map(d); } +#ifdef CONFIG_ARGO +static inline xsm_argo_register_single_source(struct domain *d, struct domain *t) +{ + return xsm_ops->argo_register_single_source(d, t); +} + +static inline xsm_argo_register_any_source(struct domain *d, bool strict) +{ +return xsm_ops->argo_register_any_source (d); +} + +#endif /* CONFIG_ARGO */ +#endif /* XSM_NO_WRAPPERS */ #ifdef
CONFIG_MULTIBOOTxen/xsm/dummy.c +4
-0@@ -152,4 +152,8 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)#endif set_to_dummy_if_null (ops, xen_version);set_to_dummy_if_null (ops, domain_resource_map); +#ifdef CONFIG_ARGO +set_to_dummy_if_null (ops, argo_register_single_source); +set_to_dummy_if_null (ops, argo_register_any_source); +#endif }xen/xsm/flask/hooks.c +19
-0@@ -1717,6 +1717,21 @@ static int flask_domain_resource_map(struct domain *d)return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__RESOURCE_MAP ); } +#ifdef CONFIG_ARGO +static int flask_argo_register_single_source (struct domain *d, +struct domain *t) +{ +return domain_has_perm (d, t,SECCLASS_ARGO , +ARGO__REGISTER_SINGLE_SOURCE ); +} + +static int flask_argo_register_any_source (struct domain *d) +{ +return avc_has_perm (domain_sid (d),SECINITSID_XEN ,SECCLASS_ARGO , +ARGO__REGISTER_ANY_SOURCE ,NULL ); +} +#endif +long do_flask_op (XEN_GUEST_HANDLE_PARAM (xsm_op_t) u_flask_op);int compat_flask_op (XEN_GUEST_HANDLE_PARAM (xsm_op_t) u_flask_op);@@ -1851,6 +1866,10 @@ static struct xsm_operations flask_ops = {#endif .xen_version = flask_xen_version, .domain_resource_map = flask_domain_resource_map, +#ifdef CONFIG_ARGO + .argo_register_single_source = flask_argo_register_single_source, + .argo_register_any_source = flask_argo_register_any_source, +#endif };void __init flask_init (const void *policy_buffer,size_t policy_size)xen/xsm/flask/policy/access_vectors +2
-0@@ -531,3 +531,14 @@ class version # Xen build id xen_build_id } + +# Class argo is used to describe the Argo interdomain communicationsystem.
xen/xsm/flask/policy/access_vectors +9
-0+class argo +{ + # Domain requesting registration of a communication ring + # to receive messages from a specific other domain. + register_single_source + # Domain requesting registration of a communication ring + # to receive messages from any other domain. + register_any_source +}xen/xsm/flask/policy/security_classes +1
-1@@ -19,5 +19,6 @@ class event class grant class security class version +class argo # FLASK --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 11/25] xsm, argo: XSM control for argo register operation, argo_mac bootparam
Quoted 67 lines · Paul Durrant
Quoted 67 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Daniel De Graaf <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 11/25] xsm, argo: XSM control for argo register operation,argo_macbootparamXSM hooks implement distinct permissions for these two distinct cases of Argo ring registration:
* Single source: registering a ring for communication to receive messages from a specified single other domain. Default policy: allow.
* Any source: registering a ring for communication to receive messages
from any, or all, other domains (ie. wildcard).
Default policy: deny, with runtime policy configuration via new
bootparam.The reason why the default for wildcard rings is 'deny' is that there is currently no means other than XSM to protect the ring from DoS by a noisy domain spamming the ring, reducing the ability of other domains to send to it.
Using XSM at least allows per-domain control over access to the send permission, to limit communication to domains that can be trusted.Since denying access to any-sender rings unless a flask XSM policy is active
will prevent many users from using a key Argo feature, also introduce a bootparam
that can override this constraint:
"argo_mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order
to make it obvious to an administrator that this has potential security impact.Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 15 +++++++++++++++ xen/include/xsm/dummy.h | 15 +++++++++++++++ xen/include/xsm/xsm.h | 17 +++++++++++++++++ xen/xsm/dummy.c | 4 ++++ xen/xsm/flask/hooks.c | 19 +++++++++++++++++++ xen/xsm/flask/policy/access_vectors | 11 +++++++++++ xen/xsm/flask/policy/security_classes | 1 + 7 files changed, 82 insertions(+)xen/common/argo.c +0
-0@@ -32,6 +32,21 @@ DEFINE_XEN_GUEST_HANDLE(argo_ring_t);static bool __read_mostly opt_argo_enabled =0 ;boolean_param ("argo" , opt_argo_enabled);
Quoted 12 lines
xen/common/argo.c DEFINE_XEN_GUEST_HANDLE(argo_ring_t); · line 44
Quoted 10 lines
xen/common/argo.c +10
-0+/* Xen command line option for conservative or relaxed access control */ +bool __read_mostly argo_mac_bootparam_enforcing = true; + +static int __init parse_argo_mac_param (const char *s) +{ +if ( !strncmp (s,"enforcing" ,10 ) ) + argo_mac_bootparam_enforcing = true; +else if ( !strncmp (s,"permissive" ,11 ) ) + argo_mac_bootparam_enforcing = false; +else Do you really want to parse e.g. 'enforcingfoobar' as 'enforcing'?
No, I don't - and it doesn't do that because the number supplied to strncmp is large enough to include comparison of the string terminator too -- but I get the point: strncmp is just confusing and is for no clear benefit, so I've dropped it in favour of strcmp in the next revision.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 12/25 xsm, argo: XSM control for argo message send operation
In reply to Christopher ClarkDefault policy: allow.
--- xen/include/xsm/dummy.h | 5 +++++ xen/include/xsm/xsm.h | 6 ++++++ xen/xsm/dummy.c | 1 + xen/xsm/flask/hooks.c | 7 +++++++ xen/xsm/flask/policy/access_vectors | 2 ++ 5 files changed, 21 insertions(+)
xen/include/xsm/dummy.h +5 -0
@@ -733,6 +733,11 @@ static XSM_INLINE int xsm_argo_register_any_source(struct domain *d,return strict ? -EPERM :0 ; } +static XSM_INLINE int xsm_argo_send(struct domain *d, struct domain *t) +{ +return 0 ; +} +#endif /* CONFIG_ARGO */ #include <public/version.h>
xen/include/xsm/xsm.h +6 -0
@@ -184,6 +184,7 @@ struct xsm_operations {#ifdef CONFIG_ARGO int (*argo_register_single_source) (struct domain *d,struct domain *t);int (*argo_register_any_source) (struct domain *d); +int (*argo_send) (struct domain *d,struct domain *t);#endif };
@@ -713,6 +714,11 @@ static inline xsm_argo_register_any_source(struct domain *d, bool strict)return xsm_ops->argo_register_any_source (d); } +static inline int xsm_argo_send (struct domain *d,struct domain *t) +{ +return xsm_ops->argo_send (d, t); +} +#endif /* CONFIG_ARGO */ #endif /* XSM_NO_WRAPPERS */
xen/xsm/dummy.c +1 -0
@@ -155,5 +155,6 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)#ifdef CONFIG_ARGO set_to_dummy_if_null (ops, argo_register_single_source);set_to_dummy_if_null (ops, argo_register_any_source); +set_to_dummy_if_null (ops, argo_send);#endif }
xen/xsm/flask/hooks.c +7 -0
@@ -1730,6 +1730,12 @@ static int flask_argo_register_any_source(struct domain *d)return avc_has_perm (domain_sid (d),SECINITSID_XEN ,SECCLASS_ARGO , ARGO__REGISTER_ANY_SOURCE, NULL); } + +static int flask_argo_send (struct domain *d,struct domain *t) +{ +return domain_has_perm (d, t,SECCLASS_ARGO ,ARGO__SEND ); +} +#endif long do_flask_op (XEN_GUEST_HANDLE_PARAM (xsm_op_t) u_flask_op);
@@ -1869,6 +1875,7 @@ static struct xsm_operations flask_ops = {#ifdef CONFIG_ARGO .argo_register_single_source = flask_argo_register_single_source, .argo_register_any_source = flask_argo_register_any_source, + .argo_send = flask_argo_send,#endif };
xen/xsm/flask/policy/access_vectors +2 -0
@@ -541,4 +541,6 @@ class argo # Domain requesting registration of a communication ring # to receive messages from any other domain. register_any_source + # Domain sending a message to another domain. + send }
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 12/25] xsm, argo: XSM control for argo message send operation
Quoted 13 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Daniel De Graaf <[email protected]>; Paul Durrant <[email protected]>; Stefano Stabellini <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 12/25] xsm, argo: XSM control for argo message send operation
Quoted 3 lines
Default policy: allow.
Signed-off-by Christopher Clark <[email protected]>
Quoted 101 lines
--- xen/include/xsm/dummy.h | 5 +++++ xen/include/xsm/xsm.h | 6 ++++++ xen/xsm/dummy.c | 1 + xen/xsm/flask/hooks.c | 7 +++++++ xen/xsm/flask/policy/access_vectors | 2 ++ 5 files changed, 21 insertions(+)xen/include/xsm/dummy.h +5
-0@@ -733,6 +733,11 @@ static XSM_INLINE int xsm_argo_register_any_source(struct domain *d, return strict ? -EPERM : 0; } +static XSM_INLINE int xsm_argo_send(struct domain *d, struct domain *t) +{ +return 0 ; +} +#endif /* CONFIG_ARGO */ #include <
public/version.h>xen/include/xsm/xsm.h +6
-0@@ -184,6 +184,7 @@ struct xsm_operations {#ifdef CONFIG_ARGO int (*argo_register_single_source) (struct domain *d,struct domain *t);int (*argo_register_any_source) (struct domain *d); +int (*argo_send) (struct domain *d,struct domain *t);#endif };@@ -713,6 +714,11 @@ static inline xsm_argo_register_any_source(struct domain *d, bool strict) return xsm_ops->argo_register_any_source(d); } +static inline int xsm_argo_send (struct domain *d,struct domain *t) +{ +return xsm_ops->argo_send (d, t); +} +#endif /* CONFIG_ARGO */ #endif /*
XSM_NO_WRAPPERS*/xen/xsm/dummy.c +1
-0@@ -155,5 +155,6 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)#ifdef CONFIG_ARGO set_to_dummy_if_null (ops, argo_register_single_source);set_to_dummy_if_null (ops, argo_register_any_source); +set_to_dummy_if_null (ops, argo_send);#endif }xen/xsm/flask/hooks.c +6
-0@@ -1730,6 +1730,12 @@ static int flask_argo_register_any_source(struct domain *d)return avc_has_perm (domain_sid (d),SECINITSID_XEN ,SECCLASS_ARGO , ARGO__REGISTER_ANY_SOURCE, NULL); } + +static int flask_argo_send (struct domain *d,struct domain *t) +{ +return domain_has_perm (d, t,SECCLASS_ARGO ,ARGO__SEND ); +} +#endif long
do_flask_op(XEN_GUEST_HANDLE_PARAM(xsm_op_t) u_flask_op);xen/xsm/flask/hooks.c +1
-0@@ -1869,6 +1875,7 @@ static struct xsm_operations flask_ops = {#ifdef CONFIG_ARGO .argo_register_single_source = flask_argo_register_single_source, .argo_register_any_source = flask_argo_register_any_source, + .argo_send = flask_argo_send,#endif };xen/xsm/flask/policy/access_vectors +2
-1@@ -541,4 +541,6 @@ class argo # Domain requesting registration of a communication ring # to receive messages from any other domain. register_any_source + # Domain sending a message to another domain. + send } --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 13/25 argo: implement the register op
In reply to Christopher ClarkUsed by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
In this code, the p2m type of the memory supplied by the guest for the ring must be p2m_ram_rw, which is a conservative choice made to defer the need to reason about the other p2m types with this commit.
argo_pfn_t type is introduced here to create a pfn_t type that is 64-bit on all architectures, to assist with avoiding the need to add a compat ABI.
--- xen/common/argo.c | 498 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/guest_access.h | 2 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 64 +++++ 4 files changed, 566 insertions(+)
xen/common/argo.c +498 -0
@@ -25,6 +25,7 @@#include <xen/guest_access.h> #include <xen/time.h> +DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t);
@@ -98,6 +99,25 @@ struct argo_domain };/* +* Helper functions +*/ + +static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id) +{ +uint16_t ret; + + ret = (uint16_t )(id->addr.port >>16 ); + ret ^= (uint16_t )id->addr.port; + ret ^= id->addr.domain_id; + ret ^= id->partner; + + ret &= (ARGO_HTABLE_SIZE -1 ); + +return ret; +} + +/* * locks */
@@ -171,6 +191,74 @@ argo_ring_unmap(struct argo_ring_info *ring_info) } } +/* caller must have L3 or W(L2) */ +static int +argo_ring_map_page (struct argo_ring_info *ring_info,uint32_t i, +uint8_t **page) +{ +if ( i >= ring_info->nmfns ) + { +printk (XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ; + } +ASSERT (ring_info->mfns); +ASSERT (ring_info->mfn_mapping); + +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read and +* write access to the page is required by the hypervisor, but read-access +* is not needed for this mapping for the remainder of the ring. +* Since this mapping will remain resident in Xen's address space for +* the lifetime of the ring, and following the principle of least privilege, +* it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as +* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT). +*/ + ring_info->mfn_mapping[i] =map_domain_page_global (ring_info->mfns[i]); + +if ( !ring_info->mfn_mapping[i] ) + { +printk (XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ; + } +argo_dprintk ("mapping page %" PRI_mfn" to %p\n" , +mfn_x (ring_info->mfns[i]), ring_info->mfn_mapping[i]); + } + +if ( page ) + *page = ring_info->mfn_mapping[i]; +return 0 ; +} + +/* caller must have L3 or W(L2) */ +static int +argo_update_tx_ptr (struct argo_ring_info *ring_info,uint32_t tx_ptr) +{ +uint8_t *dst; +uint32_t *p; +int ret; + + ret =argo_ring_map_page (ring_info,0 , &dst); +if ( ret ) +return ret; + + p = (uint32_t *)(dst + offsetof(argo_ring_t , tx_ptr)); +write_atomic (p, tx_ptr); +mb (); +return 0 ; +} +/* * pending */
@@ -231,6 +319,388 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info); } +/* +* ring +*/ + +static int +argo_find_ring_mfn (struct domain *d,argo_pfn_t pfn,mfn_t *mfn) +{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d, pfn, &p2mt); +#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn); +#endif + +return ret; +} + +static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; + +if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ +if ( ring_info->npage != npage ) + i = ring_info->nmfns +1 ;/* forces re-register below */ +else + { +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) +break ; + +if (mfn_x (mfn) !=mfn_x (ring_info->mfns[i]) ) +break ; + } + } +if ( i != ring_info->nmfns ) + { +printk (XENLOG_INFO "argo: vm%u re-registering existing argo ring" +" (vm%u:%x vm%d), clearing MFN list\n" , + current->domain->domain_id, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +argo_ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + } + +if ( !ring_info->mfns ) + { +mfn_t *mfns; +uint8_t **mfn_mapping; + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xmalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + } +ASSERT (ring_info->npage == npage); + +if ( ring_info->nmfns == ring_info->npage ) +return 0 ; + +for ( i = ring_info->nmfns; i < ring_info->npage; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) + { +printk (XENLOG_ERR "argo: vm%u passed invalid gpfn %" PRI_xen_pfn +" ring (vm%u:%x vm%d) %p seq %d of %d\n" , + d->domain_id, pfn, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, + ring_info, i, ring_info->npage); +break ; + } + + ring_info->mfns[i] = mfn; + ring_info->nmfns = i +1 ; + +argo_dprintk ("%d: %" PRI_xen_pfn" -> %" PRI_mfn"\n" , + i, pfn,mfn_x (ring_info->mfns[i])); + + ring_info->mfn_mapping[i] =NULL ; + } + +if ( ret ) +argo_ring_remove_mfns (d, ring_info); +else + { +ASSERT (ring_info->nmfns == ring_info->npage); + +printk (XENLOG_ERR "argo: vm%u ring (vm%u:%x vm%d) %p mfn_mapping %p" +" npage %d nmfns %d\n" , current->domain->domain_id, + ring_info->id.addr.domain_id, ring_info->id.addr.port, + ring_info->id.partner, ring_info, ring_info->mfn_mapping, + ring_info->npage, ring_info->nmfns); + } +return ret; +} + +static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + hash =argo_hash_fn (id); + +argo_dprintk ("d->argo=%p, d->argo->ring_hash[%d]=%p id=%p\n" , + d->argo, hash, d->argo->ring_hash[hash].first, id); +argo_dprintk ("id.addr.port=%d id.addr.domain=vm%u" +" id.addr.partner=vm%d\n" , + id->addr.port, id->addr.domain_id, id->partner); + +hlist_for_each_entry (ring_info, node, &d->argo->ring_hash[hash], node) + { +argo_ring_id_t *cmpid = &ring_info->id; + +if ( cmpid->addr.port == id->addr.port && + cmpid->addr.domain_id == id->addr.domain_id && + cmpid->partner == id->partner ) + { +argo_dprintk ("ring_info=%p\n" , ring_info); +return ring_info; + } + } +argo_dprintk ("no ring_info found\n" ); + +return NULL ; +} + +static long +argo_register_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd, +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd,uint32_t npage, +bool fail_exist) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; +bool update_tx_ptr =0 ; +uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ; + +read_lock (&argo_lock); + +do { +if ( !d->argo ) + { + ret = -ENODEV ; +break ; + } + +if (copy_from_guest (&ring, ring_hnd,1 ) ) + { + ret = -EFAULT ; +break ; + } + +if ( ring.magic !=ARGO_RING_MAGIC ) + { + ret = -EINVAL ; +break ; + } + +if ( (ring.len < (sizeof (struct argo_ring_message_header ) + +ARGO_ROUNDUP (1 ) +ARGO_ROUNDUP (1 ))) || + (ARGO_ROUNDUP (ring.len) != ring.len) ) + { + ret = -EINVAL ; +break ; + } + +if ( ring.len >ARGO_MAX_RING_SIZE ) + { + ret = -EINVAL ; +break ; + } + +if ( ring.id.partner ==ARGO_DOMID_ANY ) + { + ret =xsm_argo_register_any_source (d, argo_mac_bootparam_enforcing); +if ( ret ) +break ; + } +else + { +struct domain *dst_d =get_domain_by_id (ring.id.partner); +if ( !dst_d ) + { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_register_single_source (d, dst_d); +if ( ret ) + { +put_domain (dst_d); +break ; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); +break ; + } + + dst_domain_cookie = dst_d->argo->domain_cookie; + +put_domain (dst_d); + } + + ring.id.addr.domain_id = d->domain_id; +if (copy_field_to_guest (ring_hnd, &ring, id) ) + { + ret = -EFAULT ; +break ; + } + +/* +* no need for a lock yet, because only we know about this +* set the tx pointer if it looks bogus (we don't reset it +* because this might be a re-register after S4) +*/ + +if ( ring.tx_ptr >= ring.len || +ARGO_ROUNDUP (ring.tx_ptr) != ring.tx_ptr ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + ring.tx_ptr =ARGO_ROUNDUP (ring.rx_ptr); + +if ( ring.tx_ptr >= ring.len ) + ring.tx_ptr =0 ; + +/* ring.tx_ptr will be written back to the guest ring below. */ + update_tx_ptr =1 ; + } + +/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock); + +do { + ring_info =argo_ring_find_info (d, &ring.id); + +if ( !ring_info ) + { +uint16_t hash; + + ring_info = xmalloc(struct argo_ring_info); +if ( !ring_info ) + { + ret = -ENOMEM ; +break ; + } + +spin_lock_init (&ring_info->lock); + + ring_info->mfns =NULL ; + ring_info->npage =0 ; + ring_info->mfn_mapping =NULL ; + ring_info->len =0 ; + ring_info->nmfns =0 ; + ring_info->tx_ptr =0 ; + ring_info->partner_cookie = dst_domain_cookie; + + ring_info->id = ring.id; +INIT_HLIST_HEAD (&ring_info->pending); + + hash =argo_hash_fn (&ring_info->id); +hlist_add_head (&ring_info->node, &d->argo->ring_hash[hash]); + +printk (XENLOG_INFO "argo: vm%u registering ring (vm%u:%x vm%d)\n" , + current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner); + } +else + { +/* +* If the caller specified that the ring must not already exist, +* fail at attempt to add a completed ring which already exists. +*/ +if ( fail_exist && ring_info->len ) + { + ret = -EEXIST ; +break ; + } + +printk (XENLOG_INFO +"argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n" , + current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner); + } + +/* Since we hold W(L2), there is no need to take L3 here */ + ring_info->tx_ptr = ring.tx_ptr; + + ret =argo_find_ring_mfns (d, ring_info, npage, pfn_hnd, ring.len); +if ( !ret ) + ret = update_tx_ptr ?argo_update_tx_ptr (ring_info, ring.tx_ptr) + :argo_ring_map_page (ring_info,0 ,NULL ); +if ( !ret ) + ring_info->len = ring.len; + + }while (0 ); + +write_unlock (&d->argo->lock); + + }while (0 ); + +read_unlock (&argo_lock); + +return ret; +} + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,
@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ; +if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ; +/* arg4: reserve currently-undefined bits, require zero. */ +if (unlikely (arg4 & ~ARGO_REGISTER_FLAG_MASK ) ) + { + rc = -EINVAL ; +break ; + } + + rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist); +break ; + }default : rc = -ENOSYS ;break ;
xen/include/asm-arm/guest_access.h +2 -0
@@ -34,6 +34,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))
xen/include/asm-x86/guest_access.h +2 -0
@@ -50,6 +50,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))
xen/include/public/argo.h +64 -0
74 lines not on this page · raw message
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 11 lines · Christopher Clark
xen/include/public/argo.h +3
-0@@ -21,6 +21,20 @@#include "xen.h" +#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID
xen/include/public/argo.h line 36
Quoted 10 lines
xen/include/public/argo.h +10
-0+ +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ;
As you always use 64-bit, can we just use an address? This would make the ABI agnostic to the hypervisor page granularity.
Cheers,
Signature · 5 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 12 lines · Julien Grall
Quoted 8 lines · Christopher Clark
xen/include/public/argo.h +0
-0@@ -21,6 +21,20 @@#include "xen.h"
Quoted 16 lines
xen/include/public/argo.h line 36
Quoted 13 lines
xen/include/public/argo.h +13
-0+#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ;As you always use 64-bit, can we just use an address? This would make the ABI agnostic to the hypervisor page granularity.
Thanks for reviewing this series.
I'm not sure yet that switching to using addresses instead would be for the best, so have been working through some reasoning about your suggestion. This interface is for the guest to identify to the hypervisor the list of frames of memory to use as the ring, and the purpose of a frame number is to uniquely identify a frame. Frame numbers, as opposed to addresses, are going to remain the same across all processors, independent of the page tables that happen to currently be in use.
Where possible, translation should be performed by the guest rather than the hypervisor, minimizing the hypervisor logic (good for several reasons) - so it would be better to avoid adding the address-to-page-number walk and granularity handling in the hypervisor here. In this case, the guest has the incentive to do that work, given that it wants to register the ring.
(Slightly out of scope, but hopefully not for long: We have a near-term interest in using argo to communicate between VMs at different levels of nesting in L0/L1 nested hypervisors, and I suspect that frame number translation will end up being easier to handle across L0/L1 than translation of guest addresses in a VM running at the other level.)
Could you give a specific scenario you have in mind that is prompting a concern?
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Hi Christoffer,
Quoted 32 lines · Christopher Clark
Quoted 12 lines · Julien Grall
Quoted 8 lines · Christopher Clark
xen/include/public/argo.h +0
-0@@ -21,6 +21,20 @@#include "xen.h" Quoted 16 lines
xen/include/public/argo.h line 36
Quoted 13 lines
xen/include/public/argo.h +13
-0+#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ;As you always use 64-bit, can we just use an address? This would make the ABI agnostic to the hypervisor page granularity.
Thanks for reviewing this series.
Quoted 8 lines
I'm not sure yet that switching to using addresses instead would be for the best, so have been working through some reasoning about your suggestion. This interface is for the guest to identify to the hypervisor the list of frames of memory to use as the ring, and the purpose of a frame number is to uniquely identify a frame. Frame numbers, as opposed to addresses, are going to remain the same across all processors, independent of the page tables that happen to currently be in use.
Sorry I wasn't clear enough about the address. By address I meant guest physical address (and not guest virtual address).
guest virtual address would indeed be a pretty bad idea as you can't promise the address would stay mapped forever. For a matter of fact, we already see some issues because of (K)PTI.
Quoted 8 lines
Where possible, translation should be performed by the guest rather than the hypervisor, minimizing the hypervisor logic (good for several reasons) - so it would be better to avoid adding the address-to-page-number walk and granularity handling in the hypervisor here. In this case, the guest has the incentive to do that work, given that it wants to register the ring.
Quoted 8 lines
(Slightly out of scope, but hopefully not for long: We have a near-term interest in using argo to communicate between VMs at different levels of nesting in L0/L1 nested hypervisors, and I suspect that frame number translation will end up being easier to handle across L0/L1 than translation of guest addresses in a VM running at the other level.)
Could you give a specific scenario you have in mind that is prompting a concern?
Arm processors may support multiple page granularity (4KB, 16KB, 64KB). The software is allowed to use different granularity at different level. This means that the hypervisor could use 4KB page while the guest kernel would use 64KB page (and vice versa). Some distros made the choice to only support one type of page granularity (i.e 64KB for REHL, 4KB for Debian...).
At the moment the hypercall interface is based on the hypervisor page granularity. Because Xen has always supported 4KB page-granularity, this assumption was also hardcoded in the kernel.
What prevent us to get 64KB page support in Xen (and therefore support for 52-bit address) is the hypercall ABI. If you upgrade Xen to 64KB then the hypercall interface would defact use 64KB frame. This would break any current guest. It is also not possible to keep 4KB pages everywhere because you can only map 64KB in Xen. So you may map a bit too much from another guest.
This makes me think that the frame is probably not the best in that situation. Instead a pair of address/size would be more suitable.
The problem is much larger than this series. But I thought I would attempt to convince the community using guest physical address over guest frame address whenever it is possible.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 29 lines · Julien Grall
Quoted 9 lines · Christopher Clark
Quoted 8 lines · Julien Grall
xen/include/public/argo.h line 36
Quoted 4 lines · Christopher Clark
xen/include/public/argo.h +2
-0+/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ;As you always use 64-bit, can we just use an address? This would make the ABI agnostic to the hypervisor page granularity.
By address I meant guest physical address (and not guest virtual address).
Arm processors may support multiple page granularity (4KB, 16KB, 64KB). The software is allowed to use different granularity at different level. This means that the hypervisor could use 4KB page while the guest kernel would use 64KB page (and vice versa). Some distros made the choice to only support one type of page granularity (i.e 64KB for REHL, 4KB for Debian...).
At the moment the hypercall interface is based on the hypervisor page granularity. Because Xen has always supported 4KB page-granularity, this assumption was also hardcoded in the kernel.
What prevent us to get 64KB page support in Xen (and therefore support for 52-bit address) is the hypercall ABI. If you upgrade Xen to 64KB then the hypercall interface would defact use 64KB frame. This would break any current guest. It is also not possible to keep 4KB pages everywhere because you can only map 64KB in Xen. So you may map a bit too much from another guest.
Quoted 6 lines
This makes me think that the frame is probably not the best in that situation. Instead a pair of address/size would be more suitable.
The problem is much larger than this series. But I thought I would attempt to convince the community using guest physical address over guest frame address whenever it is possible.
Thanks, Julien -- that explanation is very helpful and your request makes sense.
So in concrete terms, with the change that you're advocating for to this patch, the 64-bit value that is supplied by the guest in the array passed as an argument to register_ring would encode the same guest physical frame number as it currently does in the patch version presented in this thread, but it would be bit-shifted to the position used in a physical address.
In addition to that change, a page size indicator would be supplied too -- for every page address supplied in the call.
Is there a method currently used within Xen (or relevant places elsewhere) for encoding both the page address and size (ie. 4KB, 16KB or 64KB) within the same 64-bits?
ie. Knowing that the smallest granularity of page is 4KB, and that all pages are aligned to at least a 4KB boundary, there are low bits in the address that are known to be zero, and those could be used to indicate the page size when supplied to this call. It seems like such an encoding would allow for avoiding doubling the size of the argument array, but I'm not sure how inconvenient it would be to work with in practice.
If so, such an interface change looks manageable and hopefully it would be acceptable to only support 4KB pages in the current implementation behind that new ABI for the time being. Let me know what you think.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Hi Christoffer,
Quoted 60 lines · Christopher Clark
Quoted 29 lines · Julien Grall
Quoted 9 lines · Christopher Clark
Quoted 8 lines · Julien Grall
xen/include/public/argo.h line 36
Quoted 4 lines · Christopher Clark
xen/include/public/argo.h +2
-0+/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ;As you always use 64-bit, can we just use an address? This would make the ABI agnostic to the hypervisor page granularity.
By address I meant guest physical address (and not guest virtual address).
Arm processors may support multiple page granularity (4KB, 16KB, 64KB). The software is allowed to use different granularity at different level. This means that the hypervisor could use 4KB page while the guest kernel would use 64KB page (and vice versa). Some distros made the choice to only support one type of page granularity (i.e 64KB for REHL, 4KB for Debian...).
At the moment the hypercall interface is based on the hypervisor page granularity. Because Xen has always supported 4KB page-granularity, this assumption was also hardcoded in the kernel.
What prevent us to get 64KB page support in Xen (and therefore support for 52-bit address) is the hypercall ABI. If you upgrade Xen to 64KB then the hypercall interface would defact use 64KB frame. This would break any current guest. It is also not possible to keep 4KB pages everywhere because you can only map 64KB in Xen. So you may map a bit too much from another guest.
Quoted 6 lines
This makes me think that the frame is probably not the best in that situation. Instead a pair of address/size would be more suitable.
The problem is much larger than this series. But I thought I would attempt to convince the community using guest physical address over guest frame address whenever it is possible.
Thanks, Julien -- that explanation is very helpful and your request makes sense.
So in concrete terms, with the change that you're advocating for to this patch, the 64-bit value that is supplied by the guest in the array passed as an argument to
register_ringwould encode the same guest physical frame number as it currently does in the patch version presented in this thread, but it would be bit-shifted to the position used in a physical address.In addition to that change, a page size indicator would be supplied too -- for every page address supplied in the call.
Is there a method currently used within Xen (or relevant places elsewhere) for encoding both the page address and size (ie. 4KB, 16KB or 64KB) within the same 64-bits?
ie. Knowing that the smallest granularity of page is 4KB, and that all pages are aligned to at least a 4KB boundary, there are low bits in the address that are known to be zero, and those could be used to indicate the page size when supplied to this call. It seems like such an encoding would allow for avoiding doubling the size of the argument array, but I'm not sure how inconvenient it would be to work with in practice.
Quoted 4 lines
If so, such an interface change looks manageable and hopefully it would be acceptable to only support 4KB pages in the current implementation behind that new ABI for the time being. Let me know what you think.
If you let the user the choice of the granularity, then, I believe, you will prevent the hypervisor to do some optimization.
For instance, if the guest supplies only 4KB page but the hypervisor is 64KB. There are no way to easily map them contiguously in the hypervisor (e.g using vmap).
Is there a particular reason to allow the ring buffer to be non-contiguous in the guest physical address?
Depending on the answer, there are different way to handle that:
1) Request the guest to allocate memory using 64KB (on Arm) chunk and pass the
base address for each chunk
2) Request the guest to allocate contiguously the buffer and pass the base
address and size
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Christopher ClarkQuoted 99 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Roger Pau Monne <[email protected]>; Paul Durrant <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 13/25] argo: implement the register opUsed by a domain to register a region of memory for receiving messages from
either a specified other domain, or, if specifying a wildcard, any domain.This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the
hypervisor will use this mapping to copy data from a sending domain into this
registered ring, making it accessible to the domain that registered the ring to
receive data.In this code, the p2m type of the memory supplied by the guest for the ring
must bep2m_ram_rw, which is a conservative choice made to defer the need to
reason about the other p2m types with this commit.
argo_pfn_ttype is introduced here to create apfn_ttype that is 64-bit on
all architectures, to assist with avoiding the need to add a compat ABI.Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 498 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/guest_access.h | 2 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 64 +++++ 4 files changed, 566 insertions(+)xen/common/argo.c +28
-0@@ -25,6 +25,7 @@#include <xen/guest_access.h> #include <xen/time.h> +DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t);@@ -98,6 +99,25 @@ struct argo_domain };/* +* Helper functions +*/ + +static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id) +{ +uint16_t ret; + + ret = (uint16_t )(id->addr.port >>16 ); + ret ^= (uint16_t )id->addr.port; + ret ^= id->addr.domain_id; + ret ^= id->partner; + + ret &= (ARGO_HTABLE_SIZE -1 ); + +return ret; +} + +/* * locks */ @@ -171,6 +191,74 @@ argo_ring_unmap(struct argo_ring_info *ring_info) } } +/* caller must have L3 or W(L2) */ +static int +argo_ring_map_page(struct argo_ring_info *ring_info, uint32_t i, + uint8_t **page) +{ + if ( i >= ring_info->nmfns ) + { + printk(XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to mappage"
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 205
Quoted 4 lines
xen/common/argo.c +4
-0+" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ;
-ENOMEM? The seems to be the conventional errno to use when a global mapping fails.
Quoted 34 lines
xen/common/argo.c +9
-0+ } +ASSERT (ring_info->mfns); +ASSERT (ring_info->mfn_mapping); + +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read and
+ * write access to the page is required by the hypervisor, but
read-accessxen/common/argo.c +2
-0+* is not needed for this mapping for the remainder of the ring. +* Since this mapping will remain resident in Xen's address space for
+ * the lifetime of the ring, and following the principle of least
privilege,xen/common/argo.c +3
-0+* it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as
xen/common/argo.c +4
-0+* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT). +*/ + ring_info->mfn_mapping[i] = map_domain_page_global(ring_info-Quoted 1 line
mfns[i]);
xen/common/argo.c +4
-0+ + if ( !ring_info->mfn_mapping[i] ) + { + printk(XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted tomap page"
xen/common/argo.c +2
-0+" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner,
ring_info,
xen/common/argo.c
Quoted 2 lines
xen/common/argo.c +2
-0+ i, ring_info->nmfns); +return -EFAULT ;
Same here.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 240
Quoted 7 lines
xen/common/argo.c +7
-0+ } +argo_dprintk ("mapping page %" PRI_mfn" to %p\n" , +mfn_x (ring_info->mfns[i]), ring_info->mfn_mapping[i]); + } + +if ( page ) + *page = ring_info->mfn_mapping[i];
Blank line here.
Quoted 8 lines
xen/common/argo.c +8
-0+return 0 ; +} + +/* caller must have L3 or W(L2) */ +static int +argo_update_tx_ptr (struct argo_ring_info *ring_info,uint32_t tx_ptr) +{ +uint8_t *dst;
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 258
Quoted 10 lines
xen/common/argo.c +10
-0+uint32_t *p; +int ret; + + ret =argo_ring_map_page (ring_info,0 , &dst); +if ( ret ) +return ret; + + p = (uint32_t *)(dst + offsetof(argo_ring_t , tx_ptr)); +write_atomic (p, tx_ptr); +mb ();
Blank line here.
Quoted 42 lines
xen/common/argo.c +33
-0+return 0 ; +} +/* * pending */ @@ -231,6 +319,388 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info ); } +/* +* ring +*/ + +static int +argo_find_ring_mfn (struct domain *d,argo_pfn_t pfn,mfn_t *mfn) +{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d, pfn, &p2mt); +#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn); +#endif +
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct · line 363
Quoted 10 lines
xen/common/argo.c +9
-0+return ret; +} + +static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i;
unsigned?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 363
Quoted 3 lines
xen/common/argo.c +3
-0+int ret =0 ; + +if ( (npage <<PAGE_SHIFT ) < len )
Overflow check?
Quoted 17 lines
xen/common/argo.c +17
-0+return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ +if ( ring_info->npage != npage ) + i = ring_info->nmfns +1 ;/* forces re-register below */ +else + { +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn;
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 390
Quoted 10 lines
xen/common/argo.c +10
-0+ + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) +break ; + +if (mfn_x (mfn) !=mfn_x (ring_info->mfns[i]) )
Use mfn_eq()
Quoted 7 lines
xen/common/argo.c +6
-0+break ; + } + } + if ( i != ring_info->nmfns ) + { + printk(XENLOG_INFO "argo: vm%u re-registering existing argoring"
Would XENLOG_G_WARNING not be more appropriate (or using gprintk with XENLOG_WARNING)?
Quoted 52 lines
xen/common/argo.c +2
-0+" (vm%u:%x vm%d), clearing MFN list\n" , + current->domain->domain_id, ring_info-Quoted 1 line
id.addr.domain_id,
xen/common/argo.c +48
-0+ ring_info->id.addr.port, ring_info->id.partner); + +argo_ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + } + +if ( !ring_info->mfns ) + { +mfn_t *mfns; +uint8_t **mfn_mapping; + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xmalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + } +ASSERT (ring_info->npage == npage); + +if ( ring_info->nmfns == ring_info->npage ) +return 0 ; + + for ( i = ring_info->nmfns; i < ring_info->npage; i++ ) + { + argo_pfn_t pfn; + mfn_t mfn; + + ret = copy_from_guest_offset_errno(&pfn, pfn_hnd, i, 1); + if ( ret ) + break; + + ret = argo_find_ring_mfn(d, pfn, &mfn); + if ( ret ) + { + printk(XENLOG_ERR "argo: vm%u passed invalid gpfn%"PRI_xen_pfn
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 451
Quoted 5 lines
xen/common/argo.c +5
-0+" ring (vm%u:%x vm%d) %p seq %d of %d\n" , + d->domain_id, pfn, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, + ring_info, i, ring_info->npage); +break ;
gprintk()?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 455
Quoted 4 lines
xen/common/argo.c +4
-0+ } + + ring_info->mfns[i] = mfn; + ring_info->nmfns = i +1 ;
Since you don't return from within this loop (only break out), can you not set 'nmfns' to 'i' after the loop terminates rather than (re)setting it on every iteration?
Quoted 14 lines
xen/common/argo.c +14
-0+ +argo_dprintk ("%d: %" PRI_xen_pfn" -> %" PRI_mfn"\n" , + i, pfn,mfn_x (ring_info->mfns[i])); + + ring_info->mfn_mapping[i] =NULL ; + } + +if ( ret ) +argo_ring_remove_mfns (d, ring_info); + else + { + ASSERT(ring_info->nmfns == ring_info->npage); + + printk(XENLOG_ERR "argo: vm%u ring (vm%u:%x vm%d) %p mfn_mapping
Quoted 1 line
%p"
gprintk()?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 474
Quoted 5 lines
xen/common/argo.c +5
-0+" npage %d nmfns %d\n" , current->domain->domain_id, + ring_info->id.addr.domain_id, ring_info->id.addr.port, + ring_info->id.partner, ring_info, ring_info->mfn_mapping, + ring_info->npage, ring_info->nmfns); + }
Blank line here.
Quoted 46 lines
xen/common/argo.c +5
-0+return ret; +} + +static struct argo_ring_info * +argo_ring_find_info(const struct domain *d, const struct argo_ring_id*id)
xen/common/argo.c +16
-0+{ +uint16_t hash; +struct hlist_node *node; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + hash =argo_hash_fn (id); + +argo_dprintk ("d->argo=%p, d->argo->ring_hash[%d]=%p id=%p\n" , + d->argo, hash, d->argo->ring_hash[hash].first, id); +argo_dprintk ("id.addr.port=%d id.addr.domain=vm%u" +" id.addr.partner=vm%d\n" , + id->addr.port, id->addr.domain_id, id->partner); + + hlist_for_each_entry(ring_info, node, &d->argo->ring_hash[hash],node)
xen/common/argo.c +20
-0+ { +argo_ring_id_t *cmpid = &ring_info->id; + +if ( cmpid->addr.port == id->addr.port && + cmpid->addr.domain_id == id->addr.domain_id && + cmpid->partner == id->partner ) + { +argo_dprintk ("ring_info=%p\n" , ring_info); +return ring_info; + } + } +argo_dprintk ("no ring_info found\n" ); + +return NULL ; +} + +static long +argo_register_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd, + XEN_GUEST_HANDLE_PARAM(argo_pfn_t) pfn_hnd, uint32_tnpage,
xen/common/argo.c +2
-0+ bool fail_exist) +{
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 527
Quoted 10 lines
xen/common/argo.c +10
-0+struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; +bool update_tx_ptr =0 ; +uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ; + +read_lock (&argo_lock);
Stray space.
Quoted 38 lines
xen/common/argo.c +37
-0+ + do { + if ( !d->argo ) + { + ret = -ENODEV; + break; + } + + if ( copy_from_guest(&ring, ring_hnd, 1) ) + { + ret = -EFAULT; + break; + } + + if ( ring.magic != ARGO_RING_MAGIC ) + { + ret = -EINVAL; + break; + } + + if ( (ring.len < (sizeof(struct argo_ring_message_header) + + ARGO_ROUNDUP(1) + ARGO_ROUNDUP(1))) || + (ARGO_ROUNDUP(ring.len) != ring.len) ) + { + ret = -EINVAL; + break; + } + + if ( ring.len > ARGO_MAX_RING_SIZE ) + { + ret = -EINVAL; + break; + } + + if ( ring.id.partner == ARGO_DOMID_ANY ) + { + ret = xsm_argo_register_any_source(d,
argo_mac_bootparam_enforcing);
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 570
Quoted 6 lines
xen/common/argo.c +6
-0+if ( ret ) +break ; + } +else + { +struct domain *dst_d =get_domain_by_id (ring.id.partner);
Blank line here.
Quoted 96 lines
xen/common/argo.c +45
-0+if ( !dst_d ) + { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_register_single_source (d, dst_d); +if ( ret ) + { +put_domain (dst_d); +break ; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); +break ; + } + + dst_domain_cookie = dst_d->argo->domain_cookie; + +put_domain (dst_d); + } + + ring.id.addr.domain_id = d->domain_id; +if (copy_field_to_guest (ring_hnd, &ring, id) ) + { + ret = -EFAULT ; +break ; + } + +/* +* no need for a lock yet, because only we know about this +* set the tx pointer if it looks bogus (we don't reset it +* because this might be a re-register after S4) +*/ + +if ( ring.tx_ptr >= ring.len || +ARGO_ROUNDUP (ring.tx_ptr) != ring.tx_ptr ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it
+ * here by setting thetx_ptrto the next aligned message
slot past
+ * the latestrx_ptrwe have observed. Handle ring wrap
correctly.xen/common/argo.c +7
-0+*/ + ring.tx_ptr =ARGO_ROUNDUP (ring.rx_ptr); + +if ( ring.tx_ptr >= ring.len ) + ring.tx_ptr =0 ; + +/* ring.tx_ptr will be written back to the guest ring below. */
xen/common/argo.c +35
-0+ update_tx_ptr =1 ; + } + +/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock); + + do { + ring_info = argo_ring_find_info(d, &ring.id); + + if ( !ring_info ) + { + uint16_t hash; + + ring_info = xmalloc(struct argo_ring_info); + if ( !ring_info ) + { + ret = -ENOMEM; + break; + } + + spin_lock_init(&ring_info->lock); + + ring_info->mfns = NULL; + ring_info->npage = 0; + ring_info->mfn_mapping = NULL; + ring_info->len = 0; + ring_info->nmfns = 0; + ring_info->tx_ptr = 0; + ring_info->partner_cookie = dst_domain_cookie; + + ring_info->id = ring.id; + INIT_HLIST_HEAD(&ring_info->pending); + + hash = argo_hash_fn(&ring_info->id); + hlist_add_head(&ring_info->node, &d->argo-Quoted 1 line
ring_hash[hash]);
xen/common/argo.c +2
-0+ + printk(XENLOG_INFO "argo: vm%u registering ring (vm%u:%x
Quoted 4 lines
vm%d)\n",
+current->domain->domain_id,
ring.id.addr.domain_id,
+ ring.id.addr.port, ring.id.partner);
Unqualified XENLOG_INFO printk()... do we really want this?
Quoted 18 lines
xen/common/argo.c +5
-0+ } +else + { +/* +* If the caller specified that the ring must not already exist,
+ * fail at attempt to add a completed ring which already
exists.xen/common/argo.c +9
-0+*/ +if ( fail_exist && ring_info->len ) + { + ret = -EEXIST ; +break ; + } + + printk(XENLOG_INFO + "argo: vm%u re-registering existing ring (vm%u:%xvm%d)\n",
xen/common/argo.c
Quoted 2 lines
xen/common/argo.c +2
-0+ current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner);
Same here.
Quoted 12 lines
xen/common/argo.c +6
-0+ } + +/* Since we hold W(L2), there is no need to take L3 here */ + ring_info->tx_ptr = ring.tx_ptr; + + ret = argo_find_ring_mfns(d, ring_info, npage, pfn_hnd,ring.len);
xen/common/argo.c +2
-0+if ( !ret ) + ret = update_tx_ptr ? argo_update_tx_ptr(ring_info,ring.tx_ptr)
+ : argo_ring_map_page(ring_info, 0,
NULL);
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 699
Quoted 10 lines
xen/common/argo.c +10
-0+if ( !ret ) + ring_info->len = ring.len; + + }while (0 ); + +write_unlock (&d->argo->lock); + + }while (0 ); + +read_unlock (&argo_lock);
Why the use of do-while-zero constructs here? Use of goto-error-lable is far more conventional in the rest of Xen, and it would reduce indentation.
Quoted 79 lines
xen/common/argo.c +32
-0+ +return ret; +} + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, switch (cmd) { +case ARGO_MESSAGE_OP_register_ring: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ; +if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ; +/* arg4: reserve currently-undefined bits, require zero. */ +if (unlikely (arg4 & ~ARGO_REGISTER_FLAG_MASK ) ) + { + rc = -EINVAL ; +break ; + } + + rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist); +break ; + }default : rc = -ENOSYS ;break;xen/include/asm- renamed from xen/include/asm-arm/guest_access.h +0
-0
arm/guest_access.h
index 1137c54..98006f8 100644xen/include/asm-arm/guest_access.h +2
-0@@ -34,6 +34,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define
guest_handle_add_offset(hnd, nr)((hnd).p += (nr)) #defineguest_handle_subtract_offset(hnd, nr)((hnd).p -= (nr))xen/include/asm- renamed from xen/include/asm-x86/guest_access.h +0
-0
x86/guest_access.h
index 9391cd3..e9d25d6 100644xen/include/asm-x86/guest_access.h +2
-0@@ -50,6 +50,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define
guest_handle_subtract_offset(hnd, nr)((hnd).p -= (nr))xen/include/public/argo.h +0
-0
xen/include/public/argo.h line 30
Quoted 10 lines
xen/include/public/argo.h +7
-0@@ -21,6 +21,20 @@#include "xen.h" +#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes.
Why not just define as a hex value?
Paul
Quoted 68 lines
xen/include/public/argo.h +13
-0+* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ; + typedef struct argo_addr { uint32_t port;@@ -52,4 +66,54 @@ typedef struct argo_ring#endif } argo_ring_t; +/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits.
+ */xen/include/public/argo.h +23
-0+#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf) + +struct argo_ring_message_header +{ +uint32_t len; +argo_addr_t source; +uint32_t message_type; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t data[]; +#elif defined(__GNUC__) +uint8_t data[0 ]; +#endif +}; + +/* +* Hypercall operations +*/ + +/* +* ARGO_MESSAGE_OP_register_ring +* +* Register a ring using the indicated memory. +* Also used to reregister an existing ring (eg. after resume from sleep).
+ *
+ * arg1:XEN_GUEST_HANDLE(argo_ring_t)
+ * arg2:XEN_GUEST_HANDLE(argo_pfn_t)
+ * arg3:uint32_tnpages
+ * arg4:uint32_tflags
+ */xen/include/public/argo.h +14
-1+#define ARGO_MESSAGE_OP_register_ring 1 + +/* Register op flags */ +/* +* Fail exist: +* If set, reject attempts to (re)register an existing established ring. +* If clear, reregistration occurs if the ring exists, with the new ring +* taking the place of the old, preserving tx_ptr if it remains valid. +*/ +#define ARGO_REGISTER_FLAG_FAIL_EXIST 0x1 + +/* Mask for all defined flags */ +#define ARGO_REGISTER_FLAG_MASK ARGO_REGISTER_FLAG_FAIL_EXIST +#endif --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Christopher Clarkxen/common/argo.c struct argo_domain · line 106
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +2
-0+static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)
We generally prefer to avoid "inline" outside of header files. Also is there any strict need for the function to return a fixed width type? Plus what's the point of the _fn suffix?
xen/common/argo.c struct argo_domain · line 111
Quoted 5 lines
xen/common/argo.c +5
-0+{ +uint16_t ret; + + ret = (uint16_t )(id->addr.port >>16 ); + ret ^= (uint16_t )id->addr.port;
Pointless casts (with ret itself being uint16_t).
Quoted 29 lines
xen/common/argo.c +29
-0+static int +argo_ring_map_page (struct argo_ring_info *ring_info,uint32_t i, +uint8_t **page) +{ +if ( i >= ring_info->nmfns ) + { +printk (XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ; + } +ASSERT (ring_info->mfns); +ASSERT (ring_info->mfn_mapping); + +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read and +* write access to the page is required by the hypervisor, but read-access +* is not needed for this mapping for the remainder of the ring. +* Since this mapping will remain resident in Xen's address space for +* the lifetime of the ring, and following the principle of least privilege, +* it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as +* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT).
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 233
Quoted 10 lines
xen/common/argo.c +10
-0+*/ + ring_info->mfn_mapping[i] =map_domain_page_global (ring_info->mfns[i]); + +if ( !ring_info->mfn_mapping[i] ) + { +printk (XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ;
Unsuitable error code.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 240
Quoted 7 lines
xen/common/argo.c +7
-0+ } +argo_dprintk ("mapping page %" PRI_mfn" to %p\n" , +mfn_x (ring_info->mfns[i]), ring_info->mfn_mapping[i]); + } + +if ( page ) + *page = ring_info->mfn_mapping[i];
This suggests that the parameter is misnamed. "page" variables should be of types other than struct page_info * only under exceptional circumstances.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 248
Quoted 4 lines
xen/common/argo.c +4
-0+static int +argo_update_tx_ptr (struct argo_ring_info *ring_info,uint32_t tx_ptr) +{ +uint8_t *dst;
Why uint8_t when you don't mean to access bytes? For the arithmetic below void * should do just fine.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 256
Quoted 8 lines
xen/common/argo.c +8
-0+uint32_t *p; +int ret; + + ret =argo_ring_map_page (ring_info,0 , &dst); +if ( ret ) +return ret; + + p = (uint32_t *)(dst + offsetof(argo_ring_t , tx_ptr));
And then you also don't need any cast here.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 258
Quoted 2 lines
xen/common/argo.c +2
-0+write_atomic (p, tx_ptr); +mb ();
While guests need to use non-SMP barriers, I don't see why an SMP one wouldn't be sufficient here. I also don't see why this isn't smp_wmb().
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 324
Quoted 7 lines
xen/common/argo.c +3
-0@@ -231,6 +319,388 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info); } +/* +* ring +*/
I can see the point of using multi-line comments in a few cases where our style would not permit this, but a single word is imo too little to justify a style violation.
Quoted 8 lines
xen/common/argo.c +8
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; +
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 10 lines
xen/common/argo.c +10
-0+if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/
This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 36 lines
xen/common/argo.c +36
-0+if ( ring_info->npage != npage ) + i = ring_info->nmfns +1 ;/* forces re-register below */ +else + { +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) +break ; + +if (mfn_x (mfn) !=mfn_x (ring_info->mfns[i]) ) +break ; + } + } +if ( i != ring_info->nmfns ) + { +printk (XENLOG_INFO "argo: vm%u re-registering existing argo ring" +" (vm%u:%x vm%d), clearing MFN list\n" , + current->domain->domain_id, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +argo_ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + } + +if ( !ring_info->mfns ) + { +mfn_t *mfns;
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 418
Quoted 10 lines
xen/common/argo.c +10
-0+uint8_t **mfn_mapping; + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xmalloc_array(uint8_t *, npage);
Perhaps better to xzalloc_array() here than to ...
Quoted 32 lines
xen/common/argo.c +32
-0+if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + } +ASSERT (ring_info->npage == npage); + +if ( ring_info->nmfns == ring_info->npage ) +return 0 ; + +for ( i = ring_info->nmfns; i < ring_info->npage; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) + { +printk (XENLOG_ERR "argo: vm%u passed invalid gpfn %" PRI_xen_pfn +" ring (vm%u:%x vm%d) %p seq %d of %d\n" , + d->domain_id, pfn, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, + ring_info, i, ring_info->npage);
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 460
Quoted 10 lines
xen/common/argo.c +10
-0+break ; + } + + ring_info->mfns[i] = mfn; + ring_info->nmfns = i +1 ; + +argo_dprintk ("%d: %" PRI_xen_pfn" -> %" PRI_mfn"\n" , + i, pfn,mfn_x (ring_info->mfns[i])); + + ring_info->mfn_mapping[i] =NULL ;
... zap individual slots late here?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 482
Quoted 5 lines
xen/common/argo.c +5
-0+static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node;
const?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 521
Quoted 10 lines
xen/common/argo.c +10
-0+static long +argo_register_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd, +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd,uint32_t npage, +bool fail_exist) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; +bool update_tx_ptr =0 ;
bool type means true/false in initializers and assignments.
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 525
Quoted 4 lines
xen/common/argo.c +4
-0+uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ;
Why? You don't store the handle for later use (and you shouldn't). If there really is a need for a full page's worth of memory, it would better be passed in as GFN.
Quoted 13 lines
xen/common/argo.c +13
-0+read_lock (&argo_lock); + + do { + if ( !d->argo ) + { + ret = -ENODEV; + break; + } + + if ( copy_from_guest(&ring, ring_hnd, 1) ) + { + ret = -EFAULT; + break;
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 549
Quoted 10 lines
xen/common/argo.c +10
-0+ } + +if ( ring.magic !=ARGO_RING_MAGIC ) + { + ret = -EINVAL ; +break ; + } + + if ( (ring.len < (sizeof(struct argo_ring_message_header) + + ARGO_ROUNDUP(1) + ARGO_ROUNDUP(1))) ||
An expression like this wants at least a brief explaining comment attached.
Quoted 12 lines
xen/common/argo.c +12
-0+ (ARGO_ROUNDUP(ring.len) != ring.len) ) + { + ret = -EINVAL ; +break ; + } + +if ( ring.len >ARGO_MAX_RING_SIZE ) + { + ret = -EINVAL ; +break ; + } +
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 571
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ring.id.partner ==ARGO_DOMID_ANY ) + { + ret =xsm_argo_register_any_source (d, argo_mac_bootparam_enforcing); +if ( ret ) +break ; + } +else + { +struct domain *dst_d =get_domain_by_id (ring.id.partner); +if ( !dst_d )
Blank line between declaration(s) and statement(s) please.
Quoted 18 lines
xen/common/argo.c +18
-0+ { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_register_single_source (d, dst_d); +if ( ret ) + { +put_domain (dst_d); +break ; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d);
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 599
Quoted 10 lines
xen/common/argo.c +10
-0+break ; + } + + dst_domain_cookie = dst_d->argo->domain_cookie; + +put_domain (dst_d); + } + + ring.id.addr.domain_id = d->domain_id; +if (copy_field_to_guest (ring_hnd, &ring, id) )
Whenever you copy back out fields (or entire structures) that you've copied in before, __copy_* variants of the functions suffice).
Quoted 43 lines
xen/common/argo.c +43
-0+ { + ret = -EFAULT ; +break ; + } + +/* +* no need for a lock yet, because only we know about this +* set the tx pointer if it looks bogus (we don't reset it +* because this might be a re-register after S4) +*/ + +if ( ring.tx_ptr >= ring.len || +ARGO_ROUNDUP (ring.tx_ptr) != ring.tx_ptr ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + ring.tx_ptr =ARGO_ROUNDUP (ring.rx_ptr); + +if ( ring.tx_ptr >= ring.len ) + ring.tx_ptr =0 ; + +/* ring.tx_ptr will be written back to the guest ring below. */ + update_tx_ptr =1 ; + } + +/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock); + + do { + ring_info = argo_ring_find_info(d, &ring.id); + + if ( !ring_info ) + { + uint16_t hash; + + ring_info = xmalloc(struct argo_ring_info); + if ( !ring_info ) + { + ret = -ENOMEM; + break;
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 652
Quoted 10 lines
xen/common/argo.c +10
-0+ } + +spin_lock_init (&ring_info->lock); + + ring_info->mfns =NULL ; + ring_info->npage =0 ; + ring_info->mfn_mapping =NULL ; + ring_info->len =0 ; + ring_info->nmfns =0 ; + ring_info->tx_ptr =0 ;
xzalloc() used above would eliminate the need for all of these.
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 663
Quoted 11 lines
xen/common/argo.c +11
-0+ ring_info->partner_cookie = dst_domain_cookie; + + ring_info->id = ring.id; +INIT_HLIST_HEAD (&ring_info->pending); + + hash =argo_hash_fn (&ring_info->id); +hlist_add_head (&ring_info->node, &d->argo->ring_hash[hash]); + +printk (XENLOG_INFO "argo: vm%u registering ring (vm%u:%x vm%d)\n" , + current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner);
Please consider using XENLOG_G_INFO for such guest related log messages.
Quoted 5 lines
xen/common/argo.c +1
-0@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring:
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 736
Quoted 10 lines
xen/common/argo.c +10
-0+ { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ;
I don't understand the need for this and ...
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 743
Quoted 7 lines
xen/common/argo.c +7
-0+if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ;
... perhaps also this, when you use copy_from_guest() upon access.
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 754
Quoted 11 lines
xen/common/argo.c +10
-0+/* arg4: reserve currently-undefined bits, require zero. */ +if (unlikely (arg4 & ~ARGO_REGISTER_FLAG_MASK ) ) + { + rc = -EINVAL ; +break ; + } + + rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist); +break ; + }default :
Blank line above here please.
Quoted 13 lines
xen/include/public/argo.h +7
-0@@ -21,6 +21,20 @@#include "xen.h" +#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes.
xen/include/public/argo.h line 40
Quoted 10 lines
xen/include/public/argo.h +7
-0+* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ; + typedef struct argo_addr { uint32_t port;
It must have started in an earlier patch where I didn't pay
attention: Please can you make sure to prefix all public header additions to global name space with XEN_ / xen_? Unless of course it is thought that ARGO_ / argo_ are entirely impossible to be used in any other environment.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 16 lines · Jan Beulich
Quoted 16 lines
xen/common/argo.c +15
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; + +if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring,
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 3 lines
xen/common/argo.c +3
-0+* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
My understanding is that the gfn->mfn translation is not necessarily stable across entry and exit from host power state S4, suspend to disk.
I've added extra explanation to the next version of the patch series in the commit message for the suspend and resume functions which trigger guest use of this logic.
Quoted 7 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 482
Quoted 5 lines
xen/common/argo.c +5
-0+static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node;const?
I couldn't determine exactly what you were pointing towards with this one. I've applied 'const' in a lot further place in the next version; please let me know if I've missed where you intended.
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 525
Quoted 4 lines
xen/common/argo.c +4
-0+uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ;Why? You don't store the handle for later use (and you shouldn't). If there really is a need for a full page's worth of memory, it would better be passed in as GFN.
I've added this comment for this behaviour in v2:
+ /*
+ * Verify the alignment of the ring data structure supplied with the + * understanding that the ring handle supplied points to the same memory as + * the first entry in the array of pages provided via pg_descr_hnd, where + * the head of the ring will reside. + * See argo_update_tx_ptr where the location of the tx_ptr is accessed at a + * fixed offset from head of the first page in the mfn array. + */
Quoted 26 lines
Quoted 5 lines
xen/common/argo.c +1
-0@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring:xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 736
Quoted 10 lines
xen/common/argo.c +10
-0+ { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ;I don't understand the need for this and ...
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 743
Quoted 7 lines
xen/common/argo.c +7
-0+if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ;
Quoted 1 line
... perhaps also this, when you use
copy_from_guest()upon access.
This is the one piece of feedback on version 1 of this series that I haven't taken the time to address yet. The code is evidently safe, with only a possible performance decrease a concern, so I'd like to study it further before removing any of the checks rather than delay posting version two of this series.
All the other points: ack, and should be ok in v2.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 27 lines
Quoted 1 line
Quoted 1 line · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 16 lines · Jan Beulich
Quoted 16 lines
xen/common/argo.c +15
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; + +if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 3 lines
xen/common/argo.c +3
-0+* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.
How would that be? It's not stable across guest migration (or its non-live save/restore equivalent), but how would things change across S3? And there's no support for S4 (and I can't see it appearing any time soon).
Quoted 11 lines
Quoted 7 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 482
Quoted 5 lines
xen/common/argo.c +5
-0+static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node;const?
I couldn't determine exactly what you were pointing towards with this one. I've applied 'const' in a lot further place in the next version; please let me know if I've missed where you intended.
This is a pretty general rule: const should be applied to pointer target types whenever no modification is intended, to make this read-only aspect very obvious (and force people to think twice if they alter such a property).
Quoted 11 lines
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 525
Quoted 4 lines
xen/common/argo.c +4
-0+uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ;Why? You don't store the handle for later use (and you shouldn't). If there really is a need for a full page's worth of memory, it would better be passed in as GFN.
I've added this comment for this behaviour in v2:
Review
Quoted 8 lines
(quoted diff) +8
-0+ /* + * Verify the alignment of the ring data structure supplied with the + * understanding that the ring handle supplied points to the same memory as + * the first entry in the array of pages provided via pg_descr_hnd, where + * the head of the ring will reside. + * See argo_update_tx_ptr where the location of the tx_ptr is accessed at a + * fixed offset from head of the first page in the mfn array. + */
Well, this then suggests that you don't want to verify alignment, but instead you want to verify addresses match.
Quoted 26 lines
Quoted 26 lines
Quoted 5 lines
xen/common/argo.c +1
-0@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring:xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 736
Quoted 10 lines
xen/common/argo.c +10
-0+ { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ;I don't understand the need for this and ...
Quoted 7 lines
xen/common/argo.c +7
-0+if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ;
Quoted 6 lines
Quoted 1 line
... perhaps also this, when you use
copy_from_guest()upon access.This is the one piece of feedback on version 1 of this series that I haven't taken the time to address yet. The code is evidently safe, with only a possible performance decrease a concern, so I'd like to study it further before removing any of the checks rather than delay posting version two of this series.
Hmm, re-posting without all comments addressed is not ideal. It means extra work for the reviewers (unless you've clearly marked respective code fragments with some sort of TBD comment).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 9 lines
xen/common/argo.c +8
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; +xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 10 lines
xen/common/argo.c +10
-0+if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 5 lines
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.How would that be? It's not stable across guest migration (or its non-live save/restore equivalent),
Right, that's clear.
Quoted 1 line
but how would things change across S3?
I don't think that they do change in that case.
From studying the code involved above, a related item: the guest runs the same suspend and resume kernel code before entering into/exiting from either guest S3 or S4, so the guest kernel resume code needs to re-register the rings, to cover the case where it is coming up in an environment where they were dropped
- so that's what it does.
This relates to the code section above: if guest entry to S3 is aborted at the final step (eg. error or platform refuses, eg. maybe a physical device interaction with passthrough) then the hypervisor has not torn down the rings, the guest remains running within the same domain, and the guest resume logic runs, which runs through re-registration for all its rings. The check in the logic above allows the existing ring mappings within the hypervisor to be preserved.
I'm not certain that is an enormous win though; it looks like it would be ok to drop that logic and reestablish the mappings as the ring is used, as per other cases.
Quoted 1 line
And there's no support for S4 (and I can't see it appearing any time soon).
OK. oh well.
Quoted 38 lines
Quoted 9 lines
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 482
Quoted 6 lines
xen/common/argo.c +5
-0+static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node;const?
Quoted 3 lines
I couldn't determine exactly what you were pointing towards with this one. I've applied 'const' in a lot further place in the next version; please let me know if I've missed where you intended.
This is a pretty general rule: const should be applied to pointer target types whenever no modification is intended, to make this read-only aspect very obvious (and force people to think twice if they alter such a property).
Quoted 11 lines
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 525
Quoted 4 lines
xen/common/argo.c +4
-0+uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ;Why? You don't store the handle for later use (and you shouldn't). If there really is a need for a full page's worth of memory, it would better be passed in as GFN.
I've added this comment for this behaviour in v2:
Review
Quoted 8 lines
(quoted diff) +8
-0+ /* + * Verify the alignment of the ring data structure supplied with the + * understanding that the ring handle supplied points to the same memory as + * the first entry in the array of pages provided via pg_descr_hnd, where + * the head of the ring will reside. + * See argo_update_tx_ptr where the location of the tx_ptr is accessed at a + * fixed offset from head of the first page in the mfn array. + */
Quoted 2 lines
Well, this then suggests that you don't want to verify alignment, but instead you want to verify addresses match.
ack. I'll take a look at doing that.
Quoted 34 lines
Quoted 27 lines
Quoted 27 lines
Quoted 6 lines
xen/common/argo.c +1
-0@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring:xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 736
Quoted 10 lines
xen/common/argo.c +10
-0+ { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ;I don't understand the need for this and ...
Quoted 7 lines
xen/common/argo.c +7
-0+if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ;Quoted 6 lines
Quoted 1 line
... perhaps also this, when you use
copy_from_guest()upon access.This is the one piece of feedback on version 1 of this series that I haven't taken the time to address yet. The code is evidently safe, with only a possible performance decrease a concern, so I'd like to study it further before removing any of the checks rather than delay posting version two of this series.
Quoted 4 lines
Hmm, re-posting without all comments addressed is not ideal. It means extra work for the reviewers (unless you've clearly marked respective code fragments with some sort of TBD comment).
Understood.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 30 lines
Quoted 1 line
Quoted 1 line · On 21.12.18 at 02:25, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 9 lines
xen/common/argo.c +8
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; +xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 10 lines
xen/common/argo.c +10
-0+if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.
Now I'm afraid there's some confusion here: Originally you've said "host".
Quoted 15 lines
Quoted 2 lines
How would that be? It's not stable across guest migration (or its non-live save/restore equivalent),
Right, that's clear.
Quoted 1 line
but how would things change across S3?
I don't think that they do change in that case.
From studying the code involved above, a related item: the guest runs the same suspend and resume kernel code before entering into/exiting from either guest S3 or S4, so the guest kernel resume code needs to re-register the rings, to cover the case where it is coming up in an environment where they were dropped
- so that's what it does.
Quoted 7 lines
This relates to the code section above: if guest entry to S3 is aborted at the final step (eg. error or platform refuses, eg. maybe a physical device interaction with passthrough) then the hypervisor has not torn down the rings, the guest remains running within the same domain, and the guest resume logic runs, which runs through re-registration for all its rings. The check in the logic above allows the existing ring mappings within the hypervisor to be preserved.
Yet now you suddenly talk about guest S3.
Quoted 3 lines
Quoted 1 line
And there's no support for S4 (and I can't see it appearing any time soon).
OK. oh well.
Considering the original "host" context, my response here was relating to host S4. Guest S4 ought to be functional (as being mostly a guest kernel function anyway).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 60 lines · Jan Beulich
Quoted 31 lines
Quoted 2 lines
Quoted 2 lines · On 21.12.18 at 02:25, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 9 lines
xen/common/argo.c +8
-0+static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; +xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 372
Quoted 10 lines
xen/common/argo.c +10
-0+if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.Now I'm afraid there's some confusion here: Originally you've said "host".
Quoted 15 lines
Quoted 2 lines
How would that be? It's not stable across guest migration (or its non-live save/restore equivalent),
Right, that's clear.
Quoted 1 line
but how would things change across S3?
I don't think that they do change in that case.
From studying the code involved above, a related item: the guest runs the same suspend and resume kernel code before entering into/exiting from either guest S3 or S4, so the guest kernel resume code needs to re-register the rings, to cover the case where it is coming up in an environment where they were dropped
- so that's what it does.Quoted 7 lines
This relates to the code section above: if guest entry to S3 is aborted at the final step (eg. error or platform refuses, eg. maybe a physical device interaction with passthrough) then the hypervisor has not torn down the rings, the guest remains running within the same domain, and the guest resume logic runs, which runs through re-registration for all its rings. The check in the logic above allows the existing ring mappings within the hypervisor to be preserved.
Quoted 1 line
Yet now you suddenly talk about guest S3.
Well, the context is that you did just ask about S3, without specifying host or guest. Host S3 doesn't involve much at all, so I went and studied the code in both the Linux driver and the hypervisor to determine what it does in the case of guest S3, and then replied with the above since it is relevant to the code in question. I hope I was clear about referring to guest S3 above in my last reply.
That logic aims to make ring registration idempotent, to avoid the teardown of established mappings of the ring pages in the case where doing so isn't needed.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
And there's no support for S4 (and I can't see it appearing any time soon).
OK. oh well.
Considering the original "host" context, my response here was relating to host S4. Guest S4 ought to be functional (as being mostly a guest kernel function anyway).
ack.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 21.12.18 at 09:16, <[email protected]> wrote:
Quoted 31 lines · Jan Beulich
Quoted 31 lines
Quoted 2 lines
Quoted 2 lines · On 21.12.18 at 02:25, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 9 lines
(quoted diff) +8
-0+static int +argo_find_ring_mfns(struct domain *d, struct argo_ring_info *ring_info, + uint32_t npage, XEN_GUEST_HANDLE_PARAM(argo_pfn_t) pfn_hnd, + uint32_t len) +{ + int i; + int ret = 0; +Review
Quoted 10 lines
(quoted diff) +10
-0+ if ( (npage << PAGE_SHIFT) < len ) + return -EINVAL; + + if ( ring_info->mfns ) + { + /* + * Ring already existed. Check if it's the same ring, + * i.e. same number of pages and all translated gpfns still + * translating to the same mfns + */This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.
Note this ^^^ (and see below).
Quoted 26 lines
Quoted 26 lines
Now I'm afraid there's some confusion here: Originally you've said "host".
Quoted 22 lines
Quoted 2 lines
How would that be? It's not stable across guest migration (or its non-live save/restore equivalent),
Right, that's clear.
Quoted 1 line
but how would things change across S3?
I don't think that they do change in that case.
From studying the code involved above, a related item: the guest runs the same suspend and resume kernel code before entering into/exiting from either guest S3 or S4, so the guest kernel resume code needs to re-register the rings, to cover the case where it is coming up in an environment where they were dropped
- so that's what it does.This relates to the code section above: if guest entry to S3 is aborted at the final step (eg. error or platform refuses, eg. maybe a physical device interaction with passthrough) then the hypervisor has not torn down the rings, the guest remains running within the same domain, and the guest resume logic runs, which runs through re-registration for all its rings. The check in the logic above allows the existing ring mappings within the hypervisor to be preserved.
Quoted 4 lines
Quoted 1 line
Yet now you suddenly talk about guest S3.
Well, the context is that you did just ask about S3, without specifying host or guest.
I'm sorry to be picky, but no, I don't think I did. You did expicitly say "host", making me further think only about that case.
Quoted 9 lines
Host S3 doesn't involve much at all, so I
went and studied the code in both the Linux driver and the hypervisor to determine what it does in the case of guest S3, and then replied with the above since it is relevant to the code in question. I hope I was clear about referring to guest S3 above in my last reply.That logic aims to make ring registration idempotent, to avoid the teardown of established mappings of the ring pages in the case where doing so isn't needed.
You treat complexity in the kernel for complexity in the hypervisor. I'm not sure this is appropriate, as I can't judge how much more difficult it would be for the guest to look after itself. But let's look at both cases again:
- For guest S3, afaik, the domain doesn't change, and hence memory assignment remains the same. No re-registration necessary then afaict.
- For guest S4, aiui, the domain gets destroyed and a new one built upon resume. Re-registration would be needed, but due to the domain re-construction no leftovers ought to exist in Xen.
Hence to me it would seem more natural to have the guest deal with the situation, and have no extra logic for this in Xen. You want the guest to re-register anyway, yet simply avoiding to do so in the S3 case ought to be a single (or very few) conditional(s), i.e. not a whole lot of complexity.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 65 lines · Jan Beulich
Quoted 34 lines
Quoted 2 lines
Quoted 2 lines · On 21.12.18 at 09:16, <[email protected]> wrote:
Quoted 31 lines · Jan Beulich
Quoted 31 lines
Quoted 2 lines
Quoted 2 lines · On 21.12.18 at 02:25, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:29, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 9 lines
(quoted diff) +8
-0+static int +argo_find_ring_mfns(struct domain *d, struct argo_ring_info *ring_info, + uint32_t npage, XEN_GUEST_HANDLE_PARAM(argo_pfn_t) pfn_hnd, + uint32_t len) +{ + int i; + int ret = 0; +Review
Quoted 10 lines
(quoted diff) +10
-0+ if ( (npage << PAGE_SHIFT) < len ) + return -EINVAL; + + if ( ring_info->mfns ) + { + /* + * Ring already existed. Check if it's the same ring, + * i.e. same number of pages and all translated gpfns still + * translating to the same mfns + */This comment makes me wonder whether the translations are permitted to change at other times. If so I'm not sure what value verification here has. If not, this probably would want to be debugging-only code.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
My understanding is that the
gfn->mfntranslation is not necessarily stable across entry and exit from host power state S4, suspend to disk.Note this ^^^ (and see below).
Quoted 26 lines
Quoted 26 lines
Now I'm afraid there's some confusion here: Originally you've said "host".
Quoted 22 lines
Quoted 2 lines
How would that be? It's not stable across guest migration (or its non-live save/restore equivalent),
Right, that's clear.
Quoted 1 line
but how would things change across S3?
I don't think that they do change in that case.
From studying the code involved above, a related item: the guest runs the same suspend and resume kernel code before entering into/exiting from either guest S3 or S4, so the guest kernel resume code needs to re-register the rings, to cover the case where it is coming up in an environment where they were dropped
- so that's what it does.This relates to the code section above: if guest entry to S3 is aborted at the final step (eg. error or platform refuses, eg. maybe a physical device interaction with passthrough) then the hypervisor has not torn down the rings, the guest remains running within the same domain, and the guest resume logic runs, which runs through re-registration for all its rings. The check in the logic above allows the existing ring mappings within the hypervisor to be preserved.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Yet now you suddenly talk about guest S3.
Well, the context is that you did just ask about S3, without specifying host or guest.
I'm sorry to be picky, but no, I don't think I did. You did expicitly say "host", making me further think only about that case.
OK, apologies for the confusing direction of the reply. It was not intended to be so.
Quoted 5 lines
Quoted 3 lines
That logic aims to make ring registration idempotent, to avoid the teardown of established mappings of the ring pages in the case where doing so isn't needed.
You treat complexity in the kernel for complexity in the hypervisor.
(s/treat/trade/ ?) OK, that is a fair concern, yes.
Quoted 10 lines
I'm not sure this is appropriate, as I can't judge how much more difficult it would be for the guest to look after itself. But let's look at both cases again:
- For guest S3, afaik, the domain doesn't change, and hence memory assignment remains the same. No re-registration necessary then afaict.
- For guest S4, aiui, the domain gets destroyed and a new one built upon resume. Re-registration would be needed, but due to the domain re-construction no leftovers ought to exist in Xen.
I agree.
Quoted 5 lines
Hence to me it would seem more natural to have the guest deal with the situation, and have no extra logic for this in Xen. You want the guest to re-register anyway, yet simply avoiding to do so in the S3 case ought to be a single (or very few) conditional(s), i.e. not a whole lot of complexity.
OK. That looks doable. thanks.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Christopher ClarkQuoted 37 lines · Christopher Clark
Used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
In this code, the p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rw, which is a conservative choice made to defer the need to reason about the other p2m types with this commit.
argo_pfn_ttype is introduced here to create apfn_ttype that is 64-bit on all architectures, to assist with avoiding the need to add a compat ABI.Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 498 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/guest_access.h | 2 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 64 +++++ 4 files changed, 566 insertions(+)xen/common/argo.c +1
-0@@ -25,6 +25,7 @@#include <xen/guest_access.h> #include <xen/time.h> +DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t);
xen/common/argo.c struct argo_domain · line 106
Quoted 8 lines
xen/common/argo.c +5
-0@@ -98,6 +99,25 @@ struct argo_domain };/* +* Helper functions +*/ + +static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)
No need for the argo_ prefix for static functions, this is already an argo specific file.
xen/common/argo.c struct argo_domain · line 115
Quoted 9 lines
xen/common/argo.c +9
-0+{ +uint16_t ret; + + ret = (uint16_t )(id->addr.port >>16 ); + ret ^= (uint16_t )id->addr.port; + ret ^= id->addr.domain_id; + ret ^= id->partner; + + ret &= (ARGO_HTABLE_SIZE -1 );
I'm having trouble figuring out what this is supposed to do, I think a comment and the expected hash formula will help make sure the code is correct.
Also doesn't this need to be documented in the public header?
Quoted 9 lines
xen/common/argo.c +4
-0+return ret; +} + +/* * locks */ @@ -171,6 +191,74 @@ argo_ring_unmap(struct argo_ring_info *ring_info) }
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 201
Quoted 10 lines
xen/common/argo.c +8
-0} +/* caller must have L3 or W(L2) */ +static int +argo_ring_map_page (struct argo_ring_info *ring_info,uint32_t i, +uint8_t **page) +{ +if ( i >= ring_info->nmfns ) + { + printk(XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page"
You likely want to use gprintk here and below, or XENLOG_G_ERR, so that the guest cannot DoS the console.
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 208
Quoted 7 lines
xen/common/argo.c +7
-0+" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ; + } +ASSERT (ring_info->mfns); +ASSERT (ring_info->mfn_mapping);
We are trying to move away from such assertions, and instead use constructions that would prevent issues in non-debug builds. I would write the above asserts as:
if ( !ring_info->mfns || !ring_info->mfn_mapping )
{
ASSERT_UNREACHABLE();
return -E<something>;
}That way non-debug builds won't trigger page faults if there's indeed a way to get here with the wrong state, and debug builds will still hit an assert.
Quoted 57 lines
xen/common/argo.c +53
-0+ +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read and +* write access to the page is required by the hypervisor, but read-access +* is not needed for this mapping for the remainder of the ring. +* Since this mapping will remain resident in Xen's address space for +* the lifetime of the ring, and following the principle of least privilege, +* it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as +* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT). +*/ + ring_info->mfn_mapping[i] =map_domain_page_global (ring_info->mfns[i]); + +if ( !ring_info->mfn_mapping[i] ) + { +printk (XENLOG_ERR "argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %u of %u\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + i, ring_info->nmfns); +return -EFAULT ; + } +argo_dprintk ("mapping page %" PRI_mfn" to %p\n" , +mfn_x (ring_info->mfns[i]), ring_info->mfn_mapping[i]); + } + +if ( page ) + *page = ring_info->mfn_mapping[i]; +return 0 ; +} + +/* caller must have L3 or W(L2) */ +static int +argo_update_tx_ptr (struct argo_ring_info *ring_info,uint32_t tx_ptr) +{ +uint8_t *dst; +uint32_t *p; +int ret; + + ret =argo_ring_map_page (ring_info,0 , &dst); +if ( ret ) +return ret; + + p = (uint32_t *)(dst + offsetof(argo_ring_t , tx_ptr)); +write_atomic (p, tx_ptr); +mb (); +return 0 ; +} +/* * pending */
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 327
Quoted 9 lines
xen/common/argo.c +6
-0@@ -231,6 +319,388 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info); } +/* +* ring +*/ + +static int +argo_find_ring_mfn (struct domain *d,argo_pfn_t pfn,mfn_t *mfn)
I think you mean gfn instead of pfn, here and below. Also I'm unsure why you need a new type for argo, it's it fine to just use uint64_t?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 333
Quoted 6 lines
xen/common/argo.c +6
-0+{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d, pfn, &p2mt);
Is this supposed to work for PV guests?
Quoted 6 lines
xen/common/argo.c +6
-0+#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ;
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 349
Quoted 10 lines
xen/common/argo.c +10
-0+#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn);
If you do this put_gfn here, by the time you check that the gfn -> mfn matches your expectations the guest might have somehow changed the gfn -> mfn mapping already (for example by ballooning down memory?)
Quoted 162 lines
xen/common/argo.c +162
-0+#endif + +return ret; +} + +static int +argo_find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage,XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd, +uint32_t len) +{ +int i; +int ret =0 ; + +if ( (npage <<PAGE_SHIFT ) < len ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* +* Ring already existed. Check if it's the same ring, +* i.e. same number of pages and all translated gpfns still +* translating to the same mfns +*/ +if ( ring_info->npage != npage ) + i = ring_info->nmfns +1 ;/* forces re-register below */ +else + { +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) +break ; + +if (mfn_x (mfn) !=mfn_x (ring_info->mfns[i]) ) +break ; + } + } +if ( i != ring_info->nmfns ) + { +printk (XENLOG_INFO "argo: vm%u re-registering existing argo ring" +" (vm%u:%x vm%d), clearing MFN list\n" , + current->domain->domain_id, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +argo_ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + } + +if ( !ring_info->mfns ) + { +mfn_t *mfns; +uint8_t **mfn_mapping; + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xmalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + } +ASSERT (ring_info->npage == npage); + +if ( ring_info->nmfns == ring_info->npage ) +return 0 ; + +for ( i = ring_info->nmfns; i < ring_info->npage; i++ ) + { +argo_pfn_t pfn; +mfn_t mfn; + + ret =copy_from_guest_offset_errno (&pfn, pfn_hnd, i,1 ); +if ( ret ) +break ; + + ret =argo_find_ring_mfn (d, pfn, &mfn); +if ( ret ) + { +printk (XENLOG_ERR "argo: vm%u passed invalid gpfn %" PRI_xen_pfn +" ring (vm%u:%x vm%d) %p seq %d of %d\n" , + d->domain_id, pfn, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, + ring_info, i, ring_info->npage); +break ; + } + + ring_info->mfns[i] = mfn; + ring_info->nmfns = i +1 ; + +argo_dprintk ("%d: %" PRI_xen_pfn" -> %" PRI_mfn"\n" , + i, pfn,mfn_x (ring_info->mfns[i])); + + ring_info->mfn_mapping[i] =NULL ; + } + +if ( ret ) +argo_ring_remove_mfns (d, ring_info); +else + { +ASSERT (ring_info->nmfns == ring_info->npage); + +printk (XENLOG_ERR "argo: vm%u ring (vm%u:%x vm%d) %p mfn_mapping %p" +" npage %d nmfns %d\n" , current->domain->domain_id, + ring_info->id.addr.domain_id, ring_info->id.addr.port, + ring_info->id.partner, ring_info, ring_info->mfn_mapping, + ring_info->npage, ring_info->nmfns); + } +return ret; +} + +static struct argo_ring_info * +argo_ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +uint16_t hash; +struct hlist_node *node; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + hash =argo_hash_fn (id); + +argo_dprintk ("d->argo=%p, d->argo->ring_hash[%d]=%p id=%p\n" , + d->argo, hash, d->argo->ring_hash[hash].first, id); +argo_dprintk ("id.addr.port=%d id.addr.domain=vm%u" +" id.addr.partner=vm%d\n" , + id->addr.port, id->addr.domain_id, id->partner); + +hlist_for_each_entry (ring_info, node, &d->argo->ring_hash[hash], node) + { +argo_ring_id_t *cmpid = &ring_info->id; + +if ( cmpid->addr.port == id->addr.port && + cmpid->addr.domain_id == id->addr.domain_id && + cmpid->partner == id->partner ) + { +argo_dprintk ("ring_info=%p\n" , ring_info); +return ring_info; + } + } +argo_dprintk ("no ring_info found\n" ); + +return NULL ; +} +
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 521
Quoted 10 lines
xen/common/argo.c +10
-0+static long +argo_register_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd, +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd,uint32_t npage, +bool fail_exist) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; +bool update_tx_ptr =0 ;
bool uses true/false.
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 527
Quoted 6 lines
xen/common/argo.c +6
-0+uint64_t dst_domain_cookie =0 ; + +if ( !(guest_handle_is_aligned (ring_hnd, ~PAGE_MASK )) ) +return -EINVAL ; + +read_lock (&argo_lock);
^ extra space.
Quoted 33 lines
xen/common/argo.c +33
-0+ + do { + if ( !d->argo ) + { + ret = -ENODEV; + break; + } + + if ( copy_from_guest(&ring, ring_hnd, 1) ) + { + ret = -EFAULT; + break; + } + + if ( ring.magic != ARGO_RING_MAGIC ) + { + ret = -EINVAL; + break; + } + + if ( (ring.len < (sizeof(struct argo_ring_message_header) + + ARGO_ROUNDUP(1) + ARGO_ROUNDUP(1))) || + (ARGO_ROUNDUP(ring.len) != ring.len) ) + { + ret = -EINVAL; + break; + } + + if ( ring.len > ARGO_MAX_RING_SIZE ) + { + ret = -EINVAL; + break; + }
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 570
Quoted 10 lines
xen/common/argo.c +10
-0+ +if ( ring.id.partner ==ARGO_DOMID_ANY ) + { + ret =xsm_argo_register_any_source (d, argo_mac_bootparam_enforcing); +if ( ret ) +break ; + } +else + { +struct domain *dst_d =get_domain_by_id (ring.id.partner);
Missing newline.
Quoted 49 lines
xen/common/argo.c +49
-0+if ( !dst_d ) + { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_register_single_source (d, dst_d); +if ( ret ) + { +put_domain (dst_d); +break ; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); +break ; + } + + dst_domain_cookie = dst_d->argo->domain_cookie; + +put_domain (dst_d); + } + + ring.id.addr.domain_id = d->domain_id; +if (copy_field_to_guest (ring_hnd, &ring, id) ) + { + ret = -EFAULT ; +break ; + } + +/* +* no need for a lock yet, because only we know about this +* set the tx pointer if it looks bogus (we don't reset it +* because this might be a re-register after S4) +*/ + +if ( ring.tx_ptr >= ring.len || +ARGO_ROUNDUP (ring.tx_ptr) != ring.tx_ptr ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + ring.tx_ptr =ARGO_ROUNDUP (ring.rx_ptr);
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 629
Quoted 10 lines
xen/common/argo.c +10
-0+ +if ( ring.tx_ptr >= ring.len ) + ring.tx_ptr =0 ; + +/* ring.tx_ptr will be written back to the guest ring below. */ + update_tx_ptr =1 ; + } + +/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock);
I don't understand this W(L2) nomenclature, is this explain somewhere?
Also there's no such comment when you take the global argo_lock above.
Quoted 54 lines
xen/common/argo.c +54
-0+ + do { + ring_info = argo_ring_find_info(d, &ring.id); + + if ( !ring_info ) + { + uint16_t hash; + + ring_info = xmalloc(struct argo_ring_info); + if ( !ring_info ) + { + ret = -ENOMEM; + break; + } + + spin_lock_init(&ring_info->lock); + + ring_info->mfns = NULL; + ring_info->npage = 0; + ring_info->mfn_mapping = NULL; + ring_info->len = 0; + ring_info->nmfns = 0; + ring_info->tx_ptr = 0; + ring_info->partner_cookie = dst_domain_cookie; + + ring_info->id = ring.id; + INIT_HLIST_HEAD(&ring_info->pending); + + hash = argo_hash_fn(&ring_info->id); + hlist_add_head(&ring_info->node, &d->argo->ring_hash[hash]); + + printk(XENLOG_INFO "argo: vm%u registering ring (vm%u:%x vm%d)\n", + current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner); + } + else + { + /* + * If the caller specified that the ring must not already exist, + * fail at attempt to add a completed ring which already exists. + */ + if ( fail_exist && ring_info->len ) + { + ret = -EEXIST; + break; + } + + printk(XENLOG_INFO + "argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n", + current->domain->domain_id, ring.id.addr.domain_id, + ring.id.addr.port, ring.id.partner); + } + + /* Since we hold W(L2), there is no need to take L3 here */
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 693
Quoted 10 lines
xen/common/argo.c +10
-0+ ring_info->tx_ptr = ring.tx_ptr; + + ret =argo_find_ring_mfns (d, ring_info, npage, pfn_hnd, ring.len); +if ( !ret ) + ret = update_tx_ptr ?argo_update_tx_ptr (ring_info, ring.tx_ptr) + :argo_ring_map_page (ring_info,0 ,NULL ); +if ( !ret ) + ring_info->len = ring.len; + + }while (0 );
Why this useless loop? Just adds to indentation.
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 697
Quoted 4 lines
xen/common/argo.c +4
-0+ +write_unlock (&d->argo->lock); + + }while (0 );
Same here.
Quoted 75 lines
xen/common/argo.c +34
-0+ +read_unlock (&argo_lock); + +return ret; +} + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,@@ -253,6 +723,34 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case ARGO_MESSAGE_OP_register_ring: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +XEN_GUEST_HANDLE_PARAM (argo_pfn_t) pfn_hnd = +guest_handle_cast (arg2, argo_pfn_t); +uint32_t npage = arg3; +bool fail_exist = arg4 &ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ; +if (unlikely (npage > (ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +if (unlikely (!guest_handle_okay (pfn_hnd, npage)) ) +break ; +/* arg4: reserve currently-undefined bits, require zero. */ +if (unlikely (arg4 & ~ARGO_REGISTER_FLAG_MASK ) ) + { + rc = -EINVAL ; +break ; + } + + rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist); +break ; + }default : rc = -ENOSYS ;break ;xen/include/asm-arm/guest_access.h +2
-0@@ -34,6 +34,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/asm-x86/guest_access.h +2
-0@@ -50,6 +50,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/public/argo.h +0
-0
xen/include/public/argo.h line 26
Quoted 6 lines
xen/include/public/argo.h +3
-0@@ -21,6 +21,20 @@#include "xen.h" +#define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL + +#define ARGO_DOMID_ANY DOMID_INVALID
I think you should either leave 1 space between the define name and the value, or if you want to add multiple spaces please make all the define values aligned on the same col.
Quoted 16 lines
xen/include/public/argo.h +11
-0+ +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 or 16777216 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define ARGO_MAX_RING_SIZE (16777216ULL) + +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ +typedef uint64_t argo_pfn_t ; + typedef struct argo_addr { uint32_t port;@@ -52,4 +66,54 @@ typedef struct argo_ring#endif
xen/include/public/argo.h typedef struct argo_ring · line 76
Quoted 10 lines
xen/include/public/argo.h +8
-0} argo_ring_t; +/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits. +*/ +#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf)
Why not just use ROUNDUP?
And in any case this shouldn't be on the public header IMO, since it's not part of the interface AFAICT.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 7 lines · Roger Pau Monné
xen/common/argo.c struct argo_domain · line 106
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)No need for the
argo_prefix for static functions, this is already an argo specific file.
Although the compiler could live without the prefix, I'm finding it helpful to very easily determine that functions being used are not defined elsewhere within Xen; so I've left the prefix as is for version two of this series.
Quoted 5 lines
Quoted 5 lines
xen/common/argo.c +5
-0+{ +uint16_t ret; + + ret = (uint16_t )(id->addr.port >>16 ); + ret ^= (uint16_t )id->addr.port;
Quoted 8 lines
xen/common/argo.c struct argo_domain · line 115
Quoted 4 lines
xen/common/argo.c +4
-0+ ret ^= id->addr.domain_id; + ret ^= id->partner; + + ret &= (ARGO_HTABLE_SIZE -1 );I'm having trouble figuring out what this is supposed to do, I think a comment and the expected hash formula will help make sure the code is correct.
Fair point. I've added a comment with explanation in the next version.
+ * This hash function is used to distribute rings within the per-domain
+ * hash table (d->argo->ring_hash). The hash table will provide a
+ * 'ring_info' struct if a match is found with a 'xen_argo_ring_id' key:
+ * ie. the key is a (domain id, port, partner domain id) tuple.
+ * There aren't many hash table buckets, and this doesn't need to be
+ * cryptographically robust. Since port number varies the most in
+ * expected use, and the Linux driver allocates at both the high and
+ * low ends, incorporate high and low bits to help with distribution.
Quoted 1 line
Also doesn't this need to be documented in the public header?
No, it's only for internal use within the hypervisor and designed to meet the hypervisor's hashing requirement with a small table.
Quoted 8 lines
xen/common/argo.c argo_ring_unmap(struct argo_ring_info *ring_info) · line 208
Quoted 2 lines
xen/common/argo.c +2
-0+ASSERT (ring_info->mfns); +ASSERT (ring_info->mfn_mapping);We are trying to move away from such assertions, and instead use constructions that would prevent issues in non-debug builds. I would write the above asserts as:
if ( !ring_info->mfns || !ring_info->mfn_mapping )
Quoted 8 lines
{ ASSERT_UNREACHABLE(); return -E<something>; }That way non-debug builds won't trigger page faults if there's indeed a way to get here with the wrong state, and debug builds will still hit an assert.
ack, will do so in v2.
Quoted 3 lines
Quoted 1 line
+ *mfn =
get_gfn_unshare(d, pfn, &p2mt);Is this supposed to work for PV guests?
Yes -- and they seem to work OK. Am I missing something?
Quoted 12 lines
Quoted 12 lines
xen/common/argo.c +12
-0+#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) )
Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 349
Quoted 4 lines
xen/common/argo.c +4
-0+ ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn);If you do this
put_gfnhere, by the time you check that the gfn -> mfn matches your expectations the guest might have somehow changed the gfn -> mfn mapping already (for example by ballooning down memory?)
If the guest does that, I think it only harms itself. If for some reason a memory access is denied, then the op would just fail. I don't think there's a more serious consequence to be worried about.
Above, if we're going to use the mfn, then we've just done a successful:
get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page)
which should hold it in a state that we're ok with until we're done with it -- see put_page_and_type in argo_ring_remove_mfns.
Quoted 4 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 629
Quoted 2 lines
xen/common/argo.c +2
-0+/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock);I don't understand this W(L2) nomenclature, is this explain somewhere?
Yes, sort of. Lock "L2" is the per-domain argo lock, identified in a comment near the top of the file. It's a read-write lock, so 'W' means:
take the write lock on it.
Quoted 1 line
Also there's no such comment when you take the global
argo_lockabove.
L2 covers more interesting work than L1, which is why there are more comments pertaining to it than L1.
Quoted 11 lines
xen/include/public/argo.h typedef struct argo_ring · line 76
Quoted 8 lines
xen/include/public/argo.h +8
-0+/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits. +*/ +#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf)Why not just use ROUNDUP?
Quoted 2 lines
And in any case this shouldn't be on the public header IMO, since it's not part of the interface AFAICT.
Well, in version two it's now: XEN_ARGO_ROUNDUP :-)
because it does need to be in the public header because it's used within the Linux device driver, and items in that public Xen header need the 'xen' prefix (so they now do). Within the Linux code, it's used to choose a sensible ring size, and also used when manipulating the rx_ptr on the guest side.
Thanks for the feedback - almost all should be covered in v2, I think.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 20.12.18 at 06:41, <[email protected]> wrote:
Quoted 7 lines · Roger Pau Monné
xen/common/argo.c struct argo_domain · line 106
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)No need for the
argo_prefix for static functions, this is already an argo specific file.
Quoted 3 lines
Although the compiler could live without the prefix, I'm finding it helpful to very easily determine that functions being used are not defined elsewhere within Xen; so I've left the prefix as is for version two of this series.
But you realize that this needlessly increases the string table size as well as the volume of output going over the (normally low bandwidth) serial line in case of an isuse?
Quoted 11 lines
Quoted 11 lines
xen/include/public/argo.h typedef struct argo_ring · line 76
Quoted 8 lines
xen/include/public/argo.h +8
-0+/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits. +*/ +#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf)Why not just use ROUNDUP?
Quoted 8 lines
Quoted 2 lines
And in any case this shouldn't be on the public header IMO, since it's not part of the interface AFAICT.
Well, in version two it's now:
XEN_ARGO_ROUNDUP:-)
because it does need to be in the public header because it's used within the Linux device driver, and items in that public Xen header need the 'xen' prefix (so they now do). Within the Linux code, it's used to choose a sensible ring size, and also used when manipulating therx_ptron the guest side.
It's at least questionable to put constructs into the public header that the header itself doesn't use, and the naming of which may not be what consumers would prefer. I can see the "single central place" point though.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Christopher ClarkQuoted 9 lines · Christopher Clark
Quoted 7 lines · Roger Pau Monné
xen/common/argo.c struct argo_domain · line 106
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)No need for the
argo_prefix for static functions, this is already an argo specific file.
Quoted 3 lines
Although the compiler could live without the prefix, I'm finding it helpful to very easily determine that functions being used are not defined elsewhere within Xen; so I've left the prefix as is for version two of this series.
Why do you care whether they are defined elsewhere in Xen? The scope of static functions is limited to the translation unit anyway.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
+ *mfn =
get_gfn_unshare(d, pfn, &p2mt);Is this supposed to work for PV guests?
Yes -- and they seem to work OK. Am I missing something?
No, my fault, this should indeed work for both paging and non paging assisted guests, sorry for the noise.
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
xen/common/argo.c +12
-0+#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) )Quoted 8 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 349
Quoted 4 lines
xen/common/argo.c +4
-0+ ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn);If you do this
put_gfnhere, by the time you check that the gfn -> mfn matches your expectations the guest might have somehow changed the gfn -> mfn mapping already (for example by ballooning down memory?)
Quoted 3 lines
If the guest does that, I think it only harms itself. If for some reason a memory access is denied, then the op would just fail. I don't think there's a more serious consequence to be worried about.
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
Quoted 11 lines
Above, if we're going to use the mfn, then we've just done a successful:
get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page)which should hold it in a state that we're ok with until we're done with it -- see
put_page_and_typeinargo_ring_remove_mfns.Quoted 4 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 629
Quoted 2 lines
xen/common/argo.c +2
-0+/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock);I don't understand this W(L2) nomenclature, is this explain somewhere?
Quoted 8 lines
Yes, sort of. Lock "L2" is the per-domain argo lock, identified in a comment near the top of the file. It's a read-write lock, so 'W' means:
take the write lock on it.Quoted 1 line
Also there's no such comment when you take the global
argo_lockabove.L2 covers more interesting work than L1, which is why there are more comments pertaining to it than L1.
I would add such comments about which locks protect what items to the declaration of the locks, rather than the usage place. I don't see a lot of value in the comments there unless they maybe describe an exception or a corner case, but that might just be my taste.
Quoted 11 lines
Quoted 11 lines
xen/include/public/argo.h typedef struct argo_ring · line 76
Quoted 8 lines
xen/include/public/argo.h +8
-0+/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits. +*/ +#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf)Why not just use ROUNDUP?
Quoted 8 lines
Quoted 2 lines
And in any case this shouldn't be on the public header IMO, since it's not part of the interface AFAICT.
Well, in version two it's now:
XEN_ARGO_ROUNDUP:-)
because it does need to be in the public header because it's used within the Linux device driver, and items in that public Xen header need the 'xen' prefix (so they now do). Within the Linux code, it's used to choose a sensible ring size, and also used when manipulating therx_ptron the guest side.
I'm quite sure Linux (or any other OS) will have a roundup helper, or if there's indeed an OS without a roundup helper it should be added to the generic OS code. There's nothing Xen or ARGO specific in this roundup helper, hence I see no need to add it to the public header.
I think you should instead:
#define XEN_ARGO_MESSAGE_SIZE 0xf
Or some such and use that value with the OS roundup helper.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 11 lines · Roger Pau Monné
Quoted 9 lines · Christopher Clark
Quoted 7 lines · Roger Pau Monné
xen/common/argo.c struct argo_domain · line 106
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static inline uint16_t +argo_hash_fn (const struct argo_ring_id *id)No need for the
argo_prefix for static functions, this is already an argo specific file.
Quoted 6 lines
Quoted 3 lines
Although the compiler could live without the prefix, I'm finding it helpful to very easily determine that functions being used are not defined elsewhere within Xen; so I've left the prefix as is for version two of this series.
Why do you care whether they are defined elsewhere in Xen? The scope of static functions is limited to the translation unit anyway.
ok, I'll remove the prefixes - you're right that I shouldn't care whether they are defined elsewhere in Xen, and Jan's points about the string table expansion and serial line bandwidth are true - I had not considered those. Would adding a note to describe this reasoning to the CODING_STYLE document be welcome?
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
xen/common/argo.c +6
-0+#else + *mfn =p2m_lookup (d,_gfn (pfn), &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ;xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 349
Quoted 10 lines
xen/common/argo.c +10
-0+#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d, pfn);If you do this
put_gfnhere, by the time you check that the gfn -> mfn matches your expectations the guest might have somehow changed the gfn -> mfn mapping already (for example by ballooning down memory?)
Quoted 6 lines
Quoted 3 lines
If the guest does that, I think it only harms itself. If for some reason a memory access is denied, then the op would just fail. I don't think there's a more serious consequence to be worried about.
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
2) when the hypervisor writes into guest memory:
- where the tx_ptr index is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notify
or is having PGT_writable_page type and ownership by the domain sufficient?
For 1), I think there's some use in saying no to a guest that has supplied a region that appears misconfigured.
For 2), input would be appreciated. It currently works under the assumption that a p2m type check is unnecessary, which is why the put_gfn is where it is.
For further background context, here's my understanding of this section:
When the guest invokes the hypercall operation to register a ring, it identifies the memory that it owns and wants the hypervisor to use by supplying an array of gfns (or in v2, addresses which are shifted to extract their gfns).
The hypervisor translates from gfns to mfns, using the translation that exists at that time, and then refers to that memory internally by mfn from there on out. This find_ring_mfn function is where the gfn->mfn translation happens. (The variable name does needs renaming from pfn, as you noted - thanks.)
To do the translation from gfn to mfn, (on x86) it's using get_gfn_unshare. That's doing three things:
* returns the mfn, if there is one.
* returns the p2m type of that memory.
* acquires a reference to that gfn, which needs to be dropped at some point.
The p2m type type check on the gfn in find_ring_mfn at that time is possibly conservative, rejecting more types than perhaps it needs to, but the type that it accepts (p2m_ram_rw) is sane. It is a validation of the p2m type at that instant, intended to detect if the guest has supplied memory to the ring register op that does not make sense for it to use as a ring, as indicated by the current p2m type, and if so, fail early, or indicate that a retry later is needed.
Then the get_page_and_type call is where the memory identified by the mfn that was just obtained, gets locked to PGT_writable_page type, and ownership fixed to its current owner domain, by adding to its reference count.
Then the gfn reference count is dropped with the put_gfn call. This means that the guest can elect to change the p2m type afterwards, if it wants; (any change needs to be consistent with its domain ownership and PGT_writable_page type though -- not sure if that constrains possible types).
That memory can have guest-supplied data written into it, either by the domain owning the page itself, or in response to argo sendv operations by other domains that are authorized to transmit into the ring.
Your note that the "check itself is racy": ie. that a change of p2m type could occur immediately afterwards is true.
So: Do you think that a check on the current p2m type of the pages in the ring is needed at the points where the hypervisor issues writes into that ring memory?
Quoted 15 lines
Quoted 15 lines
Above, if we're going to use the mfn, then we've just done a successful:
get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page)which should hold it in a state that we're ok with until we're done with it -- see
put_page_and_typeinargo_ring_remove_mfns.Quoted 4 lines
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 629
Quoted 2 lines
xen/common/argo.c +2
-0+/* W(L2) protects all the elements of the domain's ring_info */ +write_lock (&d->argo->lock);I don't understand this W(L2) nomenclature, is this explain somewhere?
Yes, sort of. Lock "L2" is the per-domain argo lock, identified in a comment near the top of the file. It's a read-write lock, so 'W' means:
take the write lock on it.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Also there's no such comment when you take the global
argo_lockabove.L2 covers more interesting work than L1, which is why there are more comments pertaining to it than L1.
I would add such comments about which locks protect what items to the declaration of the locks, rather than the usage place.
ack, and those comments are there, introduced earlier in the series. There's a dedicated section with comments on locking near the top (titled: "locking is organized as follows") and comments within the argo_ring_info data structure definition for protection of each field.
Quoted 3 lines
I don't see a
lot of value in the comments there unless they maybe describe an exception or a corner case, but that might just be my taste.
ack. It's notable with that specific site that it's (intentionally, necessarily) a write_lock that is being taken, rather than a read_lock; maybe that's enough to fall under your exception / corner case condition. I can drop it if it's really unwanted.
Quoted 25 lines
Quoted 11 lines
Quoted 11 lines
xen/include/public/argo.h typedef struct argo_ring · line 76
Quoted 8 lines
xen/include/public/argo.h +8
-0+/* +* Messages on the ring are padded to 128 bits +* Len here refers to the exact length of the data not including the +* 128 bit header. The message uses +* ((len + 0xf) & ~0xf) + sizeof(argo_ring_message_header) bytes. +* Using typeof(a) make clear that this does not truncate any high-order bits. +*/ +#define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf)Why not just use ROUNDUP?
Quoted 8 lines
Quoted 2 lines
And in any case this shouldn't be on the public header IMO, since it's not part of the interface AFAICT.
Well, in version two it's now:
XEN_ARGO_ROUNDUP:-)
because it does need to be in the public header because it's used within the Linux device driver, and items in that public Xen header need the 'xen' prefix (so they now do). Within the Linux code, it's used to choose a sensible ring size, and also used when manipulating therx_ptron the guest side.I'm quite sure Linux (or any other OS) will have a roundup helper, or if there's indeed an OS without a roundup helper it should be added to the generic OS code. There's nothing Xen or ARGO specific in this roundup helper, hence I see no need to add it to the public header.
Quoted 5 lines
I think you should instead:
#define
XEN_ARGO_MESSAGE_SIZE0xfOr some such and use that value with the OS roundup helper.
OK, I'll go and look. Thanks for the suggestion.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 5 lines · Christopher Clark
Quoted 4 lines · Roger Pau Monné
Quoted 2 lines · Christopher Clark
Quoted 1 line · Roger Pau Monné
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of ram_rw type.
Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notify
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 5 lines
or is having
PGT_writable_pagetype and ownership by the domain sufficient?For 1), I think there's some use in saying no to a guest that has supplied a region that appears misconfigured.
Sure, when you setup the ring in Xen you should execute all the checks and make sure the ring pages are of type ram_rw and take a reference to each page.
Quoted 3 lines
For 2), input would be appreciated. It currently works under the assumption that a p2m type check is unnecessary, which is why the
put_gfnis where it is.
Right, and this should go away since it's confusing and unnecessary AFAICT.
Quoted 28 lines
For further background context, here's my understanding of this section:
When the guest invokes the hypercall operation to register a ring, it identifies the memory that it owns and wants the hypervisor to use by supplying an array of gfns (or in v2, addresses which are shifted to extract their gfns).
The hypervisor translates from gfns to mfns, using the translation that exists at that time, and then refers to that memory internally by mfn from there on out. This
find_ring_mfnfunction is where thegfn->mfntranslation happens. (The variable name does needs renaming from pfn, as you noted - thanks.)To do the translation from gfn to mfn, (on x86) it's using
get_gfn_unshare. That's doing three things:
* returns the mfn, if there is one.
* returns the p2m type of that memory.
* acquires a reference to that gfn, which needs to be dropped at some point.The p2m type type check on the gfn in
find_ring_mfnat that time is possibly conservative, rejecting more types than perhaps it needs to, but the type that it accepts (p2m_ram_rw) is sane. It is a validation of the p2m type at that instant, intended to detect if the guest has supplied memory to the ring register op that does not make sense for it to use as a ring, as indicated by the current p2m type, and if so, fail early, or indicate that a retry later is needed.
Quoted 4 lines
Then the
get_page_and_typecall is where the memory identified by the mfn that was just obtained, gets locked toPGT_writable_pagetype, and ownership fixed to its current owner domain, by adding to its reference count.
Here you should make sure the get_page is performed while the gfn is still locked, and once you have a reference to the page you can unlock the gfn. As said above, if the guest then wants to mess up with the gfn -> mfn mapping that's fine, the hypervisor already has a reference to the underlying original memory page.
Quoted 4 lines
Then the gfn reference count is dropped with the
put_gfncall. This means that the guest can elect to change the p2m type afterwards, if it wants; (any change needs to be consistent with its domain ownership andPGT_writable_pagetype though -- not sure if that constrains possible types).
Hm, not really get_gfn / put_gfn doesn't actually take references, but rather pick the p2m lock. What takes references is get_page.
Quoted 10 lines
That memory can have guest-supplied data written into it, either by the domain owning the page itself, or in response to argo sendv operations by other domains that are authorized to transmit into the ring.
Your note that the "check itself is racy": ie. that a change of p2m type could occur immediately afterwards is true.
So: Do you think that a check on the current p2m type of the pages in the ring is needed at the points where the hypervisor issues writes into that ring memory?
No, as long as you have a reference to the page (note: not the gfn) that you are writing to it should be fine.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 5 lines · Christopher Clark
Quoted 4 lines · Roger Pau Monné
Quoted 2 lines · Christopher Clark
Quoted 1 line · Roger Pau Monné
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notify
Quoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 32 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 5 lines · Christopher Clark
Quoted 4 lines · Roger Pau Monné
Quoted 2 lines · Christopher Clark
Quoted 1 line · Roger Pau Monné
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 49 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 16:35, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 4 lines · Christopher Clark
Quoted 3 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
On Wed, Dec 12, 2018 at 8:48 AM Roger Pau Monné <[email protected]>
Quoted 26 lines · wrote:
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Adding the introspection guys.
Quoted 52 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 16:35, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 4 lines · Christopher Clark
Quoted 3 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
On Wed, Dec 12, 2018 at 8:48 AM Roger Pau Monné <[email protected]>
Quoted 26 lines · wrote:
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
Quoted 6 lines
Quoted 1 line
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.
There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 68 lines · Roger Pau Monné
Adding the introspection guys.
Quoted 52 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 16:35, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 4 lines · Christopher Clark
Quoted 3 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
On Wed, Dec 12, 2018 at 8:48 AM Roger Pau Monné <[email protected]>
Quoted 26 lines · wrote:
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
Quoted 6 lines
Quoted 1 line
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Tamas
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 77 lines · Tamas K Lengyel
Quoted 68 lines · Roger Pau Monné
Adding the introspection guys.
Quoted 52 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 16:35, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 4 lines · Christopher Clark
Quoted 3 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
On Wed, Dec 12, 2018 at 8:48 AM Roger Pau Monné <[email protected]>
Quoted 26 lines · wrote:
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
Quoted 6 lines
Quoted 1 line
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Quoted 7 lines
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Indeed, that's how it goes.
Thanks,
Razvan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Tamas K LengyelQuoted 78 lines · Tamas K Lengyel
Quoted 68 lines · Roger Pau Monné
Adding the introspection guys.
Quoted 52 lines · Jan Beulich
Quoted 52 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 16:35, <[email protected]> wrote:
Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:57, <[email protected]> wrote:
Quoted 4 lines · Christopher Clark
Quoted 3 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
On Wed, Dec 12, 2018 at 8:48 AM Roger Pau Monné <[email protected]>
Quoted 26 lines · wrote:
Quoted 19 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines · Christopher Clark
Then I wonder why you need such check in any case if the code can handle such cases, the more than the check itself is racy.
OK, so at the root of the question here is: does it matter what the p2m type of the memory is at these points:
1) when the gfn is translated to mfn, at the time of ring registration
This is the important check, because that's where you should take a reference to the page. In this case you should check that the page is of
ram_rwtype.Quoted 4 lines
2) when the hypervisor writes into guest memory:
- where thetx_ptrindex is initialized in the register op
- where ringbuf data is written in sendv
- where ring description data is written in notifyQuoted 6 lines
As long as you keep a reference to the pages that are part of the ring you don't need to do any checks when writing/reading from them. If the guest messes up it's p2m and does change the gfn -> mfn mappings for pages that are part of the ring that's the guest problem, the hypervisor still has a reference to those pages so they won't be reused.
Quoted 13 lines
For use cases like introspection this may not be fully correct, but it may also be that my understanding there isn't fully correct. If introspection agents care about _any_ writes to a page, hypervisor ones (which in most cases are merely writes on behalf of the guest) might matter as well. I think to decide whether page accesses need to be accompanied by any checks (and if so, which ones) one needs to
- establish what p2m type transitions are possible for a given page,
- verify what restrictions may occur "behind the back" of the entity wanting to do the accesses,
- explore whether doing the extra checking at p2m type change time wouldn't be better than at the time of access.Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
Quoted 6 lines
Quoted 1 line
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Quoted 7 lines
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
If that's the case, and introspection doesn't care about hypervisor accesses to guest pages, then just getting a reference to the underlying page when the ring is setup should be enough. There's no need to check the gfn -> mfn relation every time there's an hypervisor access to the ring.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 31 lines · Roger Pau Monné
Quoted 23 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Quoted 7 lines
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Quoted 3 lines
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
Would these two guests be HVM guests? Introspection only works for HVM guests. I'm not sure I follow your scenario though. How would these guests collaborate to escape introspection via grants?
Quoted 5 lines
If that's the case, and introspection doesn't care about hypervisor accesses to guest pages, then just getting a reference to the underlying page when the ring is setup should be enough. There's no need to check the gfn -> mfn relation every time there's an hypervisor access to the ring.
I think so, but I might be missing something.
Thanks,
Razvan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 33 lines · Razvan Cojocaru
Quoted 31 lines · Roger Pau Monné
Quoted 30 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Quoted 7 lines
Quoted 3 lines
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
Would these two guests be HVM guests? Introspection only works for HVM guests. I'm not sure I follow your scenario though. How would these guests collaborate to escape introspection via grants?
If there are two domains acting maliciously in concert to bypass monitoring of memory writes they could achieve that with grants, yes. Say a write-monitored page is grant-shared to another domain, which then does the write on behalf of the first. I wouldn't say that's "completely bypassing introspection" though, there are many types of events that can be monitored, write-accesses are only one. I'm not aware of any mechanism that can be used to limit which pages can be shared but you can use XSM to restrict which domains can share pages to begin with. That's normally enough.
Tamas
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
Quoted 44 lines · Tamas K Lengyel
Quoted 33 lines · Razvan Cojocaru
Quoted 31 lines · Roger Pau Monné
Quoted 30 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Quoted 7 lines
Quoted 3 lines
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
Would these two guests be HVM guests? Introspection only works for HVM guests. I'm not sure I follow your scenario though. How would these guests collaborate to escape introspection via grants?
Quoted 9 lines
If there are two domains acting maliciously in concert to bypass monitoring of memory writes they could achieve that with grants, yes. Say a write-monitored page is grant-shared to another domain, which then does the write on behalf of the first. I wouldn't say that's "completely bypassing introspection" though, there are many types of events that can be monitored, write-accesses are only one. I'm not aware of any mechanism that can be used to limit which pages can be shared but you can use XSM to restrict which domains can share pages to begin with. That's normally enough.
Yes, I assumed that would be the way to protect against such attacks,
ie: limiting to which guests pages can be shared. I think just making sure the right access checks are placed in XSM (just like they are for grants) should be enough.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 13/25 argo: implement the register op
Quoted 44 lines · Tamas K Lengyel
Quoted 33 lines · Razvan Cojocaru
Quoted 31 lines · Roger Pau Monné
Quoted 30 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Quoted 7 lines
Quoted 3 lines
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
Would these two guests be HVM guests? Introspection only works for HVM guests. I'm not sure I follow your scenario though. How would these guests collaborate to escape introspection via grants?
Quoted 9 lines
If there are two domains acting maliciously in concert to bypass monitoring of memory writes they could achieve that with grants, yes. Say a write-monitored page is grant-shared to another domain, which then does the write on behalf of the first. I wouldn't say that's "completely bypassing introspection" though, there are many types of events that can be monitored, write-accesses are only one. I'm not aware of any mechanism that can be used to limit which pages can be shared but you can use XSM to restrict which domains can share pages to begin with. That's normally enough.
Yes, I assumed that would be the way to protect against such attacks,
ie: limiting to which guests pages can be shared. I think just making sure the right access checks are placed in XSM (just like they are for grants) should be enough.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 13/25] argo: implement the register op
In reply to Tamas K LengyelQuoted 43 lines · Tamas K Lengyel
Quoted 33 lines · Razvan Cojocaru
Quoted 31 lines · Roger Pau Monné
Quoted 30 lines
Quoted 15 lines
Quoted 9 lines
Quoted 4 lines
Maybe this is use-case is different, but how does introspection handle accesses to the shared info page or the runstate info for example?
I would consider argo to be the same in this regard.
Not exactly: The shared info page is special in any event. For runstate info (and alike - there's also struct
vcpu_time_info) I'd question correctness of the current handling. If that's wrong already, I'd prefer if the issue wasn't spread.There are also grants, which when used together with another guest on the same host could allow to bypass introspection AFAICT? (unless there's some policy applied that limit grant sharing to trusted domains)
Quoted 7 lines
TBH I'm not sure how to handle hypoervisor accesses with introspection. My knowledge of introspection is fairly limited, but it pauses the guest and sends a notification to an in guest agent. I'm not sure this is applicable to hypervisor writes, since it's not possible to pause hypervisor execution and wait for a response from a guest agent.
Introspection applications only care about memory accesses performed by the guest. Hypervisor accesses to monitored pages are not included when monitoring - it is actually a feature when using the emulator in Xen to continue guest execution because the hypervisor ignores EPT memory permissions that trip the guest for introspection. So having the hypervisor access memory or a grant-shared page being accessed in another domain are not a problem for introspection.
Quoted 7 lines
Quoted 3 lines
Can't then two guests running on the same host be able to completely bypass introspection? I guess you prevent this by limiting to which guests pages can be shared?
Would these two guests be HVM guests? Introspection only works for HVM guests. I'm not sure I follow your scenario though. How would these guests collaborate to escape introspection via grants?
Quoted 9 lines
If there are two domains acting maliciously in concert to bypass monitoring of memory writes they could achieve that with grants, yes. Say a write-monitored page is grant-shared to another domain, which then does the write on behalf of the first. I wouldn't say that's "completely bypassing introspection" though, there are many types of events that can be monitored, write-accesses are only one. I'm not aware of any mechanism that can be used to limit which pages can be shared but you can use XSM to restrict which domains can share pages to begin with. That's normally enough.
Right, I agree. We're not currently dealing with that case and assume that XSM (or a similar mechanism) will be used in scenarios where this level of access is possible.
Thanks,
Razvan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 14/25 argo: implement the unregister op
In reply to Christopher ClarkTakes a single argument: a handle to the registered ring.
The ring's entry is removed from the hashtable of registered rings; any entries for pending notifications are removed; and the ring is unmapped from Xen's address space.
--- xen/common/argo.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 9 +++++++ 2 files changed, 71 insertions(+)
xen/common/argo.c +62 -0
@@ -510,6 +510,59 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) }static long +argo_unregister_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; + +read_lock (&argo_lock); + +do { +if ( !d->argo ) + { + ret = -ENODEV ; +break ; + } + + ret =copy_from_guest_errno (&ring, ring_hnd,1 ); +if ( ret ) +break ; + +if ( ring.magic !=ARGO_RING_MAGIC ) + { +argo_dprintk ( +"ring.magic(%" PRIx64") != ARGO_RING_MAGIC(%llx), EINVAL\n" , + ring.magic,ARGO_RING_MAGIC ); + ret = -EINVAL ; +break ; + } + + ring.id.addr.domain_id = d->domain_id; + +write_lock (&d->argo->lock); + + ring_info =argo_ring_find_info (d, &ring.id); +if ( ring_info ) +argo_ring_remove_info (d, ring_info); + +write_unlock (&d->argo->lock); + +if ( !ring_info ) + { +argo_dprintk ("ENOENT\n" ); + ret = -ENOENT ; +break ; + } + + }while (0 ); + +read_unlock (&argo_lock); +return ret; +} + +static long argo_register_ring(struct domain *d, XEN_GUEST_HANDLE_PARAM(argo_ring_t) ring_hnd, XEN_GUEST_HANDLE_PARAM(argo_pfn_t) pfn_hnd, uint32_t npage,
@@ -751,6 +804,15 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist);break ; } +case ARGO_MESSAGE_OP_unregister_ring: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ; + rc =argo_unregister_ring (d, ring_hnd); +break ; + }default : rc = -ENOSYS ;break ;
xen/include/public/argo.h +9 -0
@@ -116,4 +116,13 @@ struct argo_ring_message_header/* Mask for all defined flags */ #define ARGO_REGISTER_FLAG_MASK ARGO_REGISTER_FLAG_FAIL_EXIST +/* +* ARGO_MESSAGE_OP_unregister_ring +* +* Unregister a previously-registered ring, ending communication. +* +* arg1: XEN_GUEST_HANDLE(argo_ring_t) +*/ +#define ARGO_MESSAGE_OP_unregister_ring 2 +#endif
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 14/25] argo: implement the unregister op
Quoted 75 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 14/25] argo: implement the unregister opTakes a single argument: a handle to the registered ring.
The ring's entry is removed from the hashtable of registered rings; any entries for pending notifications are removed; and the ring is unmapped from Xen's address space.
Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 9 +++++++ 2 files changed, 71 insertions(+)xen/common/argo.c +36
-0@@ -510,6 +510,59 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) }static long +argo_unregister_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; + +read_lock (&argo_lock); + + do { + if ( !d->argo ) + { + ret = -ENODEV; + break; + } + + ret = copy_from_guest_errno(&ring, ring_hnd, 1); + if ( ret ) + break; + + if ( ring.magic != ARGO_RING_MAGIC ) + { + argo_dprintk( + "ring.magic(%"PRIx64") != ARGO_RING_MAGIC(%llx), EINVAL\n", + ring.magic, ARGO_RING_MAGIC); + ret = -EINVAL; + break; + } + + ring.id.addr.domain_id = d->domain_id; + + write_lock(&d->argo->lock); + + ring_info = argo_ring_find_info(d, &ring.id); + if ( ring_info ) + argo_ring_remove_info(d, ring_info);
xen/common/argo.c argo_ring_find_info(const struct domain *d, const · line 560
Quoted 10 lines
xen/common/argo.c +10
-0+ +write_unlock (&d->argo->lock); + +if ( !ring_info ) + { +argo_dprintk ("ENOENT\n" ); + ret = -ENOENT ; +break ; + } +
Stray blank line?
xen/common/argo.c argo_ring_find_info(const struct domain *d, const · line 562
Quoted 2 lines
xen/common/argo.c +2
-0+ }while (0 ); +
Again, forward goto style is more conventional.
Quoted 45 lines
xen/common/argo.c +5
-0+read_unlock (&argo_lock); +return ret; +} + +static long argo_register_ring(struct domain *d,
XEN_GUEST_HANDLE_PARAM(argo_ring_t)ring_hnd,XEN_GUEST_HANDLE_PARAM(argo_pfn_t)pfn_hnd,uint32_t
npage,xen/common/argo.c +9
-0@@ -751,6 +804,15 @@ do_argo_message_op(int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1, rc =argo_register_ring (d, ring_hnd, pfn_hnd, npage, fail_exist);break ; } +case ARGO_MESSAGE_OP_unregister_ring: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd = +guest_handle_cast (arg1, argo_ring_t); +if (unlikely (!guest_handle_okay (ring_hnd,1 )) ) +break ; + rc =argo_unregister_ring (d, ring_hnd); +break ; + }default : rc = -ENOSYS ;break;xen/include/public/argo.h +9
-1@@ -116,4 +116,13 @@ struct argo_ring_message_header/* Mask for all defined flags */ #define ARGO_REGISTER_FLAG_MASK ARGO_REGISTER_FLAG_FAIL_EXIST +/* +* ARGO_MESSAGE_OP_unregister_ring +* +* Unregister a previously-registered ring, ending communication. +* +* arg1: XEN_GUEST_HANDLE(argo_ring_t) +*/ +#define ARGO_MESSAGE_OP_unregister_ring 2 +#endif --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 14/25] argo: implement the unregister op
In reply to Christopher ClarkQuoted 27 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +20
-0@@ -510,6 +510,59 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) }static long +argo_unregister_ring (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd) +{ +struct argo_ring ring; +struct argo_ring_info *ring_info; +int ret =0 ; + +read_lock (&argo_lock); + + do { + if ( !d->argo ) + { + ret = -ENODEV; + break; + } + + ret = copy_from_guest_errno(&ring, ring_hnd, 1); + if ( ret ) + break; +
xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 542
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ring.magic !=ARGO_RING_MAGIC ) + { +argo_dprintk ( +"ring.magic(%" PRIx64") != ARGO_RING_MAGIC(%llx), EINVAL\n" , + ring.magic,ARGO_RING_MAGIC ); + ret = -EINVAL ; +break ; + } + + ring.id.addr.domain_id = d->domain_id;
Why the override?
Quoted 10 lines
xen/common/argo.c +10
-0+write_lock (&d->argo->lock); + + ring_info =argo_ring_find_info (d, &ring.id); +if ( ring_info ) +argo_ring_remove_info (d, ring_info); + +write_unlock (&d->argo->lock); + +if ( !ring_info ) + {
xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 563
Quoted 10 lines
xen/common/argo.c +10
-0+argo_dprintk ("ENOENT\n" ); + ret = -ENOENT ; +break ; + } + + }while (0 ); + +read_unlock (&argo_lock); +return ret; +}
Blank line ahead of the main return statement of a function please.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 15/25 argo: implement the sendv op
In reply to Christopher Clarksendv operation is invoked to perform a synchronous send of buffers contained in iovs to a remote domain's registered ring.
It takes:
* A destination address (domid, port) for the ring to send to. It performs a most-specific match lookup, to allow for wildcard.
* A source address, used to inform the destination of where to reply.
* The address of an array of iovs containing the data to send
* .. and the length of that array of iovs
* and a 32-bit message type, available to communicate message context data (eg. kernel-to-kernel, separate from the application data).
If insufficient space exists in the destination ring, it will return -EAGAIN and Xen will notify the caller when sufficient space becomes available.
Accesses to the ring indices are appropriately atomic. The rings are mapped into Xen's private address space to write as needed and the mappings are retained for later use.
When locating the destination ring, a check is performed via a cookie installed at ring registration time, to ensure that the source domain is the same as it was when the ring was registered.
Fixed-size types are used in some areas within this code where caution around avoiding integer overflow is important.
--- xen/common/argo.c | 528 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 59 ++++++ 2 files changed, 587 insertions(+)
xen/common/argo.c +528 -0
@@ -24,10 +24,13 @@#include <xen/domain_page.h> #include <xen/guest_access.h> #include <xen/time.h> +#include <xsm/xsm.h> DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (argo_send_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (uint8_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled =0 ;
@@ -166,6 +169,21 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#endif /* +* Event channel +*/ + +static void +argo_signal_domain (struct domain *d) +{ +argo_dprintk ("signalling domid:%d\n" , d->domain_id); + +if ( !d->argo )/* This can happen if the domain is being destroyed */ +return ; + +evtchn_send (d, d->argo->evtchn_port); +} + +/* * ring buffer */
@@ -259,6 +277,333 @@ argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr)return 0 ; } +static int +argo_memcpy_to_guest_ring (struct argo_ring_info *ring_info, +uint32_t offset, +const void *src, +XEN_GUEST_HANDLE (uint8_t ) src_hnd, +uint32_t len) +{ +int page = offset >>PAGE_SHIFT ; +uint8_t *dst; +int ret; +unsigned int src_offset =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + + offset &= ~PAGE_MASK ; + +if ( (len >ARGO_MAX_RING_SIZE ) || (offset >ARGO_MAX_RING_SIZE ) ) +return -EFAULT ; + +while ( (offset + len) >PAGE_SIZE ) + { + ret =argo_ring_map_page (ring_info, page, &dst); +if ( ret ) +return ret; + +if ( src ) + { +memcpy (dst + offset, src + src_offset,PAGE_SIZE - offset); + src_offset += (PAGE_SIZE - offset); + } +else + { + ret =copy_from_guest_errno (dst + offset, src_hnd, +PAGE_SIZE - offset); +if ( ret ) +return ret; + +guest_handle_add_offset (src_hnd,PAGE_SIZE - offset); + } + + page++; + len -=PAGE_SIZE - offset; + offset =0 ; + } + + ret =argo_ring_map_page (ring_info, page, &dst); +if ( ret ) + { +argo_dprintk ("argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %d of %d\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + page, ring_info->nmfns); +return ret; + } + +if ( src ) +memcpy (dst + offset, src + src_offset, len); +else + ret =copy_from_guest_errno (dst + offset, src_hnd, len); + +return ret; +} + +static int +argo_ringbuf_get_rx_ptr (struct argo_ring_info *ring_info,uint32_t *rx_ptr) +{ +uint8_t *src; +argo_ring_t *ringp; +int ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( !ring_info->nmfns || ring_info->nmfns < ring_info->npage ) +return -EINVAL ; + + ret =argo_ring_map_page (ring_info,0 , &src); +if ( ret ) +return ret; + + ringp = (argo_ring_t *)src; + + *rx_ptr =read_atomic (&ringp->rx_ptr); + +return 0 ; +} + +/* +* argo_sanitize_ring creates a modified copy of the ring pointers +* where the rx_ptr is rounded up to ensure it is aligned, and then +* ring wrap is handled. Simplifies safe use of the rx_ptr for +* available space calculation. +*/ +static void +argo_sanitize_ring (argo_ring_t *ring,const struct argo_ring_info *ring_info) +{ +uint32_t rx_ptr = ring->rx_ptr; + + ring->tx_ptr = ring_info->tx_ptr; + ring->len = ring_info->len; + + rx_ptr =ARGO_ROUNDUP (rx_ptr); +if ( rx_ptr >= ring_info->len ) + rx_ptr =0 ; + + ring->rx_ptr = rx_ptr; +} + +/* +* argo_iov_count returns its count on success via an out variable +* to avoid potential for a negative return value to be used incorrectly +* (eg. coerced into an unsigned variable resulting in a large incorrect value) +*/ +static int +argo_iov_count (XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t *count) +{ +argo_iov_t iov; +uint32_t sum_iov_lens =0 ; +int ret; + +if ( niov >ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +return ret; + +/* check each to protect sum against integer overflow */ +if ( iov.iov_len >ARGO_MAX_RING_SIZE ) +return -EINVAL ; + + sum_iov_lens += iov.iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >ARGO_MAX_MSG_SIZE ) +return -EINVAL ; + +guest_handle_add_offset (iovs,1 ); + } + + *count = sum_iov_lens; +return 0 ; +} + +static int +argo_ringbuf_insert (struct domain *d, +struct argo_ring_info *ring_info, +const struct argo_ring_id *src_id, +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t message_type,unsigned long *out_len) +{ +argo_ring_t ring; +struct argo_ring_message_header mh = {0 }; +int32_t sp; +int32_t ret =0 ; +uint32_t len; +uint32_t iov_len; +uint32_t sum_iov_len =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( (ret =argo_iov_count (iovs, niov, &len)) ) +return ret; + +if ( ((ARGO_ROUNDUP (len) +sizeof (struct argo_ring_message_header ) ) >= + ring_info->len) + || (len >ARGO_MAX_MSG_SIZE ) ) +return -EMSGSIZE ; + +do { + ret =argo_ringbuf_get_rx_ptr (ring_info, &ring.rx_ptr); +if ( ret ) +break ; + +argo_sanitize_ring (&ring, ring_info); + +argo_dprintk ("ring.tx_ptr=%d ring.rx_ptr=%d ring.len=%d" +" ring_info->tx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr, ring.len, ring_info->tx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) + sp = ring_info->len; +else + { + sp = ring.rx_ptr - ring.tx_ptr; +if ( sp <0 ) + sp += ring.len; + } + +if ( (ARGO_ROUNDUP (len) +sizeof (struct argo_ring_message_header )) >= sp ) + { +argo_dprintk ("EAGAIN\n" ); + ret = -EAGAIN ; +break ; + } + + mh.len = len +sizeof (struct argo_ring_message_header ); + mh.source.port = src_id->addr.port; + mh.source.domain_id = src_id->addr.domain_id; + mh.message_type = message_type; + +/* +* For this copy to the guest ring, tx_ptr is always 16-byte aligned +* and the message header is 16 bytes long. +*/ +BUILD_BUG_ON (sizeof (struct argo_ring_message_header ) !=ARGO_ROUNDUP (1 )); + +if ( (ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), + &mh, +XEN_GUEST_HANDLE_NULL (uint8_t), +sizeof (mh))) ) +break ; + + ring.tx_ptr +=sizeof (mh); +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +while ( niov-- ) + { +XEN_GUEST_HANDLE_PARAM (uint8_t ) bufp_hnd; +XEN_GUEST_HANDLE (uint8_t ) buf_hnd; +argo_iov_t iov; + + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +break ; + + bufp_hnd =guest_handle_from_ptr ((uintptr_t )iov.iov_base, uint8_t); + buf_hnd =guest_handle_from_param (bufp_hnd, uint8_t); + iov_len = iov.iov_len; + +if ( !iov_len ) + { +printk (XENLOG_ERR "argo: iov.iov_len=0 iov.iov_base=%" + PRIx64" ring (vm%u:%x vm%d)\n" , + iov.iov_base, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +guest_handle_add_offset (iovs,1 ); +continue ; + } + +if ( iov_len >ARGO_MAX_MSG_SIZE ) + { + ret = -EINVAL ; +break ; + } + + sum_iov_len += iov_len; +if ( sum_iov_len > len ) + { + ret = -EINVAL ; +break ; + } + +if (unlikely (!guest_handle_okay (buf_hnd, iov_len)) ) + { + ret = -EFAULT ; +break ; + } + + sp = ring.len - ring.tx_ptr; + +if ( iov_len > sp ) + { + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, sp); +if ( ret ) +break ; + + ring.tx_ptr =0 ; + iov_len -= sp; +guest_handle_add_offset (buf_hnd, sp); + } + + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, iov_len); +if ( ret ) +break ; + + ring.tx_ptr += iov_len; + +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +guest_handle_add_offset (iovs,1 ); + } + +if ( ret ) +break ; + + ring.tx_ptr =ARGO_ROUNDUP (ring.tx_ptr); + +if ( ring.tx_ptr >= ring_info->len ) + ring.tx_ptr -= ring_info->len; + +mb (); + ring_info->tx_ptr = ring.tx_ptr; +if ( (ret =argo_update_tx_ptr (ring_info, ring.tx_ptr)) ) +break ; + + }while (0 ); + +/* +* At this point it is possible to unmap the ring_info, ie: +* argo_ring_unmap(ring_info); +* but performance should be improved by not doing so, and retaining +* the mapping. +* An XSM policy control over level of confidentiality required +* versus performance cost could be added to decide that here. +* See the similar comment in argo_ring_map_page re: write-only mappings. +*/ + +if ( !ret ) + *out_len = len; + +return ret; +} +/* * pending */
@@ -282,6 +627,47 @@ argo_pending_remove_all(struct argo_ring_info *ring_info) } } +static int +argo_pending_queue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ent = xmalloc(struct argo_pending_ent); + +if ( !ent ) +return -ENOMEM ; + + ent->len = len; + ent->id = src_id; + +hlist_add_head (&ent->node, &ring_info->pending); + +return 0 ; +} + +static int +argo_pending_requeue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct hlist_node *node; +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry (ent, node, &ring_info->pending, node) + { +if ( ent->id == src_id ) + { +if ( ent->len < len ) + ent->len = len; +return 0 ; + } + } + +return argo_pending_queue (ring_info, src_id, len); +} +static void argo_ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) {
@@ -509,6 +895,28 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id; + id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info; + + id.partner =ARGO_DOMID_ANY ; + +return argo_ring_find_info (d, &id); +} +static long argo_unregister_ring (struct domain *d,XEN_GUEST_HANDLE_PARAM (argo_ring_t) ring_hnd)
@@ -754,6 +1162,103 @@ argo_register_ring(struct domain *d,return ret; } +/* +* io +*/ + +static long +argo_sendv (struct domain *src_d,const argo_addr_t *src_addr, +const argo_addr_t *dst_addr, +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint32_t niov, +uint32_t message_type) +{ +struct domain *dst_d =NULL ; +struct argo_ring_id src_id; +struct argo_ring_info *ring_info; +int ret =0 ; +unsigned long len =0 ; + +ASSERT (src_d->domain_id == src_addr->domain_id); + +read_lock (&argo_lock); + +do { +if ( !src_d->argo ) + { + ret = -ENODEV ; +break ; + } + + src_id.addr.pad =0 ; + src_id.addr.port = src_addr->port; + src_id.addr.domain_id = src_d->domain_id; + src_id.partner = dst_addr->domain_id; + + dst_d =get_domain_by_id (dst_addr->domain_id); +if ( !dst_d || !dst_d->argo ) + { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_send (src_d, dst_d); +if ( ret ) + { +printk (XENLOG_ERR "argo: XSM REJECTED %i -> %i\n" , + src_addr->domain_id, dst_addr->domain_id); +break ; + } + +read_lock (&dst_d->argo->lock); + +do { + ring_info =argo_ring_find_info_by_match (dst_d, dst_addr->port, + src_addr->domain_id, + src_d->argo->domain_cookie); +if ( !ring_info ) + { +printk (XENLOG_ERR "argo: vm%u connection refused, " +"src (vm%u:%x) dst (vm%u:%x)\n" , + current->domain->domain_id, + src_id.addr.domain_id, src_id.addr.port, + dst_addr->domain_id, dst_addr->port); + + ret = -ECONNREFUSED ; +break ; + } + +spin_lock (&ring_info->lock); + + ret =argo_ringbuf_insert (dst_d, ring_info, &src_id, + iovs, niov, message_type, &len); +if ( ret == -EAGAIN ) + { +argo_dprintk ("argo_ringbuf_sendv failed, EAGAIN\n" ); +/* requeue to issue a notification when space is there */ +if (argo_pending_requeue (ring_info, src_addr->domain_id, len) ) + ret = -ENOMEM ; + } + +spin_unlock (&ring_info->lock); + +if ( ret >=0 ) +argo_signal_domain (dst_d); + + }while (0 ); + +read_unlock (&dst_d->argo->lock); + + }while (0 ); + +if ( dst_d ) +put_domain (dst_d); + +read_unlock (&argo_lock); + +return ( ret <0 ) ? ret : len; +} + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,
@@ -813,6 +1318,29 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, rc =argo_unregister_ring (d, ring_hnd);break ; } +case ARGO_MESSAGE_OP_sendv: + { +argo_send_addr_t send_addr; +uint32_t niov = arg3; +uint32_t message_type = arg4; + +XEN_GUEST_HANDLE_PARAM (argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs = +guest_handle_cast (arg2, argo_iov_t); + +if (unlikely (!guest_handle_okay (send_addr_hnd,1 )) ) +break ; + rc =copy_from_guest_errno (&send_addr, send_addr_hnd,1 ); +if ( rc ) +break ; + + send_addr.src.domain_id = d->domain_id; + + rc =argo_sendv (d, &send_addr.src, &send_addr.dst, + iovs, niov, message_type); +break ; + }default : rc = -ENOSYS ;break ;
xen/include/public/argo.h +59 -0
75 lines not on this page · raw message
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 94 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 01 December 2018 01:33
To: [email protected]
Cc: Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; Eric Chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: [PATCH 15/25] argo: implement the sendv opsendv operation is invoked to perform a synchronous send of buffers contained in iovs to a remote domain's registered ring.
It takes:
* A destination address (domid, port) for the ring to send to. It performs a most-specific match lookup, to allow for wildcard.
* A source address, used to inform the destination of where to reply.
* The address of an array of iovs containing the data to send
* .. and the length of that array of iovs
* and a 32-bit message type, available to communicate message context data (eg. kernel-to-kernel, separate from the application data).If insufficient space exists in the destination ring, it will return - EAGAIN
and Xen will notify the caller when sufficient space becomes available.Accesses to the ring indices are appropriately atomic. The rings are mapped into Xen's private address space to write as needed and the mappings are retained for later use.
When locating the destination ring, a check is performed via a cookie installed at ring registration time, to ensure that the source domain is the same as it was when the ring was registered.
Fixed-size types are used in some areas within this code where caution around avoiding integer overflow is important.
Signed-off-by Christopher Clark <[email protected]>--- xen/common/argo.c | 528 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 59 ++++++ 2 files changed, 587 insertions(+)xen/common/argo.c +18
-0@@ -24,10 +24,13 @@#include <xen/domain_page.h> #include <xen/guest_access.h> #include <xen/time.h> +#include <xsm/xsm.h> DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (argo_send_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (uint8_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled =0 ;@@ -166,6 +169,21 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#endif /* +* Event channel +*/ + +static void +argo_signal_domain (struct domain *d) +{ +argo_dprintk ("signalling domid:%d\n" , d->domain_id); + +if ( !d->argo )/* This can happen if the domain is being destroyed */ +return ; + +evtchn_send (d, d->argo->evtchn_port); +} + +/* * ring buffer */ @@ -259,6 +277,333 @@ argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr)return 0 ;
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, · line 288
Quoted 10 lines
xen/common/argo.c +8
-0} +static int +argo_memcpy_to_guest_ring (struct argo_ring_info *ring_info, +uint32_t offset, +const void *src, +XEN_GUEST_HANDLE (uint8_t ) src_hnd, +uint32_t len) +{ +int page = offset >>PAGE_SHIFT ;
unsigned?
Quoted 24 lines
xen/common/argo.c +24
-0+uint8_t *dst; +int ret; +unsigned int src_offset =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + + offset &= ~PAGE_MASK ; + +if ( (len >ARGO_MAX_RING_SIZE ) || (offset >ARGO_MAX_RING_SIZE ) ) +return -EFAULT ; + +while ( (offset + len) >PAGE_SIZE ) + { + ret =argo_ring_map_page (ring_info, page, &dst); +if ( ret ) +return ret; + +if ( src ) + { +memcpy (dst + offset, src + src_offset,PAGE_SIZE - offset); + src_offset += (PAGE_SIZE - offset); + } +else + {
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 321
Quoted 10 lines
xen/common/argo.c +10
-0+ ret =copy_from_guest_errno (dst + offset, src_hnd, +PAGE_SIZE - offset); +if ( ret ) +return ret; + +guest_handle_add_offset (src_hnd,PAGE_SIZE - offset); + } + + page++; + len -=PAGE_SIZE - offset;
A lot of "PAGE_SIZE - offset" here... maybe worth a local variable?
Quoted 271 lines
xen/common/argo.c +69
-0+ offset =0 ; + } + + ret =argo_ring_map_page (ring_info, page, &dst); +if ( ret ) + { +argo_dprintk ("argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %d of %d\n" , ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner, ring_info, + page, ring_info->nmfns); +return ret; + } + +if ( src ) +memcpy (dst + offset, src + src_offset, len); +else + ret =copy_from_guest_errno (dst + offset, src_hnd, len); + +return ret; +} + +static int +argo_ringbuf_get_rx_ptr (struct argo_ring_info *ring_info,uint32_t *rx_ptr) +{ +uint8_t *src; +argo_ring_t *ringp; +int ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( !ring_info->nmfns || ring_info->nmfns < ring_info->npage ) +return -EINVAL ; + + ret =argo_ring_map_page (ring_info,0 , &src); +if ( ret ) +return ret; + + ringp = (argo_ring_t *)src; + + *rx_ptr =read_atomic (&ringp->rx_ptr); + +return 0 ; +} + +/* +* argo_sanitize_ring creates a modified copy of the ring pointers +* where the rx_ptr is rounded up to ensure it is aligned, and then +* ring wrap is handled. Simplifies safe use of the rx_ptr for +* available space calculation. +*/ +static void +argo_sanitize_ring (argo_ring_t *ring,const struct argo_ring_info *ring_info) +{ +uint32_t rx_ptr = ring->rx_ptr; + + ring->tx_ptr = ring_info->tx_ptr; + ring->len = ring_info->len; + + rx_ptr =ARGO_ROUNDUP (rx_ptr); +if ( rx_ptr >= ring_info->len ) + rx_ptr =0 ; + + ring->rx_ptr = rx_ptr; +} + +/* +* argo_iov_count returns its count on success via an out variable +* to avoid potential for a negative return value to be used incorrectly +* (eg. coerced into an unsigned variable resulting in a large incorrect value)
+ */xen/common/argo.c +42
-0+static int +argo_iov_count (XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t *count) +{ +argo_iov_t iov; +uint32_t sum_iov_lens =0 ; +int ret; + +if ( niov >ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +return ret; + +/* check each to protect sum against integer overflow */ +if ( iov.iov_len >ARGO_MAX_RING_SIZE ) +return -EINVAL ; + + sum_iov_lens += iov.iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >ARGO_MAX_MSG_SIZE ) +return -EINVAL ; + +guest_handle_add_offset (iovs,1 ); + } + + *count = sum_iov_lens; +return 0 ; +} + +static int +argo_ringbuf_insert (struct domain *d, +struct argo_ring_info *ring_info, +const struct argo_ring_id *src_id, + XEN_GUEST_HANDLE_PARAM(argo_iov_t) iovs, uint8_tniov,
xen/common/argo.c +16
-0+ uint32_t message_type, unsigned long *out_len) +{ + argo_ring_t ring; +struct argo_ring_message_header mh = {0 }; +int32_t sp; +int32_t ret =0 ; +uint32_t len; +uint32_t iov_len; +uint32_t sum_iov_len =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( (ret =argo_iov_count (iovs, niov, &len)) ) +return ret; + + if ( ((ARGO_ROUNDUP(len) + sizeof (struct argo_ring_message_header) )Quoted 1 line
=
xen/common/argo.c +14
-0+ ring_info->len) + || (len > ARGO_MAX_MSG_SIZE) ) +return -EMSGSIZE ; + + do { + ret = argo_ringbuf_get_rx_ptr(ring_info, &ring.rx_ptr); + if ( ret ) + break; + + argo_sanitize_ring(&ring, ring_info); + + argo_dprintk("ring.tx_ptr=%d ring.rx_ptr=%d ring.len=%d" + " ring_info->tx_ptr=%d\n", + ring.tx_ptr, ring.rx_ptr, ring.len, ring_info-Quoted 1 line
tx_ptr);xen/common/argo.c +11
-0+ +if ( ring.rx_ptr == ring.tx_ptr ) + sp = ring_info->len; +else + { + sp = ring.rx_ptr - ring.tx_ptr; +if ( sp <0 ) + sp += ring.len; + } + + if ( (ARGO_ROUNDUP(len) + sizeof(struct
argo_ring_message_header)) >= sp )xen/common/argo.c +13
-0+ { +argo_dprintk ("EAGAIN\n" ); + ret = -EAGAIN ; +break ; + } + + mh.len = len +sizeof (struct argo_ring_message_header ); + mh.source.port = src_id->addr.port; + mh.source.domain_id = src_id->addr.domain_id; + mh.message_type = message_type; + +/* +* For this copy to the guest ring, tx_ptr is always 16-byte aligned
xen/common/argo.c +3
-0+* and the message header is 16 bytes long. +*/ + BUILD_BUG_ON(sizeof(struct argo_ring_message_header) !=
ARGO_ROUNDUP(1));xen/common/argo.c +3
-0+ + if ( (ret = argo_memcpy_to_guest_ring(ring_info, + ring.tx_ptr +
sizeof(argo_ring_t),xen/common/argo.c +2
-0+ &mh, +
XEN_GUEST_HANDLE_NULL(uint8_t),xen/common/argo.c +18
-0+ sizeof(mh))) ) +break ; + + ring.tx_ptr +=sizeof (mh); +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +while ( niov-- ) + { +XEN_GUEST_HANDLE_PARAM (uint8_t ) bufp_hnd; +XEN_GUEST_HANDLE (uint8_t ) buf_hnd; +argo_iov_t iov; + + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +break ; + + bufp_hnd = guest_handle_from_ptr((uintptr_t)iov.iov_base,
uint8_t);xen/common/argo.c +67
-0+ buf_hnd =guest_handle_from_param (bufp_hnd, uint8_t); + iov_len = iov.iov_len; + +if ( !iov_len ) + { +printk (XENLOG_ERR "argo: iov.iov_len=0 iov.iov_base=%" + PRIx64" ring (vm%u:%x vm%d)\n" , + iov.iov_base, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +guest_handle_add_offset (iovs,1 ); +continue ; + } + +if ( iov_len >ARGO_MAX_MSG_SIZE ) + { + ret = -EINVAL ; +break ; + } + + sum_iov_len += iov_len; +if ( sum_iov_len > len ) + { + ret = -EINVAL ; +break ; + } + +if (unlikely (!guest_handle_okay (buf_hnd, iov_len)) ) + { + ret = -EFAULT ; +break ; + } + + sp = ring.len - ring.tx_ptr; + +if ( iov_len > sp ) + { + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, sp); +if ( ret ) +break ; + + ring.tx_ptr =0 ; + iov_len -= sp; +guest_handle_add_offset (buf_hnd, sp); + } + + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, iov_len); +if ( ret ) +break ; + + ring.tx_ptr += iov_len; + +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +guest_handle_add_offset (iovs,1 ); + } + +if ( ret ) +break ; + + ring.tx_ptr =ARGO_ROUNDUP (ring.tx_ptr); +
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 590
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ring.tx_ptr >= ring_info->len ) + ring.tx_ptr -= ring_info->len; + +mb (); + ring_info->tx_ptr = ring.tx_ptr; +if ( (ret =argo_update_tx_ptr (ring_info, ring.tx_ptr)) ) +break ; + + }while (0 ); +
Again. Odd do-while-zero style... I won't mention this again.
Paul
Quoted 329 lines
xen/common/argo.c +8
-0+/* +* At this point it is possible to unmap the ring_info, ie: +* argo_ring_unmap(ring_info); +* but performance should be improved by not doing so, and retaining +* the mapping. +* An XSM policy control over level of confidentiality required +* versus performance cost could be added to decide that here. +* See the similar comment in argo_ring_map_page re: write-only mappings.
xen/common/argo.c +49
-0+*/ + +if ( !ret ) + *out_len = len; + +return ret; +} +/* * pending */ @@ -282,6 +627,47 @@ argo_pending_remove_all(struct argo_ring_info *ring_info) } } +static int +argo_pending_queue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ent = xmalloc(struct argo_pending_ent); + +if ( !ent ) +return -ENOMEM ; + + ent->len = len; + ent->id = src_id; + +hlist_add_head (&ent->node, &ring_info->pending); + +return 0 ; +} + +static int +argo_pending_requeue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct hlist_node *node; +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry (ent, node, &ring_info->pending, node) + { +if ( ent->id == src_id ) + { +if ( ent->len < len ) + ent->len = len; +return 0 ; + } + } + +return argo_pending_queue (ring_info, src_id, len); +} +static void argo_ring_remove_mfns(const struct domain *d,
structargo_ring_info*ring_info)
{xen/common/argo.c +22
-0@@ -509,6 +895,28 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id; + id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info; + + id.partner =ARGO_DOMID_ANY ; + +return argo_ring_find_info (d, &id); +} + static long argo_unregister_ring(struct domain *d,
XEN_GUEST_HANDLE_PARAM(argo_ring_t)ring_hnd)xen/common/argo.c +97
-0@@ -754,6 +1162,103 @@ argo_register_ring(struct domain *d,return ret; } +/* +* io +*/ + +static long +argo_sendv (struct domain *src_d,const argo_addr_t *src_addr, +const argo_addr_t *dst_addr, +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint32_t niov, +uint32_t message_type) +{ +struct domain *dst_d =NULL ; +struct argo_ring_id src_id; +struct argo_ring_info *ring_info; +int ret =0 ; +unsigned long len =0 ; + +ASSERT (src_d->domain_id == src_addr->domain_id); + +read_lock (&argo_lock); + +do { +if ( !src_d->argo ) + { + ret = -ENODEV ; +break ; + } + + src_id.addr.pad =0 ; + src_id.addr.port = src_addr->port; + src_id.addr.domain_id = src_d->domain_id; + src_id.partner = dst_addr->domain_id; + + dst_d =get_domain_by_id (dst_addr->domain_id); +if ( !dst_d || !dst_d->argo ) + { +argo_dprintk ("!dst_d, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +break ; + } + + ret =xsm_argo_send (src_d, dst_d); +if ( ret ) + { +printk (XENLOG_ERR "argo: XSM REJECTED %i -> %i\n" , + src_addr->domain_id, dst_addr->domain_id); +break ; + } + +read_lock (&dst_d->argo->lock); + +do { + ring_info =argo_ring_find_info_by_match (dst_d, dst_addr- >port, + src_addr->domain_id, + src_d->argo- >domain_cookie); +if ( !ring_info ) + { +printk (XENLOG_ERR "argo: vm%u connection refused, " +"src (vm%u:%x) dst (vm%u:%x)\n" , + current->domain->domain_id, + src_id.addr.domain_id, src_id.addr.port, + dst_addr->domain_id, dst_addr->port); + + ret = -ECONNREFUSED ; +break ; + } + +spin_lock (&ring_info->lock); + + ret =argo_ringbuf_insert (dst_d, ring_info, &src_id, + iovs, niov, message_type, &len); +if ( ret == -EAGAIN ) + { +argo_dprintk ("argo_ringbuf_sendv failed, EAGAIN\n" ); +/* requeue to issue a notification when space is there */ +if (argo_pending_requeue (ring_info, src_addr->domain_id, len) ) + ret = -ENOMEM ; + } + +spin_unlock (&ring_info->lock); + +if ( ret >=0 ) +argo_signal_domain (dst_d); + + }while (0 ); + +read_unlock (&dst_d->argo->lock); + + }while (0 ); + +if ( dst_d ) +put_domain (dst_d); + +read_unlock (&argo_lock); + +return ( ret <0 ) ? ret : len; +} +long
do_argo_message_op(int cmd,XEN_GUEST_HANDLE_PARAM(void)arg1,
XEN_GUEST_HANDLE_PARAM(void)arg2,xen/common/argo.c +23
-0@@ -813,6 +1318,29 @@ do_argo_message_op(int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1, rc =argo_unregister_ring (d, ring_hnd);break ; } +case ARGO_MESSAGE_OP_sendv: + { +argo_send_addr_t send_addr; +uint32_t niov = arg3; +uint32_t message_type = arg4; + +XEN_GUEST_HANDLE_PARAM (argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs = +guest_handle_cast (arg2, argo_iov_t); + +if (unlikely (!guest_handle_okay (send_addr_hnd,1 )) ) +break ; + rc =copy_from_guest_errno (&send_addr, send_addr_hnd,1 ); +if ( rc ) +break ; + + send_addr.src.domain_id = d->domain_id; + + rc =argo_sendv (d, &send_addr.src, &send_addr.dst, + iovs, niov, message_type); +break ; + }default : rc = -ENOSYS ;break;xen/include/public/argo.h +35
-0@@ -32,6 +32,28 @@*/ #define ARGO_MAX_RING_SIZE (16777216ULL) +/* +* ARGO_MAXIOV : maximum number of iovs accepted in a single sendv. +* Rationale for the value: +* The Linux argo driver never passes more than two iovs. +* Linux defines UIO_MAXIOV as 1024. +* POSIX mandates at least 16 -- not that this is a POSIX API of course. +* +* Limit the total amount of data posted in a single argo operation to +* no more than 2^31 bytes to reduce risk of integer overflow defects. +* Each argo iov can hold ~ 2^24 bytes, so set ARGO_MAXIOV to 2^(31-24), +* minus one to enable simple efficient bounds checking via masking: 127. +*/ +#define ARGO_MAXIOV 127U + +typedef struct argo_iov +{ +uint64_t iov_base; +uint32_t iov_len; +uint32_t pad; +}argo_iov_t ; +DEFINE_XEN_GUEST_HANDLE (argo_iov_t); +/* pfn type: 64-bit on all architectures to aid avoiding a compat ABI */ typedef uint64_t argo_pfn_t ;@@ -42,6 +64,12 @@ typedef struct argo_addruint16_t pad; } argo_addr_t; +typedef struct argo_send_addr +{ +argo_addr_t src; +argo_addr_t dst; +}argo_send_addr_t ; + typedef struct argo_ring_id { struct argo_addr addr;@@ -125,4 +153,35 @@ struct argo_ring_message_header*/ #define ARGO_MESSAGE_OP_unregister_ring 2 +/* +* ARGO_MESSAGE_OP_sendv +* +* Send a list of buffers contained in iovs. +* +* The send address struct specifies the source and destination addresses +* for the message being sent, which are used to find the destination ring:
+ * Xen first looks for a most-specific match with a registered ring with
+ * (id.addr == dst) and (id.partner ==sending_domain) ;
+ * if that fails, it then looks for a wildcard match (aka multicast receiver)
+ * where (id.addr == dst) and (id.partner ==DOMID_ANY).
+ *
+ * For each iov entry, sendiov_lenbytes fromiov_baseto the destination ring.
+ * If insufficient space exists in the destination ring, it will return - EAGAIN
+ * and Xen will notify the caller when sufficient space becomes available.
+ *
+ * The message type is a 32-bit data field available to communicate message
+ * context data (eg. kernel-to-kernel, rather than application layer).
+ *
+ * arg1:XEN_GUEST_HANDLE(argo_send_addr_t)source and dest addresses
+ * arg2:XEN_GUEST_HANDLE(argo_iov_t)iovs
+ * arg3:uint32_tniov
+ * arg4:uint32_tmessage type
+ */xen/include/public/argo.h +3
-0+#define ARGO_MESSAGE_OP_sendv 5 + +/* The maximum size of a guest message that may be sent on an Argo ring. */
xen/include/public/argo.h +4
-1+#define ARGO_MAX_MSG_SIZE ((ARGO_MAX_RING_SIZE) - \ + (sizeof(struct argo_ring_message_header)) - \ + ARGO_ROUNDUP(1)) +#endif --
Quoted 1 line
2.1.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
In reply to Christopher Clarkxen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 181
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +7
-0+static void +argo_signal_domain (struct domain *d) +{ +argo_dprintk ("signalling domid:%d\n" , d->domain_id); + +if ( !d->argo )/* This can happen if the domain is being destroyed */ +return ;
If such a precaution is necessary, how is it guaranteed that the pointer doesn't change to NULL between the check above and ...
Quoted 1 line
+
evtchn_send(d, d->argo->evtchn_port);
... the use here?
Quoted 19 lines
xen/common/argo.c +19
-0+static int +argo_iov_count (XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t *count) +{ +argo_iov_t iov; +uint32_t sum_iov_lens =0 ; +int ret; + +if ( niov >ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +return ret; + +/* check each to protect sum against integer overflow */ +if ( iov.iov_len >ARGO_MAX_RING_SIZE )
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 420
Quoted 10 lines
xen/common/argo.c +10
-0+return -EINVAL ; + + sum_iov_lens += iov.iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >ARGO_MAX_MSG_SIZE ) +return -EINVAL ;
So you do overflow checks here. But how does this help when ...
Quoted 78 lines
xen/common/argo.c +78
-0+guest_handle_add_offset (iovs,1 ); + } + + *count = sum_iov_lens; +return 0 ; +} + +static int +argo_ringbuf_insert (struct domain *d, +struct argo_ring_info *ring_info, +const struct argo_ring_id *src_id, +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t message_type,unsigned long *out_len) +{ +argo_ring_t ring; +struct argo_ring_message_header mh = {0 }; +int32_t sp; +int32_t ret =0 ; +uint32_t len; +uint32_t iov_len; +uint32_t sum_iov_len =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( (ret =argo_iov_count (iovs, niov, &len)) ) +return ret; + +if ( ((ARGO_ROUNDUP (len) +sizeof (struct argo_ring_message_header ) ) >= + ring_info->len) + || (len >ARGO_MAX_MSG_SIZE ) ) +return -EMSGSIZE ; + + do { + ret = argo_ringbuf_get_rx_ptr(ring_info, &ring.rx_ptr); + if ( ret ) + break; + + argo_sanitize_ring(&ring, ring_info); + + argo_dprintk("ring.tx_ptr=%d ring.rx_ptr=%d ring.len=%d" + " ring_info->tx_ptr=%d\n", + ring.tx_ptr, ring.rx_ptr, ring.len, ring_info->tx_ptr); + + if ( ring.rx_ptr == ring.tx_ptr ) + sp = ring_info->len; + else + { + sp = ring.rx_ptr - ring.tx_ptr; + if ( sp < 0 ) + sp += ring.len; + } + + if ( (ARGO_ROUNDUP(len) + sizeof(struct argo_ring_message_header)) >= sp ) + { + argo_dprintk("EAGAIN\n"); + ret = -EAGAIN; + break; + } + + mh.len = len + sizeof(struct argo_ring_message_header); + mh.source.port = src_id->addr.port; + mh.source.domain_id = src_id->addr.domain_id; + mh.message_type = message_type; + + /* + * For this copy to the guest ring, tx_ptr is always 16-byte aligned + * and the message header is 16 bytes long. + */ + BUILD_BUG_ON(sizeof(struct argo_ring_message_header) != ARGO_ROUNDUP(1)); + + if ( (ret = argo_memcpy_to_guest_ring(ring_info, + ring.tx_ptr + sizeof(argo_ring_t), + &mh, + XEN_GUEST_HANDLE_NULL(uint8_t), + sizeof(mh))) ) + break; + + ring.tx_ptr += sizeof(mh);
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 509
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +while ( niov-- ) + { +XEN_GUEST_HANDLE_PARAM (uint8_t ) bufp_hnd; +XEN_GUEST_HANDLE (uint8_t ) buf_hnd; +argo_iov_t iov; + + ret =copy_from_guest_errno (&iov, iovs,1 );
... here you copy the structure again from guest memory, at which point it may have changed? I see you do some checks further down, but the question then is - is the checking in argo_iov_count() redundant and hence unnecessary? Are you really safe here against inconsistencies between the first and second reads? If so, a thorough explanation in a comment is needed here.
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 513
Quoted 4 lines
xen/common/argo.c +4
-0+if ( ret ) +break ; + + bufp_hnd =guest_handle_from_ptr ((uintptr_t )iov.iov_base, uint8_t);
Please use a handle in the public interface instead of such a cast.
Quoted 41 lines
xen/common/argo.c +41
-0+ buf_hnd =guest_handle_from_param (bufp_hnd, uint8_t); + iov_len = iov.iov_len; + +if ( !iov_len ) + { +printk (XENLOG_ERR "argo: iov.iov_len=0 iov.iov_base=%" + PRIx64" ring (vm%u:%x vm%d)\n" , + iov.iov_base, ring_info->id.addr.domain_id, + ring_info->id.addr.port, ring_info->id.partner); + +guest_handle_add_offset (iovs,1 ); +continue ; + } + +if ( iov_len >ARGO_MAX_MSG_SIZE ) + { + ret = -EINVAL ; +break ; + } + + sum_iov_len += iov_len; +if ( sum_iov_len > len ) + { + ret = -EINVAL ; +break ; + } + +if (unlikely (!guest_handle_okay (buf_hnd, iov_len)) ) + { + ret = -EFAULT ; +break ; + } + + sp = ring.len - ring.tx_ptr; + +if ( iov_len > sp ) + { + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, sp); +if ( ret )
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 564
Quoted 10 lines
xen/common/argo.c +10
-0+break ; + + ring.tx_ptr =0 ; + iov_len -= sp; +guest_handle_add_offset (buf_hnd, sp); + } + + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, iov_len);
Extending the remark on double guest memory read above, is it certain you won't overrun the ring here?
Quoted 11 lines
xen/common/argo.c +11
-0+if ( ret ) +break ; + + ring.tx_ptr += iov_len; + +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +guest_handle_add_offset (iovs,1 ); + } +
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 585
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ret ) +break ; + + ring.tx_ptr =ARGO_ROUNDUP (ring.tx_ptr); + +if ( ring.tx_ptr >= ring_info->len ) + ring.tx_ptr -= ring_info->len; + +mb (); + ring_info->tx_ptr = ring.tx_ptr;
What does the above barrier guard against? It's all hypervisor local memory which gets altered afaict.
Quoted 4 lines
xen/common/argo.c +4
-0+static int +argo_pending_requeue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct hlist_node *node;
xen/common/argo.c argo_pending_remove_all(struct argo_ring_info *ring_info) · line 663
Quoted 10 lines
xen/common/argo.c +10
-0+struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry (ent, node, &ring_info->pending, node) + { +if ( ent->id == src_id ) + { +if ( ent->len < len ) + ent->len = len;
What does this achieve? I.e. why is this not either a plain assignment or a check that the length is the same?
Quoted 6 lines
xen/common/argo.c +6
-0+static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info;
xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 913
Quoted 10 lines
xen/common/argo.c +10
-0+ +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id; + id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info;
Such a cookie makes mismatches unlikely, but it doesn't exclude them. If there are other checks, is the cookie useful at all?
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1325
Quoted 9 lines
xen/common/argo.c +5
-0@@ -813,6 +1318,29 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, rc =argo_unregister_ring (d, ring_hnd);break ; } +case ARGO_MESSAGE_OP_sendv: + { +argo_send_addr_t send_addr; +uint32_t niov = arg3; +uint32_t message_type = arg4;
At the example of these (perhaps I've again overlooked earlier instances), what about the upper halves on 64-bit? Given the rather generic interface of the actual hypercall, I don't think it is a good idea to ignore the bits. The situation is different for the "cmd" parameter, which is uniformly 32-bit for all sub-ops.
Talking of "cmd" and its type: In case it wasn't said by anyone else yet, please use unsigned types wherever negative values are impossible.
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1338
Quoted 12 lines
xen/common/argo.c +12
-0+XEN_GUEST_HANDLE_PARAM (argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs = +guest_handle_cast (arg2, argo_iov_t); + +if (unlikely (!guest_handle_okay (send_addr_hnd,1 )) ) +break ; + rc =copy_from_guest_errno (&send_addr, send_addr_hnd,1 ); +if ( rc ) +break ; + + send_addr.src.domain_id = d->domain_id;
What use is the field if you override it like this?
Quoted 15 lines
xen/include/public/argo.h +9
-0@@ -32,6 +32,28 @@*/ #define ARGO_MAX_RING_SIZE (16777216ULL) +/* +* ARGO_MAXIOV : maximum number of iovs accepted in a single sendv. +* Rationale for the value: +* The Linux argo driver never passes more than two iovs. +* Linux defines UIO_MAXIOV as 1024. +* POSIX mandates at least 16 -- not that this is a POSIX API of course. +* +* Limit the total amount of data posted in a single argo operation to +* no more than 2^31 bytes to reduce risk of integer overflow defects.
xen/include/public/argo.h line 53
Quoted 10 lines
xen/include/public/argo.h +10
-0+* Each argo iov can hold ~ 2^24 bytes, so set ARGO_MAXIOV to 2^(31-24), +* minus one to enable simple efficient bounds checking via masking: 127. +*/ +#define ARGO_MAXIOV 127U + +typedef struct argo_iov +{ + uint64_t iov_base; + uint32_t iov_len; + uint32_t pad;
I don't think I've found any checking of this field to be zero, to allow for future re-use.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 14 lines · Jan Beulich
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 181
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +7
-0+static void +argo_signal_domain (struct domain *d) +{ +argo_dprintk ("signalling domid:%d\n" , d->domain_id); + +if ( !d->argo )/* This can happen if the domain is being destroyed */ +return ;If such a precaution is necessary, how is it guaranteed that the pointer doesn't change to NULL between the check above and ...
Quoted 3 lines
Quoted 1 line
+
evtchn_send(d, d->argo->evtchn_port);... the use here?
ack, this code is gone in v2.d->argo is safe to access when holding either read or write of L1, the global argo lock, so won't switch to NULL as a surprise.
Quoted 121 lines
Quoted 19 lines
xen/common/argo.c +19
-0+static int +argo_iov_count (XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t *count) +{ +argo_iov_t iov; +uint32_t sum_iov_lens =0 ; +int ret; + +if ( niov >ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { + ret =copy_from_guest_errno (&iov, iovs,1 ); +if ( ret ) +return ret; + +/* check each to protect sum against integer overflow */ +if ( iov.iov_len >ARGO_MAX_RING_SIZE )xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 420
Quoted 10 lines
xen/common/argo.c +10
-0+return -EINVAL ; + + sum_iov_lens += iov.iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >ARGO_MAX_MSG_SIZE ) +return -EINVAL ;So you do overflow checks here. But how does this help when ...
Quoted 78 lines
xen/common/argo.c +78
-0+guest_handle_add_offset (iovs,1 ); + } + + *count = sum_iov_lens; +return 0 ; +} + +static int +argo_ringbuf_insert (struct domain *d, +struct argo_ring_info *ring_info, +const struct argo_ring_id *src_id, +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs,uint8_t niov, +uint32_t message_type,unsigned long *out_len) +{ +argo_ring_t ring; +struct argo_ring_message_header mh = {0 }; +int32_t sp; +int32_t ret =0 ; +uint32_t len; +uint32_t iov_len; +uint32_t sum_iov_len =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( (ret =argo_iov_count (iovs, niov, &len)) ) +return ret; + +if ( ((ARGO_ROUNDUP (len) +sizeof (struct argo_ring_message_header ) ) >= + ring_info->len) + || (len >ARGO_MAX_MSG_SIZE ) ) +return -EMSGSIZE ; + + do { + ret = argo_ringbuf_get_rx_ptr(ring_info, &ring.rx_ptr); + if ( ret ) + break; + + argo_sanitize_ring(&ring, ring_info); + + argo_dprintk("ring.tx_ptr=%d ring.rx_ptr=%d ring.len=%d" + " ring_info->tx_ptr=%d\n", + ring.tx_ptr, ring.rx_ptr, ring.len, ring_info->tx_ptr); + + if ( ring.rx_ptr == ring.tx_ptr ) + sp = ring_info->len; + else + { + sp = ring.rx_ptr - ring.tx_ptr; + if ( sp < 0 ) + sp += ring.len; + } + + if ( (ARGO_ROUNDUP(len) + sizeof(struct argo_ring_message_header)) >= sp ) + { + argo_dprintk("EAGAIN\n"); + ret = -EAGAIN; + break; + } + + mh.len = len + sizeof(struct argo_ring_message_header); + mh.source.port = src_id->addr.port; + mh.source.domain_id = src_id->addr.domain_id; + mh.message_type = message_type; + + /* + * For this copy to the guest ring, tx_ptr is always 16-byte aligned + * and the message header is 16 bytes long. + */ + BUILD_BUG_ON(sizeof(struct argo_ring_message_header) != ARGO_ROUNDUP(1)); + + if ( (ret = argo_memcpy_to_guest_ring(ring_info, + ring.tx_ptr + sizeof(argo_ring_t), + &mh, + XEN_GUEST_HANDLE_NULL(uint8_t), + sizeof(mh))) ) + break; + + ring.tx_ptr += sizeof(mh);xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 509
Quoted 10 lines
xen/common/argo.c +10
-0+if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +while ( niov-- ) + { +XEN_GUEST_HANDLE_PARAM (uint8_t ) bufp_hnd; +XEN_GUEST_HANDLE (uint8_t ) buf_hnd; +argo_iov_t iov; + + ret =copy_from_guest_errno (&iov, iovs,1 );
Quoted 7 lines
... here you copy the structure again from guest memory, at which point it may have changed? I see you do some checks further down, but the question then is - is the checking in
argo_iov_count()redundant and hence unnecessary? Are you really safe here against inconsistencies between the first and second reads? If so, a thorough explanation in a comment is needed here.
Fair point and comments have been added to v2.
Quoted 8 lines
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 513
Quoted 5 lines
xen/common/argo.c +4
-0+if ( ret ) +break ; + + bufp_hnd =guest_handle_from_ptr ((uintptr_t )iov.iov_base, uint8_t);Please use a handle in the public interface instead of such a cast.
ack.
Quoted 13 lines
Quoted 13 lines
xen/common/argo.c +13
-0+ sp = ring.len - ring.tx_ptr; + +if ( iov_len > sp ) + { + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, sp); +if ( ret ) +break ; + + ring.tx_ptr =0 ; + iov_len -= sp; +guest_handle_add_offset (buf_hnd, sp);
Quoted 8 lines
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 564
Quoted 5 lines
xen/common/argo.c +5
-0+ } + + ret =argo_memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (argo_ring_t), +NULL , buf_hnd, iov_len);Extending the remark on double guest memory read above, is it certain you won't overrun the ring here?
Yes, certain it's ok. Comments added to explain.
Quoted 17 lines
Quoted 17 lines
xen/common/argo.c +16
-0+if ( ret ) +break ; + + ring.tx_ptr += iov_len; + +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + +guest_handle_add_offset (iovs,1 ); + } + +if ( ret ) +break ; + + ring.tx_ptr =ARGO_ROUNDUP (ring.tx_ptr); +
Quoted 8 lines
xen/common/argo.c argo_update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 585
Quoted 5 lines
xen/common/argo.c +5
-0+if ( ring.tx_ptr >= ring_info->len ) + ring.tx_ptr -= ring_info->len; + +mb (); + ring_info->tx_ptr = ring.tx_ptr;What does the above barrier guard against? It's all hypervisor local memory which gets altered afaict.
ack, dropped.
Quoted 10 lines
Quoted 10 lines
xen/common/argo.c +9
-0+static int +argo_pending_requeue (struct argo_ring_info *ring_info,domid_t src_id,int len) +{ +struct hlist_node *node; +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry (ent, node, &ring_info->pending, node)
Quoted 8 lines
xen/common/argo.c argo_pending_remove_all(struct argo_ring_info *ring_info) · line 663
Quoted 5 lines
xen/common/argo.c +5
-0+ { +if ( ent->id == src_id ) + { +if ( ent->len < len ) + ent->len = len;What does this achieve? I.e. why is this not either a plain assignment or a check that the length is the same?
New comment added:
/*
* Reuse an existing queue entry for a notification rather than add
* another. If the existing entry is waiting for a smaller size than
* the current message then adjust the record to wait for the
* current (larger) size to be available before triggering a
* notification.
* This assists the waiting sender by ensuring that whenever a
* notification is triggered, there is sufficient space available
* for (at least) any one of the messages awaiting transmission. */
Quoted 11 lines
Quoted 11 lines
xen/common/argo.c +11
-0+static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id;
Quoted 8 lines
xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 913
Quoted 5 lines
xen/common/argo.c +5
-0+ id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info;Such a cookie makes mismatches unlikely, but it doesn't exclude them. If there are other checks, is the cookie useful at all?
Yes, I think so and it's proved useful elsewhere in the second version of the series: it helps avoid sending signals to incorrect domains that may not be argo-enabled.
Quoted 7 lines
Quoted 7 lines
xen/common/argo.c +3
-0@@ -813,6 +1318,29 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, rc =argo_unregister_ring (d, ring_hnd);break ; } +case ARGO_MESSAGE_OP_sendv: + { +argo_send_addr_t send_addr;
Quoted 8 lines
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1325
Quoted 2 lines
xen/common/argo.c +2
-0+uint32_t niov = arg3; +uint32_t message_type = arg4;At the example of these (perhaps I've again overlooked earlier instances), what about the upper halves on 64-bit? Given the rather generic interface of the actual hypercall, I don't think it is a good idea to ignore the bits. The situation is different for the "cmd" parameter, which is uniformly 32-bit for all sub-ops.
ack.
Quoted 4 lines
Talking of "cmd" and its type: In case it wasn't said by anyone else yet, please use unsigned types wherever negative values are impossible.
Quoted 14 lines
xen/common/argo.c do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1338
Quoted 12 lines
xen/common/argo.c +12
-0+XEN_GUEST_HANDLE_PARAM (argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (argo_iov_t) iovs = +guest_handle_cast (arg2, argo_iov_t); + +if (unlikely (!guest_handle_okay (send_addr_hnd,1 )) ) +break ; + rc =copy_from_guest_errno (&send_addr, send_addr_hnd,1 ); +if ( rc ) +break ; + + send_addr.src.domain_id = d->domain_id;What use is the field if you override it like this?
ack, have switched to a correct match check in v2.
Quoted 2 lines
I don't think I've found any checking of this field to be zero, to allow for future re-use.
ack.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 20.12.18 at 06:58, <[email protected]> wrote:
Quoted 12 lines · Jan Beulich
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +11
-0+static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id;Quoted 8 lines
xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 913
Quoted 5 lines
xen/common/argo.c +5
-0+ id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info;Such a cookie makes mismatches unlikely, but it doesn't exclude them. If there are other checks, is the cookie useful at all?
Quoted 3 lines
Yes, I think so and it's proved useful elsewhere in the second version of the series: it helps avoid sending signals to incorrect domains that may not be argo-enabled.
"It helps avoid" still isn't "it allows to avoid", i.e. it still sounds like an approach reducing likelihood instead of one excluding mistakes altogether.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 24 lines · Jan Beulich
Quoted 24 lines
Quoted 2 lines
Quoted 2 lines · On 20.12.18 at 06:58, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +6
-0+static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie) +{ +argo_ring_id_t id; +struct argo_ring_info *ring_info;xen/common/argo.c argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 913
Quoted 10 lines
xen/common/argo.c +10
-0+ +ASSERT (rw_is_locked (&d->argo->lock)); + + id.addr.port = port; + id.addr.domain_id = d->domain_id; + id.partner = partner_id; + + ring_info =argo_ring_find_info (d, &id); +if ( ring_info && (partner_cookie == ring_info->partner_cookie) ) +return ring_info;Such a cookie makes mismatches unlikely, but it doesn't exclude them. If there are other checks, is the cookie useful at all?
Quoted 7 lines
Quoted 3 lines
Yes, I think so and it's proved useful elsewhere in the second version of the series: it helps avoid sending signals to incorrect domains that may not be argo-enabled.
"It helps avoid" still isn't "it allows to avoid", i.e. it still sounds like an approach reducing likelihood instead of one excluding mistakes altogether.
ok, I'm at the point where I'm close to having a version three of the series to post that addresses all the feedback so far, plus some additional improvements, with the following two items remaining to discuss:
1) the domain_cookie, with Jan's question about a) its exclusion of mismatches and b) its utility.
Given the expressed concern that the timer-based cookie initialization does not necessarily exclude mismatches, I've reimplemented it as a simple 128-bit counter protected by the L1 lock: this does now exclude mismatches.
The utility of the cookie follows from this:
domid, despite its name, is not a unique domain identifier; it's a temporally unique id: Xen will ensure that no two domains that execute concurrently have the same domid. Domain authentication needs to take this into account.
With Argo, it affects these points:
* ring registration: when the partner domain domid is specified, argo finds the currently executing domain with that domid, and needs to be able to confirm that it is the same domain later when a sendv is issued.
* sendv: needs to confirm that the domain sending a message is the same as the single domain authorized to transmit when the ring was first registered.
* notify: the querying domain asks about free space, and if there's not enough then a record is kept internal to the hypervisor, and a signal will be sent to the caller later when sufficient space becomes available. Before sending the signal, Xen needs to confirm that the current domain with the domid it remembered is the same as the one that issued the query, otherwise Xen is sending spurious signals to domains that are not expecting it (and unless it checks, may not even be argo-enabled).
* domain teardown: in the absence of the domain cookie, or an alternative data structure that achieves the same ability to distinguish a reincarnated domain, all the rings that are registered that authorize the dying domid to send need to be torn down with suitable notification to their owners, and all the pending signals for that domain about available free space need to be nullified, to prevent a later domain inheriting these credentials and signals.
Doing so either entails a potentially-expensive walk of all rings of all domains, plus all the pending notifications on all rings the domain can access, or additional complexity with new data structures storing further metadata on the authorized domain on ring registration, etc. The domain cookie which enables identity confirmation on a domid is a reasonable alternative solution.
So: if the switch to a simple counter is sufficient to mitigate the mismatch concern, and the utility of the cookie is potentially acceptable, I'll post a v3 series for review with that present.
2) the p2m type of the guest-supplied memory for the ring.
Roger raised a query about not pinning the p2m type of memory used for the ring, and my response on 21st December is here:
https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg02204.html
At the moment, I haven't changed that code. If the p2m type is changed after ring registration, is it a problem? If not, then I think the code is OK; but if so then a pointer to what makes it problematic would be helpful to determine an appropriate next step.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 11 lines · Christopher Clark
2) the p2m type of the guest-supplied memory for the ring.
Roger raised a query about not pinning the p2m type of memory used for the ring, and my response on 21st December is here:
https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg02204.html
At the moment, I haven't changed that code. If the p2m type is changed after ring registration, is it a problem? If not, then I think the code is OK; but if so then a pointer to what makes it problematic would be helpful to determine an appropriate next step.
My point was that you don't need to check every time that the gfn -> mfn translations for the ring are the same (as is done in argo_find_ring_mfns). AFAICT you take a reference to each page in the ring, so there's no need to check that the p2m mapping is still the same.
I think the p2m type checks that you do are correct, and you should only allow p2m_ram_rw pages to be used for the ring.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
In reply to Christopher ClarkQuoted 6 lines
Quoted 1 line
Quoted 1 line · On 04.01.19 at 09:13, <[email protected]> wrote:
ok, I'm at the point where I'm close to having a version three of the series to post that addresses all the feedback so far, plus some additional improvements, with the following two items remaining to discuss:
Quoted 7 lines
1) the
domain_cookie, with Jan's question about a) its exclusion of mismatches and b) its utility.Given the expressed concern that the timer-based cookie initialization does not necessarily exclude mismatches, I've reimplemented it as a simple 128-bit counter protected by the L1 lock: this does now exclude mismatches.
... for all practical purposes, I assume you mean. In which case I'd then immediately ask whether a 64-bit counter wouldn't do as well.
Quoted 6 lines
The utility of the cookie follows from this:
domid, despite its name, is not a unique domain identifier; it's a temporally unique id: Xen will ensure that no two domains that execute concurrently have the same domid. Domain authentication needs to take this into account.
Correct, at which point the question arises whether domain IDs aren't too narrow. After all this isn't the first time we run into such a restriction - see the opt_ibpb related code in context_switch().
Quoted 28 lines
With Argo, it affects these points:
* ring registration: when the partner domain domid is specified, argo finds the currently executing domain with that domid, and needs to be able to confirm that it is the same domain later when a sendv is issued.
* sendv: needs to confirm that the domain sending a message is the same as the single domain authorized to transmit when the ring was first registered.
* notify: the querying domain asks about free space, and if there's not enough then a record is kept internal to the hypervisor, and a signal will be sent to the caller later when sufficient space becomes available. Before sending the signal, Xen needs to confirm that the current domain with the domid it remembered is the same as the one that issued the query, otherwise Xen is sending spurious signals to domains that are not expecting it (and unless it checks, may not even be argo-enabled).
* domain teardown: in the absence of the domain cookie, or an alternative data structure that achieves the same ability to distinguish a reincarnated domain, all the rings that are registered that authorize the dying domid to send need to be torn down with suitable notification to their owners, and all the pending signals for that domain about available free space need to be nullified, to prevent a later domain inheriting these credentials and signals.
Quoted 6 lines
Doing so either entails a potentially-expensive walk of all rings of all domains, plus all the pending notifications on all rings the domain can access, or additional complexity with new data structures storing further metadata on the authorized domain on ring registration, etc. The domain cookie which enables identity confirmation on a domid is a reasonable alternative solution.
For all of these the question then is whether holding a reference to the other domain (which has been looked up during ring registration) wouldn't help. Furthermore this isn't a new problem, see e.g. how event channel code deals with the ECS_INTERDOMAIN case - without acquiring extra references, but instead with suitable (and mutual) cleanup during domain destruction.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 15/25] argo: implement the sendv op
Quoted 65 lines · Jan Beulich
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines · On 04.01.19 at 09:13, <[email protected]> wrote:
ok, I'm at the point where I'm close to having a version three of the series to post that addresses all the feedback so far, plus some additional improvements, with the following two items remaining to discuss:
Quoted 7 lines
1) the
domain_cookie, with Jan's question about a) its exclusion of mismatches and b) its utility.Given the expressed concern that the timer-based cookie initialization does not necessarily exclude mismatches, I've reimplemented it as a simple 128-bit counter protected by the L1 lock: this does now exclude mismatches.
... for all practical purposes, I assume you mean. In which case I'd then immediately ask whether a 64-bit counter wouldn't do as well.
Quoted 6 lines
The utility of the cookie follows from this:
domid, despite its name, is not a unique domain identifier; it's a temporally unique id: Xen will ensure that no two domains that execute concurrently have the same domid. Domain authentication needs to take this into account.
Correct, at which point the question arises whether domain IDs aren't too narrow. After all this isn't the first time we run into such a restriction - see the
opt_ibpbrelated code incontext_switch().Quoted 28 lines
With Argo, it affects these points:
* ring registration: when the partner domain domid is specified, argo finds the currently executing domain with that domid, and needs to be able to confirm that it is the same domain later when a sendv is issued.
* sendv: needs to confirm that the domain sending a message is the same as the single domain authorized to transmit when the ring was first registered.
* notify: the querying domain asks about free space, and if there's not enough then a record is kept internal to the hypervisor, and a signal will be sent to the caller later when sufficient space becomes available. Before sending the signal, Xen needs to confirm that the current domain with the domid it remembered is the same as the one that issued the query, otherwise Xen is sending spurious signals to domains that are not expecting it (and unless it checks, may not even be argo-enabled).
* domain teardown: in the absence of the domain cookie, or an alternative data structure that achieves the same ability to distinguish a reincarnated domain, all the rings that are registered that authorize the dying domid to send need to be torn down with suitable notification to their owners, and all the pending signals for that domain about available free space need to be nullified, to prevent a later domain inheriting these credentials and signals.
Quoted 6 lines
Doing so either entails a potentially-expensive walk of all rings of all domains, plus all the pending notifications on all rings the domain can access, or additional complexity with new data structures storing further metadata on the authorized domain on ring registration, etc. The domain cookie which enables identity confirmation on a domid is a reasonable alternative solution.
Quoted 6 lines
For all of these the question then is whether holding a reference to the other domain (which has been looked up during ring registration) wouldn't help. Furthermore this isn't a new problem, see e.g. how event channel code deals with the
ECS_INTERDOMAINcase - without acquiring extra references, but instead with suitable (and mutual) cleanup during domain destruction.
Just to close on this thread: the v3 series posted last night has added state teardown and mutual cleanup as requested, with the cookie removed. Thanks for the pointers.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 16/25 argo: implement the notify op
In reply to Christopher ClarkQueries for data about space availability in registered rings and causes notification to be sent when space has become available.
The hypercall op populates a supplied data structure with information about ring state, and if insufficent space is currently available in a given ring, the hypervisor will record the domain's expressed interest and notify it when it observes that space has become available.
Checks for free space occur when this notify op is invoked, so it may be intentionally invoked with no data structure to populate (ie. NULL argument), to trigger such a check and consequent notifications.
--- xen/common/argo.c | 326 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 62 +++++++++ 2 files changed, 388 insertions(+)
xen/common/argo.c +326 -0
@@ -30,6 +30,8 @@ DEFINE_XEN_GUEST_HANDLE(argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_send_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (argo_ring_data_t); +DEFINE_XEN_GUEST_HANDLE (argo_ring_data_ent_t);DEFINE_XEN_GUEST_HANDLE (uint8_t);/* Xen command line option to enable argo */
@@ -120,6 +122,10 @@ argo_hash_fn(const struct argo_ring_id *id)return ret; } +static struct argo_ring_info * +argo_ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id,uint64_t partner_cookie); +/* * locks */
@@ -183,6 +189,19 @@ argo_signal_domain(struct domain *d)evtchn_send (d, d->argo->evtchn_port); } +static void +argo_signal_domid (domid_t id) +{ +struct domain *d =get_domain_by_id (id); + +if ( !d ) +return ; + +argo_signal_domain (d); + +put_domain (d); +} +/* * ring buffer */
@@ -363,6 +382,39 @@ argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr)return 0 ; } +static uint32_t +argo_ringbuf_payload_space (struct domain *d,struct argo_ring_info *ring_info) +{ +argo_ring_t ring; +int32_t ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ring.len = ring_info->len; +if ( !ring.len ) +return 0 ; + + ring.tx_ptr = ring_info->tx_ptr; + +if (argo_ringbuf_get_rx_ptr (ring_info, &ring.rx_ptr) ) +return 0 ; + +argo_dprintk ("argo_ringbuf_payload_space: tx_ptr=%d rx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) +return ring.len -sizeof (struct argo_ring_message_header ); + + ret = ring.rx_ptr - ring.tx_ptr; +if ( ret <0 ) + ret += ring.len; + + ret -=sizeof (struct argo_ring_message_header ); + ret -=ARGO_ROUNDUP (1 ); + +return (ret <0 ) ?0 : ret; +} +/* * argo_sanitize_ring creates a modified copy of the ring pointers * where the rx_ptr is rounded up to ensure it is aligned, and then
@@ -627,6 +679,43 @@ argo_pending_remove_all(struct argo_ring_info *ring_info) } } +static void +argo_pending_notify (struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *pending_ent; + +ASSERT (rw_is_locked (&argo_lock)); + +hlist_for_each_entry_safe (pending_ent, node, next, to_notify, node) + { +hlist_del (&pending_ent->node); +argo_signal_domid (pending_ent->id); +xfree (pending_ent); + } +} + +static void +argo_pending_find (const struct domain *d,struct argo_ring_info *ring_info, +uint32_t payload_space,struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *ent; + +ASSERT (rw_is_locked (&d->argo->lock)); + +spin_lock (&ring_info->lock); +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( payload_space >= ent->len ) + { +hlist_del (&ent->node); +hlist_add_head (&ent->node, to_notify); + } + } +spin_unlock (&ring_info->lock); +} +static int argo_pending_queue (struct argo_ring_info *ring_info,domid_t src_id,int len) {
@@ -668,6 +757,24 @@ argo_pending_requeue(struct argo_ring_info *ring_info, domid_t src_id, int len)return argo_pending_queue (ring_info, src_id, len); } +static void +argo_pending_cancel (struct argo_ring_info *ring_info,domid_t src_id) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( ent->id == src_id) + { +hlist_del (&ent->node); +xfree (ent); + } + } +} +static void argo_ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) {
@@ -705,6 +812,107 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info); } +/*ring data*/ + +static int +argo_fill_ring_data (struct domain *src_d, +XEN_GUEST_HANDLE (argo_ring_data_ent_t) data_ent_hnd) +{ +argo_ring_data_ent_t ent; +domid_t src_id; +struct domain *dst_d; +struct argo_ring_info *ring_info; +int ret; + +ASSERT (rw_is_locked (&argo_lock)); + + ret =copy_from_guest_errno (&ent, data_ent_hnd,1 ); +if ( ret ) +return ret; + +argo_dprintk ("argo_fill_ring_data: ent.ring.domain=%u,ent.ring.port=%u\n" , + ent.ring.domain_id, ent.ring.port); + + src_id = src_d->domain_id; + ent.flags =0 ; + + dst_d =get_domain_by_id (ent.ring.domain_id); + +if ( dst_d && dst_d->argo ) + { +read_lock (&dst_d->argo->lock); + + ring_info =argo_ring_find_info_by_match (dst_d, ent.ring.port, src_id, + src_d->argo->domain_cookie); + +if ( ring_info ) + { +uint32_t space_avail; + + ent.flags |=ARGO_RING_DATA_F_EXISTS ; + ent.max_message_size = + ring_info->len -sizeof (struct argo_ring_message_header ) - +ARGO_ROUNDUP (1 ); + +spin_lock (&ring_info->lock); + + space_avail =argo_ringbuf_payload_space (dst_d, ring_info); + +argo_dprintk ("argo_fill_ring_data: port=%d space_avail=%d" +" space_wanted=%d\n" , + ring_info->id.addr.port, space_avail, + ent.space_required); + +if ( space_avail >= ent.space_required ) + { +argo_pending_cancel (ring_info, src_id); + ent.flags |=ARGO_RING_DATA_F_SUFFICIENT ; + } +else + { +argo_pending_requeue (ring_info, src_id, ent.space_required); + ent.flags |=ARGO_RING_DATA_F_PENDING ; + } + +spin_unlock (&ring_info->lock); + +if ( space_avail == ent.max_message_size ) + ent.flags |=ARGO_RING_DATA_F_EMPTY ; + + } +read_unlock (&dst_d->argo->lock); + } + +if ( dst_d ) +put_domain (dst_d); + + ret =copy_field_to_guest_errno (data_ent_hnd, &ent, flags); +if ( ret ) +return ret; + ret =copy_field_to_guest_errno (data_ent_hnd, &ent, max_message_size); +if ( ret ) +return ret; + +return 0 ; +} + +static int +argo_fill_ring_data_array (struct domain *d,int nent, +XEN_GUEST_HANDLE (argo_ring_data_ent_t) data_ent_hnd) +{ +int ret =0 ; + +ASSERT (rw_is_locked (&argo_lock)); + +while ( !ret && nent-- ) + { + ret =argo_fill_ring_data (d, data_ent_hnd); +guest_handle_add_offset (data_ent_hnd,1 ); + } + +return ret; +} +/* * ring */
@@ -1166,6 +1374,116 @@ argo_register_ring(struct domain *d,* io */ +static void +argo_notify_ring (struct domain *d,struct argo_ring_info *ring_info, +struct hlist_head *to_notify) +{ +uint32_t space; + +ASSERT (rw_is_locked (&argo_lock)); +ASSERT (rw_is_locked (&d->argo->lock)); + +spin_lock (&ring_info->lock); + +if ( ring_info->len ) + space =argo_ringbuf_payload_space (d, ring_info); +else + space =0 ; + +spin_unlock (&ring_info->lock); + +if ( space ) +argo_pending_find (d, ring_info, space, to_notify); +} + +static void +argo_notify_check_pending (struct domain *d) +{ +int i; +HLIST_HEAD (to_notify); + +ASSERT (rw_is_locked (&argo_lock)); + +read_lock (&d->argo->lock); + +mb (); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; i++ ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) + { +argo_notify_ring (d, ring_info, &to_notify); + } + } +read_unlock (&d->argo->lock); + +if ( !hlist_empty (&to_notify) ) +argo_pending_notify (&to_notify); +} + +static long +argo_notify (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_data_t) ring_data_hnd) +{ +argo_ring_data_t ring_data; +int ret =0 ; + +read_lock (&argo_lock); + +if ( !d->argo ) + { +read_unlock (&argo_lock); +argo_dprintk ("!d->argo, ENODEV\n" ); +return -ENODEV ; + } + +argo_notify_check_pending (d); + +do { +if ( !guest_handle_is_null (ring_data_hnd) ) + { +/* Quick sanity check on ring_data_hnd */ + ret =copy_field_from_guest_errno (&ring_data, ring_data_hnd, magic); +if ( ret ) +break ; + +if ( ring_data.magic !=ARGO_RING_DATA_MAGIC ) + { +argo_dprintk ( +"ring.magic(%" PRIx64") != ARGO_RING_MAGIC(%llx), EINVAL\n" , + ring_data.magic,ARGO_RING_MAGIC ); + ret = -EINVAL ; +break ; + } + + ret =copy_from_guest_errno (&ring_data, ring_data_hnd,1 ); +if ( ret ) +break ; + + { +/* +* This is a guest pointer passed as a field in a struct +* so XEN_GUEST_HANDLE is used. +*/ +XEN_GUEST_HANDLE (argo_ring_data_ent_t) ring_data_ent_hnd; + ring_data_ent_hnd =guest_handle_for_field (ring_data_hnd, + argo_ring_data_ent_t, + data[0 ]); + ret =argo_fill_ring_data_array (d, ring_data.nent, + ring_data_ent_hnd); + } + } + }while (0 ); + +read_unlock (&argo_lock); + +return ret; +} + static long argo_sendv(struct domain *src_d, const argo_addr_t *src_addr, const argo_addr_t *dst_addr,
@@ -1341,6 +1659,14 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, iovs, niov, message_type);break ; } +case ARGO_MESSAGE_OP_notify: + { +XEN_GUEST_HANDLE_PARAM (argo_ring_data_t) ring_data_hnd = +guest_handle_cast (arg1, argo_ring_data_t); + + rc =argo_notify (d, ring_data_hnd); +break ; + }default : rc = -ENOSYS ;break ;
xen/include/public/argo.h +62 -0
80 lines not on this page · raw message
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 16/25] argo: implement the notify op
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +16
-0+static uint32_t +argo_ringbuf_payload_space (struct domain *d,struct argo_ring_info *ring_info) +{ +argo_ring_t ring; +int32_t ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ring.len = ring_info->len; +if ( !ring.len ) +return 0 ; + + ring.tx_ptr = ring_info->tx_ptr; + +if (argo_ringbuf_get_rx_ptr (ring_info, &ring.rx_ptr) ) +return 0 ;
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 410
Quoted 10 lines
xen/common/argo.c +10
-0+ +argo_dprintk ("argo_ringbuf_payload_space: tx_ptr=%d rx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) +return ring.len -sizeof (struct argo_ring_message_header ); + + ret = ring.rx_ptr - ring.tx_ptr; +if ( ret <0 ) + ret += ring.len;
Seeing these two if()-s - how is an empty ring distinguished from a completely full one? I'm getting the impression that ring.rx_ptr == ring.tx_ptr in both cases.
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 413
Quoted 2 lines
xen/common/argo.c +2
-0+ ret -=sizeof (struct argo_ring_message_header ); + ret -=ARGO_ROUNDUP (1 );
Wouldn't you instead better round ret to a suitable multiple of whatever granularity you try to arrange for here? Otherwise what is this extra subtraction supposed to do?
Quoted 28 lines
xen/common/argo.c +24
-0@@ -627,6 +679,43 @@ argo_pending_remove_all(struct argo_ring_info *ring_info) } } +static void +argo_pending_notify (struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *pending_ent; + +ASSERT (rw_is_locked (&argo_lock)); + +hlist_for_each_entry_safe (pending_ent, node, next, to_notify, node) + { +hlist_del (&pending_ent->node); +argo_signal_domid (pending_ent->id); +xfree (pending_ent); + } +} + +static void +argo_pending_find (const struct domain *d,struct argo_ring_info *ring_info, +uint32_t payload_space,struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *ent; + +ASSERT (rw_is_locked (&d->argo->lock));
xen/common/argo.c argo_pending_remove_all(struct argo_ring_info *ring_info) · line 715
Quoted 10 lines
xen/common/argo.c +10
-0+ +spin_lock (&ring_info->lock); +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( payload_space >= ent->len ) + { +hlist_del (&ent->node); +hlist_add_head (&ent->node, to_notify); + } + }
So if there's space available to fit e.g. just the first pending entry, you'd continue the loop and also signal all others, provided their lengths aren't too big? What good does producing such a burst of notifications do, when only one of the interested parties is actually going to be able to put something on the ring?
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 815
Quoted 5 lines
xen/common/argo.c +1
-0@@ -705,6 +812,107 @@ argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info)xfree (ring_info); } +/*ring data*/
Now this comment is malformed in any event.
xen/common/argo.c argo_ring_remove_info(struct domain *d, struct argo_ring_info *ring_info) · line 818
Quoted 2 lines
xen/common/argo.c +2
-0+static int +argo_fill_ring_data(struct domain *src_d,
const?
Quoted 41 lines
xen/common/argo.c +41
-0+static long +argo_notify (struct domain *d, +XEN_GUEST_HANDLE_PARAM (argo_ring_data_t) ring_data_hnd) +{ +argo_ring_data_t ring_data; +int ret =0 ; + +read_lock (&argo_lock); + +if ( !d->argo ) + { +read_unlock (&argo_lock); +argo_dprintk ("!d->argo, ENODEV\n" ); +return -ENODEV ; + } + +argo_notify_check_pending (d); + + do { + if ( !guest_handle_is_null(ring_data_hnd) ) + { + /* Quick sanity check on ring_data_hnd */ + ret = copy_field_from_guest_errno(&ring_data, ring_data_hnd, magic); + if ( ret ) + break; + + if ( ring_data.magic != ARGO_RING_DATA_MAGIC ) + { + argo_dprintk( + "ring.magic(%"PRIx64") != ARGO_RING_MAGIC(%llx), EINVAL\n", + ring_data.magic, ARGO_RING_MAGIC); + ret = -EINVAL; + break; + } + + ret = copy_from_guest_errno(&ring_data, ring_data_hnd, 1); + if ( ret ) + break; + + { + /*
xen/common/argo.c argo_register_ring(struct domain *d, · line 1478
Quoted 10 lines
xen/common/argo.c +10
-0+* This is a guest pointer passed as a field in a struct +* so XEN_GUEST_HANDLE is used. +*/ +XEN_GUEST_HANDLE (argo_ring_data_ent_t) ring_data_ent_hnd; + ring_data_ent_hnd =guest_handle_for_field (ring_data_hnd, + argo_ring_data_ent_t, + data[0 ]); + ret =argo_fill_ring_data_array (d, ring_data.nent, + ring_data_ent_hnd); + }
Stray braces and bogus indentation: The declaration can move up and then you don't need the braces and the extra level of indentation.
Quoted 18 lines
xen/include/public/argo.h +14
-0@@ -103,6 +104,40 @@ typedef struct argo_ring*/ #define ARGO_ROUNDUP (a) (((a) + 0xf) & ~(typeof(a))0xf) +/* +* Notify flags +*/ +/* Ring is empty */ +#define ARGO_RING_DATA_F_EMPTY (1U << 0) +/* Ring exists */ +#define ARGO_RING_DATA_F_EXISTS (1U << 1) +/* Pending interrupt exists. Do not rely on this field - for profiling only */ +#define ARGO_RING_DATA_F_PENDING (1U << 2) +/* Sufficient space to queue space_required bytes exists */ +#define ARGO_RING_DATA_F_SUFFICIENT (1U << 3) + +typedef struct argo_ring_data_ent +{
xen/include/public/argo.h typedef struct argo_ring · line 130
Quoted 10 lines
xen/include/public/argo.h +10
-0+argo_addr_t ring; +uint16_t flags; +uint16_t pad; +uint32_t space_required; +uint32_t max_message_size; +} argo_ring_data_ent_t; + +typedef struct argo_ring_data +{ + uint64_t magic;
What is this good for?
Quoted 18 lines
xen/include/public/argo.h +14
-0@@ -179,6 +214,33 @@ struct argo_ring_message_header*/ #define ARGO_MESSAGE_OP_sendv 5 +/* +* ARGO_MESSAGE_OP_notify +* +* Asks Xen for information about other rings in the system. +* +* ent->ring is the argo_addr_t of the ring you want information on. +* Uses the same ring matching rules as ARGO_MESSAGE_OP_sendv. +* +* ent->space_required : if this field is not null then Xen will check +* that there is space in the destination ring for this many bytes of payload. +* If sufficient space is available, it will set ARGO_RING_DATA_F_SUFFICIENT +* and CANCEL any pending notification for that ent->ring; otherwise it +* will schedule a notification event and the flag will not be set. +*
xen/include/public/argo.h struct argo_ring_message_header · line 240
Quoted 10 lines
xen/include/public/argo.h +10
-0+* These flags are set by Xen when notify replies: +* ARGO_RING_DATA_F_EMPTY ring is empty +* ARGO_RING_DATA_F_PENDING notify event is pending - * don't rely on this * +* ARGO_RING_DATA_F_SUFFICIENT sufficient space for space_required is there +* ARGO_RING_DATA_F_EXISTS ring exists +* +* arg1: XEN_GUEST_HANDLE(argo_ring_data_t) ring_data (may be NULL) +* arg2: NULL +* arg3: 0 (ZERO) +* arg4: 0 (ZERO)
Another observation I probably should have made earlier: You don't check that the NULL/ZERO specified argument are indeed so. Just like for padding fields, please do.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 16/25] argo: implement the notify op
Quoted 24 lines · Jan Beulich
Quoted 24 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +22
-0+static uint32_t +argo_ringbuf_payload_space (struct domain *d,struct argo_ring_info *ring_info) +{ +argo_ring_t ring; +int32_t ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ring.len = ring_info->len; +if ( !ring.len ) +return 0 ; + + ring.tx_ptr = ring_info->tx_ptr; + +if (argo_ringbuf_get_rx_ptr (ring_info, &ring.rx_ptr) ) +return 0 ; + +argo_dprintk ("argo_ringbuf_payload_space: tx_ptr=%d rx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) +return ring.len -sizeof (struct argo_ring_message_header );
Quoted 8 lines
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 410
Quoted 4 lines
xen/common/argo.c +4
-0+ + ret = ring.rx_ptr - ring.tx_ptr; +if ( ret <0 ) + ret += ring.len;Seeing these two if()-s - how is an empty ring distinguished from a completely full one? I'm getting the impression that ring.rx_ptr == ring.tx_ptr in both cases.
The subtraction from ring.len above is missing an additional subtraction of ARGO_ROUNDUP(1), which doesn't help reasoning about this. (Fixed in v2.)
If rx_ptr == tx_ptr, then the ring is empty. The ring insertion functions won't allow filling the ring, and I've added more comments in the v2 code to explain.
Quoted 6 lines
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 413
Quoted 2 lines
xen/common/argo.c +2
-0+ ret -=sizeof (struct argo_ring_message_header ); + ret -=ARGO_ROUNDUP (1 );Wouldn't you instead better round ret to a suitable multiple of whatever granularity you try to arrange for here? Otherwise what is this extra subtraction supposed to do?
re: subtraction, have added new comment:
/*
* The maximum size payload for a message that will be accepted is:
* (the available space between the ring indexes)
* minus (space for a message header)
* minus (space for one message slot)
* since argo_ringbuf_insert requires that one message slot be left
* unfilled, to avoid filling the ring to capacity and confusing a full
* ring with an empty one.
*/
re: rounding: Possibly. Not sure. In practice, both sides are updating the indexes in quantized steps matching the ARGO_ROUNDUP unit. Not sure it needs to change.
Quoted 37 lines
Quoted 37 lines
xen/common/argo.c +32
-0@@ -627,6 +679,43 @@ argo_pending_remove_all(struct argo_ring_info *ring_info) } } +static void +argo_pending_notify (struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *pending_ent; + +ASSERT (rw_is_locked (&argo_lock)); + +hlist_for_each_entry_safe (pending_ent, node, next, to_notify, node) + { +hlist_del (&pending_ent->node); +argo_signal_domid (pending_ent->id); +xfree (pending_ent); + } +} + +static void +argo_pending_find (const struct domain *d,struct argo_ring_info *ring_info, +uint32_t payload_space,struct hlist_head *to_notify) +{ +struct hlist_node *node, *next; +struct argo_pending_ent *ent; + +ASSERT (rw_is_locked (&d->argo->lock)); + +spin_lock (&ring_info->lock); +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( payload_space >= ent->len ) + { +hlist_del (&ent->node); +hlist_add_head (&ent->node, to_notify);
Quoted 8 lines
Review argo_pending_remove_all(struct argo_ring_info *ring_info) · line 715
Quoted 2 lines
(quoted diff) +2
-0+ } + }So if there's space available to fit e.g. just the first pending entry, you'd continue the loop and also signal all others, provided their lengths aren't too big? What good does producing such a burst of notifications do, when only one of the interested parties is actually going to be able to put something on the ring?
Added new comment:
/*
* TODO: Current policy here is to signal _all_ of the waiting domains
* interested in sending a message of size less than payload_space.
*
* This is likely to be suboptimal, since once one of them has added
* their message to the ring, there may well be insufficient room
* available for any of the others to transmit, meaning that they were
* woken in vain, which created extra work just to requeue their wait. *
* Retain this simple policy for now since it at least avoids starving a
* domain of available space notifications because of a policy that only
* notified other domains instead. Improvement may be possible;
* investigation required.
*/
Quoted 5 lines
xen/include/public/argo.h typedef struct argo_ring · line 130
Quoted 3 lines
xen/include/public/argo.h +3
-0+typedef struct argo_ring_data +{ + uint64_t magic;What is this good for?
New comment added:
/*
* Contents of the 'magic' field are inspected to verify that they contain
* an expected value before the hypervisor will perform writes into this
* structure in guest-supplied memory.
*/
Quoted 25 lines
Quoted 25 lines
xen/include/public/argo.h +20
-0@@ -179,6 +214,33 @@ struct argo_ring_message_header*/ #define ARGO_MESSAGE_OP_sendv 5 +/* +* ARGO_MESSAGE_OP_notify +* +* Asks Xen for information about other rings in the system. +* +* ent->ring is the argo_addr_t of the ring you want information on. +* Uses the same ring matching rules as ARGO_MESSAGE_OP_sendv. +* +* ent->space_required : if this field is not null then Xen will check +* that there is space in the destination ring for this many bytes of payload. +* If sufficient space is available, it will set ARGO_RING_DATA_F_SUFFICIENT +* and CANCEL any pending notification for that ent->ring; otherwise it +* will schedule a notification event and the flag will not be set. +* +* These flags are set by Xen when notify replies: +* ARGO_RING_DATA_F_EMPTY ring is empty +* ARGO_RING_DATA_F_PENDING notify event is pending - * don't rely on this * +* ARGO_RING_DATA_F_SUFFICIENT sufficient space for space_required is there +* ARGO_RING_DATA_F_EXISTS ring exists +*
Quoted 8 lines
xen/include/public/argo.h struct argo_ring_message_header · line 240
Quoted 4 lines
xen/include/public/argo.h +4
-0+* arg1: XEN_GUEST_HANDLE(argo_ring_data_t) ring_data (may be NULL) +* arg2: NULL +* arg3: 0 (ZERO) +* arg4: 0 (ZERO) Another observation I probably should have made earlier: You don't check that the NULL/ZERO specified argument are indeed so. Just like for padding fields, please do.
ack, thanks.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 16/25] argo: implement the notify op
Quoted 60 lines
Quoted 1 line
Quoted 1 line · On 20.12.18 at 07:12, <[email protected]> wrote:
Quoted 24 lines · Jan Beulich
Quoted 24 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:32, <[email protected]> wrote:
xen/common/argo.c +22
-0+static uint32_t +argo_ringbuf_payload_space (struct domain *d,struct argo_ring_info *ring_info) +{ +argo_ring_t ring; +int32_t ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + + ring.len = ring_info->len; +if ( !ring.len ) +return 0 ; + + ring.tx_ptr = ring_info->tx_ptr; + +if (argo_ringbuf_get_rx_ptr (ring_info, &ring.rx_ptr) ) +return 0 ; + +argo_dprintk ("argo_ringbuf_payload_space: tx_ptr=%d rx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) +return ring.len -sizeof (struct argo_ring_message_header );Quoted 8 lines
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 410
Quoted 4 lines
xen/common/argo.c +4
-0+ + ret = ring.rx_ptr - ring.tx_ptr; +if ( ret <0 ) + ret += ring.len;Seeing these two if()-s - how is an empty ring distinguished from a completely full one? I'm getting the impression that ring.rx_ptr == ring.tx_ptr in both cases.
The subtraction from ring.len above is missing an additional subtraction of
ARGO_ROUNDUP(1), which doesn't help reasoning about this. (Fixed in v2.)If
rx_ptr==tx_ptr, then the ring is empty. The ring insertion functions won't allow filling the ring, and I've added more comments in the v2 code to explain.Quoted 6 lines
xen/common/argo.c argo_ringbuf_get_rx_ptr(struct argo_ring_info *ring_info, uint32_t *rx_ptr) · line 413
Quoted 2 lines
xen/common/argo.c +2
-0+ ret -=sizeof (struct argo_ring_message_header ); + ret -=ARGO_ROUNDUP (1 );Wouldn't you instead better round ret to a suitable multiple of whatever granularity you try to arrange for here? Otherwise what is this extra subtraction supposed to do?
re: subtraction, have added new comment:
/*
* The maximum size payload for a message that will be accepted is:
* (the available space between the ring indexes)
* minus (space for a message header)
* minus (space for one message slot)
* sinceargo_ringbuf_insertrequires that one message slot be left
* unfilled, to avoid filling the ring to capacity and confusing a full
* ring with an empty one.
*/
Quoted 3 lines
re: rounding: Possibly. Not sure. In practice, both sides are updating the indexes in quantized steps matching the
ARGO_ROUNDUPunit. Not sure it needs to change.
Here you appear to talk about both sides being well behaved. Did you also consider misbehaving partners?
Quoted 6 lines
Quoted 5 lines
xen/include/public/argo.h typedef struct argo_ring · line 130
Quoted 3 lines
xen/include/public/argo.h +3
-0+typedef struct argo_ring_data +{ + uint64_t magic;What is this good for?
Quoted 6 lines
New comment added:
/*
* Contents of the 'magic' field are inspected to verify that they contain
* an expected value before the hypervisor will perform writes into this
* structure in guest-supplied memory.
*/
But this does not help understand what this verification is good for (or what it guards against). This again looks to be a reduction of likelihood of misbehavior, instead of its exclusion.
As things accumulate: Personally I'd consider it better to wait with posting a new version until discussions have settled. At this point I'm already uncertain whether it'll be worthwhile for me to thoroughly look at v2, when I'm likely to re-encounter things I've already commented on in v1.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 17/25 xsm, argo: XSM control for any access to argo by a domain
In reply to Christopher ClarkWill inhibit initialization of the domain's argo data structure to prevent receiving any messages or notifications and access to any of the argo hypercall operations.
--- xen/common/argo.c | 4 ++-- xen/include/xsm/dummy.h | 5 +++++ xen/include/xsm/xsm.h | 6 ++++++ xen/xsm/dummy.c | 1 + xen/xsm/flask/hooks.c | 7 +++++++ xen/xsm/flask/policy/access_vectors | 3 +++ 6 files changed, 24 insertions(+), 2 deletions(-)
xen/common/argo.c +2 -2
@@ -1588,7 +1588,7 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,argo_dprintk ("->do_argo_message_op(%d,%p,%p,%d,%d)\n" , cmd, (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); -if (unlikely (!opt_argo_enabled) ) +if (unlikely (!opt_argo_enabled ||xsm_argo_enable (d)) ) { rc = -ENOSYS ;argo_dprintk ("<-do_argo_message_op()=%ld\n" , rc);
@@ -1685,7 +1685,7 @@ argo_init(struct domain *d)int i;int rc; -if ( !opt_argo_enabled ) +if ( !opt_argo_enabled ||xsm_argo_enable (d) ) {argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id);return 0 ;
xen/include/xsm/dummy.h +5 -0
@@ -721,6 +721,11 @@ static XSM_INLINE int xsm_dm_op(XSM_DEFAULT_ARG struct domain *d)#endif /* CONFIG_X86 */ #ifdef CONFIG_ARGO +static XSM_INLINE int xsm_argo_enable(struct domain *d) +{ +return 0 ; +} + static XSM_INLINE int xsm_argo_register_single_source(struct domain *d,struct domain *t) {
xen/include/xsm/xsm.h +6 -0
@@ -182,6 +182,7 @@ struct xsm_operations {int (*xen_version) (uint32_t cmd);int (*domain_resource_map) (struct domain *d);#ifdef CONFIG_ARGO +int (*argo_enable) (struct domain *d);int (*argo_register_single_source) (struct domain *d,struct domain *t);int (*argo_register_any_source) (struct domain *d);int (*argo_send) (struct domain *d,struct domain *t);
@@ -704,6 +705,11 @@ static inline int xsm_domain_resource_map(xsm_default_t def, struct domain *d) }#ifdef CONFIG_ARGO +static inline xsm_argo_enable(struct domain *d) +{ +return xsm_ops->argo_enable (d); +} + static inline xsm_argo_register_single_source(struct domain *d, struct domain *t) { return xsm_ops->argo_register_single_source(d, t);
xen/xsm/dummy.c +1 -0
@@ -153,6 +153,7 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)set_to_dummy_if_null (ops, xen_version);set_to_dummy_if_null (ops, domain_resource_map);#ifdef CONFIG_ARGO +set_to_dummy_if_null (ops, argo_enable);set_to_dummy_if_null (ops, argo_register_single_source);set_to_dummy_if_null (ops, argo_register_any_source);set_to_dummy_if_null (ops, argo_send);
xen/xsm/flask/hooks.c +7 -0
@@ -1718,6 +1718,12 @@ static int flask_domain_resource_map(struct domain *d) }#ifdef CONFIG_ARGO +static int flask_argo_enable (struct domain *d) +{ +return avc_has_perm (domain_sid (d),SECINITSID_XEN ,SECCLASS_ARGO , +ARGO__ENABLE ,NULL ); +} +static int flask_argo_register_single_source (struct domain *d,struct domain *t) {
@@ -1873,6 +1879,7 @@ static struct xsm_operations flask_ops = { .xen_version = flask_xen_version, .domain_resource_map = flask_domain_resource_map, #ifdef CONFIG_ARGO + .argo_enable = flask_argo_enable, .argo_register_single_source = flask_argo_register_single_source, .argo_register_any_source = flask_argo_register_any_source, .argo_send = flask_argo_send,
xen/xsm/flask/policy/access_vectors +3 -0
@@ -535,6 +535,9 @@ class version # Class argo is used to describe the Argo interdomain communication system. class argo { + # Enable initialization of a domain's argo subsystem and + # permission to access the argo hypercall operations. + enable # Domain requesting registration of a communication ring # to receive messages from a specific other domain. register_single_source
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 18/25 argo: limit the max number of rings that a domain may register.
In reply to Christopher ClarkVery basic implementation: a fixed limit of 128.
--- xen/common/argo.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
xen/common/argo.c +17 -0
@@ -26,6 +26,8 @@#include <xen/time.h> #include <xsm/xsm.h> +#define ARGO_MAX_RINGS_PER_DOMAIN 128U +DEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);DEFINE_XEN_GUEST_HANDLE (argo_send_addr_t);
@@ -101,6 +103,8 @@ struct argo_domainstruct hlist_head ring_hash[ARGO_HTABLE_SIZE ];/* id cookie, written only at init, so readable with R(L1) */ uint64_t domain_cookie; +/* counter of rings registered by this domain, protected by L2 */ +uint32_t ring_count; };/*
@@ -1161,7 +1165,10 @@ argo_unregister_ring(struct domain *d, ring_info =argo_ring_find_info (d, &ring.id);if ( ring_info ) + {argo_ring_remove_info (d, ring_info); + d->argo->ring_count--; + }write_unlock (&d->argo->lock);
@@ -1298,6 +1305,12 @@ argo_register_ring(struct domain *d,write_lock (&d->argo->lock); do { + if ( d->argo->ring_count >= ARGO_MAX_RINGS_PER_DOMAIN ) + { + ret = -ENOSPC; + break; + } + ring_info = argo_ring_find_info(d, &ring.id); if ( !ring_info )
@@ -1357,7 +1370,10 @@ argo_register_ring(struct domain *d, ret = update_tx_ptr ?argo_update_tx_ptr (ring_info, ring.tx_ptr) :argo_ring_map_page (ring_info,0 ,NULL );if ( !ret ) + { ring_info->len = ring.len; + d->argo->ring_count++; + } }while (0 );
@@ -1698,6 +1714,7 @@ argo_init(struct domain *d)return -ENOMEM ;rwlock_init (&argo->lock); + argo->ring_count =0 ;for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i )INIT_HLIST_HEAD (&argo->ring_hash[i]);
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 18/25] argo: limit the max number of rings that a domain may register.
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:32, <[email protected]> wrote:
Very basic implementation: a fixed limit of 128.
Such restrictions to limit resource use would better be implemented right away for code that can be used (in a limited way) already with just the initial parts of the series applied.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 19/25 argo: limit the max number of notify requests in a single operation.
In reply to Christopher ClarkVery basic implementation: a fixed limit of 256.
--- xen/common/argo.c | 7 +++++++ 1 file changed, 7 insertions(+)
xen/common/argo.c +7 -0
@@ -27,6 +27,7 @@#include <xsm/xsm.h> #define ARGO_MAX_RINGS_PER_DOMAIN 128U +#define ARGO_MAX_NOTIFY_COUNT 256UDEFINE_XEN_GUEST_HANDLE (argo_pfn_t);DEFINE_XEN_GUEST_HANDLE (argo_addr_t);
@@ -1480,6 +1481,12 @@ argo_notify(struct domain *d,if ( ret )break ; +if ( ring_data.nent >ARGO_MAX_NOTIFY_COUNT ) + { + ret = -EACCES ; +break ; + } + {/* * This is a guest pointer passed as a field in a struct
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 20/25 argo, xsm: notify: don't describe rings that cannot be sent to
In reply to Christopher Clark--- xen/common/argo.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
xen/common/argo.c +11 -0
@@ -845,6 +845,17 @@ argo_fill_ring_data(struct domain *src_d,if ( dst_d && dst_d->argo ) { +/* +* Don't supply information about rings that a guest is not +* allowed to send to. +*/ + ret =xsm_argo_send (src_d, dst_d); +if ( ret ) + { +put_domain (dst_d); +return ret; + } +read_lock (&dst_d->argo->lock); ring_info = argo_ring_find_info_by_match(dst_d, ent.ring.port, src_id,
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 21/25 argo: add array_index_nospec to guard the result of the hash func
In reply to Christopher ClarkThis is out of an abundance of caution, since this is a very basic hash function, chosen more for its bucket distribution properties to cluster related rings rather than for cryptographic strength or any uniformness of output, and it operates upon values supplied by the guest just before being used as an array index.
--- xen/common/argo.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
xen/common/argo.c +4 -2
@@ -23,6 +23,7 @@#include <xen/event.h> #include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/nospec.h> #include <xen/time.h> #include <xsm/xsm.h>
@@ -1094,7 +1095,7 @@ argo_ring_find_info(const struct domain *d, const struct argo_ring_id *id)ASSERT (rw_is_locked (&d->argo->lock)); - hash =argo_hash_fn (id); + hash =array_index_nospec (argo_hash_fn (id),ARGO_HTABLE_SIZE );argo_dprintk ("d->argo=%p, d->argo->ring_hash[%d]=%p id=%p\n" , d->argo, hash, d->argo->ring_hash[hash].first, id);
@@ -1349,7 +1350,8 @@ argo_register_ring(struct domain *d, ring_info->id = ring.id;INIT_HLIST_HEAD (&ring_info->pending); - hash =argo_hash_fn (&ring_info->id); + hash =array_index_nospec (argo_hash_fn (&ring_info->id), +ARGO_HTABLE_SIZE );hlist_add_head (&ring_info->node, &d->argo->ring_hash[hash]); printk(XENLOG_INFO "argo: vm%u registering ring (vm%u:%x vm%d)\n",
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 21/25] argo: add array_index_nospec to guard the result of the hash func
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:33, <[email protected]> wrote:
This is out of an abundance of caution, since this is a very basic hash function, chosen more for its bucket distribution properties to cluster related rings rather than for cryptographic strength or any uniformness of output, and it operates upon values supplied by the guest just before being used as an array index.
Same here: Better to put this in place right away for new code than to incrementally add it later.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 22/25 xen/evtchn: expose send_guest_global_virq for use within Xen
In reply to Christopher ClarkTo be used by Argo for delivery of notifications to some guests.
--- xen/common/event_channel.c | 2 +- xen/include/xen/event.h | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-)
xen/common/event_channel.c +1 -1
@@ -769,7 +769,7 @@ void send_guest_vcpu_virq(struct vcpu *v, uint32_t virq)spin_unlock_irqrestore (&v->virq_lock, flags); } -static void send_guest_global_virq (struct domain *d,uint32_t virq) +void send_guest_global_virq (struct domain *d,uint32_t virq) {unsigned long flags;int port;
xen/include/xen/event.h +7 -0
@@ -29,6 +29,13 @@ void send_guest_vcpu_virq(struct vcpu *v, uint32_t virq);void send_global_virq (uint32_t virq);/* +* send_guest_global_virq: +* @d: Domain to which VIRQ should be sent +* @virq: Virtual IRQ number (VIRQ_*), must be global +*/ +void send_guest_global_virq (struct domain *d,uint32_t virq); + +/* * sent_global_virq_handler: Set a global VIRQ handler. * @d: New target domain for this VIRQ * @virq: Virtual IRQ number (VIRQ_*), must be global
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 22/25] xen/evtchn: expose send_guest_global_virq for use within Xen
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:33, <[email protected]> wrote:
To be used by Argo for delivery of notifications to some guests.
Better not to make this a separate patch: By folding it into where it's needed it is easier for everyone to judge whether the exposure is indeed necessary, and it also eliminates the risk of the series getting committed up to here, and the function then being pointlessly non- static for an extended period of time.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 23/25 argo: signal x86 HVM and ARM via VIRQ
In reply to Christopher Clark* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
--- xen/common/argo.c | 39 ++++++++++++++++++++++++++++++++++++--- xen/include/public/argo.h | 3 +++ xen/include/public/xen.h | 2 +- 3 files changed, 40 insertions(+), 4 deletions(-)
xen/common/argo.c +36 -3
@@ -181,18 +181,51 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#endif /* -* Event channel +* Signalling */ +static unsigned int argo_signal_method (const struct domain *d) +{ +unsigned int method; +#ifdef CONFIG_X86 +if (is_hvm_domain (d) ) + method =ARGO_SIGNAL_METHOD_VIRQ ; +else + method =ARGO_SIGNAL_METHOD_EVTCHN ; +#else + method =ARGO_SIGNAL_METHOD_VIRQ ; +#endif +return method; +} +static void argo_signal_domain (struct domain *d) { -argo_dprintk ("signalling domid:%d\n" , d->domain_id); +unsigned int method =argo_signal_method (d);if ( !d->argo )/* This can happen if the domain is being destroyed */ return ; -evtchn_send (d, d->argo->evtchn_port); +argo_dprintk ("signalling domid:%d via method:%u\n" , d->domain_id, method); + +switch ( method ) + { +case ARGO_SIGNAL_METHOD_EVTCHN : + { +evtchn_send (d, d->argo->evtchn_port); +break ; + } +case ARGO_SIGNAL_METHOD_VIRQ : + { +send_guest_global_virq (d,VIRQ_ARGO ); +break ; + } +default : + { +BUG (); +break ; + } + } } static void
xen/include/public/argo.h +3 -0
@@ -150,6 +150,9 @@ struct argo_ring_message_header#endif }; +#define ARGO_SIGNAL_METHOD_EVTCHN 1 +#define ARGO_SIGNAL_METHOD_VIRQ 2 +/* * Hypercall operations */
xen/include/public/xen.h +1 -1
@@ -178,7 +178,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);#define VIRQ_CON_RING 8/* G. (DOM0) Bytes received on console */ #define VIRQ_PCPU_STATE 9/* G. (DOM0) PCPU state changed */ #define VIRQ_MEM_EVENT 10/* G. (DOM0) A memory event has occurred */ -#define VIRQ_XC_RESERVED 11/* G. Reserved for XenClient */ +#define VIRQ_ARGO 11/* G. Argo interdomain message notification */ #define VIRQ_ENOMEM 12/* G. (DOM0) Low on heap memory */ #define VIRQ_XENPMU 13/* V. PMC interrupt */
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
Hi,
Quoted 11 lines · Christopher Clark
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
I am a bit confused. vIRQs are based on event channel, so how do you remove the requirement on event channel?
Cheers,
Signature · 5 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
Quoted 11 lines · Julien Grall
Hi,
Quoted 7 lines · Christopher Clark
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
* x86 HVM domains and all ARM domains are notified via VIRQ.
Quoted 7 lines
Quoted 4 lines
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
I am a bit confused. vIRQs are based on event channel, so how do you remove the requirement on event channel?
Are VIRQs always delivered via events in all cases? I was under the impression that was not necessarily so with HVM guests but I haven't checked and could well be incorrect.
A bit of context might help with how this multiple-method logic (as submitted) was arrived at:
1) Both XenClient's original version of v4v, and that used in OpenXT, deliver notifications to guests via VIRQ.
This logic has been performing fine for our uses cases, so there hasn't really been a push to switch away from it.
2) The last version of v4v that was submitted to xen-devel for iteration with the Xen community was intended to use event channels instead, in response to a request from Jan at the time. Given that expressed preference, I've added that, plumbing it in through via the IPI event method exposed in patch #01, and then used in patch #05, of the submitted series.
3) Bromium's uxen uses different logic for delivery of events to non-PV guests: an edge-triggered, ISA IRQ, along these lines:
#define ARGO_SIGNAL_ISA_IRQ 8
hvm_isa_irq_assert(d, ARGO_SIGNAL_ISA_IRQ, NULL);
hvm_isa_irq_deassert(d, ARGO_SIGNAL_ISA_IRQ);I'm told that this avoids the need to EOI in the guest, reducing the VMEXIT load, and using an ISA IRQ avoids some logic in Windows that requires that a device be detected. I briefly looked into adding this to Argo, but Linux wasn't immediately happy and I haven't had time to look into it further given the proximity of the 4.12 release, with other work still to complete.
Anyway: since method 3 isn't ready to submit, and if VIRQs don't have an advantage over using event channels directly wrt. to needing in-guest support to function, then I can drop this patch (#23) and simplify the get_config op (#25), which will leave all notifications being delivered as events.
Alternatively, if this is about which is the right delivery method for ARM, with some valid reason to retain use of VIRQ for HVM x86, then I'm happy to switch ARM over to deliver by the event method rather than VIRQ if that makes more sense.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
Quoted 44 lines
-----Original Message-----
From: Christopher Clark [mailto:[email protected]]
Sent: 04 December 2018 09:03
To: Julien Grall <[email protected]>
Cc: xen-devel <[email protected]>; [email protected]; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; Stefano Stabellini <[email protected]>; Tim (Xen.org) <[email protected]>; Wei Liu <[email protected]>; Paul Durrant <[email protected]>; Rich Persaud <[email protected]>; Ross Philipson <[email protected]>; eric chanudet <[email protected]>; James McKenzie <[email protected]>; Jason Andryuk <[email protected]>; Daniel Smith <[email protected]>
Subject: Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQQuoted 7 lines · Julien Grall
Hi,
Quoted 3 lines · Christopher Clark
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the
guest
Quoted 5 lines
Quoted 5 lines
kernel, so it is fine to depend on and use it.
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to
be
Quoted 1 line
Quoted 1 line
installed within these guests in order to use Argo. VIRQ signalling is
also
Quoted 1 line
Quoted 1 line
the method that has been in use for the longest period with this
hypercall
Quoted 4 lines
Quoted 1 line
in both XenClient and OpenXT.
I am a bit confused. vIRQs are based on event channel, so how do you remove the requirement on event channel?
Are VIRQs always delivered via events in all cases? I was under the impression that was not necessarily so with HVM guests but I haven't checked and could well be incorrect.
Quoted 7 lines
A bit of context might help with how this multiple-method logic (as submitted) was arrived at:
1) Both XenClient's original version of v4v, and that used in OpenXT, deliver notifications to guests via VIRQ.
This logic has been performing fine for our uses cases, so there hasn't really been a push to switch away from it.
I'm not aware of any way to map VIRQs to vectors directly so I think they have to be dealt with as any other event channel.
Paul
Quoted 28 lines
2) The last version of v4v that was submitted to xen-devel for iteration with the Xen community was intended to use event channels instead, in response to a request from Jan at the time. Given that expressed preference, I've added that, plumbing it in through via the IPI event method exposed in patch #01, and then used in patch #05, of the submitted series.
3) Bromium's uxen uses different logic for delivery of events to non-PV guests: an edge-triggered, ISA IRQ, along these lines:
#define ARGO_SIGNAL_ISA_IRQ 8 hvm_isa_irq_assert(d, ARGO_SIGNAL_ISA_IRQ, NULL); hvm_isa_irq_deassert(d, ARGO_SIGNAL_ISA_IRQ);I'm told that this avoids the need to EOI in the guest, reducing the VMEXIT load, and using an ISA IRQ avoids some logic in Windows that requires that a device be detected. I briefly looked into adding this to Argo, but Linux wasn't immediately happy and I haven't had time to look into it further given the proximity of the 4.12 release, with other work still to complete.
Anyway: since method 3 isn't ready to submit, and if VIRQs don't have an advantage over using event channels directly wrt. to needing in-guest support to function, then I can drop this patch (#23) and simplify the
get_configop (#25), which will leave all notifications being delivered as events.
Quoted 6 lines
Alternatively, if this is about which is the right delivery method for ARM, with some valid reason to retain use of VIRQ for HVM x86, then I'm happy to switch ARM over to deliver by the event method rather than VIRQ if that makes more sense.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
I think there are two issues:
1) VIRQ vs some other sort of event channel
For PV guests we originally chose a VIRQ in order to have a well known number against which the kernel driver could bind, so that it wasn't dependent on any of the other interdomain communication systems (such as xenstore) to find the correct channel.
VIRQs and events in general make sense for PV domains since the up call mechanism fits well into the way argo expects scheduling to work.
I dont see any pressing reason to not use a VIRQ for PV or PVH domains, perhaps I've missed something.
2) VIRQ vs direct injection of vector in hvm case.
for HVM guests - you can make the argument for injection via a (potentially hardware) emulated LAPIC. In this case the best performance is obtained by not having to clear the interrupt (requiring another VMEXIT). The only two sorts of interrupts that have that property are 1) MSIs, and 2) edge interrupts via the IOAPIC:
Not all operating systems had/have good MSI support, PCI doesn't [really] support edge interrupts on LNK[ABCD], and even if you go the PCI route almost no OSes do the right thing anyway. However it is possible to specify a device via ACPI with a fixed GSI with an edge interrupt. Windows from at least XP, and linux both handle that well and it's the perfect fit for sending the idempotent "you have work to do" type interrupt that argo needs.
One of the design goals was that multiple independent actors in a VM should be able to run without knowledge of each other, and using a standard GSI allows something like an EDK II, grub and linux to all use it without having to hand forward state.
J.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
In reply to Christopher ClarkHi Christopher,
Quoted 17 lines · Christopher Clark
Quoted 16 lines · Julien Grall
Hi,
Quoted 11 lines · Christopher Clark
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
Quoted 6 lines
Quoted 2 lines
I am a bit confused. vIRQs are based on event channel, so how do you remove the requirement on event channel?
Are VIRQs always delivered via events in all cases? I was under the impression that was not necessarily so with HVM guests but I haven't checked and could well be incorrect.
It depends on your meaning of vIRQs. We seem to use it for two cases in the hypervisor.
In the context of send_guest_global_virq(), the interrupt will be para-virtualized as we delivered via events.
On Arm most of the virtual interrupts will goes through the virtual interrupt controller. They can be raised using vgic_inject_irq() and event channel are therefore not required. I think this is fairly similar on PVH/HVM but I will let the x86 folks confirm here.
Quoted 8 lines
A bit of context might help with how this multiple-method logic (as submitted) was arrived at:
1) Both XenClient's original version of v4v, and that used in OpenXT, deliver notifications to guests via VIRQ.
This logic has been performing fine for our uses cases, so there hasn't really been a push to switch away from it.
From my understanding, VIRQ is just a convenience alias for the guest to receive the associated event. The guest only need to say "I want to bind VIRQ foo". In the other case, you would need to allocate the event channel in the hypervisor and then pass the information somehow to the guest.
Quoted 28 lines
2) The last version of v4v that was submitted to xen-devel for iteration with the Xen community was intended to use event channels instead, in response to a request from Jan at the time. Given that expressed preference, I've added that, plumbing it in through via the IPI event method exposed in patch #01, and then used in patch #05, of the submitted series.
3) Bromium's uxen uses different logic for delivery of events to non-PV guests: an edge-triggered, ISA IRQ, along these lines:
#define ARGO_SIGNAL_ISA_IRQ 8 hvm_isa_irq_assert(d, ARGO_SIGNAL_ISA_IRQ, NULL); hvm_isa_irq_deassert(d, ARGO_SIGNAL_ISA_IRQ);I'm told that this avoids the need to EOI in the guest, reducing the VMEXIT load, and using an ISA IRQ avoids some logic in Windows that requires that a device be detected. I briefly looked into adding this to Argo, but Linux wasn't immediately happy and I haven't had time to look into it further given the proximity of the 4.12 release, with other work still to complete.
Anyway: since method 3 isn't ready to submit, and if VIRQs don't have an advantage over using event channels directly wrt. to needing in-guest support to function, then I can drop this patch (#23) and simplify the
get_configop (#25), which will leave all notifications being delivered as events.
Quoted 4 lines
Alternatively, if this is about which is the right delivery method for ARM, with some valid reason to retain use of VIRQ for HVM x86, then I'm happy to switch ARM over to deliver by the event method rather than VIRQ if that makes more sense.
For Arm, 3) would look the right approach if you want to avoid the dependencies on the event channel driver.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
In reply to Christopher ClarkQuoted 6 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:33, <[email protected]> wrote:
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
Quoted 6 lines
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
I'm afraid I don't follow: send_guest_global_virq() uses, well, evtchn_port_set_pending(), just like evtchn_send() does. Therefore how does sending a vIRQ help with a guest without event channel awareness?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 23/25] argo: signal x86 HVM and ARM via VIRQ
Quoted 14 lines · Jan Beulich
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:33, <[email protected]> wrote:
* x86 PV domains are notified via event channel.
PV guests are known to have the event channel software present in the guest kernel, so it is fine to depend on and use it.
Quoted 6 lines
* x86 HVM domains and all ARM domains are notified via VIRQ.
The intent is to remove the requirement for event channel software to be installed within these guests in order to use Argo. VIRQ signalling is also the method that has been in use for the longest period with this hypercall in both XenClient and OpenXT.
Quoted 4 lines
I'm afraid I don't follow:
send_guest_global_virq()uses, well,evtchn_port_set_pending(), just likeevtchn_send()does. Therefore how does sending a vIRQ help with a guest without event channel awareness?
On this topic, signal delivery to guests, I'm simplifying the next version of the patch series: I'm just going to use VIRQs.
It doesn't remove the dependency on event channel software in the guest, and it doesn't optimize efficiency of notifications with HVM guests but I'd like to come back and address that in a subsequent patch once this series has been accepted. It'll follow from the explanation that James's message provides.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 24/25 argo: unmap rings on suspend and send signal to ring-owners on resume
In reply to Christopher Clarkso that the guest may re-register the rings on resume with current mappings.
--- xen/common/argo.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++ xen/common/domain.c | 9 +++++++ xen/include/xen/argo.h | 2 ++ 3 files changed, 80 insertions(+)
xen/common/argo.c +69 -0
@@ -840,6 +840,16 @@ static void argo_ring_remove_mfns(const struct domain *d, }static void +argo_ring_reset (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock)); + +argo_ring_remove_mfns (d, ring_info); + ring_info->len =0 ; + ring_info->tx_ptr =0 ; +} + +static void argo_ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) {ASSERT (rw_is_write_locked (&d->argo->lock));
@@ -1832,3 +1842,62 @@ argo_destroy(struct domain *d)* domain. */ } + +void +argo_shutdown_for_suspend (struct domain *d) +{ +int i; + +if ( !d ) +return ; + +if (get_domain (d) ) + { +read_lock (&argo_lock); + +if ( d->argo ) + { +write_lock (&d->argo->lock); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; i++ ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; +hlist_for_each_entry_safe (ring_info, node, + next, &d->argo->ring_hash[i], node) +argo_ring_reset (d, ring_info); + } + +write_unlock (&d->argo->lock); + } + +read_unlock (&argo_lock); + +put_domain (d); + } +} + +void +argo_resume (struct domain *d) +{ +bool send_wakeup; + +if ( !d ) +return ; + +if ( !get_domain (d) ) +return ; + +read_lock (&argo_lock); + +read_lock (&d->argo->lock); + send_wakeup = ( d->argo->ring_count >0 ); +read_unlock (&d->argo->lock); + +if ( send_wakeup ) +argo_signal_domain (d); + +read_unlock (&argo_lock); + +put_domain (d); +}
xen/common/domain.c +9 -0
@@ -88,6 +88,11 @@ static void __domain_finalise_shutdown(struct domain *d)if ( !v->paused_for_shutdown )return ; +#ifdef CONFIG_ARGO +if ( d->shutdown_code == SHUTDOWN_suspend ) +argo_shutdown_for_suspend (d); +#endif + d->is_shut_down =1 ;if ( (d->shutdown_code == SHUTDOWN_suspend) && d->suspend_evtchn )evtchn_send (d, d->suspend_evtchn);
@@ -856,6 +861,10 @@ void domain_resume(struct domain *d)spin_unlock (&d->shutdown_lock);domain_unpause (d); + +#ifdef CONFIG_ARGO +argo_resume (d); +#endif }int vcpu_start_shutdown_deferral (struct vcpu *v)
xen/include/xen/argo.h +2 -0
@@ -26,5 +26,7 @@ struct argo_domain;int argo_init (struct domain *d);void argo_destroy (struct domain *d); +void argo_shutdown_for_suspend (struct domain *d); +void argo_resume (struct domain *d);#endif
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 24/25] argo: unmap rings on suspend and send signal to ring-owners on resume
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:33, <[email protected]> wrote:
so that the guest may re-register the rings on resume with current mappings.
Is this something guests really need help with, rather than managing it on their own? What does "current mappings" here mean, i.e. why do rings need re-registration in the first place?
Quoted 14 lines
xen/common/argo.c +14
-0+void +argo_resume (struct domain *d) +{ +bool send_wakeup; + +if ( !d ) +return ; + +if ( !get_domain (d) ) +return ; + +read_lock (&argo_lock); + +read_lock (&d->argo->lock);
xen/common/argo.c argo_destroy(struct domain *d) · line 1903
Quoted 10 lines
xen/common/argo.c +10
-0+ send_wakeup = ( d->argo->ring_count >0 ); +read_unlock (&d->argo->lock); + +if ( send_wakeup ) +argo_signal_domain (d); + +read_unlock (&argo_lock); + +put_domain (d); +}
domain_resume() also gets called from domain_soft_reset(). Do you really want such handling in that case as well, when after a soft-reset the domain is supposed to be "blank"?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 24/25] argo: unmap rings on suspend and send signal to ring-owners on resume
Quoted 7 lines · Jan Beulich
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines · On 01.12.18 at 02:33, <[email protected]> wrote:
so that the guest may re-register the rings on resume with current mappings.
Is this something guests really need help with, rather than managing it on their own? What does "current mappings" here mean, i.e. why do rings need re-registration in the first place?
My understanding is that the gfn->mfn mapping is not necessarily stable across entry and exit to host S4, suspend to disk, so the rings need to be torn down before suspend to stop further writes into those pages after resume.
When the guest gets the notification after resume, it can re-register the rings with its list of gfns, which can then be re-translated into the (possibly) new mfns needed for the ring.
Quoted 20 lines
Quoted 20 lines
xen/common/argo.c +20
-0+void +argo_resume (struct domain *d) +{ +bool send_wakeup; + +if ( !d ) +return ; + +if ( !get_domain (d) ) +return ; + +read_lock (&argo_lock); + +read_lock (&d->argo->lock); + send_wakeup = ( d->argo->ring_count >0 ); +read_unlock (&d->argo->lock); + +if ( send_wakeup ) +argo_signal_domain (d); +
Quoted 8 lines
xen/common/argo.c argo_destroy(struct domain *d) · line 1877
Quoted 4 lines
xen/common/argo.c +4
-0+read_unlock (&argo_lock); + +put_domain (d); +}
domain_resume()also gets called fromdomain_soft_reset(). Do you really want such handling in that case as well, when after a soft-reset the domain is supposed to be "blank"?
Thanks for the pointer to soft reset: I've added implementation for this to the next version of the patch series, and it'll be fine with resume then.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 25/25 argo: implement the get_config op to query notification config
In reply to Christopher ClarkNeeded by a guest to obtain the evtchn port to use, if notifications are via event channel, so: this operation will return the current notification method active for the domain, and method-specific configuration data:
* event channel: port number
* VIRQ: virq number
Return structure has reserved space intentionally to for future alternative notification mechanism to return data about both an IRQ number and bound VCPU.
--- xen/common/argo.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 28 +++++++++++++++++++++++ 2 files changed, 85 insertions(+)
xen/common/argo.c +57 -0
@@ -1656,6 +1656,46 @@ argo_sendv(struct domain *src_d, const argo_addr_t *src_addr,return ( ret <0 ) ? ret : len; } +static void +argo_get_config (struct domain *d,argo_get_config_t *get_config) +{ +unsigned int method =argo_signal_method (d); + + get_config->signal_method = method; + +switch ( method ) + { +case ARGO_SIGNAL_METHOD_EVTCHN : + { +read_lock (&argo_lock); +read_lock (&d->argo->lock); + + get_config->signal.evtchn = d->argo->evtchn_port; + +read_unlock (&d->argo->lock); +read_unlock (&argo_lock); + +argo_dprintk ("signal for dom:%d evtchn %u\n" , d->domain_id, + get_config->signal.evtchn); + +break ; + } +case ARGO_SIGNAL_METHOD_VIRQ : + { + get_config->signal.virq =VIRQ_ARGO ; + +argo_dprintk ("signal for dom:%d virq %u\n" , d->domain_id, + get_config->signal.virq); +break ; + } +default : + { +BUG (); +break ; + } + } +} + long do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2,
@@ -1746,6 +1786,23 @@ do_argo_message_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, rc =argo_notify (d, ring_data_hnd);break ; } +case ARGO_MESSAGE_OP_get_config: + { +XEN_GUEST_HANDLE_PARAM (argo_get_config_t) get_config_hnd = +guest_handle_cast (arg1, argo_get_config_t); +argo_get_config_t get_config; + +if (unlikely (!guest_handle_okay (get_config_hnd,1 )) ) +break ; + +argo_get_config (d, &get_config); + +if (__copy_to_guest (get_config_hnd, &get_config,1 ) ) +break ; + + rc =0 ; +break ; + }default : rc = -ENOSYS ;break ;
xen/include/public/argo.h +28 -0
@@ -20,6 +20,7 @@#define __XEN_PUBLIC_ARGO_H__#include "xen.h" +#include "event_channel.h" #define ARGO_RING_MAGIC 0xbd67e163e7777f2fULL#define ARGO_RING_DATA_MAGIC 0xcce4d30fbc82e92aULL
@@ -153,6 +154,18 @@ struct argo_ring_message_header#define ARGO_SIGNAL_METHOD_EVTCHN 1#define ARGO_SIGNAL_METHOD_VIRQ 2 +typedef struct argo_get_config +{ +uint32_t signal_method; +union + { +evtchn_port_t evtchn; +uint32_t virq; + } signal; +uint32_t reserved; +}argo_get_config_t ; +DEFINE_XEN_GUEST_HANDLE (argo_get_config_t); +/* * Hypercall operations */
@@ -244,6 +257,21 @@ struct argo_ring_message_header*/ #define ARGO_MESSAGE_OP_notify 4 +/* +* ARGO_MESSAGE_OP_get_config +* +* Queries Xen for argo configuration values. +* +* Used by a guest to obtain the signal method in use for Argo notifications +* and the event channel port or isa irq in use. +* +* arg1: XEN_GUEST_HANDLE(argo_get_config_t) +* arg2: NULL +* arg3: 0 (ZERO) +* arg4: 0 (ZERO) +*/ +#define ARGO_MESSAGE_OP_get_config 6 +/* The maximum size of a guest message that may be sent on an Argo ring. */ #define ARGO_MAX_MSG_SIZE ((ARGO_MAX_RING_SIZE) - \ (sizeof(struct argo_ring_message_header)) - \
Signature · 7 lines
2.1.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 25/25] argo: implement the get_config op to query notification config
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 01.12.18 at 02:33, <[email protected]> wrote:
xen/common/argo.c +27
-0@@ -1656,6 +1656,46 @@ argo_sendv(struct domain *src_d, const argo_addr_t *src_addr,return ( ret <0 ) ? ret : len; } +static void +argo_get_config (struct domain *d,argo_get_config_t *get_config) +{ +unsigned int method =argo_signal_method (d); + + get_config->signal_method = method; + +switch ( method ) + { +case ARGO_SIGNAL_METHOD_EVTCHN : + { +read_lock (&argo_lock); +read_lock (&d->argo->lock); + + get_config->signal.evtchn = d->argo->evtchn_port; + +read_unlock (&d->argo->lock); +read_unlock (&argo_lock); + +argo_dprintk ("signal for dom:%d evtchn %u\n" , d->domain_id, + get_config->signal.evtchn); + +break ; + } +case ARGO_SIGNAL_METHOD_VIRQ : + { + get_config->signal.virq =VIRQ_ARGO ;
xen/common/argo.c argo_sendv(struct domain *src_d, const argo_addr_t *src_addr, · line 1695
Quoted 10 lines
xen/common/argo.c +10
-0+ +argo_dprintk ("signal for dom:%d virq %u\n" , d->domain_id, + get_config->signal.virq); +break ; + } +default : + { +BUG (); +break ; + }
There are quite a few stray braces here.
xen/common/argo.c argo_sendv(struct domain *src_d, const argo_addr_t *src_addr,
Quoted 9 lines
xen/common/argo.c +9
-0+typedef struct argo_get_config +{ + uint32_t signal_method; + union + { + evtchn_port_t evtchn; + uint32_t virq; + } signal; + uint32_t reserved;
Judging from the description, did you perhaps mean to put uint32_t reserved[2] inside the union?
Then again "get_config" sounds much more generic than just obtaining the notification method.
xen/common/argo.c struct argo_ring_message_header · line 266
Quoted 11 lines
xen/common/argo.c +7
-0@@ -244,6 +257,21 @@ struct argo_ring_message_header*/ #define ARGO_MESSAGE_OP_notify 4 +/* +* ARGO_MESSAGE_OP_get_config +* +* Queries Xen for argo configuration values. +* +* Used by a guest to obtain the signal method in use for Argo notifications +* and the event channel port or isa irq in use.
ISA IRQ? It's a vIRQ that you have as alternative to bare event-channel signaling.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 10 lines
== Future items
The Linux device driver used to test this software is derived from the OpenXT v4v Linux device driver, available at:
https://github.com/OpenXT/v4v
The Argo implementation is not yet ready to publish (focus has been on the hypervisor code to this point). A Linux device driver suitable for inclusion in Xen will be submitted for a future Xen release and incorporation into OpenXT.
Hey Christopher, I am glad you are tackling this. While the Linux driver is not ready to publish, is there a version you can share for someone who wants to test this series? Or is the v4v driver compatible as-is?
Cheers,
-Chris
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 12 lines · Chris Patterson
Quoted 11 lines
== Future items
The Linux device driver used to test this software is derived from the OpenXT v4v Linux device driver, available at:
https://github.com/OpenXT/v4v
The Argo implementation is not yet ready to publish (focus has been on the hypervisor code to this point). A Linux device driver suitable for inclusion in Xen will be submitted for a future Xen release and incorporation into OpenXT.
Quoted 4 lines
Hey Christopher, I am glad you are tackling this. While the Linux driver is not ready to publish, is there a version you can share for someone who wants to test this series? Or is the v4v driver compatible as-is?
Hi Chris,
Thanks for the interest -- so: ok, for you to take a look and to enable testing by anyone who would like to: I've just pushed a copy of the Argo ported Linux driver and userspace interposer, etc., with some OpenEmbedded build integration and instructions, to my github account here:
https://github.com/dozylynx/meta-argo-linux
This a pretty fast port of the v4v Linux software to use the argo interfaces -- the existing OpenXT v4v interface is not quite the same -- plus metadata in there to turn it into a new OpenEmbedded layer in the same repo with recipes to work with meta-virtualization. I've been building with the rocko release, just to pick a stable reference point, so it's the rocko branch in meta-argo-linux that you'll want to look at, and there are instructions in the README.md in that branch.
If you build that per the instructions, just a heads up that the Xen recipe in there will pull from a recent snapshot of Xen's staging branch, with the posted Argo series applied, from a copy on my github account.
If you give it a spin, let me know how it goes.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 37 lines · Christopher Clark
Quoted 12 lines · Chris Patterson
Quoted 11 lines
== Future items
The Linux device driver used to test this software is derived from the OpenXT v4v Linux device driver, available at:
https://github.com/OpenXT/v4v
The Argo implementation is not yet ready to publish (focus has been on the hypervisor code to this point). A Linux device driver suitable for inclusion in Xen will be submitted for a future Xen release and incorporation into OpenXT.Quoted 4 lines
Hey Christopher, I am glad you are tackling this. While the Linux driver is not ready to publish, is there a version you can share for someone who wants to test this series? Or is the v4v driver compatible as-is?
Hi Chris,
Thanks for the interest -- so: ok, for you to take a look and to enable testing by anyone who would like to: I've just pushed a copy of the Argo ported Linux driver and userspace interposer, etc., with some OpenEmbedded build integration and instructions, to my github account here:
https://github.com/dozylynx/meta-argo-linux
This a pretty fast port of the v4v Linux software to use the argo interfaces -- the existing OpenXT v4v interface is not quite the same -- plus metadata in there to turn it into a new OpenEmbedded layer in the same repo with recipes to work with meta-virtualization. I've been building with the rocko release, just to pick a stable reference point, so it's the rocko branch in meta-argo-linux that you'll want to look at, and there are instructions in the README.md in that branch.
Quoted 7 lines
If you build that per the instructions, just a heads up that the Xen recipe in there will pull from a recent snapshot of Xen's staging branch, with the posted Argo series applied, from a copy on my github account.
If you give it a spin, let me know how it goes.
Thank you! Will do. :D
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel