Argo: hypervisor-mediated interdomain communication
v3 cover Argo: hypervisor-mediated interdomain communication
Version three of this patch series:
* Teardown of rings and pending notifications is implemented for domain destroy, removing need to avoid state for defunct domains. Data structures added to track rings that a domain is the single partner for and pending notifications about wildcard-sender rings.
* Register and unregister ops take dedicated argument structs rather than a handle to the ring struct in ring memory as a simpler interface for this upstreaming effort. Ring data structure now has fewer member fields. Interface may need revision later with development of support for communication in L0/L1 nested hypervisor configuration.
* Added constraints to the notify op: limits number of pending notifications on a ring to a simple threshold value. Validates the space query to ensure within achievable bounds.
* Disallows resize of existing rings via re-registration. Could be added later; needs work to handle pending notifications where resized ring would make space availability unachievable.
* Reordered series: XSM patches after main implementation.
* Improved hypercall arg validation; using faster __copy ops where ok.
* Guest memory region validation via fixed constant value fields removed.
Christopher Clark (15):
argo: Introduce the Kconfig option to govern inclusion of Argo
argo: introduce the argo_op hypercall boilerplate
argo: define argo_dprintk for subsystem debugging
argo: init, destroy and soft-reset, with enable command line opt
errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI xen/arm: introduce guest_handle_for_field()
argo: implement the register op
argo: implement the unregister op
argo: implement the sendv op; evtchn: expose send_guest_global_virq
argo: implement the notify op
xsm, argo: XSM control for argo register
xsm, argo: XSM control for argo message send operation xsm, argo: XSM control for any access to argo by a domain xsm, argo: notify: don't describe rings that cannot be sent to
argo: validate hypercall arg structures via compat machinery
docs/misc/xen-command-line.pandoc | 26 + xen/arch/x86/guest/hypercall_page.S | 2 +- xen/arch/x86/hvm/hypercall.c | 3 + xen/arch/x86/hypercall.c | 3 + xen/arch/x86/pv/hypercall.c | 3 + xen/common/Kconfig | 19 + xen/common/Makefile | 3 +- xen/common/argo.c | 2214 +++++++++++++++++++++++++++++++++ xen/common/compat/argo.c | 61 + xen/common/domain.c | 20 + xen/common/event_channel.c | 2 +- xen/include/Makefile | 1 + xen/include/asm-arm/guest_access.h | 5 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 277 +++++ xen/include/public/errno.h | 2 + xen/include/public/xen.h | 4 +- xen/include/xen/argo.h | 23 + xen/include/xen/event.h | 7 + xen/include/xen/hypercall.h | 9 + xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 8 + xen/include/xsm/dummy.h | 26 + xen/include/xsm/xsm.h | 31 + xen/xsm/dummy.c | 6 + xen/xsm/flask/hooks.c | 41 +- xen/xsm/flask/policy/access_vectors | 16 + xen/xsm/flask/policy/security_classes | 1 + 28 files changed, 2813 insertions(+), 8 deletions(-) create mode 100644 xen/common/argo.c create mode 100644 xen/common/compat/argo.c create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 1/15 argo: Introduce the Kconfig option to govern inclusion of Argo
Defines CONFIG_ARGO when enabled. Default: disabled.
When the Kconfig option is enabled, the Argo hypercall implementation will be included, allowing use of the hypervisor-mediated interdomain communication mechanism.
Argo is implemented for x86 and ARM hardware platforms.
Availability of the option depends on EXPERT and Argo is currently an experimental feature.
---
v2 #01 feedback, Jan: replace def_bool/prompt with bool
v1 #02 feedback, Jan: default Kconfig off, use EXPERT, fix whitespace
xen/common/Kconfig | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
xen/common/Kconfig +19 -0
@@ -200,6 +200,25 @@ config LATE_HWDOM If unsure, say N. +config ARGO + bool "Argo: hypervisor-mediated interdomain communication" if EXPERT = "y" + ---help--- + Enables a hypercall for domains to ask the hypervisor to perform + data transfer of messages between domains. + + This allows communication channels to be established that do not + require any shared memory between domains; the hypervisor is the + entity that each domain interacts with. The hypervisor is able to + enforce Mandatory Access Control policy over the communication. + + If XSM_FLASK is enabled, XSM policy can govern which domains may + communicate via the Argo system. + + This feature does nothing if the "argo" boot parameter is not present. + Argo is disabled at runtime by default. + + If unsure, say N. + menu "Schedulers" visible if EXPERT = "y"
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 01/15] argo: Introduce the Kconfig option to govern inclusion of Argo
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
Defines
CONFIG_ARGOwhen enabled. Default: disabled.When the
Kconfigoption is enabled, the Argo hypercall implementation will be included, allowing use of the hypervisor-mediated interdomain communication mechanism.
Quoted 6 lines
Argo is implemented for x86 and ARM hardware platforms.
Availability of the option depends on EXPERT and Argo is currently an experimental feature.
Signed-off-by Christopher Clark <[email protected]>
but only for committing together with at least one patch actually using the symbol.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 2/15 argo: introduce the argo_op hypercall boilerplate
In reply to Christopher ClarkPresence is gated upon CONFIG_ARGO.
Registers the hypercall previously reserved for this. Takes 5 arguments, does nothing and returns -ENOSYS.
Will be avoiding a compat ABI by using fixed-size types in hypercall ops so HYPERCALL, rather than COMPAT_CALL, is the correct macro for the hypercall tables.
Even though handles will be used for (up to) two of the arguments to the hypercall, there will be no need for any XLAT_* translation functions because the referenced data structures have been constructed to be exactly the same size and bit pattern on both 32-bit and 64-bit guests, and padded to be integer multiples of 32 bits in size. This means that the same copy_to_guest and copy_from_guest logic can be relied upon to perform as required without any further intervention. Testing communication with 32 and 64 bit guests has confirmed this works as intended.
---
v2 Copyright line: add 2019
v2 feedback #3 Jan: drop "message" from argo_message_op v2 feedback #3 Jan: add Acked-by
v1 feedback #15 Jan: handle upper-halves of hypercall args v1 feedback #15 Jan: use unsigned where negative values impossible
xen/arch/x86/guest/hypercall_page.S | 2 +- xen/arch/x86/hvm/hypercall.c | 3 +++ xen/arch/x86/hypercall.c | 3 +++ xen/arch/x86/pv/hypercall.c | 3 +++ xen/common/Makefile | 1 + xen/common/argo.c | 28 ++++++++++++++++++++++++++++ xen/include/public/xen.h | 2 +- xen/include/xen/hypercall.h | 9 +++++++++ 8 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 xen/common/argo.c
xen/arch/x86/guest/hypercall_page.S +1 -1
@@ -59,7 +59,7 @@ DECLARE_HYPERCALL(sysctl) DECLARE_HYPERCALL(domctl) DECLARE_HYPERCALL(kexec_op) DECLARE_HYPERCALL(tmem_op) -DECLARE_HYPERCALL(xc_reserved_op) +DECLARE_HYPERCALL(argo_op) DECLARE_HYPERCALL(xenpmu_op) DECLARE_HYPERCALL(arch_0)
xen/arch/x86/hvm/hypercall.c +3 -0
@@ -134,6 +134,9 @@ static const hypercall_table_t hvm_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO + HYPERCALL(argo_op), +#endif COMPAT_CALL(platform_op),#ifdef CONFIG_PV COMPAT_CALL(mmuext_op),
xen/arch/x86/hypercall.c +3 -0
@@ -64,6 +64,9 @@ const hypercall_args_t hypercall_args_table[NR_hypercalls] =ARGS (domctl,1 ),ARGS (kexec_op,2 ),ARGS (tmem_op,1 ), +#ifdef CONFIG_ARGO +ARGS (argo_op,5 ), +#endifARGS (xenpmu_op,2 ), #ifdef CONFIG_HVM ARGS(hvm_op, 2),
xen/arch/x86/pv/hypercall.c +3 -0
@@ -77,6 +77,9 @@ const hypercall_table_t pv_hypercall_table[] = {#ifdef CONFIG_TMEM HYPERCALL(tmem_op),#endif +#ifdef CONFIG_ARGO + HYPERCALL(argo_op), +#endif HYPERCALL(xenpmu_op),#ifdef CONFIG_HVM HYPERCALL(hvm_op),
xen/common/Makefile +1 -0
@@ -1,3 +1,4 @@ +obj-$(CONFIG_ARGO) += argo.o obj-y += bitmap.o obj-y += bsearch.o obj-$(CONFIG_CORE_PARKING) += core_parking.o
xen/common/argo.c new file +28 -0
@@ -0,0 +1,28 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018-2019 BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#include <xen/errno.h> +#include <xen/guest_access.h> + +long +do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3, +unsigned long arg4) +{ +return -ENOSYS ; +}
xen/include/public/xen.h +1 -1
@@ -118,7 +118,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);#define __HYPERVISOR_domctl 36#define __HYPERVISOR_kexec_op 37#define __HYPERVISOR_tmem_op 38 -#define __HYPERVISOR_xc_reserved_op 39/* reserved for XenClient */ +#define __HYPERVISOR_argo_op 39#define __HYPERVISOR_xenpmu_op 40#define __HYPERVISOR_dm_op 41
xen/include/xen/hypercall.h +9 -0
@@ -136,6 +136,15 @@ do_tmem_op( XEN_GUEST_HANDLE_PARAM(tmem_op_t) uops);#endif +#ifdef CONFIG_ARGO +extern long do_argo_op ( +unsigned int cmd, +XEN_GUEST_HANDLE_PARAM (void ) arg1, +XEN_GUEST_HANDLE_PARAM (void ) arg2, +unsigned long arg3, +unsigned long arg4); +#endif +extern long do_xenoprof_op (int op,XEN_GUEST_HANDLE_PARAM (void ) arg);
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 3/15 argo: define argo_dprintk for subsystem debugging
In reply to Christopher ClarkA convenience for working on development of the argo subsystem:
setting a #define variable enables additional debug messages.
---
v2 #03 feedback, Jan: fix ifdef/define confusion error v1 #04 feedback, Jan: fix dprintk implementation
xen/common/argo.c | 9 +++++++++ 1 file changed, 9 insertions(+)
xen/common/argo.c +9 -0
@@ -19,6 +19,15 @@#include <xen/errno.h> #include <xen/guest_access.h> +/* Change this to #define ARGO_DEBUG here to enable more debug messages */ +#undef ARGO_DEBUG + +#ifdef ARGO_DEBUG +#define argo_dprintk(format, args...) printk("argo: " format, ## args ) +#else +#define argo_dprintk (format, ... ) ((void)0) +#endif + long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2, unsigned long arg3,
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 03/15] argo: define argo_dprintk for subsystem debugging
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
A convenience for working on development of the argo subsystem:
setting a #define variable enables additional debug messages.Signed-off-by Christopher Clark <[email protected]>
with one further remark:
xen/common/argo.c line 28
Quoted 13 lines
xen/common/argo.c +7
-0@@ -19,6 +19,15 @@#include <xen/errno.h> #include <xen/guest_access.h> +/* Change this to #define ARGO_DEBUG here to enable more debug messages */ +#undef ARGO_DEBUG + +#ifdef ARGO_DEBUG +#define argo_dprintk(format, args...) printk("argo: " format, ## args ) +#else +#define argo_dprintk (format, ... ) ((void)0)
This would better be an inline function, such that arguments passed in actually get evaluated. Otherwise you risk overlooking variables used for such logging only, and in particular the compiler then issuing warnings (breaking the build due to -Werror).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 03/15] argo: define argo_dprintk for subsystem debugging
In reply to Christopher ClarkQuoted 5 lines · Christopher Clark
A convenience for working on development of the argo subsystem:
setting a #define variable enables additional debug messages.Signed-off-by Christopher Clark <[email protected]>
Quoted 12 lines
---
v2 #03 feedback, Jan: fix ifdef/define confusion error v1 #04 feedback, Jan: fix dprintk implementationxen/common/argo.c | 9 +++++++++ 1 file changed, 9 insertions(+)xen/common/argo.c +0
-0
xen/common/argo.c line 26
Quoted 8 lines
xen/common/argo.c +5
-0@@ -19,6 +19,15 @@#include <xen/errno.h> #include <xen/guest_access.h> +/* Change this to #define ARGO_DEBUG here to enable more debug messages */ +#undef ARGO_DEBUG + +#ifdef ARGO_DEBUG +#define argo_dprintk(format, args...) printk("argo: " format, ## args )
I would maybe consider prefixing this with XENLOG_DEBUG, but since it's a hidden compile-time debug I'm not sure it's relevant to set the log level.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 4/15 argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkInitialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on CONFIG_ARGO.
Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures for domain_create.
argo_destroy: teardown for domain_destroy and the error exit
path of domain_create.
argo_soft_reset: reset of domain state for domain_soft_reset.
Adds two new fields to struct domain:
rwlock_t argo_lock;
struct argo_domain *argo;In accordance with recent work on _domain_destroy, argo_destroy is idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.
A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.html
The following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).
---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: use ring_info backpointer in pending_ent to maintain npending v2 self: rename all_rings_remove_info to domain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2 cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: drop pending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' from do_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Use mfn_eq for comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: drop opt_argo_enabled initializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include from sched.h
v1. self, Jan: drop inclusion of public argo.h in private one
v1. self, Jan: add include of public argo.h to argo.c
v1. self, Jan: drop fwd decl of argo_domain in priv header
v1. Paul/self/Jan: add data structures to xlat.lst and compat/argo.h to Makefile v1. self: removed allocation of event channel since switching to VIRQ v1. self: drop types.h include from private argo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: add domain_cookie to ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: use ASSERT_UNREACHABLE
docs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.h
docs/misc/xen-command-line.pandoc +11 -0
@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware. +### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. + ### asid (x86) > `= <boolean>`
xen/common/argo.c +460 -1
477 lines not on this page · raw message
xen/common/domain.c +20 -0
@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h>
@@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif +rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);
@@ -376,6 +381,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;
@@ -445,6 +453,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );
@@ -717,6 +730,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);
@@ -1175,6 +1191,10 @@ int domain_soft_reset(struct domain *d)grant_table_warn_active_grants (d); +#ifdef CONFIG_ARGO +argo_soft_reset (d); +#endif + for_each_vcpu ( d, v ) {set_xen_guest_handle (runstate_guest (v),NULL );
xen/include/Makefile +1 -0
@@ -5,6 +5,7 @@ ifneq ($(CONFIG_COMPAT),) compat-arch-$(CONFIG_X86) := x86_32 headers-y := \ + compat/argo.h \ compat/callback.h \ compat/elfnote.h \ compat/event_channel.h \
xen/include/public/argo.h new file +59 -0
59 lines not on this page · raw message
xen/include/xen/argo.h new file +23 -0
@@ -0,0 +1,23 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); +void argo_soft_reset (struct domain *d); + +#endif
xen/include/xen/sched.h +6 -0
@@ -490,6 +490,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_ARGO +/* Argo interdomain communication support */ +rwlock_t argo_lock; +struct argo_domain *argo; +#endif };/* Protect updates/reads (resp.) of domain_list and domain_hash. */
xen/include/xlat.lst +2 -0
@@ -148,3 +148,5 @@ ? flask_setenforce xsm/flask_op.h ! flask_sid_context xsm/flask_op.h ? flask_transition xsm/flask_op.h +? argo_addr argo.h +? argo_ring argo.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 36 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.html
Quoted 8 lines
The following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>
Other than an indentation issue in domain_rings_remove_all, this LGTM.
Quoted 759 lines
---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: usering_infobackpointer inpending_entto maintain npending v2 self: renameall_rings_remove_infotodomain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: droppending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' fromdo_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Usemfn_eqfor comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: dropopt_argo_enabledinitializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include fromsched.h
v1. self, Jan: drop inclusion of publicargo.hin private one
v1. self, Jan: add include of publicargo.htoargo.c
v1. self, Jan: drop fwd decl ofargo_domainin priv header
v1. Paul/self/Jan: add data structures to xlat.lst andcompat/argo.htoMakefilev1. self: removed allocation of event channel since switching to VIRQ v1. self: droptypes.hinclude from privateargo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: adddomain_cookieto ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: useASSERT_UNREACHABLEdocs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hdocs/misc/xen-command-line.pandoc +11
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware. +### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. + ### asid (x86) > `= <boolean>`xen/common/argo.c +460
-1@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> + +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +/* number of pending entries queued for this ring, protected by L3 */ +uint32_t npending; +}; + +/* Data about a single-sender ring, held by the sender (partner) domain */ +struct argo_send_info +{ +/* next node in the hash, protected by Lsend */ +struct hlist_node node; +/* this ring's id, protected by Lsend */ +struct argo_ring_id id; +}; + +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ +struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. +*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad; +/* minimum ring space available that this signal is waiting upon */ +uint32_t len; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1) +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend. +* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/ /* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id); +if ( !d ) +return ; + +if ( d->argo ) + { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + } +put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node); + ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} + +static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +pending_remove_all (ring_info); +hlist_del (&ring_info->node); +ring_remove_mfns (d, ring_info); +xfree (ring_info); +} + +static void +domain_rings_remove_all (struct domain *d) +{ +unsigned int i; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) +ring_remove_info (d, ring_info); + } + d->argo->ring_count =0 ; +} + +/* +* Tear down all rings of other domains where src_d domain is the partner. +* (ie. it is the single domain that can send to those rings.) +* This will also cancel any pending notifications about those rings. +*/ +static void +partner_rings_remove (struct domain *src_d) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&argo_lock)); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_send_info *send_info; + +hlist_for_each_entry_safe (send_info, node, next, + &src_d->argo->send_hash[i], node) + { +struct argo_ring_info *ring_info; +struct domain *dst_d; + + dst_d =get_domain_by_id (send_info->id.domain_id); +if ( dst_d ) + { + ring_info =ring_find_info (dst_d, &send_info->id); +if ( ring_info ) + { +ring_remove_info (dst_d, ring_info); + dst_d->argo->ring_count--; + } + +put_domain (dst_d); + } + +hlist_del (&send_info->node); +xfree (send_info); + } + } +} +long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ; + } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +20
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> @@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif +rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);@@ -376,6 +381,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;@@ -445,6 +453,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );@@ -717,6 +730,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);@@ -1175,6 +1191,10 @@ int domain_soft_reset(struct domain *d)grant_table_warn_active_grants (d); +#ifdef CONFIG_ARGO +argo_soft_reset (d); +#endif + for_each_vcpu ( d, v ) {set_xen_guest_handle (runstate_guest (v),NULL );xen/include/Makefile +1
-0@@ -5,6 +5,7 @@ ifneq ($(CONFIG_COMPAT),) compat-arch-$(CONFIG_X86) := x86_32 headers-y := \ + compat/argo.h \ compat/callback.h \ compat/elfnote.h \ compat/event_channel.h \xen/include/public/argo.h new file +59
-0@@ -0,0 +1,59 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018-2019, BAE Systems +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +* +*/ + +#ifndef __XEN_PUBLIC_ARGO_H__ +#define __XEN_PUBLIC_ARGO_H__ + +#include "xen.h" + +typedef struct xen_argo_addr +{ +uint32_t port; +domid_t domain_id; +uint16_t pad; +}xen_argo_addr_t ; + +typedef struct xen_argo_ring +{ +/* Guests should use atomic operations to access rx_ptr */ +uint32_t rx_ptr; +/* Guests should use atomic operations to access tx_ptr */ +uint32_t tx_ptr; +/* +* Header space reserved for later use. Align the start of the ring to a +* multiple of the message slot size. +*/ +uint8_t reserved[56 ]; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t ring[]; +#elif defined(__GNUC__) +uint8_t ring[0 ]; +#endif +}xen_argo_ring_t ; + +#endif xen/include/xen/argo.h new file +23
-0@@ -0,0 +1,23 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); +void argo_soft_reset (struct domain *d); + +#endif xen/include/xen/sched.h +6
-0@@ -490,6 +490,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_ARGO +/* Argo interdomain communication support */ +rwlock_t argo_lock; +struct argo_domain *argo; +#endif };/* Protect updates/reads (resp.) of domain_list and domain_hash. */ xen/include/xlat.lst +0
-0
xen/include/xlat.lst line 152
Quoted 5 lines
xen/include/xlat.lst +2
-0@@ -148,3 +148,5 @@ ? flask_setenforce xsm/flask_op.h ! flask_sid_context xsm/flask_op.h ? flask_transition xsm/flask_op.h +? argo_addr argo.h +? argo_ring argo.h
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 44 lines · Ross Philipson
Quoted 42 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.htmlThe following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).
Quoted 6 lines
Quoted 2 lines
Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>Other than an indentation issue in
domain_rings_remove_all, this LGTM.Reviewed-by Ross Philipson <[email protected]>
Thanks for the review, Ross.
I don't think there is an indentation issue there -- it's just the hlist_for_each_entry_safe macro, which is operating on ring_remove_info, so it should be ok.
thanks,
Christopher
Quoted 753 lines
Quoted 753 lines
---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: usering_infobackpointer inpending_entto maintain npending v2 self: renameall_rings_remove_infotodomain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: droppending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' fromdo_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Usemfn_eqfor comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: dropopt_argo_enabledinitializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include fromsched.h
v1. self, Jan: drop inclusion of publicargo.hin private one
v1. self, Jan: add include of publicargo.htoargo.c
v1. self, Jan: drop fwd decl ofargo_domainin priv header
v1. Paul/self/Jan: add data structures to xlat.lst andcompat/argo.htoMakefilev1. self: removed allocation of event channel since switching to VIRQ v1. self: droptypes.hinclude from privateargo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: adddomain_cookieto ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: useASSERT_UNREACHABLEdocs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hdocs/misc/xen-command-line.pandoc +11
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware. +### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. + ### asid (x86) > `= <boolean>`xen/common/argo.c +460
-1@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> + +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +/* number of pending entries queued for this ring, protected by L3 */ +uint32_t npending; +}; + +/* Data about a single-sender ring, held by the sender (partner) domain */ +struct argo_send_info +{ +/* next node in the hash, protected by Lsend */ +struct hlist_node node; +/* this ring's id, protected by Lsend */ +struct argo_ring_id id; +}; + +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ +struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. +*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad; +/* minimum ring space available that this signal is waiting upon */ +uint32_t len; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1) +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend. +* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/ /* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id); +if ( !d ) +return ; + +if ( d->argo ) + { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + } +put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node); + ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} + +static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +pending_remove_all (ring_info); +hlist_del (&ring_info->node); +ring_remove_mfns (d, ring_info); +xfree (ring_info); +} + +static void +domain_rings_remove_all (struct domain *d) +{ +unsigned int i; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) +ring_remove_info (d, ring_info); + } + d->argo->ring_count =0 ; +} + +/* +* Tear down all rings of other domains where src_d domain is the partner. +* (ie. it is the single domain that can send to those rings.) +* This will also cancel any pending notifications about those rings. +*/ +static void +partner_rings_remove (struct domain *src_d) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&argo_lock)); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_send_info *send_info; + +hlist_for_each_entry_safe (send_info, node, next, + &src_d->argo->send_hash[i], node) + { +struct argo_ring_info *ring_info; +struct domain *dst_d; + + dst_d =get_domain_by_id (send_info->id.domain_id); +if ( dst_d ) + { + ring_info =ring_find_info (dst_d, &send_info->id); +if ( ring_info ) + { +ring_remove_info (dst_d, ring_info); + dst_d->argo->ring_count--; + } + +put_domain (dst_d); + } + +hlist_del (&send_info->node); +xfree (send_info); + } + } +} +long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ; + } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +20
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> @@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif +rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);@@ -376,6 +381,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;@@ -445,6 +453,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );@@ -717,6 +730,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);@@ -1175,6 +1191,10 @@ int domain_soft_reset(struct domain *d)grant_table_warn_active_grants (d); +#ifdef CONFIG_ARGO +argo_soft_reset (d); +#endif + for_each_vcpu ( d, v ) {set_xen_guest_handle (runstate_guest (v),NULL );xen/include/Makefile +1
-0@@ -5,6 +5,7 @@ ifneq ($(CONFIG_COMPAT),) compat-arch-$(CONFIG_X86) := x86_32 headers-y := \ + compat/argo.h \ compat/callback.h \ compat/elfnote.h \ compat/event_channel.h \xen/include/public/argo.h new file +59
-0@@ -0,0 +1,59 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Derived from v4v, the version 2 of v2v. +* +* Copyright (c) 2010, Citrix Systems +* Copyright (c) 2018-2019, BAE Systems +* +* Permission is hereby granted, free of charge, to any person obtaining a copy +* of this software and associated documentation files (the "Software"), to +* deal in the Software without restriction, including without limitation the +* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +* sell copies of the Software, and to permit persons to whom the Software is +* furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +* DEALINGS IN THE SOFTWARE. +* +*/ + +#ifndef __XEN_PUBLIC_ARGO_H__ +#define __XEN_PUBLIC_ARGO_H__ + +#include "xen.h" + +typedef struct xen_argo_addr +{ +uint32_t port; +domid_t domain_id; +uint16_t pad; +}xen_argo_addr_t ; + +typedef struct xen_argo_ring +{ +/* Guests should use atomic operations to access rx_ptr */ +uint32_t rx_ptr; +/* Guests should use atomic operations to access tx_ptr */ +uint32_t tx_ptr; +/* +* Header space reserved for later use. Align the start of the ring to a +* multiple of the message slot size. +*/ +uint8_t reserved[56 ]; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t ring[]; +#elif defined(__GNUC__) +uint8_t ring[0 ]; +#endif +}xen_argo_ring_t ; + +#endif xen/include/xen/argo.h new file +23
-0@@ -0,0 +1,23 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); +void argo_soft_reset (struct domain *d); + +#endif xen/include/xen/sched.h +6
-0@@ -490,6 +490,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_ARGO +/* Argo interdomain communication support */ +rwlock_t argo_lock; +struct argo_domain *argo; +#endif };
Quoted 13 lines
xen/include/xlat.lst line 152
Quoted 12 lines
xen/include/xen/sched.h +0
-0/* Protect updates/reads (resp.) of domain_list and domain_hash. */ xen/include/xlat.lst +2
-0@@ -148,3 +148,5 @@ ? flask_setenforce xsm/flask_op.h ! flask_sid_context xsm/flask_op.h ? flask_transition xsm/flask_op.h +? argo_addr argo.h +? argo_ring argo.h
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkQuoted 113 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.htmlThe following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: usering_infobackpointer inpending_entto maintain npending v2 self: renameall_rings_remove_infotodomain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: droppending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' fromdo_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Usemfn_eqfor comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: dropopt_argo_enabledinitializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include fromsched.h
v1. self, Jan: drop inclusion of publicargo.hin private one
v1. self, Jan: add include of publicargo.htoargo.c
v1. self, Jan: drop fwd decl ofargo_domainin priv header
v1. Paul/self/Jan: add data structures to xlat.lst andcompat/argo.htoMakefilev1. self: removed allocation of event channel since switching to VIRQ v1. self: droptypes.hinclude from privateargo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: adddomain_cookieto ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: useASSERT_UNREACHABLEdocs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hdocs/misc/xen-command-line.pandoc +1
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware. +### argo
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 195
Quoted 10 lines
docs/misc/xen-command-line.pandoc +10
-0+> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. +
Do we want to say it's only available when Xen is compiled with CONFIG_ARGO?
Quoted 76 lines
docs/misc/xen-command-line.pandoc +0
-0### asid (x86) > `= <boolean>`xen/common/argo.c +64
-0@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> + +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +/* number of pending entries queued for this ring, protected by L3 */ +uint32_t npending; +}; + +/* Data about a single-sender ring, held by the sender (partner) domain */ +struct argo_send_info +{ +/* next node in the hash, protected by Lsend */ +struct hlist_node node; +/* this ring's id, protected by Lsend */ +struct argo_ring_id id; +}; + +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/*
xen/common/argo.c line 94
Quoted 10 lines
xen/common/argo.c +10
-0+* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ +struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this.
It would be clearer if this stated the correct locks.
xen/common/argo.c line 99
Quoted 5 lines
xen/common/argo.c +5
-0+*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;
Can we order domain_id after len and drop the pad?
Quoted 159 lines
xen/common/argo.c +152
-0+/* minimum ring space available that this signal is waiting upon */ +uint32_t len; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1) +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend. +* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/ /* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id); +if ( !d ) +return ; + +if ( d->argo ) + { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + } +put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); +
xen/common/argo.c line 271
Quoted 10 lines
xen/common/argo.c +10
-0+hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node); + ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} +
Maybe move ring_unmap() here so it's closer to where it is used?
Quoted 206 lines
xen/common/argo.c +200
-1+static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +pending_remove_all (ring_info); +hlist_del (&ring_info->node); +ring_remove_mfns (d, ring_info); +xfree (ring_info); +} + +static void +domain_rings_remove_all (struct domain *d) +{ +unsigned int i; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) +ring_remove_info (d, ring_info); + } + d->argo->ring_count =0 ; +} + +/* +* Tear down all rings of other domains where src_d domain is the partner. +* (ie. it is the single domain that can send to those rings.) +* This will also cancel any pending notifications about those rings. +*/ +static void +partner_rings_remove (struct domain *src_d) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&argo_lock)); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_send_info *send_info; + +hlist_for_each_entry_safe (send_info, node, next, + &src_d->argo->send_hash[i], node) + { +struct argo_ring_info *ring_info; +struct domain *dst_d; + + dst_d =get_domain_by_id (send_info->id.domain_id); +if ( dst_d ) + { + ring_info =ring_find_info (dst_d, &send_info->id); +if ( ring_info ) + { +ring_remove_info (dst_d, ring_info); + dst_d->argo->ring_count--; + } + +put_domain (dst_d); + } + +hlist_del (&send_info->node); +xfree (send_info); + } + } +} +long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock);
xen/common/argo.c line 486
Quoted 10 lines
xen/common/argo.c +10
-0+ +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled )
Shouldn't this function just exit early if argo is disabled?
xen/common/argo.c line 496
Quoted 10 lines
xen/common/argo.c +9
-0+ { +xfree (d->argo); + d->argo =NULL ; + } + else + argo_domain_init(d->argo); + } + +write_unlock (&argo_lock); }
<snip>
Regards,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 115 lines · Jason Andryuk
Quoted 112 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.htmlThe following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: usering_infobackpointer inpending_entto maintain npending v2 self: renameall_rings_remove_infotodomain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: droppending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' fromdo_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Usemfn_eqfor comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: dropopt_argo_enabledinitializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include fromsched.h
v1. self, Jan: drop inclusion of publicargo.hin private one
v1. self, Jan: add include of publicargo.htoargo.c
v1. self, Jan: drop fwd decl ofargo_domainin priv header
v1. Paul/self/Jan: add data structures to xlat.lst andcompat/argo.htoMakefilev1. self: removed allocation of event channel since switching to VIRQ v1. self: droptypes.hinclude from privateargo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: adddomain_cookieto ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: useASSERT_UNREACHABLEdocs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hdocs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.
Quoted 13 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 195
Quoted 11 lines
docs/misc/xen-command-line.pandoc +11
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. +Do we want to say it's only available when Xen is compiled with
CONFIG_ARGO?
It's a fair question and I can respin with that added, if needed. However, there are other boot options that are dependent on KCONFIG options that don't make such a statement (altp2m, efi, ept, flask, hardware_dom, tmem, ...), so I'm not sure if it is required.
Quoted 81 lines
Quoted 81 lines
docs/misc/xen-command-line.pandoc +0
-0### asid (x86) > `= <boolean>`xen/common/argo.c +68
-0@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> + +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage; +/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr; +/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping; +/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +/* number of pending entries queued for this ring, protected by L3 */ +uint32_t npending; +}; + +/* Data about a single-sender ring, held by the sender (partner) domain */ +struct argo_send_info +{ +/* next node in the hash, protected by Lsend */ +struct hlist_node node; +/* this ring's id, protected by Lsend */ +struct argo_ring_id id; +}; + +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/
Quoted 8 lines
xen/common/argo.c line 94
Quoted 6 lines
xen/common/argo.c +6
-0+struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. It would be clearer if this stated the correct locks.
ok - it would mean duplicating the statement about which locks are needed though, since it is explained elsewhere in the file, which means it will need updating in two places if the locking requirements change. That was why I worded it that way, as an indicator to go and find where it is already described, to avoid that.
Quoted 8 lines
xen/common/argo.c line 99
Quoted 6 lines
xen/common/argo.c +5
-0+*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;Can we order
domain_idafter len and drop the pad?
I'm not sure that would be right to do that. I think that the pad ensures that len is word aligned to 32-bit boundary. I was asked to insert a pad field for such a struct like this in an earlier review here:
https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00239.html
Quoted 164 lines
Quoted 164 lines
xen/common/argo.c +156
-0+/* minimum ring space available that this signal is waiting upon */ +uint32_t len; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1) +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ + +/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend. +* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/ /* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + } +} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id); +if ( !d ) +return ; + +if ( d->argo ) + { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + } +put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node) + { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node);
Quoted 8 lines
xen/common/argo.c line 271
Quoted 6 lines
xen/common/argo.c +6
-0+ ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} +Maybe move
ring_unmap()here so it's closer to where it is used?
I'm fine with moving it if it needs it, but it's located where it is in order to put it right next to the corresponding ring_map_page function - the two are paired really, with one doing map_domain_page_global and the other undoing it with unmap_domain_page_global. That's how it ends up when the full series is applied.
Quoted 211 lines
Quoted 211 lines
xen/common/argo.c +204
-1+static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info) +{ +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +pending_remove_all (ring_info); +hlist_del (&ring_info->node); +ring_remove_mfns (d, ring_info); +xfree (ring_info); +} + +static void +domain_rings_remove_all (struct domain *d) +{ +unsigned int i; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) +ring_remove_info (d, ring_info); + } + d->argo->ring_count =0 ; +} + +/* +* Tear down all rings of other domains where src_d domain is the partner. +* (ie. it is the single domain that can send to those rings.) +* This will also cancel any pending notifications about those rings. +*/ +static void +partner_rings_remove (struct domain *src_d) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&argo_lock)); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_send_info *send_info; + +hlist_for_each_entry_safe (send_info, node, next, + &src_d->argo->send_hash[i], node) + { +struct argo_ring_info *ring_info; +struct domain *dst_d; + + dst_d =get_domain_by_id (send_info->id.domain_id); +if ( dst_d ) + { + ring_info =ring_find_info (dst_d, &send_info->id); +if ( ring_info ) + { +ring_remove_info (dst_d, ring_info); + dst_d->argo->ring_count--; + } + +put_domain (dst_d); + } + +hlist_del (&send_info->node); +xfree (send_info); + } + } +} +long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo )
Quoted 8 lines
xen/common/argo.c line 486
Quoted 6 lines
xen/common/argo.c +6
-0+ { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled )Shouldn't this function just exit early if argo is disabled?
There has been support added to Xen with a hypercall to make a subset of boot parameters modifiable at runtime. Argo-enabled isn't currently one of them, but that may be changed later so I did not want to bake into this function the assumption that the enabled/disabled configuration could not change after being initially evaluated at the time the domain was launched. That's possibly a conservative choice though.
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=82cf78468e96de1e4d1400bbf5508f8b111650c3
Thanks for the review.
Christopher
Quoted 12 lines
xen/common/argo.c line 496
Quoted 11 lines
xen/common/argo.c +9
-0+ { +xfree (d->argo); + d->argo =NULL ; + } + else + argo_domain_init(d->argo); + } + +write_unlock (&argo_lock); }
Quoted 4 lines
<snip>
Regards,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 26 lines · Christopher Clark
Quoted 15 lines · Jason Andryuk
Quoted 11 lines · Christopher Clark
xen/common/argo.c +11
-0+ +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ Quoted 8 lines
xen/common/argo.c line 94
Quoted 6 lines
xen/common/argo.c +6
-0+struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. It would be clearer if this stated the correct locks.
Quoted 5 lines
ok - it would mean duplicating the statement about which locks are needed though, since it is explained elsewhere in the file, which means it will need updating in two places if the locking requirements change. That was why I worded it that way, as an indicator to go and find where it is already described, to avoid that.
"Caution" made me think *ring_info points from domain A's pending_ent to domain B's ring_info. Reading patch 10 (notify op) I see that it really just points back to domain A's ring_info. So the "Caution" is just that you still have to lock ring_info (L3) even though you can get to the pointer via L2. Is that correct?
I agree a single location for the locking documentation is better than splitting or duplicating. As long as no cross-domain locking is required, this is fine.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
xen/common/argo.c +5
-0+*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;
Quoted 7 lines
Quoted 1 line
Can we order
domain_idafter len and drop the pad?I'm not sure that would be right to do that. I think that the pad ensures that len is word aligned to 32-bit boundary. I was asked to insert a pad field for such a struct like this in an earlier review here:
https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00239.html
I'll respond to this in the other email from Jan.
Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
xen/common/argo.c +13
-0+ +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node);Quoted 8 lines
xen/common/argo.c line 271
Quoted 6 lines
xen/common/argo.c +6
-0+ ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} +Maybe move
ring_unmap()here so it's closer to where it is used?
Quoted 5 lines
I'm fine with moving it if it needs it, but it's located where it is in order to put it right next to the corresponding
ring_map_pagefunction - the two are paired really, with one doingmap_domain_page_globaland the other undoing it withunmap_domain_page_global. That's how it ends up when the full series is applied.
Okay. My comment came only from reading only this single patch.
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
xen/common/argo.c +8
-0+void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo )Quoted 8 lines
xen/common/argo.c line 486
Quoted 6 lines
xen/common/argo.c +6
-0+ { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled )Shouldn't this function just exit early if argo is disabled?
Quoted 8 lines
There has been support added to Xen with a hypercall to make a subset of boot parameters modifiable at runtime. Argo-enabled isn't currently one of them, but that may be changed later so I did not want to bake into this function the assumption that the enabled/disabled configuration could not change after being initially evaluated at the time the domain was launched. That's possibly a conservative choice though.
http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=82cf78468e96de1e4d1400bbf5508f8b111650c3
Okay. I was not aware of this functionality.
Regards,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 22 lines · Jason Andryuk
Quoted 15 lines · Christopher Clark
Quoted 13 lines · Jason Andryuk
xen/common/argo.c line 94
Quoted 9 lines · Christopher Clark
xen/common/argo.c +9
-0+ +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. It would be clearer if this stated the correct locks.
Quoted 5 lines
ok - it would mean duplicating the statement about which locks are needed though, since it is explained elsewhere in the file, which means it will need updating in two places if the locking requirements change. That was why I worded it that way, as an indicator to go and find where it is already described, to avoid that.
Quoted 5 lines
"Caution" made me think
*ring_infopoints from domain A'spending_entto domain B'sring_info. Reading patch 10 (notify op) I see that it really just points back to domain A'sring_info. So the "Caution" is just that you still have to lockring_info(L3) even though you can get to the pointer via L2. Is that correct?
Yes, exactly.
Quoted 3 lines
I agree a single location for the locking documentation is better than splitting or duplicating. As long as no cross-domain locking is required, this is fine.
OK - thanks.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Jason AndryukQuoted 1 line
Quoted 1 line
Quoted 1 line · On 08.01.19 at 23:54, <[email protected]> wrote:
First of all - please trim your replies.
Quoted 7 lines · Christopher Clark
Quoted 6 lines
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.
Quoted 13 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 195
Quoted 11 lines
docs/misc/xen-command-line.pandoc +11
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. +Do we want to say it's only available when Xen is compiled with
CONFIG_ARGO?
We don't do so elsewhere, so I'm with Christopher.
Quoted 7 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware
Quoted 5 lines
docs/misc/xen-command-line.pandoc +5
-0+ */ + struct argo_ring_info *ring_info; + /* domain to be notified when space is available */ + domid_t domain_id; + uint16_t pad;Can we order
domain_idafter len and drop the pad?
That would still call for a pad field - we prefer to have explicit padding, and also to check it's zero, the latter to allow for assigning meaning to the field down the road.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 4 lines · Jan Beulich
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · On 08.01.19 at 23:54, <[email protected]> wrote:
First of all - please trim your replies.
Sorry. Will do.
Quoted 21 lines
Quoted 7 lines · Christopher Clark
Quoted 6 lines
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.Quoted 13 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 195
Quoted 11 lines
docs/misc/xen-command-line.pandoc +11
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. +Do we want to say it's only available when Xen is compiled with
CONFIG_ARGO?
Quoted 1 line
We don't do so elsewhere, so I'm with Christopher.
Okay.
Quoted 11 lines
Quoted 7 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware
Quoted 5 lines
docs/misc/xen-command-line.pandoc +5
-0+ */ + struct argo_ring_info *ring_info; + /* domain to be notified when space is available */ + domid_t domain_id; + uint16_t pad;Can we order
domain_idafter len and drop the pad?That would still call for a pad field - we prefer to have explicit padding, and also to check it's zero, the latter to allow for assigning meaning to the field down the road.
This struct is internal to Xen and argo, so do we still need explicit padding?
There are other public argo structs with padding fields. I haven't gotten through all the patches, but I think at least some of those are missing zero checks.
Regards,
Jason
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 09.01.19 at 15:26, <[email protected]> wrote:
Quoted 8 lines · Jan Beulich
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 08.01.19 at 23:54, <[email protected]> wrote:
Quoted 5 lines · Christopher Clark
xen/common/argo.c +5
-0+*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Can we order
domain_idafter len and drop the pad?That would still call for a pad field - we prefer to have explicit padding, and also to check it's zero, the latter to allow for assigning meaning to the field down the road.
This struct is internal to Xen and argo, so do we still need explicit padding?
Oh, internal structures don't need any explicit padding. Where the domain_id field gets placed still doesn't matter then, though.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 2 lines
Quoted 2 lines · On 09.01.19 at 15:26, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 08.01.19 at 23:54, <[email protected]> wrote:
xen/common/argo.c line 99
Quoted 5 lines · Christopher Clark
xen/common/argo.c +5
-0+*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;Can we order
domain_idafter len and drop the pad?That would still call for a pad field - we prefer to have explicit padding, and also to check it's zero, the latter to allow for assigning meaning to the field down the road.
Quoted 5 lines
Quoted 2 lines
This struct is internal to Xen and argo, so do we still need explicit padding?
Oh, internal structures don't need any explicit padding. Where the
domain_idfield gets placed still doesn't matter then, though.
ok, I've switched the len field here from uint32_t to unsigned int (part of moving away from fixed-width types where not required, as requested later in this thread) and so while at it have dropped the unneeded pad field too.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkaOn Mon, Jan 7, 2019 at 8:44 AM Christopher Clark
Quoted 132 lines · <[email protected]> wrote:
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.htmlThe following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>---
v2 rewrite locking explanation comment
v2 header copyright line now includes 2019
v2 self: usering_infobackpointer inpending_entto maintain npending v2 self: renameall_rings_remove_infotodomain_rings_remove_all
v2 feedback Jan: drop cookie, implement teardown
v2 self: add npending to track number of pending entries per ring v2 self: amend comment on locking; drop section comments
v2cookie_eq: test low bits first and use likely on high bits
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 self: droppending_remove_ent, inline at single call site
v1 feedback Roger, Jan: drop argo prefix on static functions
v2 #4 Lars: add Acked-by and details to commit message.
v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v2 bugfix: xsm use in soft-reset prior to introduction
v2 feedback #9 Jan: drop 'message' fromdo_argo_message_op
v1 #5 feedback Paul: init/destroy unsigned, brackets and whitespace fixes v1 #5 feedback Paul: Usemfn_eqfor comparing mfns.
v1 #5 feedback Paul: init/destroy : use currd
v1 #6 (#5) feedback Jan: init/destroy: s/ENOSYS/EOPNOTSUPP/
v1 #6 feedback Paul: Folded patch 6 into patch 5.
v1 #6 feedback Jan: dropopt_argo_enabledinitializer
v1 $6 feedback Jan: s/ENOSYS/EOPNOTSUPP/g and drop useless dprintk v1. #5 feedback Paul: change the license on public header to BSD
- ack from Lars at Citrix.
v1. self, Jan: drop unnecessary xen include fromsched.h
v1. self, Jan: drop inclusion of publicargo.hin private one
v1. self, Jan: add include of publicargo.htoargo.c
v1. self, Jan: drop fwd decl ofargo_domainin priv header
v1. Paul/self/Jan: add data structures to xlat.lst andcompat/argo.htoMakefilev1. self: removed allocation of event channel since switching to VIRQ v1. self: droptypes.hinclude from privateargo.h
v1: reorder public argo include position
v1: #13 feedback Jan: public namespace: prefix with xen
v1: self: rename pending ent "id" to "domain_id"
v1: self: adddomain_cookieto ent struct
v1. #15 feedback Jan: make cmd unsigned
v1. #15 feedback Jan: make i loop variable unsigned
v1: self: adjust dprintks in init, destroy
v1: #18 feedback Jan: meld max ring count limit
v1: self: use type not struct in public defn, affects compat gen header
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1: add comment explaining the 'magic' field
v1: self + Jan feedback: implement soft reset
v1: feedback #13 Roger: useASSERT_UNREACHABLEdocs/misc/xen-command-line.pandoc | 11 + xen/common/argo.c | 461 +++++++++++++++++++++++++++++++++++++- xen/common/domain.c | 20 ++ xen/include/Makefile | 1 + xen/include/public/argo.h | 59 +++++ xen/include/xen/argo.h | 23 ++ xen/include/xen/sched.h | 6 + xen/include/xlat.lst | 2 + 8 files changed, 582 insertions(+), 1 deletion(-) create mode 100644 xen/include/public/argo.h create mode 100644 xen/include/xen/argo.hdocs/misc/xen-command-line.pandoc +11
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware. +### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains. + ### asid (x86) > `= <boolean>`xen/common/argo.c +0
-0@@ -17,7 +17,177 @@*/
xen/common/argo.c line 27
Quoted 10 lines
xen/common/argo.c +7
-0#include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h>
We usually try to sort header includes alphabetically, and I would add a newline between the xen/* and the public/* header includes.
xen/common/argo.c line 34
Quoted 7 lines
xen/common/argo.c +7
-0+ +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled);
I would drop the opt_* prefix, new options added recently don't include the prefix already.
Quoted 11 lines
xen/common/argo.c +11
-0+ +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{
xen/common/argo.c line 55
Quoted 10 lines
xen/common/argo.c +10
-0+/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage;
Can you infer number of pages form the length of the ring, or the other way around?
I'm not sure why both need to be stored here.
xen/common/argo.c line 59
Quoted 4 lines
xen/common/argo.c +4
-0+/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr;
All this fields are not part of any public structure, so I wonder if it would be better to simply use unsigned int for those, or size_t.
xen/common/argo.c line 61
Quoted 2 lines
xen/common/argo.c +2
-0+/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping;
Why 'uint8_t *', wouldn't it be better to just use 'void *' if it's a mapping?
Quoted 28 lines
xen/common/argo.c +28
-0+/* list of mfns of guest ring, protected by L3 */ +mfn_t *mfns; +/* list of struct pending_ent for this ring, protected by L3 */ +struct hlist_head pending; +/* number of pending entries queued for this ring, protected by L3 */ +uint32_t npending; +}; + +/* Data about a single-sender ring, held by the sender (partner) domain */ +struct argo_send_info +{ +/* next node in the hash, protected by Lsend */ +struct hlist_node node; +/* this ring's id, protected by Lsend */ +struct argo_ring_id id; +}; + +/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ +struct hlist_node wildcard_node;
xen/common/argo.c line 99
Quoted 10 lines
xen/common/argo.c +10
-0+/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy. +* Caution: Must hold the correct locks before accessing ring_info via this. +*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;
No need for the pad in internal structures.
Quoted 47 lines
xen/common/argo.c +47
-0+/* minimum ring space available that this signal is waiting upon */ +uint32_t len; +}; + +/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system.
xen/common/argo.c line 156
Quoted 10 lines
xen/common/argo.c +10
-0+* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1) +* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */
You also add an argo_lock to each domain struct which doesn't seem to be mentioned here at all. Shouldn't that lock be the one that protects d->argo? (instead of this global lock?)
Quoted 24 lines
xen/common/argo.c +24
-0+ +/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend.
xen/common/argo.c line 190
Quoted 10 lines
xen/common/argo.c +10
-0+* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/
IMO I think the locking is overly complicated, and there's no reasoning why so many locks are needed. Wouldn't it be enough to start with a single lock that protects the whole d->argo existence and contents?
I would start with a very simple (as simple as possible) locking structure and go improving from there if there are performance bottlenecks.
Quoted 15 lines
xen/common/argo.c +9
-0/* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ )
xen/common/argo.c line 219
Quoted 10 lines
xen/common/argo.c +10
-0+ { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) +argo_dprintk (XENLOG_ERR "argo: unmapping page %" PRI_mfn" from %p\n" , +mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + }
As noted in another patch, I would consider mapping this in contiguous virtual address space using vmap, but I'm not going to insist.
xen/common/argo.c line 225
Quoted 6 lines
xen/common/argo.c +6
-0+} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id);
Newline.
xen/common/argo.c line 229
Quoted 4 lines
xen/common/argo.c +4
-0+if ( !d ) +return ; + +if ( d->argo )
Don't you need to pick d->argo_lock here to prevent d->argo from being removed under your feet?
Quoted 5 lines
xen/common/argo.c +5
-0+ { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + }
xen/common/argo.c line 244
Quoted 10 lines
xen/common/argo.c +10
-0+put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node)
As a side note, it might be interesting to introduce a helper like hlist_first_entry_or_null, that would remove the need to have an extra *next element, and would be the more natural way to drain a hlist (seeing that you have the same pattern in wildcard_rings_pending_remove).
Quoted 24 lines
xen/common/argo.c +24
-0+ { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node); + ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent);
xen/common/argo.c line 278
Quoted 10 lines
xen/common/argo.c +10
-0+ } +} + +static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock));
I think the above requires a comment of why two different locks are used to protect the ring mfns, and why just having one of them locked is enough.
Quoted 8 lines
xen/common/argo.c +8
-0+ +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ;
xen/common/argo.c line 296
Quoted 10 lines
xen/common/argo.c +10
-0+ } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ;
Xen has a handly macro for this, XFREE. That would free the memory and assign the pointer to NULL.
xen/common/argo.c line 304
Quoted 8 lines
xen/common/argo.c +8
-0+ ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info)
I think the domain parameter can be constifyed here, since it's only used by ring_remove_mfnsnd that function already expects a const domain struct.
Quoted 75 lines
xen/common/argo.c +69
-1+{ +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +pending_remove_all (ring_info); +hlist_del (&ring_info->node); +ring_remove_mfns (d, ring_info); +xfree (ring_info); +} + +static void +domain_rings_remove_all (struct domain *d) +{ +unsigned int i; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_ring_info *ring_info; + +hlist_for_each_entry_safe (ring_info, node, next, + &d->argo->ring_hash[i], node) +ring_remove_info (d, ring_info); + } + d->argo->ring_count =0 ; +} + +/* +* Tear down all rings of other domains where src_d domain is the partner. +* (ie. it is the single domain that can send to those rings.) +* This will also cancel any pending notifications about those rings. +*/ +static void +partner_rings_remove (struct domain *src_d) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&argo_lock)); + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +struct hlist_node *node, *next; +struct argo_send_info *send_info; + +hlist_for_each_entry_safe (send_info, node, next, + &src_d->argo->send_hash[i], node) + { +struct argo_ring_info *ring_info; +struct domain *dst_d; + + dst_d =get_domain_by_id (send_info->id.domain_id); +if ( dst_d ) + { + ring_info =ring_find_info (dst_d, &send_info->id); +if ( ring_info ) + { +ring_remove_info (dst_d, ring_info); + dst_d->argo->ring_count--; + } + +put_domain (dst_d); + } + +hlist_del (&send_info->node); +xfree (send_info); + } + } +} +long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ;
xen/common/argo.c line 388
Quoted 10 lines
xen/common/argo.c +10
-0+struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc;
Why not just 'return -EOPNOTSUPP;'?
Quoted 48 lines
xen/common/argo.c +48
-0+ } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id);
xen/common/argo.c line 446
Quoted 10 lines
xen/common/argo.c +10
-0+ + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo;
Where's the d->argo_lock initialization?
Quoted 33 lines
xen/common/argo.c +33
-0+ +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); +
xen/common/argo.c line 489
Quoted 10 lines
xen/common/argo.c +10
-0+if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ;
Can opt_argo_enabled change during runtime?
Quoted 20 lines
xen/common/argo.c +6
-0+ } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +1
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h>
xen/common/domain.c static void _domain_destroy(struct domain *d) · line 283
Quoted 6 lines
xen/common/domain.c +3
-0@@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif
Instead of adding such ifdefs you could provide dummy argo_destroy inline functions in argo.h when CONFIG_ARGO is not set.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 10.01.19 at 11:19, <[email protected]> wrote:
xen/common/argo.c line 34
Quoted 4 lines · Christopher Clark
xen/common/argo.c +3
-0+/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled);I would drop the opt_* prefix, new options added recently don't include the prefix already.
Would you mind pointing out examples? Especially for boolean ones I think we've tried to consistently name them opt_*. But in the case here (it being static) I'm not overly fussed.
Quoted 27 lines
Quoted 11 lines
xen/common/argo.c +11
-0+ +typedef struct argo_ring_id +{ +uint32_t port; +domid_t partner_id; +domid_t domain_id; +}argo_ring_id ; + +/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{xen/common/argo.c line 55
Quoted 10 lines
xen/common/argo.c +10
-0+/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage;Can you infer number of pages form the length of the ring, or the other way around?
I'm not sure why both need to be stored here.
Quoted 7 lines
xen/common/argo.c line 59
Quoted 4 lines
xen/common/argo.c +4
-0+/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr;All this fields are not part of any public structure, so I wonder if it would be better to simply use unsigned int for those, or
size_t.
Yes indeed - there's way too much use of fixed width types here.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines · On 10.01.19 at 11:19, <[email protected]> wrote:
xen/common/argo.c line 34
Quoted 4 lines · Christopher Clark
xen/common/argo.c +3
-0+/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled);I would drop the opt_* prefix, new options added recently don't include the prefix already.
Quoted 3 lines
Would you mind pointing out examples? Especially for boolean ones I think we've tried to consistently name them opt_*. But in the case here (it being static) I'm not overly fussed.
I was mostly thinking about the dom0_pvh boolean option that I added. I'm not overly fuzzed, I just think it's not adding any value to the variable name, but if you prefer to keep the opt_ prefix that's fine.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 10.01.19 at 13:26, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines · On 10.01.19 at 11:19, <[email protected]> wrote:
xen/common/argo.c line 34
Quoted 4 lines · Christopher Clark
xen/common/argo.c +3
-0+/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled);I would drop the opt_* prefix, new options added recently don't include the prefix already.
Quoted 7 lines
Quoted 3 lines
Would you mind pointing out examples? Especially for boolean ones I think we've tried to consistently name them opt_*. But in the case here (it being static) I'm not overly fussed.
I was mostly thinking about the
dom0_pvhboolean option that I added. I'm not overly fuzzed, I just think it's not adding any value to the variable name, but if you prefer to keep theopt_prefix that's fine.
Well, one value is that at use sites of such variables you immediately notice that the value is (potentially) admin controlled. At least in some cases this is helpful to be obvious.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Roger Pau MonnéQuoted 14 lines · Roger Pau Monné
Quoted 11 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
xen/common/argo.c +0
-0@@ -17,7 +17,177 @@*/
Quoted 12 lines
xen/common/argo.c line 27
Quoted 9 lines
xen/common/argo.c +7
-0#include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> We usually try to sort header includes alphabetically, and I would add a newline between the xen/* and the public/* header includes.
ack, thanks, done.
Quoted 11 lines
xen/common/argo.c line 34
Quoted 8 lines
xen/common/argo.c +7
-0+ +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled);I would drop the opt_* prefix, new options added recently don't include the prefix already.
after the later in-thread discussion w/ yourself and Jan, I've retained this prefix
Quoted 17 lines
xen/common/argo.c line 55
Quoted 13 lines
xen/common/argo.c +13
-0+/* Data about a domain's own ring that it has registered */ +struct argo_ring_info +{ +/* next node in the hash, protected by L2 */ +struct hlist_node node; +/* this ring's id, protected by L2 */ +struct argo_ring_id id; +/* L3 */ +spinlock_t lock; +/* length of the ring, protected by L3 */ +uint32_t len; +/* number of pages in the ring, protected by L3 */ +uint32_t npage;Can you infer number of pages form the length of the ring, or the other way around?
Quoted 1 line
I'm not sure why both need to be stored here.
Yes, you're right. I've removed the npage struct member and calculated the value where needed from the len.
Quoted 7 lines
xen/common/argo.c line 59
Quoted 4 lines
xen/common/argo.c +4
-0+/* number of pages translated into mfns, protected by L3 */ +uint32_t nmfns; +/* cached tx pointer location, protected by L3 */ +uint32_t tx_ptr;All this fields are not part of any public structure, so I wonder if it would be better to simply use unsigned int for those, or
size_t.
ack, done.
Quoted 4 lines
xen/common/argo.c line 61
Quoted 2 lines
xen/common/argo.c +2
-0+/* mapped ring pages protected by L3 */ +uint8_t **mfn_mapping;Why '
uint8_t*', wouldn't it be better to just use 'void *' if it's a mapping?
Yes. Have switched it to be void*.
Quoted 15 lines
Quoted 15 lines
xen/common/argo.c +15
-0+/* A space-available notification that is awaiting sufficient space */ +struct pending_ent +{ +/* List node within argo_ring_info's pending list */ +struct hlist_node node; +/* +* List node within argo_domain's wildcard_pend_list. Only used if the +* ring is one with a wildcard partner (ie. that any domain may send to) +* to enable cancelling signals on wildcard rings on domain destroy. +*/ +struct hlist_node wildcard_node; +/* +* Pointer to the ring_info that this ent pertains to. Used to ensure that +* ring_info->npending is decremented when ents for wildcard rings are +* cancelled for domain destroy.
Quoted 8 lines
xen/common/argo.c line 99
Quoted 6 lines
xen/common/argo.c +6
-0+* Caution: Must hold the correct locks before accessing ring_info via this. +*/ +struct argo_ring_info *ring_info; +/* domain to be notified when space is available */ +domid_t domain_id; +uint16_t pad;No need for the pad in internal structures.
ack - removed pad.
Quoted 48 lines
Quoted 48 lines
xen/common/argo.c +48
-0+/* +* The value of the argo element in a struct domain is +* protected by the global lock argo_lock: L1 +*/ +#define ARGO_HTABLE_SIZE 32 +struct argo_domain +{ +/* L2 */ +rwlock_t lock; +/* +* Hash table of argo_ring_info about rings this domain has registered. +* Protected by L2. +*/ +struct hlist_head ring_hash[ARGO_HTABLE_SIZE ]; +/* Counter of rings registered by this domain. Protected by L2. */ +uint32_t ring_count; + +/* Lsend */ +spinlock_t send_lock; +/* +* Hash table of argo_send_info about rings other domains have registered +* for this domain to send to. Single partner, non-wildcard rings. +* Protected by Lsend. +*/ +struct hlist_head send_hash[ARGO_HTABLE_SIZE ]; + +/* Lwildcard */ +spinlock_t wildcard_lock; +/* +* List of pending space-available signals for this domain about wildcard +* rings registered by other domains. Protected by Lwildcard. +*/ +struct hlist_head wildcard_pend_list; +}; + +/* +* Locking is organized as follows: +* +* Terminology: R(<lock>) means taking a read lock on the specified lock; +* W(<lock>) means taking a write lock on it. +* +* L1 : The global lock: argo_lock +* Protects the argo elements of all struct domain *d in the system. +* It does not protect any of the elements of d->argo, only their +* addresses. +* +* By extension since the destruction of a domain with a non-NULL +* d->argo will need to free the d->argo pointer, holding W(L1)
Quoted 8 lines
xen/common/argo.c line 156
Quoted 5 lines
xen/common/argo.c +5
-0+* guarantees that no domains pointers that argo is interested in +* become invalid whilst this lock is held. +*/ + +static DEFINE_RWLOCK (argo_lock);/* L1 */ You also add an
argo_lockto each domain struct which doesn't seem to be mentioned here at all.
You're right! Thanks - that's a nice find. That lock is not used at all. I'd missed it since it's just not referenced anywhere in the argo.c file. I've removed it.
Quoted 1 line
Shouldn't that lock be the one that protects
d->argo? (instead of this global lock?)
According the design that is in place at the moment, no, but I need to study that option a bit before I can comment further on whether it would make sense to add it in order to do so.
I imagine not though because we're not looking to add any more locks.
Quoted 34 lines
Quoted 23 lines
xen/common/argo.c +23
-0+/* +* L2 : The per-domain ring hash lock: d->argo->lock +* Holding a read lock on L2 protects the ring hash table and +* the elements in the hash_table d->argo->ring_hash, and +* the node and id fields in struct argo_ring_info in the +* hash table. +* Holding a write lock on L2 protects all of the elements of +* struct argo_ring_info. +* +* To take L2 you must already have R(L1). W(L1) implies W(L2) and L3. +* +* L3 : The ringinfo lock: argo_ring_info *ringinfo; ringinfo->lock +* Protects all the fields within the argo_ring_info, aside from the ones that +* L2 already protects: node, id, lock. +* +* To aquire L3 you must already have R(L2). W(L2) implies L3. +* +* Lsend : The per-domain single-sender partner rings lock: d->argo->send_lock +* Protects the per-domain send hash table : d->argo->send_hash +* and the elements in the hash table, and the node and id fields +* in struct argo_send_info in the hash table. +* +* To take Lsend, you must already have R(L1). W(L1) implies Lsend. xen/common/argo.c line 190
Quoted 10 lines
xen/common/argo.c +10
-0+* Do not attempt to acquire a L2 on any domain after taking and while +* holding a Lsend lock -- acquire the L2 (if one is needed) beforehand. +* +* Lwildcard : The per-domain wildcard pending list lock: d->argo->wildcard_lock +* Protects the per-domain list of outstanding signals for space availability +* on wildcard rings. +* +* To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard. +* No other locks are acquired after obtaining Lwildcard. +*/
Quoted 8 lines
IMO I think the locking is overly complicated, and there's no reasoning why so many locks are needed. Wouldn't it be enough to start with a single lock that protects the whole
d->argoexistence and contents?I would start with a very simple (as simple as possible) locking structure and go improving from there if there are performance bottlenecks.
It definitely doesn't help when there's an extra lock lying around just to be confusing. Sorry.
The locking discipline in this code is challenging and you are right that there hasn't a explanation given as to _why_ there are the locks that there are. I will fix that. I can also review the placement of the ASSERTs that check (and document) the locks within the code, if that helps.
The current locking comments describe the how, but the why hasn't been covered so far and it is needed. The unreasonably-short version is: this code is *hot* when the communication paths are in use -- it operates the data path -- and there needs to be isolation for paths using rings from the potentially malicious or disruptive activities of other domains, or even other vcpus of the same domain operating other rings.
I am confident that the locking (that actually gets operated) is correct and justified though, and I hope that adding some new clear documentation for it can address this.
Quoted 20 lines
Quoted 20 lines
xen/common/argo.c +14
-0/* Change this to #define ARGO_DEBUG here to enable more debug messages */ #undef ARGO_DEBUG@@ -28,10 +198,299 @@#define argo_dprintk (format, ... ) ((void)0)#endif +static void +ring_unmap (struct argo_ring_info *ring_info) +{ +unsigned int i; + +if ( !ring_info->mfn_mapping ) +return ; + +for ( i =0 ; i < ring_info->nmfns; i++ ) + { +if ( !ring_info->mfn_mapping[i] ) +continue ; +if ( ring_info->mfns ) + argo_dprintk(XENLOG_ERR "argo: unmapping page %"PRI_mfn" from %p\n",
Quoted 8 lines
xen/common/argo.c line 219
Quoted 5 lines
xen/common/argo.c +5
-0+mfn_x (ring_info->mfns[i]), + ring_info->mfn_mapping[i]); +unmap_domain_page_global (ring_info->mfn_mapping[i]); + ring_info->mfn_mapping[i] =NULL ; + }As noted in another patch, I would consider mapping this in contiguous virtual address space using vmap, but I'm not going to insist.
Thanks - I will look at what is involved to do it, but I may well have to leave it as-is for now.
Quoted 9 lines
xen/common/argo.c line 225
Quoted 7 lines
xen/common/argo.c +6
-0+} + +static void +wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id);Newline.
ack
Quoted 8 lines
xen/common/argo.c line 229
Quoted 5 lines
xen/common/argo.c +4
-0+if ( !d ) +return ; + +if ( d->argo )Don't you need to pick
d->argo_lockhere to preventd->argofrom being removed under your feet?
No, because wildcard_pending_list_remove is called from:
* pending_find and pending_cancel:
with both R(L2) and L3 held (of a different domain, but that's ok), which means R(L1) is held,
so d->argo is safe in wildcard_pending_list_remove.
* pending_remove_all:
which is only called from ring_remove_info,
which has ASSERTS that either R(L1) or R(L2) is held, and R(L2) means R(L1) must already be held (following protocol). so d->argo is safe in wildcard_pending_list_remove.
Quoted 53 lines
Quoted 6 lines
xen/common/argo.c +5
-0+ { +spin_lock (&d->argo->wildcard_lock); +hlist_del (&ent->wildcard_node); +spin_unlock (&d->argo->wildcard_lock); + }xen/common/argo.c line 244
Quoted 10 lines
xen/common/argo.c +10
-0+put_domain (d); +} + +static void +pending_remove_all (struct argo_ring_info *ring_info) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +hlist_for_each_entry_safe (ent, node, next, &ring_info->pending, node)As a side note, it might be interesting to introduce a helper like
hlist_first_entry_or_null, that would remove the need to have an extra *next element, and would be the more natural way to drain a hlist (seeing that you have the same pattern inwildcard_rings_pending_remove).Quoted 30 lines
xen/common/argo.c +30
-0+ { +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_remove (ent->domain_id, ent); +hlist_del (&ent->node); +xfree (ent); + } + ring_info->npending =0 ; +} + +static void +wildcard_rings_pending_remove (struct domain *d) +{ +struct hlist_node *node, *next; +struct pending_ent *ent; + +ASSERT (rw_is_write_locked (&argo_lock)); + +hlist_for_each_entry_safe (ent, node, next, &d->argo->wildcard_pend_list, + node) + { +hlist_del (&ent->node); + ent->ring_info->npending--; +hlist_del (&ent->wildcard_node); +xfree (ent); + } +} + +static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{
Quoted 8 lines
xen/common/argo.c line 278
Quoted 4 lines
xen/common/argo.c +4
-0+unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock));I think the above requires a comment of why two different locks are used to protect the ring mfns, and why just having one of them locked is enough.
ack, I need to add further locking docs.
Quoted 14 lines
Quoted 14 lines
xen/common/argo.c +13
-0+ +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ )
Quoted 8 lines
xen/common/argo.c line 296
Quoted 5 lines
xen/common/argo.c +5
-0+if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ;Xen has a handly macro for this, XFREE. That would free the memory and assign the pointer to NULL.
thanks, that's great - done
Quoted 13 lines
xen/common/argo.c line 304
Quoted 9 lines
xen/common/argo.c +8
-0+ ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ; +} + +static void +ring_remove_info (struct domain *d,struct argo_ring_info *ring_info)I think the domain parameter can be constifyed here, since it's only used by
ring_remove_mfnsndthat function already expects a const domain struct.
ack, yes, thanks.
Quoted 11 lines
Quoted 11 lines
xen/common/argo.c +5
-1+long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + + argo_dprintk("->do_argo_op(%u,%p,%p,%d,%d)\n", cmd,
Quoted 8 lines
xen/common/argo.c line 388
Quoted 6 lines
xen/common/argo.c +6
-0+ (void *)arg1.p, (void *)arg2.p, (int) arg3, (int) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc;Why not just 'return
-EOPNOTSUPP;'?
good point - will do
Quoted 17 lines
Quoted 17 lines
xen/common/argo.c +17
-0+ +int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ;
Quoted 8 lines
xen/common/argo.c line 446
Quoted 6 lines
xen/common/argo.c +6
-0+ +write_lock (&argo_lock); + +argo_domain_init (argo); + + d->argo = argo;Where's the
d->argo_lockinitialization?
It was added to domain.c in this patch, but there is now no need, that lock is gone. Thanks for the catch.
Quoted 38 lines
Quoted 38 lines
xen/common/argo.c +37
-0+ +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d) +{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d);
Quoted 8 lines
xen/common/argo.c line 489
Quoted 6 lines
xen/common/argo.c +6
-0+wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ;Can
opt_argo_enabledchange during runtime?
Not at the moment, no. It should be made changeable later, but keeping it fixed assists with derisking this for release consideration.
Quoted 22 lines
Quoted 22 lines
xen/common/argo.c +6
-0+ } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +1
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> @@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)
Quoted 8 lines
xen/common/domain.c static void _domain_destroy(struct domain *d) · line 283
Quoted 5 lines
xen/common/domain.c +3
-0xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif Instead of adding such ifdefs you could provide dummy
argo_destroyinline functions inargo.hwhenCONFIG_ARGOis not set.
ack, have done this.
Thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 31 lines · Christopher Clark
Quoted 16 lines · Roger Pau Monné
Quoted 13 lines · Christopher Clark
+/*
+ * Locking is organized as follows:
+ *
+ * Terminology: R(<lock>) means taking a read lock on the specified lock;
+ * W(<lock>) means taking a write lock on it.
+ *
+ * L1 : The global lock:argo_lock
+ * Protects the argo elements of all struct domain *d in the system.
+ * It does not protect any of the elements ofd->argo, only their
+ * addresses.
+ *
+ * By extension since the destruction of a domain with a non-NULL
+ *d->argowill need to free thed->argopointer, holding W(L1)Quoted 8 lines
Review
Quoted 5 lines
+ * guarantees that no domains pointers that argo is interested in
+ * become invalid whilst this lock is held.
+ */(quoted diff) +2
-0+ +static DEFINE_RWLOCK(argo_lock); /* L1 */You also add an
argo_lockto each domain struct which doesn't seem to be mentioned here at all.You're right! Thanks - that's a nice find. That lock is not used at all. I'd missed it since it's just not referenced anywhere in the
argo.cfile. I've removed it.
Quoted 6 lines
Quoted 1 line
Shouldn't that lock be the one that protects
d->argo? (instead of this global lock?)According the design that is in place at the moment, no, but I need to study that option a bit before I can comment further on whether it would make sense to add it in order to do so.
I imagine not though because we're not looking to add any more locks.
I'm wondering why a global argo_lock shared with all domains is used to protect d->argo, instead of using a per-domain lock (d->argo_lock for example). This global argo_lock shared between all domains is going to introduce contention with no specific benefit AFAICT.
I would recommend an initial implementation that uses a single per-domain lock (ie: d->argo_lock) to protect the whole contents of d->argo, and then go adding more fine grained locking as required, providing evidence that such fine grainer locking is actually improving performance (or required for some other reason). IMO, the current locking scheme is overly complicated, and it's very hard for me to reason about it's correctness.
Quoted 51 lines
Quoted 34 lines
Quoted 25 lines
+/*
+ * L2 : The per-domain ring hash lock:d->argo->lock
+ * Holding a read lock on L2 protects the ring hash table and
+ * the elements in thehash_tabled->argo->ring_hash, and
+ * the node and id fields in structargo_ring_infoin the
+ * hash table.
+ * Holding a write lock on L2 protects all of the elements of
+ * structargo_ring_info.
+ *
+ * To take L2 you must already have R(L1). W(L1) implies W(L2) and L3.
+ *
+ * L3 : The ringinfo lock:argo_ring_info*ringinfo;ringinfo->lock
+ * Protects all the fields within theargo_ring_info, aside from the ones that
+ * L2 already protects: node, id, lock.
+ *
+ * To aquire L3 you must already have R(L2). W(L2) implies L3.
+ *
+ * Lsend : The per-domain single-sender partner rings lock:d->argo->send_lock
+ * Protects the per-domain send hash table :d->argo->send_hash
+ * and the elements in the hash table, and the node and id fields
+ * in structargo_send_infoin the hash table.
+ *
+ * To take Lsend, you must already have R(L1). W(L1) implies Lsend.
+ * Do not attempt to acquire a L2 on any domain after taking and while
+ * holding a Lsend lock -- acquire the L2 (if one is needed) beforehand.Quoted 8 lines
+ *
+ * Lwildcard : The per-domain wildcard pending list lock:d->argo->wildcard_lock
+ * Protects the per-domain list of outstanding signals for space availability
+ * on wildcard rings.
+ *
+ * To take Lwildcard, you must already have R(L1). W(L1) implies Lwildcard.
+ * No other locks are acquired after obtaining Lwildcard.
+ */Quoted 8 lines
IMO I think the locking is overly complicated, and there's no reasoning why so many locks are needed. Wouldn't it be enough to start with a single lock that protects the whole
d->argoexistence and contents?I would start with a very simple (as simple as possible) locking structure and go improving from there if there are performance bottlenecks.
It definitely doesn't help when there's an extra lock lying around just to be confusing. Sorry.
The locking discipline in this code is challenging and you are right that there hasn't a explanation given as to _why_ there are the locks that there are. I will fix that. I can also review the placement of the ASSERTs that check (and document) the locks within the code, if that helps.
Quoted 6 lines
The current locking comments describe the how, but the why hasn't been covered so far and it is needed. The unreasonably-short version is: this code is *hot* when the communication paths are in use -- it operates the data path -- and there needs to be isolation for paths using rings from the potentially malicious or disruptive activities of other domains, or even other vcpus of the same domain operating other rings.
Yes, that’s fine, but as said above I wonder why for example a global argo_lock is used to protect d->argo, instead of a per-domain lock. At first sight this doesn’t look like the best approach performance wise.
Quoted 3 lines
I am confident that the locking (that actually gets operated) is correct and justified though, and I hope that adding some new clear documentation for it can address this.
I’m not saying otherwise, but I cannot assert it either.
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
xen/common/argo.c +11
-0+void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d);Quoted 8 lines
xen/common/argo.c line 489
Quoted 6 lines
xen/common/argo.c +6
-0+wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ;Can
opt_argo_enabledchange during runtime?
Quoted 3 lines
Not at the moment, no. It should be made changeable later, but keeping it fixed assists with derisking this for release consideration.
Then if d->argo is set opt_argo_enabled must be true, and thus this condition is never true?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 22 lines · Roger Pau Monné
Quoted 20 lines · Christopher Clark
Quoted 18 lines · Roger Pau Monné
Quoted 15 lines · Christopher Clark
+/*
+ * Locking is organized as follows:
+ *
+ * Terminology: R(<lock>) means taking a read lock on the specified lock;
+ * W(<lock>) means taking a write lock on it.
+ *
+ * L1 : The global lock:argo_lock
+ * Protects the argo elements of all struct domain *d in the system.
+ * It does not protect any of the elements ofd->argo, only their
+ * addresses.
+ *
+ * By extension since the destruction of a domain with a non-NULL
+ *d->argowill need to free thed->argopointer, holding W(L1)
+ * guarantees that no domains pointers that argo is interested in
+ * become invalid whilst this lock is held.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
+ */
(quoted diff) +2
-0+ +static DEFINE_RWLOCK(argo_lock); /* L1 */I'm wondering why a global
argo_lockshared with all domains is used to protectd->argo, instead of using a per-domain lock (d->argo_lockfor example). This globalargo_lockshared between all domains is going to introduce contention with no specific benefit AFAICT.
The granular locking structure is motivated by:
1) Performance isolation / DoS avoidance
2) Teardown of state across multiple domains on domain destroy
3) Performance via concurrent operation of rings
Using the single global lock avoids the need for sequencing the acquisition of multiple individual per-domain locks (and lower level data structure locks) to prevent deadlock: taking W(L1) grants access to all and taking R(L1) ensures that teardown of any domain will not interfere with any Argo hypercall operation. It supports using the granular locks across domains without complicated or fragile lock acquisition logic.
I've written a document about the locking to add to the tree with the series, and a copy is at github here:
Quoted 7 lines
I would recommend an initial implementation that uses a single per-domain lock (ie:
d->argo_lock) to protect the whole contents ofd->argo, and then go adding more fine grained locking as required, providing evidence that such fine grainer locking is actually improving performance (or required for some other reason). IMO, the current locking scheme is overly complicated, and it's very hard for me to reason about it's correctness.
I've now implemented some macros to describe and document locking requirements in the code, and simplify ASSERTing the correct status. The majority of functions have a single annotation at entry indicating and verifying their locking status. They are described in the doc.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 8 lines
xen/common/argo.c +7
-0+void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); +xen/common/argo.c line 489
Quoted 10 lines
xen/common/argo.c +10
-0+if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ;Can
opt_argo_enabledchange during runtime?
Quoted 6 lines
Quoted 3 lines
Not at the moment, no. It should be made changeable later, but keeping it fixed assists with derisking this for release consideration.
Then if
d->argois setopt_argo_enabledmust be true, and thus this condition is never true?
Yes; ack, have removed this logic.
I'm holding off on posting v4 of the series but my latest tree, with the new macros applied, is on github at:
branch:
https://github.com/dozylynx/xen/tree/staging-argo-2019-01-13
main file within that branch:
https://github.com/dozylynx/xen/blob/staging-argo-2019-01-13/xen/common/argo.c
Given the limited time remaining for the merge for 4.12 under consideration, if there are any further reservations, please let me know.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 38 lines · Christopher Clark
Quoted 22 lines · Roger Pau Monné
Quoted 20 lines · Christopher Clark
Quoted 18 lines · Roger Pau Monné
Quoted 15 lines · Christopher Clark
+/*
+ * Locking is organized as follows:
+ *
+ * Terminology: R(<lock>) means taking a read lock on the specified lock;
+ * W(<lock>) means taking a write lock on it.
+ *
+ * L1 : The global lock:argo_lock
+ * Protects the argo elements of all struct domain *d in the system.
+ * It does not protect any of the elements ofd->argo, only their
+ * addresses.
+ *
+ * By extension since the destruction of a domain with a non-NULL
+ *d->argowill need to free thed->argopointer, holding W(L1)
+ * guarantees that no domains pointers that argo is interested in
+ * become invalid whilst this lock is held.Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Review
Quoted 3 lines
+ */
(quoted diff) +2
-0+ +static DEFINE_RWLOCK(argo_lock); /* L1 */I'm wondering why a global
argo_lockshared with all domains is used to protectd->argo, instead of using a per-domain lock (d->argo_lockfor example). This globalargo_lockshared between all domains is going to introduce contention with no specific benefit AFAICT.The granular locking structure is motivated by:
1) Performance isolation / DoS avoidance
2) Teardown of state across multiple domains on domain destroy
3) Performance via concurrent operation of rings
Quoted 7 lines
Using the single global lock avoids the need for sequencing the acquisition of multiple individual per-domain locks (and lower level data structure locks) to prevent deadlock: taking W(L1) grants access to all and taking R(L1) ensures that teardown of any domain will not interfere with any Argo hypercall operation. It supports using the granular locks across domains without complicated or fragile lock acquisition logic.
I'm not sure such global lock is needed. Isn't it going to introduce a non-trivial amount of contention?
Please bear with me, iff instead of the global lock a d->arg_lock is used, is there any use-case where you would need to lock multiple d->arg_locks in sequence? I'm not sure I see which use-case would require this, since I expect you take the d->arg_lock, perform the needed operations on that domain argo data/rings and move to the next one.
Quoted 4 lines
I've written a document about the locking to add to the tree with the series, and a copy is at github here:
Thanks. It would have been better to send the contents of the document to the list, so inline comments can be added. It's hard to comment on the document now since it's only on github AFAICT.
As a general comment, I think the "Hierarchical Locking Model and Protocol" is too complex, and the fact that there are multiple interchangeable lock sequences to write to the rings for example is not a good locking scheme because it makes it hard to reason about.
For example you can write to the ring by simply write-locking L1, or by read-locking L1 and L2, and then locking the ring lock (L3). IMO I would state that writing to the ring _always_ requires L3 to be locked, regardless of the other locks. This makes it easier to reason about locking correctness, and which locks protect what data in the argos structure, if indeed so many different locks are required.
There are also several claims that fine-grainer locking provides better performance in order to justify the need of such locks. IMO without providing any evidence of such performance benefit it's hard to be convinced so many locks are needed.
Quoted 43 lines
Quoted 7 lines
I would recommend an initial implementation that uses a single per-domain lock (ie:
d->argo_lock) to protect the whole contents ofd->argo, and then go adding more fine grained locking as required, providing evidence that such fine grainer locking is actually improving performance (or required for some other reason). IMO, the current locking scheme is overly complicated, and it's very hard for me to reason about it's correctness.I've now implemented some macros to describe and document locking requirements in the code, and simplify ASSERTing the correct status. The majority of functions have a single annotation at entry indicating and verifying their locking status. They are described in the doc.
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 8 lines
xen/common/argo.c +7
-0+void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); +xen/common/argo.c line 489
Quoted 10 lines
xen/common/argo.c +10
-0+if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ;Can
opt_argo_enabledchange during runtime?Quoted 6 lines
Quoted 3 lines
Not at the moment, no. It should be made changeable later, but keeping it fixed assists with derisking this for release consideration.
Then if
d->argois setopt_argo_enabledmust be true, and thus this condition is never true?Yes; ack, have removed this logic.
Quoted 7 lines
I'm holding off on posting v4 of the series but my latest tree, with the new macros applied, is on github at:
branch:
https://github.com/dozylynx/xen/tree/staging-argo-2019-01-13
main file within that branch:
https://github.com/dozylynx/xen/blob/staging-argo-2019-01-13/xen/common/argo.c
IMO, I'm finding the patch series slightly difficult to review. For example patch 4 introduces a full-blown argo_domain structure with all the sub-structures and fields, which I'm not sure all are actually used in that patch. It would be easier to review if a simpler set of operations is first introduced, like starting by only allowing single domain rings, and then introducing multicast rings.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 6 lines · Roger Pau Monné
Quoted 1 line · Christopher Clark
Quoted 8 lines
Quoted 4 lines
I've written a document about the locking to add to the tree with the series, and a copy is at github here:
Thanks. It would have been better to send the contents of the document to the list, so inline comments can be added. It's hard to comment on the document now since it's only on github AFAICT.
Here's an inline copy of the doc:
=== begin document ===
# Argo: Locking
## Introduction
Argo is an interdomain communication mechanism. It has requirements for performance isolation between domains, to prevent negative performance impact from malicious or disruptive activity of other domains, or even other vcpus of the same domain operating other rings.
Since Argo operates a data path between domains, sections of this code are *hot* when the communication paths are in use. To encourage high performance, a goal is to limit mutual exclusion to only where required and enable significant concurrency.
Avoidance of deadlock is essential and since state must frequently be updated that pertains to more than one domain, a locking protocol defines which locks are needed and the order of their acquistion.
## Structure
The granular locking structure of Argo enables:
1. Performance isolation of guests
2. Avoidance of DoS of rings by domains that are not authorized to send to them
3. Deadlock-free teardown of state across multiple domains on domain destroy
4. Performance of guests using Argo with concurrent operation of rings.
Argo uses three per-domain locks to protect three separate data structures. Access to the ring_hash data structure is confined to domains that a ring-registering domain has authorized to send data via the ring. The complete set of Argo locks is:
* Global : L1_global_argo_rwlock
* Per-domain: rings_L2_rwlock
* Per-domain: send_L2_lock
* Per-domain: wildcard_L2_lock
* Per-ring: L3_lock
## Protected State
The data structures being protected by the locks are all per-domain. The only global Argo state is the L1_global_argo_rwlock used to coordinate access to data structures of other domains.
### State: Rings registered and owned by a domain
This includes the state to run that ring, such as memory frame numbers and established mappings. Per-ring state is protected by its own lock, so that multiple VCPUs of the same domain operating different rings do not inhibit the performance of each other.
The per-domain ring state also includes the list of pending notifications for other domains that are waiting for ring space availability.
### State: Partner rings registered by other domains that this domain is the single allowed sender
This state belonging to the permitted sender is written to when a ring is registered by another domain. The lock that protects this state is subject to locking at arbitrary frequency by those foreign domains when registering rings -- which do not need any permission granted by this domain in order to register a ring to communicate with it -- so it must not inhibit the domain's own ability to use its own rings, to protect them from DoS. For this reason, this state is protected by its own lock.
### State: Pending notifications to this domain about wildcard rings registered by other domains
This data structure is needed when a domain is destroyed, to cancel the outstanding space availability notifications about the wildcard rings of other domains that this domain has queried.
Data is entered into this data structure by the domain that owns it, either by a space-inhibited sendv or a notify operation.
Data is removed from this data structure in one of three cases: when space becomes available in the destination ring and the notification is sent, when the ring is torn down, or when the awaiting domain is destroyed.
In the case where a notification is sent, access to the data structure is triggered by the ring owner domain, rather than the domain waiting for notification. This data structure is protected by its own lock since doing so entails less contention than the alternative of reusing an existing lock owned by the domain.
## Hierarchical Locking Model and Protocol
The locking discipline within the Argo code is heirarchical and utilizes reader/writer locks to enable increased concurrency when operations do not conflict. None of the Argo locks are reentrant.
The hierarchy:
* There is a global rwlock (L1) to protect access to all of the per-domain argo data structures.
* There is a rwlock per-domain (rings_L2) to protect the hashtable of the per-ring data structures.
* There is a lock per ring (L3) to protect the per-ring data structure, struct argo_ring_info.
There are a two other per-domain L2 locks; their operation is similar and they are described later.
The protocol to safely acquire write access to the per-ring data structure, struct argo_ring_info, is:
1) Acquire a Read lock on L1.
2) Acquire a Read lock on L2.
3) Acquire L3.
An alternative valid sequence is:
1) Acquire a Read lock on L1.
2) Acquire a Write lock on L2.
This second sequence grants write access to _all_ of the argo_ring_info structs belonging to the domain, but at the expense of less concurrency: no other operation can access those structs while the locks are held, which will inhibit operations on those rings until the locks are released.
Another alternative valid sequence is:
1) Acquire a Write lock on L1.
This grants write access to _all_ of the argo_ring_info structs belonging to _all domains_, but again at the expense of far less concurrency: no other operation can operate on Argo rings until the locks are released.
## Lock Definitions
The full set of locks that are directly operated upon by the Argo code are described in the following section.
### The global singleton lock:
* L1_global_argo_rwlock
The rationale for having a global lock is to be able to enforce system-wide exclusion for a critical region and simplify the logic required to avoid deadlock, for teardown of state across multiple domains when a domain is destroyed.
The majority of operations take a read-lock on this lock, allowing concurrent Argo operations by many domains.
The pointer d->argo on every domain is protected by this lock. A set of more granular per-domain locks could be used to do that, but since domain start and stop is expected to be a far less frequent operation than the other argo operations, acquiring a single read lock to enable access to all the argo structs of all domains simplifies the protocol.
Points of write-locking on this lock:
* argo_destroy, where:
* All of the domain's own rings are destroyed.
* All of the notifications pending for other domains are cancelled.
* All of the unicast partner rings owned by other domains for this domain to send to, are destroyed.
* All of the notifications pending on those rings are cancelled.
* All of the notifications pending for this domain on wildcard rings owned by other domains are cancelled.
* argo_soft_reset, for similar teardown operations as argo_destroy.
* argo_init, where the d->argo pointer is first populated.
* Since the write lock is taken here, there is serialization all concurrent Argo operations around this single pointer write; this is the cost of using the simpler one global lock approach.
Enforcing that the write_lock is acquired on L1_global_argo_rwlock before executing teardown, ensures that no teardown operations act concurrently and no other Argo operations happen concurrently with a teardown. The teardown logic is free to safely modify the Argo state across all domains without having to acquire per-domain locks and deadlock cannot occur.
### Per-Domain: Ring hash lock
rings_L2_rwlock
Protects: the per-domain ring hash table of argo_ring_info structs.
Holding a read lock on rings_L2 protects the ring hash table and the elements in the hash table d->argo->ring_hash, and the node and id fields in struct argo_ring_info in the hash table.
Holding a write lock on rings_L2 protects all of the elements of all the struct argo_ring_info belonging to this domain.
To take rings_L2 you must already have R(L1). W(L1) implies W(rings_L2) and L3.
Prerequisites:
* R(L1_global_argo_rwlock) must be acquired before taking either read or write on rings_L2_rwlock.
* W(L1_global_argo_rwlock) implies W(rings_L2_rwlock), so if W(L1_global_argo_rwlock) is held, then rings_L2_rwlock does not need to be acquired, and all the data structures that rings_L2_rwlock protects can be accessed as if W(ring_L2_rwlock) was held.
Is accessed by the hypervisor on behalf of:
* The domain that registered the ring.
* Any domain that is allowed to send to the ring -- so that's the partner domain, for unicast rings, or any domain, for wildcard rings.
### Send hash lock
send_L2_lock
Protects: the per-domain send hash table of argo_send_info structs.
Is accessed by the hypervisor on behalf of:
* Any domain that registers a ring that specifies the domain as the unicast sender.
* The domain that has been allowed to send, as part of teardown when the domain is being destroyed.
### Wildcard pending list lock
wildcard_L2_lock
Protects: the per-domain list of pending notifications to the domain from wildcard rings owned by other domains.
Is accessed by the hypervisor on behalf of:
* The domain that issued a query to another about space availability in one of its wildcard rings - this can be done by attempting a send operation when there is insufficient ring space available at the time.
* Any domain that the domain has issued a query to about space availability in one of their wildcard rings.
### Per-Ring locks:
* L3_lock
This lock protects the members of a struct ring_info which is the primary state for a domain's own registered ring.
## Reasoning Model
A common model for reasoning about concurrent code focusses on accesses to individual variables: if code touches this variable, see that it first acquires the corresponding lock and then drops it afterwards. A challenge with this model is in ensuring that the sequence of locks acquired within nested functions, when operating on data from multiple domains with concurrent operations, is safe from deadlock.
An alternative method that is better suited to the Argo software is to consider the execution path, the full sequence of locks acquired, accesses performed, and locks released, from entering an operation, to the completion of the work.
An example code path for an operation:
[entry] > -- [ take R(L1) ] -- [ take R(L2) ] -- loop [ take a L3 / drop L3 ] -- [ drop R(L2) ] -- [ drop R(L1)] -- > [exit]
If a function implements a section of the path, it is important to know not only what variables the function itself operates upon, but also the locking state that will already have been established at the point when the function is invoked, since this will affect what data the function can access. For this reason, comments in the code, or ASSERTs that explicitly check lock state, communicate what the locking state is expected and intended to be when that code is invoked. See the macros defined to support this for Argo later in this document.
## Macros to Validate and Document Lock State
These macros encode the logic to verify that the locking has adhered to the locking discipline.
eg. On entry to logic that requires holding at least R(rings_L2), this:
ASSERT(LOCKING_Read_rings_L2(d));
checks that the lock state is sufficient, validating that one of the following must be true when executed:
R(rings_L2) && R(L1)
or: `W(rings_L2) && R(L1)` or: `W(L1)`
The macros are defined thus:
``` /* RAW macros here are only used to assist defining the other macros below */ #define RAW_LOCKING_Read_L1 (rw_is_locked(&L1_global_argo_rwlock)) #define RAW_LOCKING_Read_rings_L2(d) \ (rw_is_locked(&d->argo->rings_L2_rwlock) && RAW_LOCKING_Read_L1) /* The LOCKING macros defined below here are for use at verification points */ #define LOCKING_Write_L1 (rw_is_write_locked(&L1_global_argo_rwlock)) #define LOCKING_Read_L1 (RAW_LOCKING_Read_L1 || LOCKING_Write_L1) #define LOCKING_Write_rings_L2(d) \ ((RAW_LOCKING_Read_L1 && rw_is_write_locked(&d->argo->rings_L2_rwlock)) || \ LOCKING_Write_L1) #define LOCKING_Read_rings_L2(d) \ ((RAW_LOCKING_Read_L1 && rw_is_locked(&d->argo->rings_L2_rwlock)) || \ LOCKING_Write_rings_L2(d) || LOCKING_Write_L1) #define LOCKING_L3(d, r) \ ((RAW_LOCKING_Read_rings_L2(d) && spin_is_locked(&r->L3_lock)) || \ LOCKING_Write_rings_L2(d) || LOCKING_Write_L1) #define LOCKING_send_L2(d) \ ((RAW_LOCKING_Read_L1 && spin_is_locked(&d->argo->send_L2_lock)) || \ LOCKING_Write_L1) ```
Here is an example of a macro in use:
```
static void
notify_ring(const struct domain *d, struct argo_ring_info *ring_info,
struct hlist_head *to_notify)
{
uint32_t space;
ASSERT(LOCKING_Read_rings_L2(d));
spin_lock(&ring_info->L3_lock);
if ( ring_info->len )
space = ringbuf_payload_space(d, ring_info);
else
space = 0;
spin_unlock(&ring_info->L3_lock);
if ( space )
pending_find(d, ring_info, space, to_notify);
}
```In the above example, it can be seen that it is safe to acquire the L3 lock because _at least_ R(rings_L2) is already held, as documented and verified by the macro.
## Appendix: FAQ / Other Considerations
### Why not have a single per-domain lock?
Due to performance isolation / DoS avoidance: if there is a single per-domain lock, acquiring this lock will stall operations on other active rings owned by the domain. A malicious domain can loop registering and unregistering rings, without any consent by the targetted domain, which would experience decreased throughput due to the contention on the single per-domain lock. The granular locking structure of Argo prevents this. It also allows concurrent operation of different rings by multiple VCPUs of the same domain without contention, to avoid negative application performance interaction.
## Rationale for Using a Singleton Global Lock: L1
### Teardown on domain destroy
The single global lock enables exclusive access to the argo data structures across domains when a domain is destroyed. Every unicast ring that the dying domain is the authorized sender is torn down and any pending space-available notifications in other domain's wildcard rings are cancelled. This requires gaining safe access to the data structures on each of the domains involved.
The 'send hashtable' data structure is needed in order to perform the teardown of rings when a domain is destroyed. To populate it, whenever a unicast ring is registered, the lock that protects that data structure must be taken exclusively.
There are granular per-domain locks which protect the per-domain data structures. The global singleton L1 lock operates with-and-above the per-domain locks and is used to obtain exclusive access to multiple domain's argo data structures in the infrequent case where it is used -- for domain destroy -- whilst otherwise allowing concurrent access, via acquiring it with 'read' access, for the majority of the time.
To perform the required state teardown on domain destruction, which can require removing state from the data structures of multiple domains, a locking protocol to obtain mutual exclusion and safe access to the state is required, without deadlocking.
Using the single global lock avoids the need for sequencing the acquisition of multiple individual per-domain locks (and lower level data structure locks) to prevent deadlock: taking W(L1) grants access to all and taking R(L1) ensures that teardown of any domain will not interfere with any Argo hypercall operation. It enables introducing granular locking without complex or error-prone lock acquisition logic.
=== end document ===
Quoted 4 lines
There are also several claims that fine-grainer locking provides better performance in order to justify the need of such locks. IMO without providing any evidence of such performance benefit it's hard to be convinced so many locks are needed.
Benchmarks would be useful for regression testing. We can investigate resourcing for Argo synthetic benchmarks in the Xen 4.13 release cycle. In the meantime, we can cite the shipment of Citrix XenClient in 2011, followed by customer production deployments of v4v in OpenXT and Bromium uXen (including HP business laptops). Argo is derived from v4v.
Ian Pratt's PSEC 2018 presentation [1] on hypervisor security indirectly referenced v4v and uXen isolation/performance requirements, it may illustrate the scope of possible Argo use cases. Click the "uXen" button below the video to navigate to the clip. There's also a uXen source code link. An excerpt (my annotations in []):
"PV device interfaces all built on simple hypervisor copy-based primitive
We came up with a very simple primitive [v4v] for communication between VMs, between VMs and the host, and then just built everything on that primitive. It's a simple copy-based primitive. We didn't want any memory sharing of any kind. Anything involving memory sharing ends up being complex ... Grant tables being a huge example ... a copy-based approach is just much simpler, and actually, performance-wise, it turns out being equivalent from a performance point of view ...
Getting rid of other terrible ideas, like Xenstore ... we still want the primitive [v4v] interface to be able to support things like device reconnection. That was a good idea, this idea that you can restart these driver domains, have VMs reconnect and be able to continue, that's very useful. We want to enable this simple primitive [v4v] to do that. Then just build everything using very simple, narrow interfaces."
[1] https://www.platformsecuritysummit.com/2018/speaker/pratt/
Regards,
Rich
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkQuoted 1 line · Christopher Clark
+partner_rings_remove(struct domain
*src_d)
<snip>
Quoted 1 line
+
ring_info=ring_find_info(dst_d, &send_info->id);
ring_find_info is defined later (PATCH 07/15), should it be moved to this patch since it is now used here?
Signature · 6 lines
Eric Chanudet _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 7 lines · Eric Chanudet
Quoted 1 line · Christopher Clark
+partner_rings_remove(struct domain
*src_d)<snip>
Quoted 1 line
+
ring_info=ring_find_info(dst_d, &send_info->id);
ring_find_infois defined later (PATCH 07/15), should it be moved to this patch since it is now used here?
Yes, you're right -- thanks -- and I should've caught that. Fixed and will be thorough with testing the series individually before posting again.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkQuoted 16 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +8
-0@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> +
xen/common/argo.c line 38
Quoted 10 lines
xen/common/argo.c +10
-0+DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */ +static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ + uint32_t port;
evtchn_port_t?
Quoted 19 lines
xen/common/argo.c +19
-0+static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); +
xen/common/argo.c line 300
Quoted 10 lines
xen/common/argo.c +10
-0+for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ; + ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ;
While it shouldn't matter with locking in use, I generally would consider it better if counts got set to zero before freeing the arrays.
Quoted 9 lines
xen/common/argo.c +3
-1long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; +
xen/common/argo.c line 391
Quoted 10 lines
xen/common/argo.c +10
-0+argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc; + } + +domain_lock (currd);
What is the rationale for using the domain lock here? We're trying to limit its use as much as possible, due to the otherwise heavy contention which can result, as it may be held for comparably long periods of time.
Quoted 10 lines
xen/common/argo.c +10
-0+int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + }
xen/common/argo.c line 444
Quoted 10 lines
xen/common/argo.c +10
-0+ +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain); +if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo);
I doubt the lock needs to be held for this function call.
xen/include/xlat.lst line 152
Quoted 8 lines
xen/include/xlat.lst +2
-0@@ -148,3 +148,5 @@ ? flask_setenforce xsm/flask_op.h ! flask_sid_context xsm/flask_op.h ? flask_transition xsm/flask_op.h +? argo_addr argo.h +? argo_ring argo.h
Did I overlook the use of what these cause to be generated?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 2 lines
Quoted 2 lines · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +12
-0@@ -17,7 +17,177 @@*/ #include <xen/errno.h> +#include <xen/sched.h> +#include <xen/domain.h> +#include <xen/argo.h> +#include <xen/event.h> +#include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/time.h> +#include <public/argo.h> + +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); + +/* Xen command line option to enable argo */
Quoted 8 lines
xen/common/argo.c line 38
Quoted 6 lines
xen/common/argo.c +6
-0+static bool __read_mostly opt_argo_enabled; +boolean_param ("argo" , opt_argo_enabled); + +typedef struct argo_ring_id +{ + uint32_t port;
evtchn_port_t?
No; so to avoid the potential for confusion, I've renamed that (and other places where 'port' was used) to 'aport', for "argo port", and added a definition for the type: xen_argo_port_t as uint32_t, so the distinction is clearer.
Quoted 26 lines
Quoted 26 lines
xen/common/argo.c +25
-0+static void +ring_remove_mfns (const struct domain *d,struct argo_ring_info *ring_info) +{ +unsigned int i; + +ASSERT (rw_is_write_locked (&d->argo->lock) || +rw_is_write_locked (&argo_lock)); + +if ( !ring_info->mfns ) +return ; + +if ( !ring_info->mfn_mapping ) + { +ASSERT_UNREACHABLE (); +return ; + } + +ring_unmap (ring_info); + +for ( i =0 ; i < ring_info->nmfns; i++ ) +if ( !mfn_eq (ring_info->mfns[i],INVALID_MFN ) ) +put_page_and_type (mfn_to_page (ring_info->mfns[i])); + +xfree (ring_info->mfns); + ring_info->mfns =NULL ;
Quoted 8 lines
xen/common/argo.c line 300
Quoted 4 lines
xen/common/argo.c +4
-0+ ring_info->npage =0 ; +xfree (ring_info->mfn_mapping); + ring_info->mfn_mapping =NULL ; + ring_info->nmfns =0 ;While it shouldn't matter with locking in use, I generally would consider it better if counts got set to zero before freeing the arrays.
ack, done.
Quoted 17 lines
Quoted 17 lines
xen/common/argo.c +10
-1long do_argo_op (unsigned int cmd,XEN_GUEST_HANDLE_PARAM (void ) arg1,XEN_GUEST_HANDLE_PARAM (void ) arg2,unsigned long arg3,unsigned long arg4) { -return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4); + +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ; +return rc;
Quoted 8 lines
xen/common/argo.c line 391
Quoted 3 lines
xen/common/argo.c +3
-0+ } + +domain_lock (currd);What is the rationale for using the domain lock here? We're trying to limit its use as much as possible, due to the otherwise heavy contention which can result, as it may be held for comparably long periods of time.
My inference is that was intended for avoiding interaction between the hypercall ops and domain destroy, but it is not necessary. I've removed it. Thanks.
Quoted 15 lines
Quoted 15 lines
xen/common/argo.c +14
-0+int +argo_init (struct domain *d) +{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain);
Quoted 8 lines
xen/common/argo.c line 444
Quoted 6 lines
xen/common/argo.c +6
-0+if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo);I doubt the lock needs to be held for this function call.
ack; have reordered it to not do that.
Quoted 11 lines
xen/include/xlat.lst line 152
Quoted 9 lines
xen/include/xlat.lst +2
-0@@ -148,3 +148,5 @@ ? flask_setenforce xsm/flask_op.h ! flask_sid_context xsm/flask_op.h ? flask_transition xsm/flask_op.h +? argo_addr argo.h +? argo_ring argo.hDid I overlook the use of what these cause to be generated?
The last commit in the v3 series added a file to make use of it but I've now melded that into the commit series now, so coverage will be introduced as hypercall argument types are added in the next series.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkHi all
The locking scheme seems to be remaining sticking point. The rest are mostly cosmetic issues (FAOD, they still need to be addressed). Frankly I don't think there is enough time to address all the technical details, but let me sum up each side's position and see if we can reach an amicable solution.
From maintainers and reviewers' point of view:
1. Maintainers / reviewers don't like complexity unless absolutely necessary.
2. Maintainers / reviewers feel they have a responsibility to understand the code and algorithm.
Yet being the gatekeepers doesn't necessarily mean we understand every technical details and every usecase. We would like to, but most of the time it is unrealistic.
Down to this specific patch series:
Roger thinks the locking scheme is too complex. Christopher argues that's necessary for short-live channels to be performant.
Both have their point.
I think having a complex locking scheme is inevitable, just like we did for performant grant table several years ago. Regardless of the timing issue we have at hand, asking Christopher to implement a stripped down version creates more work for him.
Yet ignoring Roger's concerns is unfair to him as well, since he put in so much time and effort to understand the algorithm and provide suggestions. It is in fact unreasonable to ask anyone to fully understand the locking mechanism and check the implementation is correct in a few days (given the series was posted in Dec and there were major holidays in between, plus everyone had other commitments).
To unblock this, how about we make Christopher maintainer of Argo? He and OpenXT will be on the hook for further improvement. And I believe it would be in their best interest to keep Argo bug-free and eventually make it become supported.
So:
1. Make sure Argo is self-contained -- this requires careful review for interaction between Argo and other parts of the hypervisor.
2. Argo is going to be experimental and off-by-default -- this is the default status for new feature anyway.
3. Make Christopher maintainer of Argo -- this would be a natural thing to do anyway.
Does this work for everyone?
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Adding Juergen,
to make sure he is aware of the discussion, as this is an excellent summary of the status of this series.
Quoted 9 lines · Wei Liu
Hi all
The locking scheme seems to be remaining sticking point. The rest are mostly cosmetic issues (FAOD, they still need to be addressed). Frankly I don't think there is enough time to address all the technical details, but let me sum up each side's position and see if we can reach an amicable solution.
snip
Quoted 7 lines
To unblock this, how about we make Christopher maintainer of Argo? He and OpenXT will be on the hook for further improvement. And I believe it would be in their best interest to keep Argo bug-free and eventually make it become supported.
So:
Quoted 8 lines
1. Make sure Argo is self-contained -- this requires careful review for interaction between Argo and other parts of the hypervisor.
2. Argo is going to be experimental and off-by-default -- this is the default status for new feature anyway.
3. Make Christopher maintainer of Argo -- this would be a natural thing to do anyway.Does this work for everyone?
I think this is a good way forward. Thank you Wei for putting this mail together.
Best Regards
Lars
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Wei LiuQuoted 46 lines · Wei Liu
Hi all
The locking scheme seems to be remaining sticking point. The rest are mostly cosmetic issues (FAOD, they still need to be addressed). Frankly I don't think there is enough time to address all the technical details, but let me sum up each side's position and see if we can reach an amicable solution.
From maintainers and reviewers' point of view:
1. Maintainers / reviewers don't like complexity unless absolutely necessary.
2. Maintainers / reviewers feel they have a responsibility to understand the code and algorithm.Yet being the gatekeepers doesn't necessarily mean we understand every technical details and every usecase. We would like to, but most of the time it is unrealistic.
Down to this specific patch series:
Roger thinks the locking scheme is too complex. Christopher argues that's necessary for short-live channels to be performant.
Both have their point.
I think having a complex locking scheme is inevitable, just like we did for performant grant table several years ago. Regardless of the timing issue we have at hand, asking Christopher to implement a stripped down version creates more work for him.
Yet ignoring Roger's concerns is unfair to him as well, since he put in so much time and effort to understand the algorithm and provide suggestions. It is in fact unreasonable to ask anyone to fully understand the locking mechanism and check the implementation is correct in a few days (given the series was posted in Dec and there were major holidays in between, plus everyone had other commitments).
To unblock this, how about we make Christopher maintainer of Argo? He and OpenXT will be on the hook for further improvement. And I believe it would be in their best interest to keep Argo bug-free and eventually make it become supported.
So:
Quoted 8 lines
1. Make sure Argo is self-contained -- this requires careful review for interaction between Argo and other parts of the hypervisor.
2. Argo is going to be experimental and off-by-default -- this is the default status for new feature anyway.
3. Make Christopher maintainer of Argo -- this would be a natural thing to do anyway.Does this work for everyone?
Yes, this will work for me. Thanks for the summary and proposal.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Wei LiuQuoted 46 lines · Wei Liu
Hi all
The locking scheme seems to be remaining sticking point. The rest are mostly cosmetic issues (FAOD, they still need to be addressed). Frankly I don't think there is enough time to address all the technical details, but let me sum up each side's position and see if we can reach an amicable solution.
From maintainers and reviewers' point of view:
1. Maintainers / reviewers don't like complexity unless absolutely necessary.
2. Maintainers / reviewers feel they have a responsibility to understand the code and algorithm.Yet being the gatekeepers doesn't necessarily mean we understand every technical details and every usecase. We would like to, but most of the time it is unrealistic.
Down to this specific patch series:
Roger thinks the locking scheme is too complex. Christopher argues that's necessary for short-live channels to be performant.
Both have their point.
I think having a complex locking scheme is inevitable, just like we did for performant grant table several years ago. Regardless of the timing issue we have at hand, asking Christopher to implement a stripped down version creates more work for him.
Yet ignoring Roger's concerns is unfair to him as well, since he put in so much time and effort to understand the algorithm and provide suggestions. It is in fact unreasonable to ask anyone to fully understand the locking mechanism and check the implementation is correct in a few days (given the series was posted in Dec and there were major holidays in between, plus everyone had other commitments).
To unblock this, how about we make Christopher maintainer of Argo? He and OpenXT will be on the hook for further improvement. And I believe it would be in their best interest to keep Argo bug-free and eventually make it become supported.
So:
Quoted 8 lines
1. Make sure Argo is self-contained -- this requires careful review for interaction between Argo and other parts of the hypervisor.
2. Argo is going to be experimental and off-by-default -- this is the default status for new feature anyway.
3. Make Christopher maintainer of Argo -- this would be a natural thing to do anyway.Does this work for everyone?
I think this is fine.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Wei LiuQuoted 44 lines · Wei Liu
Hi all
The locking scheme seems to be remaining sticking point. The rest are mostly cosmetic issues (FAOD, they still need to be addressed). Frankly I don't think there is enough time to address all the technical details, but let me sum up each side's position and see if we can reach an amicable solution.
From maintainers and reviewers' point of view:
1. Maintainers / reviewers don't like complexity unless absolutely necessary.
2. Maintainers / reviewers feel they have a responsibility to understand the code and algorithm.Yet being the gatekeepers doesn't necessarily mean we understand every technical details and every usecase. We would like to, but most of the time it is unrealistic.
Down to this specific patch series:
Roger thinks the locking scheme is too complex. Christopher argues that's necessary for short-live channels to be performant.
Both have their point.
I think having a complex locking scheme is inevitable, just like we did for performant grant table several years ago. Regardless of the timing issue we have at hand, asking Christopher to implement a stripped down version creates more work for him.
Yet ignoring Roger's concerns is unfair to him as well, since he put in so much time and effort to understand the algorithm and provide suggestions. It is in fact unreasonable to ask anyone to fully understand the locking mechanism and check the implementation is correct in a few days (given the series was posted in Dec and there were major holidays in between, plus everyone had other commitments).
To unblock this, how about we make Christopher maintainer of Argo? He and OpenXT will be on the hook for further improvement. And I believe it would be in their best interest to keep Argo bug-free and eventually make it become supported.
Quoted 9 lines
So:
1. Make sure Argo is self-contained -- this requires careful review for interaction between Argo and other parts of the hypervisor.
2. Argo is going to be experimental and off-by-default -- this is the default status for new feature anyway.
3. Make Christopher maintainer of Argo -- this would be a natural thing to do anyway.
Wei,
do you have any feedback on the latest argo MAINTAINERS patch?
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 4 lines · Christopher Clark
Wei,
do you have any feedback on the latest argo
MAINTAINERSpatch?
It looks fine to me.
Wei.
Quoted 2 lines
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Christopher ClarkQuoted 36 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.html
Quoted 8 lines
The following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>
I hope I've not trodden on the toes of any other reviews. I've got some minor requests, but hopefully its all fairly trivial.
Quoted 8 lines
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 194
Quoted 10 lines
docs/misc/xen-command-line.pandoc +10
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains.
Please do include a note about CONFIG_ARGO. I know this doc is inconsistent on the matter (as Kconfig postdates the written entries here), but I have been trying to fix up, and now about half of the documentation does mention appropriate Kconfig information.
xen/common/argo.c line 375
Quoted 6 lines
xen/common/argo.c +0
-0long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,
I know I'm commenting on the wrong patch, but please use unsigned long cmd, so the type definition here doesn't truncate the caller provided value. We have similar buggy code all over Xen, but its too late to fix that, and I'd prefer not to propagate the error.
xen/common/argo.c line 383
Quoted 9 lines
XEN_GUEST_HANDLE_PARAM(void)arg2, unsigned long arg3, unsigned long arg4)
{xen/common/argo.c +5
-1-return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4);
For debugging purposes, you don't want to truncate any of these values, or you'll have a print message which doesn't match what the guest provided. I'd use %ld for arg3 and arg4.
xen/common/argo.c line 387
Quoted 4 lines
xen/common/argo.c +4
-0+ +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ;
Shouldn't this be ENOSYS instead? There isn't a conceptual difference between CONFIG_ARGO compiled out, and opt_argo clear on the command line, and I don't think a guest should be able to tell the difference.
Quoted 29 lines
xen/common/argo.c +29
-0+return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; +
xen/common/argo.c line 426
Quoted 10 lines
xen/common/argo.c +10
-0+for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +} + +int +argo_init (struct domain *d)
Are there any per-vcpu argo resources? I don't see any in the series, but I'd be tempted to name the external functions as argo_domain_{init,destroy}() which is slightly clearer in the caller context.
xen/common/argo.c line 438
Quoted 12 lines
xen/common/argo.c +12
-0+{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain);
For sanity sake, I'd suggest xzalloc() here, not that I can spot anything wrong with the current code.
xen/common/argo.c line 444
Quoted 6 lines
xen/common/argo.c +6
-0+if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo);
This call doesn't need to be within the global argo_lock critical region, because it exclusively operates on state which is inaccessible to the rest of the system until d->argo is written. This then shrinks the critical region to a single pointer write. (Further, with a patch I haven't posted yet, the memset(0) in zxalloc() can be write-merged with the setup code to avoid repeated writes, which can't happen with a spinlock in between.)
xen/common/argo.c line 454
Quoted 10 lines
xen/common/argo.c +10
-0+ + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d)
Is this function fully idempotent? Given its current calling context, it needs to be. (I think it is, but I just want to double check, because it definitely wants to be.)
I ask, because...
Quoted 55 lines
xen/common/argo.c +41
-0+{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ; + } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +1
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h>
xen/common/domain.c static void _domain_destroy(struct domain *d) · line 283
Quoted 6 lines
xen/common/domain.c +3
-0@@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif
... given this call (which is correct), ...
Quoted 27 lines
xen/common/domain.c +9
-0+rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);@@ -376,6 +381,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;@@ -445,6 +453,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );
xen/common/domain.c int domain_kill(struct domain *d) · line 735
Quoted 6 lines
xen/common/domain.c +3
-0@@ -717,6 +730,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif
... this one isn't necessary.
I'm in the middle of fixing all this destruction logic, and _domain_destroy() is called below.
The rule is that everything in _domain_destroy() should be idempotent, and all destruction logic needs moving there, so I can remove DOMCTL_setmaxvcpus and fix a load of toolstack-triggerable NULL pointer dereferences in Xen.
Eventually, everything in this hunk will disappear.
Quoted 20 lines
xen/common/domain.c +0
-0evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);xen/include/xen/argo.h new file +11
-0@@ -0,0 +1,23 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License
xen/include/xen/argo.h line 21
Quoted 10 lines
xen/include/xen/argo.h new file +10
-0+* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__ + +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); +void argo_soft_reset (struct domain *d);
Instead of the #ifdefary in the calling code, please could you stub these out in this file? See the tail of include/asm-x86/pv/domain.h for an example based on CONFIG_PV.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 14.01.19 at 15:58, <[email protected]> wrote:
xen/common/argo.c line 375
Quoted 4 lines · Christopher Clark
xen/common/argo.c +0
-0long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,I know I'm commenting on the wrong patch, but please use unsigned long cmd, so the type definition here doesn't truncate the caller provided value. We have similar buggy code all over Xen, but its too late to fix that, and I'd prefer not to propagate the error.
Why buggy? It all depends on how the interface is specified. If the input is 32 bits wide, it is clear that higher bits are supposed to be ignored. Nothing says that the full register width is significant.
Quoted 10 lines
xen/common/argo.c line 383
Quoted 9 lines
XEN_GUEST_HANDLE_PARAM(void)arg2, unsigned long arg3, unsigned long arg4)
{xen/common/argo.c +5
-1-return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4);
Quoted 3 lines
For debugging purposes, you don't want to truncate any of these values, or you'll have a print message which doesn't match what the guest provided. I'd use %ld for arg3 and arg4.
Perhaps better %lx, for the output being easier to recognize for both bitmaps (e.g. flag values) and sufficiently large values.
Quoted 8 lines
xen/common/argo.c line 387
Quoted 4 lines
xen/common/argo.c +4
-0+ +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ;Shouldn't this be ENOSYS instead? There isn't a conceptual difference between
CONFIG_ARGOcompiled out, andopt_argoclear on the command line, and I don't think a guest should be able to tell the difference.
I admit it's a boundary case, but I think ENOSYS should strictly only ever be (and have been) returned for unrecognized major hypercall numbers.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 13 lines · Jan Beulich
Quoted 12 lines
Quoted 2 lines
Quoted 2 lines · On 14.01.19 at 15:58, <[email protected]> wrote:
xen/common/argo.c line 375
Quoted 4 lines · Christopher Clark
xen/common/argo.c +0
-0long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,I know I'm commenting on the wrong patch, but please use unsigned long cmd, so the type definition here doesn't truncate the caller provided value. We have similar buggy code all over Xen, but its too late to fix that, and I'd prefer not to propagate the error.
Quoted 4 lines
Why buggy? It all depends on how the interface is specified. If the input is 32 bits wide, it is clear that higher bits are supposed to be ignored. Nothing says that the full register width is significant.
I've left this as is (ie. unsigned int) but I can change it if it should change.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
XEN_GUEST_HANDLE_PARAM(void)arg2, unsigned long arg3, unsigned long arg4)
{xen/common/argo.c +5
-1-return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4);
Quoted 6 lines
Quoted 3 lines
For debugging purposes, you don't want to truncate any of these values, or you'll have a print message which doesn't match what the guest provided. I'd use %ld for arg3 and arg4.
Perhaps better %lx, for the output being easier to recognize for both bitmaps (e.g. flag values) and sufficiently large values.
ack, done
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
xen/common/argo.c +4
-0+ +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ;
Quoted 7 lines
Quoted 3 lines
Shouldn't this be ENOSYS instead? There isn't a conceptual difference between
CONFIG_ARGOcompiled out, andopt_argoclear on the command line, and I don't think a guest should be able to tell the difference.I admit it's a boundary case, but I think ENOSYS should strictly only ever be (and have been) returned for unrecognized major hypercall numbers.
ack
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
In reply to Andrew CooperQuoted 58 lines · Andrew Cooper
Quoted 36 lines · Christopher Clark
Initialises basic data structures and performs teardown of argo state for domain shutdown.
Inclusion of the Argo implementation is dependent on
CONFIG_ARGO.Introduces a new Xen command line parameter 'argo': bool to enable/disable the argo hypercall. Defaults to disabled.
New headers:
public/argo.h: with definions of addresses and ring structure, including indexes for atomic update for communication between domain and hypervisor.
xen/argo.h: to expose the hooks for integration into domain lifecycle:
argo_init: per-domain init of argo data structures fordomain_create.
argo_destroy: teardown fordomain_destroyand the error exit
path ofdomain_create.
argo_soft_reset: reset of domain state fordomain_soft_reset.Adds two new fields to struct domain:
rwlock_t argo_lock; struct argo_domain *argo;In accordance with recent work on
_domain_destroy,argo_destroyis idempotent. It will tear down: all rings registered by this domain, all rings where this domain is the single sender (ie. specified partner, non-wildcard rings), and all pending notifications where this domain is awaiting signal about available space in the rings of other domains.A count will be maintained of the number of rings that a domain has registered in order to limit it below the fixed maximum limit defined here.
The software license on the public header is the BSD license, standard procedure for the public Xen headers. The public header was originally posted under a GPL license at: [1]:
https://lists.xenproject.org/archives/html/xen-devel/2013-05/msg02710.htmlQuoted 8 lines
The following ACK by Lars Kurth is to confirm that only people being employees of Citrix contributed to the header files in the series posted at
[1] and that thus the copyright of the files in question is fully owned by Citrix. The ACK also confirms that Citrix is happy for the header files to be published under a BSD license in this series (which is based on [1]).Signed-off-by Christopher Clark <[email protected]>Acked-by Lars Kurth <[email protected]>I hope I've not trodden on the toes of any other reviews. I've got some minor requests, but hopefully its all fairly trivial.
Quoted 8 lines
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.
Quoted 15 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 194
Quoted 10 lines
docs/misc/xen-command-line.pandoc +10
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains.Please do include a note about
CONFIG_ARGO. I know this doc is inconsistent on the matter (asKconfigpostdates the written entries here), but I have been trying to fix up, and now about half of the documentation does mention appropriateKconfiginformation.
Ack, note added.
Quoted 12 lines
xen/common/argo.c line 375
Quoted 7 lines
xen/common/argo.c +0
-0long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,I know I'm commenting on the wrong patch, but please use unsigned long cmd, so the type definition here doesn't truncate the caller provided value. We have similar buggy code all over Xen, but its too late to fix that, and I'd prefer not to propagate the error.
On this one, given Jan's reply, I've left it as is for the series that I'll push tonight. That patch is carrying an Ack from Jan at the moment, so I wasn't going to touch it if it's not required. If there's consensus that it should change, let me know and I'll switch it.
Quoted 11 lines
xen/common/argo.c line 383
Quoted 10 lines
XEN_GUEST_HANDLE_PARAM(void)arg2, unsigned long arg3, unsigned long arg4)
{xen/common/argo.c +5
-1-return -ENOSYS ; +struct domain *currd = current->domain; +long rc = -EFAULT ; + +argo_dprintk ("->do_argo_op(%u,%p,%p,%d,%d)\n" , cmd, + (void *)arg1.p, (void *)arg2.p, (int ) arg3, (int ) arg4);
Quoted 3 lines
For debugging purposes, you don't want to truncate any of these values, or you'll have a print message which doesn't match what the guest provided. I'd use %ld for arg3 and arg4.
I've gone with %lu for arg3 and %lx for arg4, for the next round, but I could be missing something: is there a reason to prefer '%ld' over '%lu' for using with an unsigned type? Should I be using %d elsewhere, eg. for a domid?
Quoted 9 lines
xen/common/argo.c line 387
Quoted 5 lines
xen/common/argo.c +4
-0+ +if (unlikely (!opt_argo_enabled) ) + { + rc = -EOPNOTSUPP ;Shouldn't this be ENOSYS instead? There isn't a conceptual difference between
CONFIG_ARGOcompiled out, andopt_argoclear on the command line, and I don't think a guest should be able to tell the difference.
EOPNOTSUPP has been retained here, as per an earlier review and also a response later in this thread.
Quoted 37 lines
Quoted 37 lines
xen/common/argo.c +36
-0+return rc; + } + +domain_lock (currd); + +switch (cmd) + { +default : + rc = -EOPNOTSUPP ; +break ; + } + +domain_unlock (currd); + +argo_dprintk ("<-do_argo_op(%u)=%ld\n" , cmd, rc); + +return rc; +} + +static void +argo_domain_init (struct argo_domain *argo) +{ +unsigned int i; + +rwlock_init (&argo->lock); +spin_lock_init (&argo->send_lock); +spin_lock_init (&argo->wildcard_lock); + argo->ring_count =0 ; + +for ( i =0 ; i <ARGO_HTABLE_SIZE ; ++i ) + { +INIT_HLIST_HEAD (&argo->ring_hash[i]); +INIT_HLIST_HEAD (&argo->send_hash[i]); + } +INIT_HLIST_HEAD (&argo->wildcard_pend_list); +}
Quoted 8 lines
xen/common/argo.c line 426
Quoted 3 lines
xen/common/argo.c +3
-0+ +int +argo_init (struct domain *d)Are there any per-vcpu argo resources? I don't see any in the series, but I'd be tempted to name the external functions as argo_domain_{init,destroy}() which is slightly clearer in the caller context.
I haven't renamed this as I'm out of time today but I slightly wary about whether it ends up helping clarity, given 1) the name of an existing data type 'argo_domain' and 2) the other similar init functions that are invoked in domain_create don't seem to follow the <subsystem>_domain_init naming scheme. Are you sure I should do this?
Quoted 16 lines
xen/common/argo.c line 438
Quoted 13 lines
xen/common/argo.c +12
-0+{ +struct argo_domain *argo; + +if ( !opt_argo_enabled ) + { +argo_dprintk ("argo disabled, domid: %d\n" , d->domain_id); +return 0 ; + } + +argo_dprintk ("init: domid: %d\n" , d->domain_id); + + argo = xmalloc(struct argo_domain);For sanity sake, I'd suggest
xzalloc()here, not that I can spot anything wrong with the current code.
ack, done.
Quoted 12 lines
xen/common/argo.c line 444
Quoted 7 lines
xen/common/argo.c +6
-0+if ( !argo ) +return -ENOMEM ; + +write_lock (&argo_lock); + +argo_domain_init (argo);This call doesn't need to be within the global
argo_lockcritical region, because it exclusively operates on state which is inaccessible to the rest of the system untild->argois written. This then shrinks the critical region to a single pointer write.
ack, done
Quoted 5 lines
(Further, with a patch I
haven't posted yet, the memset(0) inzxalloc()can be write-merged with the setup code to avoid repeated writes, which can't happen with a spinlock in between.)
Quoted 14 lines
xen/common/argo.c line 454
Quoted 10 lines
xen/common/argo.c +10
-0+ + d->argo = argo; + +write_unlock (&argo_lock); + +return 0 ; +} + +void +argo_destroy (struct domain *d)Is this function fully idempotent? Given its current calling context, it needs to be. (I think it is, but I just want to double check, because it definitely wants to be.)
I think so and it is intended to be - it takes a lock, only does work if a pointer isn't NULL, and then NULLs it before dropping the lock, so should be ok.
Quoted 106 lines
I ask, because...
Quoted 55 lines
xen/common/argo.c +41
-0+{ +BUG_ON (!d->is_dying); + +write_lock (&argo_lock); + +argo_dprintk ("destroy: domid %d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); +xfree (d->argo); + d->argo =NULL ; + } +write_unlock (&argo_lock); +} + +void +argo_soft_reset (struct domain *d) +{ +write_lock (&argo_lock); + +argo_dprintk ("soft reset d=%d d->argo=%p\n" , d->domain_id, d->argo); + +if ( d->argo ) + { +domain_rings_remove_all (d); +partner_rings_remove (d); +wildcard_rings_pending_remove (d); + +if ( !opt_argo_enabled ) + { +xfree (d->argo); + d->argo =NULL ; + } +else +argo_domain_init (d->argo); + } + +write_unlock (&argo_lock); }xen/common/domain.c +1
-0@@ -32,6 +32,7 @@#include <xen/grant_table.h> #include <xen/xenoprof.h> #include <xen/irq.h> +#include <xen/argo.h> #include <asm/debugger.h> #include <asm/p2m.h> #include <asm/processor.h> xen/common/domain.c static void _domain_destroy(struct domain *d) · line 283
Quoted 6 lines
xen/common/domain.c +3
-0@@ -277,6 +278,10 @@ static void _domain_destroy(struct domain *d)xfree (d->pbuf); +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif ... given this call (which is correct), ...
Quoted 27 lines
xen/common/domain.c +9
-0+rangeset_domain_destroy (d);free_cpumask_var (d->dirty_cpumask);@@ -376,6 +381,9 @@ struct domain *domain_create(domid_t domid,spin_lock_init (&d->hypercall_deadlock_mutex);INIT_PAGE_LIST_HEAD (&d->page_list);INIT_PAGE_LIST_HEAD (&d->xenpage_list); +#ifdef CONFIG_ARGO +rwlock_init (&d->argo_lock); +#endif spin_lock_init (&d->node_affinity_lock); d->node_affinity =NODE_MASK_ALL ;@@ -445,6 +453,11 @@ struct domain *domain_create(domid_t domid, goto fail; init_status |= INIT_gnttab; +#ifdef CONFIG_ARGO +if ( (err =argo_init (d)) !=0 ) + goto fail; +#endif + err = -ENOMEM ; d->pbuf =xzalloc_array (char,DOMAIN_PBUF_SIZE );xen/common/domain.c int domain_kill(struct domain *d) · line 735
Quoted 6 lines
xen/common/domain.c +3
-0@@ -717,6 +730,9 @@ int domain_kill(struct domain *d)if ( d->is_dying != DOMDYING_alive )return domain_kill (d); d->is_dying = DOMDYING_dying; +#ifdef CONFIG_ARGO +argo_destroy (d); +#endif ... this one isn't necessary.
I'm in the middle of fixing all this destruction logic, and
_domain_destroy()is called below.
Quoted 6 lines
The rule is that everything in
_domain_destroy()should be idempotent, and all destruction logic needs moving there, so I can removeDOMCTL_setmaxvcpusand fix a load of toolstack-triggerable NULL pointer dereferences in Xen.Eventually, everything in this hunk will disappear.
Thanks for the guidance. I've added a FIXME for this for the series I'm about to push and will work on resolving it tomorrow.
Quoted 27 lines
Quoted 27 lines
xen/common/domain.c +0
-0evtchn_destroy (d);gnttab_release_mappings (d);tmem_destroy (d->tmem_client);xen/include/xen/argo.h new file +17
-0@@ -0,0 +1,23 @@ +/****************************************************************************** +* Argo : Hypervisor-Mediated data eXchange +* +* Copyright (c) 2018, BAE Systems +* +* This program is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU General Public License for more details. +* +* You should have received a copy of the GNU General Public License +* along with this program; if not, write to the Free Software +* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA +*/ + +#ifndef __XEN_ARGO_H__ +#define __XEN_ARGO_H__
Quoted 8 lines
xen/include/xen/argo.h line 21
Quoted 4 lines
xen/include/xen/argo.h new file +4
-0+ +int argo_init (struct domain *d); +void argo_destroy (struct domain *d); +void argo_soft_reset (struct domain *d);Instead of the #ifdefary in the calling code, please could you stub these out in this file? See the tail of
include/asm-x86/pv/domain.hfor an example based onCONFIG_PV.
ack, done at Roger's earlier request.
Thanks for the review,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 15.01.19 at 08:21, <[email protected]> wrote:
Quoted 7 lines · Andrew Cooper
Quoted 6 lines · Christopher Clark
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.Quoted 15 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 194
Quoted 10 lines
docs/misc/xen-command-line.pandoc +10
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains.Please do include a note about
CONFIG_ARGO. I know this doc is inconsistent on the matter (asKconfigpostdates the written entries here), but I have been trying to fix up, and now about half of the documentation does mention appropriateKconfiginformation.
Quoted 1 line
Ack, note added.
Just to voice my view here: While I agree that some form of indication should be added, I don't think CONFIG_ARGO should be mentioned. CONFIG_* in general are likely meaningless to the main audience of this file (admins rather than developers). Hence the wording should be mostly independent of the precise config option name; there may then be an annotation naming the option. Omitting the option name, otoh, has the benefit of not bearing the risk of going stale.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 15.01.19 at 08:21, <[email protected]> wrote:
Quoted 21 lines · Andrew Cooper
Quoted 6 lines · Christopher Clark
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" support on compatible hardware in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newer hardware.docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 194
Quoted 10 lines
docs/misc/xen-command-line.pandoc +10
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supports registration +of memory rings with the hypervisor to receive messages, sending messages to +other domains by hypercall and querying the ring status of other domains.Please do include a note about
CONFIG_ARGO. I know this doc is inconsistent on the matter (asKconfigpostdates the written entries here), but I have been trying to fix up, and now about half of the documentation does mention appropriateKconfiginformation.
Quoted 8 lines
Quoted 1 line
Ack, note added.
Just to voice my view here: While I agree that some form of indication should be added, I don't think
CONFIG_ARGOshould be mentioned. CONFIG_* in general are likely meaningless to the main audience of this file (admins rather than developers). Hence the wording should be mostly independent of the precise config option name; there may then be an annotation naming the option. Omitting the option name, otoh, has the benefit of not bearing the risk of going stale.
I completely disagree. The exact CONFIG_ name is very important for the end user who is looking at the documentation wondering "why can't I seem to enable ARGO?"
"This option is only available when CONFIG_ARGO is compiled in" isn't going to put anyone off reading the document, but is useful for some who are reading it.
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 04/15] argo: init, destroy and soft-reset, with enable command line opt
Quoted 39 lines
Quoted 1 line
Quoted 1 line · On 15.01.19 at 10:06, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 15.01.19 at 08:21, <[email protected]> wrote:
On Mon, Jan 14, 2019 at 6:58 AM Andrew Cooper <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines · Christopher Clark
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support
Quoted 3 lines
docs/misc/xen-command-line.pandoc +0
-0@@ -182,6 +182,17 @@ Permit Xen to use "Always Running APIC Timer" supporton compatible hardware
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support · line 183
Quoted 2 lines
docs/misc/xen-command-line.pandoc +0
-0in combination with cpuidle. This option is only expected to be useful for developers wishing Xen to fall back to older timing methods on newerhardware.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware
Quoted 9 lines
docs/misc/xen-command-line.pandoc +8
-0+### argo +> `= <boolean>` + +> Default: `false` + +Enable the Argo hypervisor-mediated interdomain communication mechanism. + +This allows domains access to the Argo hypercall, which supportsregistration
Quoted 1 line
Quoted 1 line
Quoted 1 line
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support · line 192
Quoted 1 line
docs/misc/xen-command-line.pandoc +1
-0+of memory rings with the hypervisor to receive messages, sending messagesto
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
docs/misc/xen-command-line.pandoc Permit Xen to use "Always Running APIC Timer" support on compatible hardware · line 194
Quoted 1 line
docs/misc/xen-command-line.pandoc +1
-0+other domains by hypercall and querying the ring status of other domains.Please do include a note about
CONFIG_ARGO. I know this doc is inconsistent on the matter (asKconfigpostdates the written entries here), but I have been trying to fix up, and now about half of the documentation does mention appropriateKconfiginformation.Quoted 8 lines
Quoted 1 line
Ack, note added.
Just to voice my view here: While I agree that some form of indication should be added, I don't think
CONFIG_ARGOshould be mentioned. CONFIG_* in general are likely meaningless to the main audience of this file (admins rather than developers). Hence the wording should be mostly independent of the precise config option name; there may then be an annotation naming the option. Omitting the option name, otoh, has the benefit of not bearing the risk of going stale.
Quoted 7 lines
I completely disagree. The exact
CONFIG_name is very important for the end user who is looking at the documentation wondering "why can't I seem to enable ARGO?""This option is only available when
CONFIG_ARGOis compiled in" isn't going to put anyone off reading the document, but is useful for some who are reading it.
But what's wrong with e.g. "The functionality this option controls is dependent on a build time condition (the ARGO config option)"? This avoids the technical detail in the main part of the statement. In addition I'm also deliberately leaving out the CONFIG_ part of the option name, as that's entirely an implementation detail.
In no case is "CONFIG_ARGO is compiled in" a sensible statement to me - "CONFIG_ARGO" simply can't be "compiled in", it can only be enabled or disabled, whereas what can be compiled in is the code controlled by CONFIG_ARGO. (Caveat: I may wrongly apply too much German interpretation here.)
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 5/15 errno: add POSIX error codes EMSGSIZE, ECONNREFUSED to the ABI
In reply to Christopher ClarkEMSGSIZE: Argo's sendv operation will return EMSGSIZE when an excess amount of data, across all iovs, has been supplied, exceeding either the statically configured maximum size of a transmittable message, or the (variable) size of the ring registered by the destination domain.
ECONNREFUSED: Argo's register operation will return ECONNREFUSED if a ring is being registered to communicate with a specific remote domain that does exist but is not argo-enabled.
These codes are described by POSIX here:
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/errno.h.html
EMSGSIZE : "Message too large"
ECONNREFUSED : "Connection refused".
The numeric values assigned to each are taken from Linux, as is the case for the existing error codes.
EMSGSIZE : 90
ECONNREFUSED : 111
--- xen/include/public/errno.h | 2 ++ 1 file changed, 2 insertions(+)
xen/include/public/errno.h +2 -0
@@ -102,6 +102,7 @@ XEN_ERRNO(EILSEQ, 84) /* Illegal byte sequence */ XEN_ERRNO(ERESTART, 85) /* Interrupted system call should be restarted */#endif XEN_ERRNO(ENOTSOCK, 88) /* Socket operation on non-socket */ +XEN_ERRNO(EMSGSIZE, 90) /* Message too large. */ XEN_ERRNO(EOPNOTSUPP, 95) /* Operation not supported on transport endpoint */ XEN_ERRNO(EADDRINUSE, 98) /* Address already in use */XEN_ERRNO (EADDRNOTAVAIL ,99 )/* Cannot assign requested address */
@@ -109,6 +110,7 @@ XEN_ERRNO(ENOBUFS, 105) /* No buffer space available */ XEN_ERRNO(EISCONN, 106) /* Transport endpoint is already connected */ XEN_ERRNO(ENOTCONN, 107) /* Transport endpoint is not connected */ XEN_ERRNO(ETIMEDOUT, 110) /* Connection timed out */ +XEN_ERRNO(ECONNREFUSED, 111) /* Connection refused */#undef XEN_ERRNO#endif /* XEN_ERRNO */
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 6/15 xen/arm: introduce guest_handle_for_field()
In reply to Christopher ClarkARM port of c/s bb544585: "introduce guest_handle_for_field()"
This helper turns a field of a GUEST_HANDLE into a GUEST_HANDLE.
--- xen/include/asm-arm/guest_access.h | 3 +++ 1 file changed, 3 insertions(+)
xen/include/asm-arm/guest_access.h +3 -0
@@ -63,6 +63,9 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, _y; \ }) +#define guest_handle_for_field (hnd, type, fld) \ + ((XEN_GUEST_HANDLE(type)) { &(hnd).p->fld }) +#define guest_handle_from_ptr (ptr, type) \ ((XEN_GUEST_HANDLE_PARAM(type)) { (type *)ptr })#define const_guest_handle_from_ptr (ptr, type) \
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 06/15] xen/arm: introduce guest_handle_for_field()
Quoted 6 lines · Christopher Clark
ARM port of c/s
bb544585: "introduceguest_handle_for_field()"This helper turns a field of a
GUEST_HANDLEinto aGUEST_HANDLE.Signed-off-by Christopher Clark <[email protected]>Reviewed-by Paul Durrant <[email protected]>
Quoted 21 lines
--- xen/include/asm-arm/guest_access.h | 3 +++ 1 file changed, 3 insertions(+)xen/include/asm-arm/guest_access.h +3
-0@@ -63,6 +63,9 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, _y; \ }) +#define guest_handle_for_field (hnd, type, fld) \ + ((XEN_GUEST_HANDLE(type)) { &(hnd).p->fld }) +#define guest_handle_from_ptr (ptr, type) \ ((XEN_GUEST_HANDLE_PARAM(type)) { (type *)ptr })#define const_guest_handle_from_ptr (ptr, type) \Signature · 1 line
2.7.4
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 7/15 argo: implement the register op
In reply to Christopher ClarkThe register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to protect the ring from DoS by a noisy domain spamming the ring, affecting other domains ability to send to it. This will be addressed with XSM policy controls in subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order to make it obvious to an administrator that this has potential security impact.
The p2m type of the memory supplied by the guest for the ring must be p2m_ram_rw and the memory will be pinned as PGT_writable_page while the ring is registered.
xen_argo_page_descr_t type is introduced as a page descriptor, to convey both the physical address of the start of the page and its granularity. The smallest granularity page is assumed to be 4096 bytes and the lower twelve bits of the type are used to indicate the size of page of memory supplied. The implementation of the hypercall op currently only supports 4K pages.
array_index_nospec is used to guard the result of the ring id hash function. This is out of an abundance of caution, since this is a very basic hash function and it operates upon values supplied by the guest just before being used as an array index.
---
v2 self: disallow ring resize via reregister
v2 feedback Jan: drop cookie, implement teardown
v2 feedback Jan: drop message from argo_message_op
v2 self: move hash_index function below locking comment
v2 self: OVERHAUL
v2 self/Jan: remove use of magic verification field and tidy up v2 self: merge max and min ring size check clauses
v2 feedback v1#13 Roger: use OS-supplied roundup; drop from public header v2 feedback #9, Jan: use the argo-mac bootparam at point of introduction v2 feedback #9, Jan: rename boot opt variable to comply with convention v2 feedback #9, Jan: rename the argo_mac bootparam to argo-mac v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 feedback Roger, Jan: drop argo prefix on static functions v1 feedback Roger: s/pfn/gfn/ and retire always-64-bit type v2. feedback Jan: document the argo-mac boot opt
v2. feedback Jan: simplify re-register, drop mappings
v1 #13 feedback Jan: revise use of guest_handle_okay vs __copy ops
v1 #13 feedback, Jan: register op : s/ECONNREFUSED/ESRCH/
v1 #5 (#13) feedback Paul: register op: use currd in do_message_op v1 #13 feedback, Paul: register op: use mfn_eq comparator
v1 #5 (#13) feedback Paul: register op: use currd in argo_register_ring v1 #13 feedback Paul: register op: whitespace, unsigned, bounds check v1 #13 feedback Paul: use of hex in limit constant definition v1 #13 feedback Paul, register op: set nmfns on loop termination v1 #13 feedback Paul: register op: do/while -> gotos, reindent v1 argo_ring_map_page: drop uint32_t for unsigned int
v1. #13 feedback Julien: use page descriptors instead of gpfns.
- adds ABI support for pages with different granularity.
v1 feedback #13, Paul: adjust log level of message
v1 feedback #13, Paul: use gprintk for guest-triggered warning v1 feedback #13, Paul: gprintk and XENLOG_DEBUG for ring registration v1 feedback #13, Paul: use gprintk for errs in argo_ring_map_page v1 feedback #13, Paul: use ENOMEM if global mapping fails
v1 feedback Paul: overflow check before shift
v1: add define for copy_field_to_guest_errno
v1: fix gprintk use for ARM as its defn dislikes split format strings
v1: use copy_field_to_guest_errno
v1 feedback #13, Jan: argo_hash_fn: no inline, rename, change type v1 feedback #13, Paul, Jan: EFAULT -> ENOMEM in argo_ring_map_page v1 feedback #13, Jan: rename page var in argo_ring_map_page
v1 feedback #13, Jan: switch uint8_t* to void* and drop cast
v1 feedback #13, Jan: switch memory barrier to smp_wmb
v1 feedback #13, Jan: make 'ring' comment comply with single-line style v1 feedback #13, Jan: use xzalloc_array, drop loop NULL init
v1 feedback #13, Jan: init bool with false rather than 0
v1 feedback #13 Jan: use __copy; define and use __copy_field_to_guest_errno v1 feedback #13, Jan: use xzalloc, drop individual init zeroes v1 feedback #13, Jan: prefix public namespace with xen
v1 feedback #13, Jan: blank line after op case in do_argo_message_op v1 self: reflow comment in argo_ring_map_page to within 80 char len v1 feedback #13, Roger: use true not 1 in assign to update_tx_ptr bool v1 feedback #21, Jan: fold in the array_index_nospec hash function guards v1 feedback #18, Jan: fold the max ring count limit into the series v1 self: use unsigned long type for XEN_ARGO_REGISTER_FLAG_MASK
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1. feedback #13 Jan: add comment re: page alignment
v1. self: confirm ring magic presence in supplied page array v1. feedback #13 Jan: add comment re: minimum ring size
v1. feedback #13 Roger: use ASSERT_UNREACHABLE
v1. feedback Roger: add comment to hash function
docs/misc/xen-command-line.pandoc | 15 + xen/common/argo.c | 566 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/guest_access.h | 2 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 72 +++++ xen/include/xlat.lst | 1 + 6 files changed, 658 insertions(+)
docs/misc/xen-command-line.pandoc +15 -0
@@ -193,6 +193,21 @@ This allows domains access to the Argo hypercall, which supports registration of memory rings with the hypervisor to receive messages, sending messages to other domains by hypercall and querying the ring status of other domains. +### argo-mac +> `= permissive | enforcing` + +> Default: `enforcing` + +Constrain the access control applied to the Argo communication mechanism. + +When `enforcing`, domains may not register rings that have wildcard specified +for the sender which would allow messages to be sent to the ring by any domain. +This is to protect rings and the services that utilize them against DoS by a +malicious or buggy domain spamming the ring. + +When the boot option is set to `permissive`, this constraint is relaxed and +wildcard any-sender rings are allowed to be registered. + ### asid (x86) > `= <boolean>`
xen/common/argo.c +566 -0
606 lines not on this page · raw message
xen/include/asm-arm/guest_access.h +2 -0
@@ -29,6 +29,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))
xen/include/asm-x86/guest_access.h +2 -0
@@ -41,6 +41,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))
xen/include/public/argo.h +72 -0
82 lines not on this page · raw message
xen/include/xlat.lst +1 -0
@@ -150,3 +150,4 @@ ? flask_transition xsm/flask_op.h ? argo_addr argo.h ? argo_ring argo.h +? argo_register_ring argo.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 29 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to protect the ring from DoS by a noisy domain spamming the ring, affecting other domains ability to send to it. This will be addressed with XSM policy controls in subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order to make it obvious to an administrator that this has potential security impact.The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhile the ring is registered.
Quoted 6 lines
xen_argo_page_descr_ttype is introduced as a page descriptor, to convey both the physical address of the start of the page and its granularity. The smallest granularity page is assumed to be 4096 bytes and the lower twelve bits of the type are used to indicate the size of page of memory supplied. The implementation of the hypercall op currently only supports 4K pages.
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
Wei.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 38 lines · Wei Liu
Quoted 29 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to protect the ring from DoS by a noisy domain spamming the ring, affecting other domains ability to send to it. This will be addressed with XSM policy controls in subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order to make it obvious to an administrator that this has potential security impact.The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhile the ring is registered.Quoted 6 lines
xen_argo_page_descr_ttype is introduced as a page descriptor, to convey both the physical address of the start of the page and its granularity. The smallest granularity page is assumed to be 4096 bytes and the lower twelve bits of the type are used to indicate the size of page of memory supplied. The implementation of the hypercall op currently only supports 4K pages.
Quoted 3 lines
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Hi,
Sorry for the formatting. Sending it from my phone.
Quoted 54 lines · On Wed, 9 Jan 2019, 11:03 Christopher Clark, <[email protected]>…
Quoted 4 lines · Wei Liu
Quoted 2 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if
specifying a
Quoted 3 lines
Quoted 3 lines
wildcard, any domain.
This operation creates a mapping within Xen's private address space
that
Quoted 1 line
Quoted 1 line
will remain resident for the lifetime of the ring. In subsequent
commits,
Quoted 1 line
Quoted 1 line
the hypervisor will use this mapping to copy data from a sending
domain into
Quoted 1 line
Quoted 1 line
this registered ring, making it accessible to the domain that
registered the
Quoted 6 lines
Quoted 6 lines
ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to
protect the
Quoted 1 line
Quoted 1 line
ring from DoS by a noisy domain spamming the ring, affecting other
domains
Quoted 1 line
Quoted 1 line
ability to send to it. This will be addressed with XSM policy controls
in
Quoted 6 lines
Quoted 6 lines
subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive stringsin
Quoted 5 lines
Quoted 5 lines
order to make it obvious to an administrator that this has potential security impact.
The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhilethe ring
Quoted 3 lines
Quoted 3 lines
is registered.
xen_argo_page_descr_ttype is introduced as a page descriptor, toconvey
Quoted 1 line
Quoted 1 line
both the physical address of the start of the page and its
granularity. The
Quoted 1 line
Quoted 1 line
smallest granularity page is assumed to be 4096 bytes and the lower
twelve
Quoted 1 line
Quoted 1 line
bits of the type are used to indicate the size of page of memory
supplied.
Quoted 1 line
Quoted 1 line
The implementation of the hypercall op currently only supports 4K
pages.
Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You are adding unnecessary burden in the hypervisor and would prevent optimization i the hypervisor and unwanted side effect.
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you ever wanted to).
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
Cheers,
Quoted 8 lines
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 53 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.On Wed, 9 Jan 2019, 11:03 Christopher Clark, <[email protected]> wrote:
On Wed, Jan 9, 2019 at 7:56 AM Wei Liu <[email protected]> wrote:
>
> On Sun, Jan 06, 2019 at 11:42:40PM -0800, Christopher Clark wrote:
> > The register op is used by a domain to register a region of memory for > > receiving messages from either a specified other domain, or, if specifying a > > wildcard, any domain.
> >
> > This operation creates a mapping within Xen's private address space that > > will remain resident for the lifetime of the ring. In subsequent commits, > > the hypervisor will use this mapping to copy data from a sending domain into > > this registered ring, making it accessible to the domain that registered the > > ring to receive data.
> >
> > Wildcard any-sender rings are default disabled and registration will be > > refused with EPERM unless they have been specifically enabled with the > > argo-mac boot option introduced here. The reason why the default for > > wildcard rings is 'deny' is that there is currently no means to protect the > > ring from DoS by a noisy domain spamming the ring, affecting other domains > > ability to send to it. This will be addressed with XSM policy controls in > > subsequent work.
> >
> > Since denying access to any-sender rings is a significant functional > > constraint, a new bootparam is provided to enable overriding this:
> > "argo-mac" variable has allowed values: 'permissive' and 'enforcing'. > > Even though this is a boolean variable, use these descriptive strings in > > order to make it obvious to an administrator that this has potential > > security impact.
> >
> > The p2m type of the memory supplied by the guest for the ring must be > >p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhile the ring > > is registered.
> >
> >xen_argo_page_descr_ttype is introduced as a page descriptor, to convey > > both the physical address of the start of the page and its granularity. The > > smallest granularity page is assumed to be 4096 bytes and the lower twelve > > bits of the type are used to indicate the size of page of memory supplied. > > The implementation of the hypercall op currently only supports 4K pages. > >
>
> What is the resolution for the Arm issues mentioned by Julien? I read > the conversation in previous thread. A solution seemed to have been > agreed upon, but the changelog doesn't say anything about it.I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
Quoted 7 lines
I still don't think allowing 4K or 64K is the right solution to go. You are adding unnecessary burden in the hypervisor and would prevent optimization i the hypervisor and unwanted side effect.
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you ever wanted to).
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
Hi Julien!
I don't think we should force 64K as the only granularity on ARM. It causes unnecessary overhead and confusion on 4K-only deployments that are almost all our use-cases today.
One option is to make the granularity configurable at the guest side, like Christopher did, letting the guest specifying the granularity. The hypervisor could return -ENOSYS if the specified granularity is not supported.
The other option is having the hypervisor export the granularity it supports for this interface: Xen would say "I support only 4K". Tomorrow, it could change and Xen could say "I support only 64K". Then, it would be up to the guest passing a page of the right granularity to the hypervisor. I think this is probably the best option, but it would require the addition of one hypercall to retrieve the supported granularity from Xen.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Ki
Quoted 92 lines · On Wed, 9 Jan 2019, 12:18 Stefano Stabellini, <[email protected]> wrote:
Quoted 4 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.On Wed, 9 Jan 2019, 11:03 Christopher Clark, <
Quoted 1 line · [email protected]> wrote:
On Wed, Jan 9, 2019 at 7:56 AM Wei Liu <[email protected]>Quoted 2 lines · wrote:
> > On Sun, Jan 06, 2019 at 11:42:40PM -0800, Christopher ClarkQuoted 1 line · wrote:
> > The register op is used by a domain to register a region of
memory for
Quoted 1 line
> > receiving messages from either a specified other domain, or,
if specifying a
Quoted 3 lines
> > wildcard, any domain.
> >
> > This operation creates a mapping within Xen's private addressspace that
Quoted 1 line
> > will remain resident for the lifetime of the ring. In
subsequent commits,
Quoted 1 line
> > the hypervisor will use this mapping to copy data from a
sending domain into
Quoted 1 line
> > this registered ring, making it accessible to the domain that
registered the
Quoted 3 lines
> > ring to receive data.
> >
> > Wildcard any-sender rings are default disabled andregistration will be
Quoted 1 line
> > refused with EPERM unless they have been specifically enabled
with the
Quoted 1 line
> > argo-mac boot option introduced here. The reason why the
default for
Quoted 1 line
> > wildcard rings is 'deny' is that there is currently no means
to protect the
Quoted 1 line
> > ring from DoS by a noisy domain spamming the ring, affecting
other domains
Quoted 1 line
> > ability to send to it. This will be addressed with XSM policy
controls in
Quoted 3 lines
> > subsequent work.
> >
> > Since denying access to any-sender rings is a significantfunctional
Quoted 1 line
> > constraint, a new bootparam is provided to enable overriding
this:
Quoted 1 line
> > "argo-mac" variable has allowed values: 'permissive' and
'enforcing'.
Quoted 1 line
> > Even though this is a boolean variable, use these descriptive
strings in
Quoted 1 line
> > order to make it obvious to an administrator that this has
potential
Quoted 3 lines
> > security impact.
> >
> > The p2m type of the memory supplied by the guest for the ringmust be
Quoted 1 line
> >
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhile the ring
Quoted 3 lines
> > is registered.
> >
> >xen_argo_page_descr_ttype is introduced as a page descriptor,to convey
Quoted 1 line
> > both the physical address of the start of the page and its
granularity. The
Quoted 1 line
> > smallest granularity page is assumed to be 4096 bytes and the
lower twelve
Quoted 1 line
> > bits of the type are used to indicate the size of page of
memory supplied.
Quoted 1 line
> > The implementation of the hypercall op currently only supports
4K pages.
Quoted 3 lines
> > > > What is the resolution for the Arm issues mentioned by Julien? Iread
Quoted 1 line
> the conversation in previous thread. A solution seemed to have
been
Quoted 3 lines
> agreed upon, but the changelog doesn't say anything about it.
I made the interface changes that Julien had asked for. The
register
Quoted 6 lines
op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You
are adding unnecessary burden in the hypervisor and would
Quoted 3 lines
prevent optimization i the hypervisor and unwanted side effect.
For instance a 64K hypervisor will always map 64K even when the guest is
passing 4K. You also can't map everything contiguously
Quoted 2 lines
in Xen (if you ever wanted to).
Quoted 8 lines
Quoted 1 line
We need to stick on a single chunk size. That could be different between
Arm and x86. For Arm it would need to be 64KB.
Hi Julien!
I don't think we should force 64K as the only granularity on ARM. It causes unnecessary overhead and confusion on 4K-only deployments that are almost all our use-cases today.
Why a confusion? People should read the documentation when writing a driver...
Quoted 5 lines
One option is to make the granularity configurable at the guest side, like Christopher did, letting the guest specifying the granularity. The hypervisor could return
-ENOSYSif the specified granularity is not supported.
Quoted 7 lines
The other option is having the hypervisor export the granularity it supports for this interface: Xen would say "I support only 4K". Tomorrow, it could change and Xen could say "I support only 64K". Then, it would be up to the guest passing a page of the right granularity to the hypervisor. I think this is probably the best option, but it would require the addition of one hypercall to retrieve the supported granularity from Xen.
I would recommend to read my initial answers on the first series to understand why allowing 4K is an issue.
AFAIK virtio and UEFI has restrictions to allow a guest running agnostically of the hypervisor page-granularity. An example is to mandate 64K chunk or 64KB aligned address.
With your suggestion you are going to break many use-cases if the hypervisor is moving to 64KB. At worst it could introduce security issues. At best preventing optimization in the hypervisor or prevent guest running (bad for backward compatibility).
Actually, this is not going to help moving towards 64K in Argo because you would still have to modify the kernel. So this does not meet my requirements.
I don't think requiring 64K chunk is going to be a major issue nor a concern. Unless you expect small ring... Christoffer, what is the expected size?
Another solution was to require contiguous guest physical memory. That would solve quite a few problem on Arm. But Christoffer had some convincing point to not implement this.
As I said before, I know this is not going to be the only place with that issue. I merely wanted to start tackling the problem. However, IHMO, this interface is not more suitable than what we have currently. So this raise the question on whether we should just use the usual Xen interface if 64K is not an option...
Cheers,
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 57 lines · Julien Grall
Quoted 30 lines · On Wed, 9 Jan 2019, 12:18 Stefano Stabellini, <[email protected]> wrote:
Quoted 29 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.On Wed, 9 Jan 2019, 11:03 Christopher Clark, <[email protected]> wrote:
On Wed, Jan 9, 2019 at 7:56 AM Wei Liu <[email protected]> wrote:
>
> On Sun, Jan 06, 2019 at 11:42:40PM -0800, Christopher Clark wrote:
> > The register op is used by a domain to register a region of memory for > > receiving messages from either a specified other domain, or, if specifying a > > wildcard, any domain.
> >
> >xen_argo_page_descr_ttype is introduced as a page descriptor, to convey > > both the physical address of the start of the page and its granularity. The > > smallest granularity page is assumed to be 4096 bytes and the lower twelve > > bits of the type are used to indicate the size of page of memory supplied. > > The implementation of the hypercall op currently only supports 4K pages. >
> What is the resolution for the Arm issues mentioned by Julien? I read > the conversation in previous thread. A solution seemed to have been > agreed upon, but the changelog doesn't say anything about it.I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You are adding unnecessary burden in the hypervisor and would prevent optimization i the hypervisor and unwanted side effect.
Quoted 8 lines
Quoted 4 lines
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you ever wanted to).
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
I don't think we should force 64K as the only granularity on ARM. It causes unnecessary overhead and confusion on 4K-only deployments that are almost all our use-cases today.
Why a confusion? People should read the documentation when writing a driver...
Quoted 5 lines
One option is to make the granularity configurable at the guest side, like Christopher did, letting the guest specifying the granularity. The hypervisor could return
-ENOSYSif the specified granularity is not supported.Quoted 7 lines
The other option is having the hypervisor export the granularity it supports for this interface: Xen would say "I support only 4K". Tomorrow, it could change and Xen could say "I support only 64K". Then, it would be up to the guest passing a page of the right granularity to the hypervisor. I think this is probably the best option, but it would require the addition of one hypercall to retrieve the supported granularity from Xen.
I would recommend to read my initial answers on the first series to understand why allowing 4K is an issue.
Quoted 7 lines
AFAIK virtio and UEFI has restrictions to allow a guest running agnostically of the hypervisor page-granularity. An example is to mandate 64K chunk or 64KB aligned address.
With your suggestion you are going to break many use-cases if the hypervisor is moving to 64KB. At worst it could introduce security issues. At best preventing optimization in the hypervisor or prevent guest running (bad for backward compatibility).
Actually, this is not going to help moving towards 64K in Argo because you would still have to modify the kernel. So this does not meet my requirements.
I don't think requiring 64K chunk is going to be a major issue nor a concern. Unless you expect small ring... Christoffer, what is the expected size?
The current implementation of the Linux driver has a default ring size of 128K and I would expect that to be a common case. I'm not familiar with any current use cases where smaller than 64K would be likely, but I can imagine it could potentially be useful to have the option to do so in a memory-constrained embedded environment running a service that handles large numbers of short-lived connections, so running frequent setup and teardown of many rings, with only small amounts of data exchanged on each. That's not my current use case though, so it is speculating a bit.
Quoted 3 lines
Another solution was to require contiguous guest physical memory. That would solve quite a few problem on Arm. But Christoffer had some convincing point to not implement this.
As I said before, I know this is not going to be the only place with that issue. I merely wanted to start tackling the problem. However, IHMO, this interface is not more suitable than what we have currently. So this raise the question on whether we should just use the usual Xen interface if 64K is not an option...
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Julien GrallQuoted 75 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.
Quoted 54 lines · On Wed, 9 Jan 2019, 11:03 Christopher Clark, <[email protected]>…
Quoted 4 lines · Wei Liu
Quoted 2 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if
specifying a
Quoted 3 lines
Quoted 3 lines
wildcard, any domain.
This operation creates a mapping within Xen's private address space
that
Quoted 1 line
Quoted 1 line
will remain resident for the lifetime of the ring. In subsequent
commits,
Quoted 1 line
Quoted 1 line
the hypervisor will use this mapping to copy data from a sending
domain into
Quoted 1 line
Quoted 1 line
this registered ring, making it accessible to the domain that
registered the
Quoted 6 lines
Quoted 6 lines
ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to
protect the
Quoted 1 line
Quoted 1 line
ring from DoS by a noisy domain spamming the ring, affecting other
domains
Quoted 1 line
Quoted 1 line
ability to send to it. This will be addressed with XSM policy controls
in
Quoted 6 lines
Quoted 6 lines
subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive stringsin
Quoted 5 lines
Quoted 5 lines
order to make it obvious to an administrator that this has potential security impact.
The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhilethe ring
Quoted 3 lines
Quoted 3 lines
is registered.
xen_argo_page_descr_ttype is introduced as a page descriptor, toconvey
Quoted 1 line
Quoted 1 line
both the physical address of the start of the page and its
granularity. The
Quoted 1 line
Quoted 1 line
smallest granularity page is assumed to be 4096 bytes and the lower
twelve
Quoted 1 line
Quoted 1 line
bits of the type are used to indicate the size of page of memory
supplied.
Quoted 1 line
Quoted 1 line
The implementation of the hypercall op currently only supports 4K
pages.
Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You are adding unnecessary burden in the hypervisor and would prevent optimization i the hypervisor and unwanted side effect.
Quoted 6 lines
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you ever wanted to).
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
Doesn't enforcing 64KB granularity has its own limitation as well? According to my understanding of arm (and this could be wrong), you would need to have the guest allocate (via memory exchange perhaps) 64KB machine contiguous memory even when the hypervisor doesn't need it to be 64KB (when hypervisor is running on 4KB granularity).
I think having a method to return granularity to guest, like Stefano suggested, is more sensible. Hypervisor will then reject registration request which doesn't conform to the requirement.
Wei.
Quoted 12 lines
Cheers,
Quoted 8 lines
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 94 lines · On Wed, 9 Jan 2019, 12:54 Wei Liu, <[email protected]> wrote:
Quoted 5 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.
On Wed, 9 Jan 2019, 11:03 Christopher Clark, <
Quoted 6 lines
Quoted 4 lines · wrote:
Quoted 3 lines · Wei Liu
Quoted 1 line · Christopher Clark
The register op is used by a domain to register a region of memory
for
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
receiving messages from either a specified other domain, or, if
specifying a
Quoted 3 lines
Quoted 3 lines
wildcard, any domain.
This operation creates a mapping within Xen's private address space
that
Quoted 1 line
Quoted 1 line
will remain resident for the lifetime of the ring. In subsequent
Quoted 8 lines
Quoted 8 lines
commits,
Quoted 1 line
Quoted 1 line
the hypervisor will use this mapping to copy data from a sending
domain into
Quoted 1 line
Quoted 1 line
this registered ring, making it accessible to the domain that
registered the
Quoted 3 lines
Quoted 3 lines
ring to receive data.
Wildcard any-sender rings are default disabled and registration
will be
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
refused with EPERM unless they have been specifically enabled with
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
argo-mac boot option introduced here. The reason why the default
for
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
wildcard rings is 'deny' is that there is currently no means to
protect the
Quoted 1 line
Quoted 1 line
ring from DoS by a noisy domain spamming the ring, affecting other
domains
Quoted 1 line
Quoted 1 line
ability to send to it. This will be addressed with XSM policy
controls
Quoted 4 lines
Quoted 4 lines
in
Quoted 3 lines
Quoted 3 lines
subsequent work.
Since denying access to any-sender rings is a significant
functional
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and'enforcing'.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Even though this is a boolean variable, use these descriptive
strings
Quoted 2 lines
Quoted 2 lines
in
Quoted 1 line
Quoted 1 line
order to make it obvious to an administrator that this has
potential
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
security impact.
The p2m type of the memory supplied by the guest for the ring must
be
Quoted 20 lines
Quoted 20 lines
Quoted 1 line
Quoted 1 line
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhilethe ring
Quoted 3 lines
Quoted 3 lines
is registered.
xen_argo_page_descr_ttype is introduced as a page descriptor, toconvey
Quoted 1 line
Quoted 1 line
both the physical address of the start of the page and its
granularity. The
Quoted 1 line
Quoted 1 line
smallest granularity page is assumed to be 4096 bytes and the lower
twelve
Quoted 1 line
Quoted 1 line
bits of the type are used to indicate the size of page of memory
supplied.
Quoted 1 line
Quoted 1 line
The implementation of the hypercall op currently only supports 4K
pages.
Quoted 5 lines
Quoted 1 line
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
Quoted 7 lines
Quoted 4 lines
I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You
are
Quoted 1 line
adding unnecessary burden in the hypervisor and would prevent
optimization
Quoted 4 lines
i the hypervisor and unwanted side effect.
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you
ever
Quoted 2 lines
wanted to).
Quoted 8 lines
Quoted 2 lines
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
Doesn't enforcing 64KB granularity has its own limitation as well? According to my understanding of arm (and this could be wrong), you would need to have the guest allocate (via memory exchange perhaps) 64KB machine contiguous memory even when the hypervisor doesn't need it to be 64KB (when hypervisor is running on 4KB granularity).
The 64K is just about the interface with the guest.
The hypervisor could just split the 64K in 16 4K chunk. No need for memory exchange here.
Quoted 4 lines
I think having a method to return granularity to guest, like Stefano suggested, is more sensible. Hypervisor will then reject registration request which doesn't conform to the requirement.
The problem is not that simple... For instance, 64K is required to support 52-bits PA yet you may still want to run your current Debian on that platform.
You can do that nicely on KVM but on Xen it is a pain due to the current interface. If you use 4K you may end up to expose too much to the other side.
The only viable solution here is a full re-design of the ABI for Arm. We can do that step by step or at one go.
The discussion here was to start solving it on Argo so that's one less step to do. Christoffer kindly try to tackle it. Sadly, I don't think the interface suggested is going to work.
But I don't want Argo to miss 4.12 for that. So maybe the solution is to stick with the usal Xen interface.
Best regards,
Quoted 15 lines
Wei.
Quoted 12 lines
Cheers,
Quoted 8 lines
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 106 lines · Julien Grall
Quoted 81 lines · On Wed, 9 Jan 2019, 12:54 Wei Liu, <[email protected]> wrote:
Quoted 79 lines · Julien Grall
Hi,
Sorry for the formatting. Sending it from my phone.
Quoted 54 lines · On Wed, 9 Jan 2019, 11:03 Christopher Clark, <[email protected]>…
Quoted 4 lines · Wei Liu
Quoted 2 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if
specifying a
Quoted 3 lines
Quoted 3 lines
wildcard, any domain.
This operation creates a mapping within Xen's private address space
that
Quoted 1 line
Quoted 1 line
will remain resident for the lifetime of the ring. In subsequent
commits,
Quoted 1 line
Quoted 1 line
the hypervisor will use this mapping to copy data from a sending
domain into
Quoted 1 line
Quoted 1 line
this registered ring, making it accessible to the domain that
registered the
Quoted 6 lines
Quoted 6 lines
ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to
protect the
Quoted 1 line
Quoted 1 line
ring from DoS by a noisy domain spamming the ring, affecting other
domains
Quoted 1 line
Quoted 1 line
ability to send to it. This will be addressed with XSM policy controls
in
Quoted 6 lines
Quoted 6 lines
subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive stringsin
Quoted 5 lines
Quoted 5 lines
order to make it obvious to an administrator that this has potential security impact.
The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhilethe ring
Quoted 3 lines
Quoted 3 lines
is registered.
xen_argo_page_descr_ttype is introduced as a page descriptor, toconvey
Quoted 1 line
Quoted 1 line
both the physical address of the start of the page and its
granularity. The
Quoted 1 line
Quoted 1 line
smallest granularity page is assumed to be 4096 bytes and the lower
twelve
Quoted 1 line
Quoted 1 line
bits of the type are used to indicate the size of page of memory
supplied.
Quoted 1 line
Quoted 1 line
The implementation of the hypercall op currently only supports 4K
pages.
Quoted 2 lines
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
What is the resolution for the Arm issues mentioned by Julien? I read the conversation in previous thread. A solution seemed to have been agreed upon, but the changelog doesn't say anything about it.
I made the interface changes that Julien had asked for. The register op now takes arguments that can describe the granularitity of the pages supplied, though only support for 4K pages is accepted in the current implementation. I believe it meets Julien's requirements.
I still don't think allowing 4K or 64K is the right solution to go. You are adding unnecessary burden in the hypervisor and would prevent optimization i the hypervisor and unwanted side effect.
For instance a 64K hypervisor will always map 64K even when the guest is passing 4K. You also can't map everything contiguously in Xen (if you ever wanted to).
Quoted 8 lines
Quoted 2 lines
We need to stick on a single chunk size. That could be different between Arm and x86. For Arm it would need to be 64KB.
Doesn't enforcing 64KB granularity has its own limitation as well? According to my understanding of arm (and this could be wrong), you would need to have the guest allocate (via memory exchange perhaps) 64KB machine contiguous memory even when the hypervisor doesn't need it to be 64KB (when hypervisor is running on 4KB granularity).
The 64K is just about the interface with the guest.
The hypervisor could just split the 64K in 16 4K chunk. No need for memory exchange here.Quoted 4 lines
I think having a method to return granularity to guest, like Stefano suggested, is more sensible. Hypervisor will then reject registration request which doesn't conform to the requirement.
The problem is not that simple... For instance, 64K is required to support 52-bits PA yet you may still want to run your current Debian on that platform.
Quoted 7 lines
You can do that nicely on KVM but on Xen it is a pain due to the current interface. If you use 4K you may end up to expose too much to the other side.
The only viable solution here is a full re-design of the ABI for Arm. We can do that step by step or at one go.
The discussion here was to start solving it on Argo so that's one less step to do. Christoffer kindly try to tackle it. Sadly, I don't think the interface suggested is going to work.
But I don't want Argo to miss 4.12 for that. So maybe the solution is to stick with the usal Xen interface.
Thanks for the consideration. With that understanding, I'll put the frame number -based interface back into place for a new revision of the series, aiming for 4.12.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Christopher ClarkQuoted 119 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
This operation creates a mapping within Xen's private address space that will remain resident for the lifetime of the ring. In subsequent commits, the hypervisor will use this mapping to copy data from a sending domain into this registered ring, making it accessible to the domain that registered the ring to receive data.
Wildcard any-sender rings are default disabled and registration will be refused with EPERM unless they have been specifically enabled with the argo-mac boot option introduced here. The reason why the default for wildcard rings is 'deny' is that there is currently no means to protect the ring from DoS by a noisy domain spamming the ring, affecting other domains ability to send to it. This will be addressed with XSM policy controls in subsequent work.
Since denying access to any-sender rings is a significant functional constraint, a new bootparam is provided to enable overriding this:
"argo-mac" variable has allowed values: 'permissive' and 'enforcing'.
Even though this is a boolean variable, use these descriptive strings in order to make it obvious to an administrator that this has potential security impact.The p2m type of the memory supplied by the guest for the ring must be
p2m_ram_rwand the memory will be pinned asPGT_writable_pagewhile the ring is registered.
xen_argo_page_descr_ttype is introduced as a page descriptor, to convey both the physical address of the start of the page and its granularity. The smallest granularity page is assumed to be 4096 bytes and the lower twelve bits of the type are used to indicate the size of page of memory supplied. The implementation of the hypercall op currently only supports 4K pages.
array_index_nospecis used to guard the result of the ring id hash function. This is out of an abundance of caution, since this is a very basic hash function and it operates upon values supplied by the guest just before being used as an array index.Signed-off-by Christopher Clark <[email protected]>---
v2 self: disallow ring resize via reregister
v2 feedback Jan: drop cookie, implement teardown
v2 feedback Jan: drop message fromargo_message_op
v2 self: movehash_indexfunction below locking comment
v2 self: OVERHAUL
v2 self/Jan: remove use of magic verification field and tidy up v2 self: merge max and min ring size check clauses
v2 feedback v1#13 Roger: use OS-supplied roundup; drop from public header v2 feedback #9, Jan: use the argo-mac bootparam at point of introduction v2 feedback #9, Jan: rename boot opt variable to comply with convention v2 feedback #9, Jan: rename theargo_macbootparam to argo-mac v2 feedback #9 Jan: document argo boot opt in xen-command-line.markdown v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 feedback Roger, Jan: drop argo prefix on static functions v1 feedback Roger: s/pfn/gfn/ and retire always-64-bit type v2. feedback Jan: document the argo-mac boot opt
v2. feedback Jan: simplify re-register, drop mappings
v1 #13 feedback Jan: revise use ofguest_handle_okayvs__copyopsv1 #13 feedback, Jan: register op : s/ECONNREFUSED/ESRCH/
v1 #5 (#13) feedback Paul: register op: use currd indo_message_opv1 #13 feedback, Paul: register op: usemfn_eqcomparator
v1 #5 (#13) feedback Paul: register op: use currd inargo_register_ringv1 #13 feedback Paul: register op: whitespace, unsigned, bounds check v1 #13 feedback Paul: use of hex in limit constant definition v1 #13 feedback Paul, register op: set nmfns on loop termination v1 #13 feedback Paul: register op: do/while -> gotos, reindent v1argo_ring_map_page: dropuint32_tfor unsigned int
v1. #13 feedback Julien: use page descriptors instead of gpfns.
- adds ABI support for pages with different granularity.
v1 feedback #13, Paul: adjust log level of message
v1 feedback #13, Paul: use gprintk for guest-triggered warning v1 feedback #13, Paul: gprintk andXENLOG_DEBUGfor ring registration v1 feedback #13, Paul: use gprintk for errs inargo_ring_map_pagev1 feedback #13, Paul: use ENOMEM if global mapping fails
v1 feedback Paul: overflow check before shift
v1: add define forcopy_field_to_guest_errno
v1: fix gprintk use for ARM as its defn dislikes split format strings
v1: usecopy_field_to_guest_errno
v1 feedback #13, Jan:argo_hash_fn: no inline, rename, change type v1 feedback #13, Paul, Jan: EFAULT -> ENOMEM inargo_ring_map_pagev1 feedback #13, Jan: rename page var inargo_ring_map_page
v1 feedback #13, Jan: switch uint8_t* to void* and drop cast
v1 feedback #13, Jan: switch memory barrier tosmp_wmb
v1 feedback #13, Jan: make 'ring' comment comply with single-line style v1 feedback #13, Jan: usexzalloc_array, drop loop NULL init
v1 feedback #13, Jan: init bool with false rather than 0
v1 feedback #13 Jan: use__copy; define and use__copy_field_to_guest_errnov1 feedback #13, Jan: use xzalloc, drop individual init zeroes v1 feedback #13, Jan: prefix public namespace with xen
v1 feedback #13, Jan: blank line after op case indo_argo_message_opv1 self: reflow comment inargo_ring_map_pageto within 80 char len v1 feedback #13, Roger: use true not 1 in assign toupdate_tx_ptrbool v1 feedback #21, Jan: fold in thearray_index_nospechash function guards v1 feedback #18, Jan: fold the max ring count limit into the series v1 self: use unsigned long type forXEN_ARGO_REGISTER_FLAG_MASK
v1: feedback #15 Jan: handle upper-halves of hypercall args
v1. feedback #13 Jan: add comment re: page alignment
v1. self: confirm ring magic presence in supplied page array v1. feedback #13 Jan: add comment re: minimum ring size
v1. feedback #13 Roger: useASSERT_UNREACHABLE
v1. feedback Roger: add comment to hash functiondocs/misc/xen-command-line.pandoc | 15 + xen/common/argo.c | 566 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/guest_access.h | 2 + xen/include/asm-x86/guest_access.h | 2 + xen/include/public/argo.h | 72 +++++ xen/include/xlat.lst | 1 + 6 files changed, 658 insertions(+)docs/misc/xen-command-line.pandoc +0
-0
docs/misc/xen-command-line.pandoc This allows domains access to the Argo hypercall, which supports registration · line 197
Quoted 5 lines
docs/misc/xen-command-line.pandoc +2
-0@@ -193,6 +193,21 @@ This allows domains access to the Argo hypercall, which supports registration of memory rings with the hypervisor to receive messages, sending messages to other domains by hypercall and querying the ring status of other domains. +### argo-mac +> `= permissive | enforcing`
Why not call this argo-mac-permissive and make it a boolean? Default would be 'false' and that would imply enforcing. This would get rid of parse_opt_argo_mac since you could use the default boolean parser.
Quoted 122 lines
docs/misc/xen-command-line.pandoc +13
-0+ +> Default: `enforcing` + +Constrain the access control applied to the Argo communication mechanism. + +When `enforcing`, domains may not register rings that have wildcard specified +for the sender which would allow messages to be sent to the ring by any domain. +This is to protect rings and the services that utilize them against DoS by a +malicious or buggy domain spamming the ring. + +When the boot option is set to `permissive`, this constraint is relaxed and +wildcard any-sender rings are allowed to be registered. + ### asid (x86) > `= <boolean>`xen/common/argo.c +74
-0@@ -23,16 +23,41 @@#include <xen/event.h> #include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/lib.h> +#include <xen/nospec.h> #include <xen/time.h> #include <public/argo.h> +#define MAX_RINGS_PER_DOMAIN 128U + +/* All messages on the ring are padded to a multiple of the slot size. */ +#define ROUNDUP_MESSAGE (a) (ROUNDUP((a), XEN_ARGO_MSG_SLOT_SIZE)) +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_page_descr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_register_ring_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled;boolean_param ("argo" , opt_argo_enabled); +/* Xen command line option for conservative or relaxed access control */ +bool __read_mostly opt_argo_mac_enforcing = true; + +static int __init parse_opt_argo_mac (const char *s) +{ +if ( !strcmp (s,"enforcing" ) ) + opt_argo_mac_enforcing = true; +else if ( !strcmp (s,"permissive" ) ) + opt_argo_mac_enforcing = false; +else +return -EINVAL ; + +return 0 ; +} +custom_param ("argo-mac" , parse_opt_argo_mac); + typedef struct argo_ring_id { uint32_t port;@@ -198,6 +223,31 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#define argo_dprintk (format, ... ) ((void)0)#endif +/* +* This hash function is used to distribute rings within the per-domain +* hash tables (d->argo->ring_hash and d->argo_send_hash). The hash table +* will provide a struct if a match is found with a 'argo_ring_id' key: +* ie. the key is a (domain id, port, partner domain id) tuple. +* Since port number varies the most in expected use, and the Linux driver +* allocates at both the high and low ends, incorporate high and low bits to +* help with distribution. +* Apply array_index_nospec as a defensive measure since this operates +* on user-supplied input and the array size that it indexes into is known. +*/ +static unsigned int +hash_index (const struct argo_ring_id *id) +{ +unsigned int hash; + + hash = (uint16_t )(id->port >>16 ); + hash ^= (uint16_t )id->port; + hash ^= id->domain_id; + hash ^= id->partner_id; + hash &= (ARGO_HTABLE_SIZE -1 ); + +return array_index_nospec (hash,ARGO_HTABLE_SIZE ); +} +static void ring_unmap (struct argo_ring_info *ring_info) {@@ -219,6 +269,78 @@ ring_unmap(struct argo_ring_info *ring_info) } } +static int +ring_map_page (struct argo_ring_info *ring_info,unsigned int i,void **out_ptr) +{ +if ( i >= ring_info->nmfns ) + { +gprintk (XENLOG_ERR , +"argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n" , + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); +return -ENOMEM ; + } + +if ( !ring_info->mfns || !ring_info->mfn_mapping) + { +ASSERT_UNREACHABLE (); + ring_info->len =0 ; +return -ENOMEM ; + } + +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 305
Quoted 10 lines
xen/common/argo.c +10
-0+* and write access to the page is required by the hypervisor, but +* read-access is not needed for this mapping for the remainder of the +* ring. +* Since this mapping will remain resident in Xen's address space for +* the lifetime of the ring, and following the principle of least +* privilege, it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as +* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT).
Why do Intel EPT or AMD NPT matter here?
You are mapping the page to Xen address space, which doesn't use either EPT or NPT. Writable or read-only mappings would be created by setting the right bit in the Xen page tables.
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 308
Quoted 3 lines
xen/common/argo.c +3
-0+*/ + ring_info->mfn_mapping[i] =map_domain_page_global (ring_info->mfns[i]); +
No need for the newline.
Quoted 20 lines
xen/common/argo.c +20
-0+if ( !ring_info->mfn_mapping[i] ) + { +gprintk (XENLOG_ERR , +"argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n" , + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); +return -ENOMEM ; + } +argo_dprintk ("mapping page %" PRI_mfn" to %p\n" , +mfn_x (ring_info->mfns[i]), ring_info->mfn_mapping[i]); + } + +if ( out_ptr ) + *out_ptr = ring_info->mfn_mapping[i]; + +return 0 ; +} + +static void +update_tx_ptr (struct argo_ring_info *ring_info,uint32_t tx_ptr)
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 338
Quoted 10 lines
xen/common/argo.c +10
-0+{ +void *dst; +uint32_t *p; + +ASSERT (ring_info->mfn_mapping[0 ]); + + ring_info->tx_ptr = tx_ptr; + + dst = ring_info->mfn_mapping[0 ]; + p = dst + offsetof(xen_argo_ring_t , tx_ptr);
Hm, wouldn't it be easier to cast page 0 to the layout of the ring so that you don't need to use pointer arithmetic to get the fields? Ie:
make dst be of type xen_argo_ring_t.
Quoted 86 lines
xen/common/argo.c +79
-0+ +write_atomic (p, tx_ptr); +smp_wmb (); +} +static void wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) {@@ -371,6 +493,418 @@ partner_rings_remove(struct domain *src_d) } } +static int +find_ring_mfn (struct domain *d,gfn_t gfn,mfn_t *mfn) +{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d,gfn_x (gfn), &p2mt); +#else + *mfn =p2m_lookup (d, gfn, &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d,gfn_x (gfn)); +#endif + +return ret; +} + +static int +find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage, +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd, +uint32_t len) +{ +unsigned int i; +int ret =0 ; +mfn_t *mfns; +uint8_t **mfn_mapping; + +/* +* first bounds check on npage here also serves as an overflow check +* before left shifting it +*/ +if ( (unlikely (npage > (XEN_ARGO_MAX_RING_SIZE >>PAGE_SHIFT ))) || + ((npage <<PAGE_SHIFT ) < len) ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* Ring already existed: drop the previous mapping. */ +gprintk (XENLOG_INFO , +"argo: vm%u re-register existing ring (vm%u:%x vm%d) clears mapping\n" , + d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id); + +ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xzalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } +
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 579
Quoted 10 lines
xen/common/argo.c +10
-0+ ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + +ASSERT (ring_info->npage == npage); + +if ( ring_info->nmfns == ring_info->npage ) +return 0 ; + +for ( i = ring_info->nmfns; i < ring_info->npage; i++ )
This loop seems to assume that there can be pages already added to the ring, but IIRC you said that redimensioning of the ring was removed in this version?
I think for an initial version it would be easier to don't allow redimensioning of active rings, and just allow teardown and re-initialization as the way to redimension a ring.
Quoted 65 lines
xen/common/argo.c +65
-0+ { +xen_argo_page_descr_t pg_descr; +gfn_t gfn; +mfn_t mfn; + + ret =__copy_from_guest_offset (&pg_descr, pg_descr_hnd, i,1 ) ? + -EFAULT :0 ; +if ( ret ) +break ; + +/* Implementation currently only supports handling 4K pages */ +if ( (pg_descr &XEN_ARGO_PAGE_DESCR_SIZE_MASK ) != +XEN_ARGO_PAGE_DESCR_SIZE_4K ) + { + ret = -EINVAL ; +break ; + } + gfn =_gfn (pg_descr >>PAGE_SHIFT ); + + ret =find_ring_mfn (d, gfn, &mfn); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u: invalid gfn %" PRI_gfn" r:(vm%u:%x vm%d) %p %d/%d\n" , + d->domain_id,gfn_x (gfn), ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id, + ring_info, i, ring_info->npage); +break ; + } + + ring_info->mfns[i] = mfn; + +argo_dprintk ("%d: %" PRI_gfn" -> %" PRI_mfn"\n" , + i,gfn_x (gfn),mfn_x (ring_info->mfns[i])); + } + + ring_info->nmfns = i; + +if ( ret ) +ring_remove_mfns (d, ring_info); +else + { +ASSERT (ring_info->nmfns == ring_info->npage); + +gprintk (XENLOG_DEBUG , +"argo: vm%u ring (vm%u:%x vm%d) %p mfn_mapping %p npage %d nmfns %d\n" , + d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id, ring_info, + ring_info->mfn_mapping, ring_info->npage, ring_info->nmfns); + } + +return ret; +} + +static struct argo_ring_info * +ring_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +unsigned int ring_hash_index; +struct hlist_node *node; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + ring_hash_index =hash_index (id); +
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 654
Quoted 10 lines
xen/common/argo.c +10
-0+argo_dprintk ("d->argo=%p, d->argo->ring_hash[%u]=%p id=%p\n" , + d->argo, ring_hash_index, + d->argo->ring_hash[ring_hash_index].first, id); +argo_dprintk ("id.port=%x id.domain=vm%u id.partner_id=vm%d\n" , + id->port, id->domain_id, id->partner_id); + +hlist_for_each_entry (ring_info, node, &d->argo->ring_hash[ring_hash_index], + node) + { +struct argo_ring_id *cmpid = &ring_info->id;
const?
Quoted 6 lines
xen/common/argo.c +6
-0+ +if ( cmpid->port == id->port && + cmpid->domain_id == id->domain_id && + cmpid->partner_id == id->partner_id ) + { +argo_dprintk ("ring_info=%p\n" , ring_info);
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 670
Quoted 10 lines
xen/common/argo.c +10
-0+return ring_info; + } + } +argo_dprintk ("no ring_info found\n" ); + +return NULL ; +} + +static long +register_ring(struct domain *currd,
If this is indeed the current domain (as the name suggests), why do you need to pass it around? Or else just name the parameter d.
Quoted 8 lines
xen/common/argo.c +8
-0+XEN_GUEST_HANDLE_PARAM (xen_argo_register_ring_t) reg_hnd, + XEN_GUEST_HANDLE_PARAM(xen_argo_page_descr_t) pg_descr_hnd, + uint32_t npage, bool fail_exist) +{ + xen_argo_register_ring_t reg; +struct argo_ring_id ring_id; +void *map_ringp; +xen_argo_ring_t *ringp;
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 688
Quoted 10 lines
xen/common/argo.c +10
-0+struct argo_ring_info *ring_info; +struct argo_send_info *send_info =NULL ; +struct domain *dst_d =NULL ; +int ret =0 ; +uint32_t private_tx_ptr; + +if (copy_from_guest (®, reg_hnd,1 ) ) + { + ret = -EFAULT ; + goto out;
I don't see the point of using an out label, why not just use 'return -EFAULT;' (here and below). This avoids the braces and also removes the need for the ret assignment.
Quoted 113 lines
xen/common/argo.c +113
-0+ } + +/* +* A ring must be large enough to transmit messages, so requires space for: +* * 1 message header, plus +* * 1 payload slot (payload is always rounded to a multiple of 16 bytes) +* for the message payload to be written into, plus +* * 1 more slot, so that the ring cannot be filled to capacity with a +* single message -- see the logic in ringbuf_insert -- allowing for this +* ensures that there can be space remaining when a message is present. +* The above determines the minimum acceptable ring size. +*/ +if ( (reg.len < (sizeof (struct xen_argo_ring_message_header ) + +ROUNDUP_MESSAGE (1 ) +ROUNDUP_MESSAGE (1 ))) || + (reg.len >XEN_ARGO_MAX_RING_SIZE ) || + (reg.len !=ROUNDUP_MESSAGE (reg.len)) || + (reg.pad !=0 ) ) + { + ret = -EINVAL ; + goto out; + } + + ring_id.partner_id = reg.partner_id; + ring_id.port = reg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock); + +if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +if ( reg.partner_id ==XEN_ARGO_DOMID_ANY ) + { +if ( opt_argo_mac_enforcing ) + { + ret = -EPERM ; + goto out_unlock; + } + } +else + { + dst_d =get_domain_by_id (reg.partner_id); +if ( !dst_d ) + { +argo_dprintk ("!dst_d, ESRCH\n" ); + ret = -ESRCH ; + goto out_unlock; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); + goto out_unlock; + } + + send_info = xzalloc(struct argo_send_info); +if ( !send_info ) + { + ret = -ENOMEM ; +put_domain (dst_d); + goto out_unlock; + } + send_info->id = ring_id; + } + +write_lock (&currd->argo->lock); + +if ( currd->argo->ring_count >=MAX_RINGS_PER_DOMAIN ) + { + ret = -ENOSPC ; + goto out_unlock2; + } + + ring_info =ring_find_info (currd, &ring_id); +if ( !ring_info ) + { + ring_info = xzalloc(struct argo_ring_info); +if ( !ring_info ) + { + ret = -ENOMEM ; + goto out_unlock2; + } + +spin_lock_init (&ring_info->lock); + + ring_info->id = ring_id; +INIT_HLIST_HEAD (&ring_info->pending); + +hlist_add_head (&ring_info->node, + &currd->argo->ring_hash[hash_index (&ring_info->id)]); + +gprintk (XENLOG_DEBUG ,"argo: vm%u registering ring (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } +else + { +if ( ring_info->len ) + { +/* +* If the caller specified that the ring must not already exist, +* fail at attempt to add a completed ring which already exists. +*/ +if ( fail_exist ) + { +argo_dprintk ("disallowed reregistration of existing ring\n" ); + ret = -EEXIST ; + goto out_unlock2;
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 811
Quoted 10 lines
xen/common/argo.c +10
-0+ } + +if ( ring_info->len != reg.len ) + { +/* +* Change of ring size could result in entries on the pending +* notifications list that will never trigger. +* Simple blunt solution: disallow ring resize for now. +* TODO: investigate enabling ring resize. +*/
I think ring resizing was removed on this version?
Quoted 73 lines
xen/common/argo.c +73
-0+gprintk (XENLOG_ERR , +"argo: vm%u attempted to change ring size(vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); +/* +* Could return EINVAL here, but if the ring didn't already +* exist then the arguments would have been valid, so: EEXIST. +*/ + ret = -EEXIST ; + goto out_unlock2; + } + +gprintk (XENLOG_DEBUG , +"argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } + } + + ret =find_ring_mfns (currd, ring_info, npage, pg_descr_hnd, reg.len); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to find ring mfns (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + +/* +* The first page of the memory supplied for the ring has the xen_argo_ring +* structure at its head, which is where the ring indexes reside. +*/ + ret =ring_map_page (ring_info,0 , &map_ringp); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to map ring mfn 0 (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + ringp = map_ringp; + + private_tx_ptr =read_atomic (&ringp->tx_ptr); + +if ( (private_tx_ptr >= reg.len) || + (ROUNDUP_MESSAGE (private_tx_ptr) != private_tx_ptr) ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + private_tx_ptr =ROUNDUP_MESSAGE (read_atomic (&ringp->rx_ptr)); + +if ( private_tx_ptr >= reg.len ) + private_tx_ptr =0 ; + +update_tx_ptr (ring_info, private_tx_ptr); + } + + ring_info->tx_ptr = private_tx_ptr; + ring_info->len = reg.len; + currd->argo->ring_count++; + +if ( send_info ) + { +spin_lock (&dst_d->argo->send_lock);
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 894
Quoted 10 lines
xen/common/argo.c +10
-0+ +hlist_add_head (&send_info->node, + &dst_d->argo->send_hash[hash_index (&send_info->id)]); + +spin_unlock (&dst_d->argo->send_lock); + } + + out_unlock2: +if ( !ret && send_info ) +xfree (send_info);
There's no need to check if send_info is set, xfree(NULL) is safe.
Quoted 86 lines
xen/common/argo.c +45
-0+ +if ( dst_d ) +put_domain (dst_d); + +write_unlock (&currd->argo->lock); + + out_unlock: +read_unlock (&argo_lock); + + out: +return ret; +} + long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2, unsigned long arg3,@@ -392,6 +926,38 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,switch (cmd) { +case XEN_ARGO_OP_register_ring: + { +XEN_GUEST_HANDLE_PARAM (xen_argo_register_ring_t) reg_hnd = +guest_handle_cast (arg1, xen_argo_register_ring_t); +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd = +guest_handle_cast (arg2, xen_argo_page_descr_t); +/* arg3 is npage */ +/* arg4 is flags */ +bool fail_exist = arg4 &XEN_ARGO_REGISTER_FLAG_FAIL_EXIST ; + +if (unlikely (arg3 > (XEN_ARGO_MAX_RING_SIZE >>PAGE_SHIFT )) ) + { + rc = -EINVAL ; +break ; + } +/* +* Check access to the whole array here so we can use the faster __copy +* operations to read each element later. +*/ +if (unlikely (!guest_handle_okay (pg_descr_hnd, arg3)) ) +break ; +/* arg4: reserve currently-undefined bits, require zero. */ +if (unlikely (arg4 & ~XEN_ARGO_REGISTER_FLAG_MASK ) ) + { + rc = -EINVAL ; +break ; + } + + rc =register_ring (currd, reg_hnd, pg_descr_hnd, arg3, fail_exist); +break ; + } +default : rc = -EOPNOTSUPP ;break ;xen/include/asm-arm/guest_access.h +2
-0@@ -29,6 +29,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/asm-x86/guest_access.h +2
-0@@ -41,6 +41,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/public/argo.h +0
-0
xen/include/public/argo.h line 37
Quoted 7 lines
xen/include/public/argo.h +4
-0@@ -31,6 +31,26 @@#include "xen.h" +#define XEN_ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB
Is such a big size really required as the default maximum? The size of the internal structures required to support a 16GB ring would be quite big, has this been taken into account?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 10.01.19 at 12:24, <[email protected]> wrote:
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 670
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static long +register_ring(struct domain *currd,If this is indeed the current domain (as the name suggests), why do you need to pass it around? Or else just name the parameter d.
When all (or at least most) callers already latch the pointer into a local variable, handing it through is often cheaper than re-obtaining it as current->domain. ASSERT(currd == current->domain) might be worthwhile in such cases, though.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 9 lines · Jan Beulich
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines · On 10.01.19 at 12:24, <[email protected]> wrote:
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 670
Quoted 2 lines · Christopher Clark
xen/common/argo.c +2
-0+static long +register_ring(struct domain *currd,If this is indeed the current domain (as the name suggests), why do you need to pass it around? Or else just name the parameter d.
Quoted 4 lines
When all (or at least most) callers already latch the pointer into a local variable, handing it through is often cheaper than re-obtaining it as
current->domain.ASSERT(currd == current->domain)might be worthwhile in such cases, though.
argument retained and ASSERTs added, thanks.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Roger Pau MonnéQuoted 16 lines · Roger Pau Monné
Quoted 13 lines · Christopher Clark
The register op is used by a domain to register a region of memory for receiving messages from either a specified other domain, or, if specifying a wildcard, any domain.
docs/misc/xen-command-line.pandoc +0
-0@@ -193,6 +193,21 @@ This allows domains access to the Argo hypercall, which supports registration of memory rings with the hypervisor to receive messages, sending messages to other domains by hypercall and querying the ring status of other domains.
Quoted 6 lines
docs/misc/xen-command-line.pandoc This allows domains access to the Argo hypercall, which supports registration · line 197
Quoted 2 lines
docs/misc/xen-command-line.pandoc +2
-0+### argo-mac +> `= permissive | enforcing`Why not call this argo-mac-permissive and make it a boolean? Default would be 'false' and that would imply enforcing. This would get rid of
parse_opt_argo_macsince you could use the default boolean parser.
Yes, that makes sense, thanks -- done
Quoted 28 lines
Quoted 28 lines
docs/misc/xen-command-line.pandoc +28
-0+static int +ring_map_page(struct argo_ring_info *ring_info, unsigned int i, void **out_ptr) +{ + if ( i >= ring_info->nmfns ) + { + gprintk(XENLOG_ERR, + "argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n", + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); + return -ENOMEM; + } + + if ( !ring_info->mfns || !ring_info->mfn_mapping) + { + ASSERT_UNREACHABLE(); + ring_info->len = 0; + return -ENOMEM; + } + + if ( !ring_info->mfn_mapping[i] ) + { + /* + * TODO: + * The first page of the ring contains the ring indices, so both read + * and write access to the page is required by the hypervisor, but + * read-access is not needed for this mapping for the remainder of the + * ring. + * Since this mapping will remain resident in Xen's address space for
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 6 lines
docs/misc/xen-command-line.pandoc +6
-0+ * the lifetime of the ring, and following the principle of least + * privilege, it could be preferable to: + * # add a XSM check to determine what policy is wanted here + * # depending on the XSM query, optionally create this mapping as + * _write-only_ on platforms that can support it. + * (eg. Intel EPT/AMD NPT).Why do Intel EPT or AMD NPT matter here?
I think (though could be wrong and am open to correction here) that EPT and NPT enable the construction of write-only (ie not readable) memory mappings. Standard page tables can't do that: with those, if it's writable, it's also readable.
Quoted 3 lines
You are mapping the page to Xen address space, which doesn't use either EPT or NPT. Writable or read-only mappings would be created by setting the right bit in the Xen page tables.
ok. I've dropped the comment.
Quoted 6 lines
docs/misc/xen-command-line.pandoc
Quoted 4 lines
docs/misc/xen-command-line.pandoc +3
-0+ */ + ring_info->mfn_mapping[i] = map_domain_page_global(ring_info->mfns[i]); +No need for the newline.
ack.
Quoted 8 lines
Quoted 8 lines
docs/misc/xen-command-line.pandoc +8
-0+static void +update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) +{ + void *dst; + uint32_t *p; + + ASSERT(ring_info->mfn_mapping[0]); +
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 4 lines
docs/misc/xen-command-line.pandoc +4
-0+ ring_info->tx_ptr = tx_ptr; + + dst = ring_info->mfn_mapping[0]; + p = dst + offsetof(xen_argo_ring_t, tx_ptr);Hm, wouldn't it be easier to cast page 0 to the layout of the ring so that you don't need to use pointer arithmetic to get the fields? Ie:
make dst be of typexen_argo_ring_t.
Yes, good point - and that's what's already done elsewhere with the rx_ptr, so that makes the code more consistent. Done.
Quoted 51 lines
Quoted 51 lines
docs/misc/xen-command-line.pandoc +51
-0+static int +find_ring_mfns(struct domain *d, struct argo_ring_info *ring_info, + uint32_t npage, + XEN_GUEST_HANDLE_PARAM(xen_argo_page_descr_t) pg_descr_hnd, + uint32_t len) +{ + unsigned int i; + int ret = 0; + mfn_t *mfns; + uint8_t **mfn_mapping; + + /* + * first bounds check on npage here also serves as an overflow check + * before left shifting it + */ + if ( (unlikely(npage > (XEN_ARGO_MAX_RING_SIZE >> PAGE_SHIFT))) || + ((npage << PAGE_SHIFT) < len) ) + return -EINVAL; + + if ( ring_info->mfns ) + { + /* Ring already existed: drop the previous mapping. */ + gprintk(XENLOG_INFO, + "argo: vm%u re-register existing ring (vm%u:%x vm%d) clears mapping\n", + d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id); + + ring_remove_mfns(d, ring_info); + ASSERT(!ring_info->mfns); + } + + mfns = xmalloc_array(mfn_t, npage); + if ( !mfns ) + return -ENOMEM; + + for ( i = 0; i < npage; i++ ) + mfns[i] = INVALID_MFN; + + mfn_mapping = xzalloc_array(uint8_t *, npage); + if ( !mfn_mapping ) + { + xfree(mfns); + return -ENOMEM; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping; + + ASSERT(ring_info->npage == npage); +
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 4 lines
docs/misc/xen-command-line.pandoc +4
-0+ if ( ring_info->nmfns == ring_info->npage ) + return 0; + + for ( i = ring_info->nmfns; i < ring_info->npage; i++ )This loop seems to assume that there can be pages already added to the ring, but IIRC you said that redimensioning of the ring was removed in this version?
That's correct - it's currently rejected.
Quoted 3 lines
I think for an initial version it would be easier to don't allow redimensioning of active rings, and just allow teardown and re-initialization as the way to redimension a ring.
ack. I'll look into how it affects this function and simplifying it.
Quoted 70 lines
Quoted 70 lines
docs/misc/xen-command-line.pandoc +69
-0+ { + xen_argo_page_descr_t pg_descr; + gfn_t gfn; + mfn_t mfn; + + ret = __copy_from_guest_offset(&pg_descr, pg_descr_hnd, i, 1) ? + -EFAULT : 0; + if ( ret ) + break; + + /* Implementation currently only supports handling 4K pages */ + if ( (pg_descr & XEN_ARGO_PAGE_DESCR_SIZE_MASK) != + XEN_ARGO_PAGE_DESCR_SIZE_4K ) + { + ret = -EINVAL; + break; + } + gfn = _gfn(pg_descr >> PAGE_SHIFT); + + ret = find_ring_mfn(d, gfn, &mfn); + if ( ret ) + { + gprintk(XENLOG_ERR, + "argo: vm%u: invalid gfn %"PRI_gfn" r:(vm%u:%x vm%d) %p %d/%d\n", + d->domain_id, gfn_x(gfn), ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id, + ring_info, i, ring_info->npage); + break; + } + + ring_info->mfns[i] = mfn; + + argo_dprintk("%d: %"PRI_gfn" -> %"PRI_mfn"\n", + i, gfn_x(gfn), mfn_x(ring_info->mfns[i])); + } + + ring_info->nmfns = i; + + if ( ret ) + ring_remove_mfns(d, ring_info); + else + { + ASSERT(ring_info->nmfns == ring_info->npage); + + gprintk(XENLOG_DEBUG, + "argo: vm%u ring (vm%u:%x vm%d) %p mfn_mapping %p npage %d nmfns %d\n", + d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id, ring_info, + ring_info->mfn_mapping, ring_info->npage, ring_info->nmfns); + } + + return ret; +} + +static struct argo_ring_info * +ring_find_info(const struct domain *d, const struct argo_ring_id *id) +{ + unsigned int ring_hash_index; + struct hlist_node *node; + struct argo_ring_info *ring_info; + + ASSERT(rw_is_locked(&d->argo->lock)); + + ring_hash_index = hash_index(id); + + argo_dprintk("d->argo=%p, d->argo->ring_hash[%u]=%p id=%p\n", + d->argo, ring_hash_index, + d->argo->ring_hash[ring_hash_index].first, id); + argo_dprintk("id.port=%x id.domain=vm%u id.partner_id=vm%d\n",
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 6 lines
docs/misc/xen-command-line.pandoc +6
-0+ id->port, id->domain_id, id->partner_id); + + hlist_for_each_entry(ring_info, node, &d->argo->ring_hash[ring_hash_index], + node) + { + struct argo_ring_id *cmpid = &ring_info->id;const?
yep, thanks, done.
Quoted 12 lines
Quoted 12 lines
docs/misc/xen-command-line.pandoc +11
-0+ + if ( cmpid->port == id->port && + cmpid->domain_id == id->domain_id && + cmpid->partner_id == id->partner_id ) + { + argo_dprintk("ring_info=%p\n", ring_info); + return ring_info; + } + } + argo_dprintk("no ring_info found\n"); +
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 5 lines
docs/misc/xen-command-line.pandoc +5
-0+ return NULL; +} + +static long +register_ring(struct domain *currd,If this is indeed the current domain (as the name suggests), why do you need to pass it around? Or else just name the parameter d.
After the later in-thread discussion between you and Jan, I've left the argument name 'currd' but added the ASSERT recommended by Jan, that currd matches domain->current.
I've done the same in the other functions (across the series) that take currd as an argument, except for notify_check_pending where I've just renamed the argument to 'd'; there's no reason in that function that it needs to be handling the current domain.
Quoted 15 lines
Quoted 15 lines
docs/misc/xen-command-line.pandoc +14
-0+ XEN_GUEST_HANDLE_PARAM(xen_argo_register_ring_t) reg_hnd, + XEN_GUEST_HANDLE_PARAM(xen_argo_page_descr_t) pg_descr_hnd, + uint32_t npage, bool fail_exist) +{ + xen_argo_register_ring_t reg; + struct argo_ring_id ring_id; + void *map_ringp; + xen_argo_ring_t *ringp; + struct argo_ring_info *ring_info; + struct argo_send_info *send_info = NULL; + struct domain *dst_d = NULL; + int ret = 0; + uint32_t private_tx_ptr; +
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 4 lines
docs/misc/xen-command-line.pandoc +4
-0+ if ( copy_from_guest(®, reg_hnd, 1) ) + { + ret = -EFAULT; + goto out;I don't see the point of using an out label, why not just use 'return
-EFAULT;' (here and below). This avoids the braces and also removes the need for the ret assignment.
done.
Quoted 118 lines
Quoted 118 lines
docs/misc/xen-command-line.pandoc +117
-0+ } + + /* + * A ring must be large enough to transmit messages, so requires space for: + * * 1 message header, plus + * * 1 payload slot (payload is always rounded to a multiple of 16 bytes) + * for the message payload to be written into, plus + * * 1 more slot, so that the ring cannot be filled to capacity with a + * single message -- see the logic in ringbuf_insert -- allowing for this + * ensures that there can be space remaining when a message is present. + * The above determines the minimum acceptable ring size. + */ + if ( (reg.len < (sizeof(struct xen_argo_ring_message_header) + + ROUNDUP_MESSAGE(1) + ROUNDUP_MESSAGE(1))) || + (reg.len > XEN_ARGO_MAX_RING_SIZE) || + (reg.len != ROUNDUP_MESSAGE(reg.len)) || + (reg.pad != 0) ) + { + ret = -EINVAL; + goto out; + } + + ring_id.partner_id = reg.partner_id; + ring_id.port = reg.port; + ring_id.domain_id = currd->domain_id; + + read_lock(&argo_lock); + + if ( !currd->argo ) + { + ret = -ENODEV; + goto out_unlock; + } + + if ( reg.partner_id == XEN_ARGO_DOMID_ANY ) + { + if ( opt_argo_mac_enforcing ) + { + ret = -EPERM; + goto out_unlock; + } + } + else + { + dst_d = get_domain_by_id(reg.partner_id); + if ( !dst_d ) + { + argo_dprintk("!dst_d, ESRCH\n"); + ret = -ESRCH; + goto out_unlock; + } + + if ( !dst_d->argo ) + { + argo_dprintk("!dst_d->argo, ECONNREFUSED\n"); + ret = -ECONNREFUSED; + put_domain(dst_d); + goto out_unlock; + } + + send_info = xzalloc(struct argo_send_info); + if ( !send_info ) + { + ret = -ENOMEM; + put_domain(dst_d); + goto out_unlock; + } + send_info->id = ring_id; + } + + write_lock(&currd->argo->lock); + + if ( currd->argo->ring_count >= MAX_RINGS_PER_DOMAIN ) + { + ret = -ENOSPC; + goto out_unlock2; + } + + ring_info = ring_find_info(currd, &ring_id); + if ( !ring_info ) + { + ring_info = xzalloc(struct argo_ring_info); + if ( !ring_info ) + { + ret = -ENOMEM; + goto out_unlock2; + } + + spin_lock_init(&ring_info->lock); + + ring_info->id = ring_id; + INIT_HLIST_HEAD(&ring_info->pending); + + hlist_add_head(&ring_info->node, + &currd->argo->ring_hash[hash_index(&ring_info->id)]); + + gprintk(XENLOG_DEBUG, "argo: vm%u registering ring (vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } + else + { + if ( ring_info->len ) + { + /* + * If the caller specified that the ring must not already exist, + * fail at attempt to add a completed ring which already exists. + */ + if ( fail_exist ) + { + argo_dprintk("disallowed reregistration of existing ring\n"); + ret = -EEXIST; + goto out_unlock2; + } + + if ( ring_info->len != reg.len ) + {
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 6 lines
docs/misc/xen-command-line.pandoc +6
-0+ /* + * Change of ring size could result in entries on the pending + * notifications list that will never trigger. + * Simple blunt solution: disallow ring resize for now. + * TODO: investigate enabling ring resize. + */I think ring resizing was removed on this version?
Yes: This is the code that was introduced to prevent it.
Quoted 78 lines
Quoted 78 lines
docs/misc/xen-command-line.pandoc +77
-0+ gprintk(XENLOG_ERR, + "argo: vm%u attempted to change ring size(vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + /* + * Could return EINVAL here, but if the ring didn't already + * exist then the arguments would have been valid, so: EEXIST. + */ + ret = -EEXIST; + goto out_unlock2; + } + + gprintk(XENLOG_DEBUG, + "argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } + } + + ret = find_ring_mfns(currd, ring_info, npage, pg_descr_hnd, reg.len); + if ( ret ) + { + gprintk(XENLOG_ERR, + "argo: vm%u failed to find ring mfns (vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + + ring_remove_info(currd, ring_info); + goto out_unlock2; + } + + /* + * The first page of the memory supplied for the ring has the xen_argo_ring + * structure at its head, which is where the ring indexes reside. + */ + ret = ring_map_page(ring_info, 0, &map_ringp); + if ( ret ) + { + gprintk(XENLOG_ERR, + "argo: vm%u failed to map ring mfn 0 (vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + + ring_remove_info(currd, ring_info); + goto out_unlock2; + } + ringp = map_ringp; + + private_tx_ptr = read_atomic(&ringp->tx_ptr); + + if ( (private_tx_ptr >= reg.len) || + (ROUNDUP_MESSAGE(private_tx_ptr) != private_tx_ptr) ) + { + /* + * Since the ring is a mess, attempt to flush the contents of it + * here by setting the tx_ptr to the next aligned message slot past + * the latest rx_ptr we have observed. Handle ring wrap correctly. + */ + private_tx_ptr = ROUNDUP_MESSAGE(read_atomic(&ringp->rx_ptr)); + + if ( private_tx_ptr >= reg.len ) + private_tx_ptr = 0; + + update_tx_ptr(ring_info, private_tx_ptr); + } + + ring_info->tx_ptr = private_tx_ptr; + ring_info->len = reg.len; + currd->argo->ring_count++; + + if ( send_info ) + { + spin_lock(&dst_d->argo->send_lock); + + hlist_add_head(&send_info->node, + &dst_d->argo->send_hash[hash_index(&send_info->id)]); +
Quoted 8 lines
docs/misc/xen-command-line.pandoc
Quoted 6 lines
docs/misc/xen-command-line.pandoc +6
-0+ spin_unlock(&dst_d->argo->send_lock); + } + + out_unlock2: + if ( !ret && send_info ) + xfree(send_info);There's no need to check if
send_infois set, xfree(NULL) is safe.
done.
Quoted 90 lines
Quoted 90 lines
docs/misc/xen-command-line.pandoc +45
-0+ + if ( dst_d ) + put_domain(dst_d); + + write_unlock(&currd->argo->lock); + + out_unlock: + read_unlock(&argo_lock); + + out: + return ret; +} + long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2, unsigned long arg3,@@ -392,6 +926,38 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, switch (cmd) { + case XEN_ARGO_OP_register_ring: + { + XEN_GUEST_HANDLE_PARAM(xen_argo_register_ring_t) reg_hnd = + guest_handle_cast(arg1, xen_argo_register_ring_t); + XEN_GUEST_HANDLE_PARAM(xen_argo_page_descr_t) pg_descr_hnd = + guest_handle_cast(arg2, xen_argo_page_descr_t); + /* arg3 is npage */ + /* arg4 is flags */ + bool fail_exist = arg4 & XEN_ARGO_REGISTER_FLAG_FAIL_EXIST; + + if ( unlikely(arg3 > (XEN_ARGO_MAX_RING_SIZE >> PAGE_SHIFT)) ) + { + rc = -EINVAL; + break; + } + /* + * Check access to the whole array here so we can use the faster __copy + * operations to read each element later. + */ + if ( unlikely(!guest_handle_okay(pg_descr_hnd, arg3)) ) + break; + /* arg4: reserve currently-undefined bits, require zero. */ + if ( unlikely(arg4 & ~XEN_ARGO_REGISTER_FLAG_MASK) ) + { + rc = -EINVAL; + break; + } + + rc = register_ring(currd, reg_hnd, pg_descr_hnd, arg3, fail_exist); + break; + } + default: rc = -EOPNOTSUPP; break;xen/include/asm-arm/guest_access.h +2
-0@@ -29,6 +29,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/asm-x86/guest_access.h +2
-0@@ -41,6 +41,8 @@/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask))) +/* Offset the given guest handle into the array it refers to. */ #define guest_handle_add_offset (hnd, nr) ((hnd).p += (nr))#define guest_handle_subtract_offset (hnd, nr) ((hnd).p -= (nr))xen/include/public/argo.h +0
-0@@ -31,6 +31,26 @@#include "xen.h"
Quoted 8 lines
xen/include/public/argo.h line 37
Quoted 4 lines
xen/include/public/argo.h +4
-0+#define XEN_ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB Is such a big size really required as the default maximum? The size of the internal structures required to support a 16GB ring would be quite big, has this been taken into account?
Yes, that was incorrect. The comment is now fixed. 16MB is much more reasonable.
thanks,
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 42 lines · Christopher Clark
Quoted 31 lines · Roger Pau Monné
Quoted 28 lines · Christopher Clark
xen/common/argo.c +28
-0+static int +ring_map_page (struct argo_ring_info *ring_info,unsigned int i,void **out_ptr) +{ +if ( i >= ring_info->nmfns ) + { +gprintk (XENLOG_ERR , +"argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n" , + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); +return -ENOMEM ; + } + +if ( !ring_info->mfns || !ring_info->mfn_mapping) + { +ASSERT_UNREACHABLE (); + ring_info->len =0 ; +return -ENOMEM ; + } + +if ( !ring_info->mfn_mapping[i] ) + { +/* +* TODO: +* The first page of the ring contains the ring indices, so both read +* and write access to the page is required by the hypervisor, but +* read-access is not needed for this mapping for the remainder of the +* ring. +* Since this mapping will remain resident in Xen's address space for Quoted 8 lines
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 305
Quoted 6 lines
xen/common/argo.c +6
-0+* the lifetime of the ring, and following the principle of least +* privilege, it could be preferable to: +* # add a XSM check to determine what policy is wanted here +* # depending on the XSM query, optionally create this mapping as +* _write-only_ on platforms that can support it. +* (eg. Intel EPT/AMD NPT). Why do Intel EPT or AMD NPT matter here?
Quoted 4 lines
I think (though could be wrong and am open to correction here) that EPT and NPT enable the construction of write-only (ie not readable) memory mappings. Standard page tables can't do that: with those, if it's writable, it's also readable.
The hypervisor itself doesn't run on EPT or NPT second stage translation, that's used exclusively for (HVM) guests. So even if there's such support in EPT or NPT it's not relevant here. x86 page tables don't have the capability to create write-only mappings.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Christopher Clarkxen/include/public/argo.h line 41
Quoted 6 lines · Christopher Clark
xen/include/public/argo.h +6
-0+/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define XEN_ARGO_MAX_RING_SIZE (0x1000000ULL)
It looks like 16MB. Did I miss a <<10 somewhere or is it a typo in the comment?
Signature · 6 lines
Eric Chanudet _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 10 lines · Eric Chanudet
xen/include/public/argo.h line 41
Quoted 6 lines · Christopher Clark
xen/include/public/argo.h +6
-0+/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define XEN_ARGO_MAX_RING_SIZE (0x1000000ULL)It looks like 16MB. Did I miss a <<10 somewhere or is it a typo in the comment?
Yeah, it's an error in the comment. Thanks.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Christopher ClarkQuoted 6 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +0
-0@@ -23,16 +23,41 @@#include <xen/event.h> #include <xen/domain_page.h>
xen/common/argo.c line 34
Quoted 10 lines
xen/common/argo.c +6
-0#include <xen/guest_access.h> +#include <xen/lib.h> +#include <xen/nospec.h> #include <xen/time.h> #include <public/argo.h> +#define MAX_RINGS_PER_DOMAIN 128U + +/* All messages on the ring are padded to a multiple of the slot size. */ +#define ROUNDUP_MESSAGE (a) (ROUNDUP((a), XEN_ARGO_MSG_SLOT_SIZE))
Pointless outermost pair of parentheses.
Quoted 12 lines
xen/common/argo.c +8
-0@@ -198,6 +223,31 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#define argo_dprintk (format, ... ) ((void)0)#endif +/* +* This hash function is used to distribute rings within the per-domain +* hash tables (d->argo->ring_hash and d->argo_send_hash). The hash table +* will provide a struct if a match is found with a 'argo_ring_id' key: +* ie. the key is a (domain id, port, partner domain id) tuple. +* Since port number varies the most in expected use, and the Linux driver +* allocates at both the high and low ends, incorporate high and low bits to +* help with distribution.
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 243
Quoted 10 lines
xen/common/argo.c +10
-0+* Apply array_index_nospec as a defensive measure since this operates +* on user-supplied input and the array size that it indexes into is known. +*/ +static unsigned int +hash_index (const struct argo_ring_id *id) +{ +unsigned int hash; + + hash = (uint16_t )(id->port >>16 ); + hash ^= (uint16_t )id->port;
I may have asked this before, but are the casts really needed with ...
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 246
Quoted 3 lines
xen/common/argo.c +3
-0+ hash ^= id->domain_id; + hash ^= id->partner_id; + hash &= (ARGO_HTABLE_SIZE -1 );
... the masking done here?
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 248
Quoted 1 line
xen/common/argo.c +1
-0+return array_index_nospec (hash,ARGO_HTABLE_SIZE );
With the masking above - is this really needed?
And then the question is whether the quality of the hash is
sufficient: There won't be more set bits in the result than are in any of the three input values, so if they're all small, higher hash table entries won't be used at all. I would assume the goal to be that by the time 32 entities appear, chances be good that at least about 30 of the hash table entries are in use.
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 278
Quoted 11 lines
xen/common/argo.c +7
-0@@ -219,6 +269,78 @@ ring_unmap(struct argo_ring_info *ring_info) } } +static int +ring_map_page (struct argo_ring_info *ring_info,unsigned int i,void **out_ptr) +{ +if ( i >= ring_info->nmfns ) + { + gprintk(XENLOG_ERR, + "argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n",
ring_info->id.{domain,partner}_id look to be of the same type - why once %u and once %d? Same elsewhere.
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 282
Quoted 4 lines
xen/common/argo.c +4
-0+ ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); +return -ENOMEM ; + }
i = array_index_nospec(i, ring_info->nmfns);
considering the array indexes here? Of course at this point only zero can be passed in, but I assume this changes in later patches and the index is at least indirectly guest controlled.
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 497
Quoted 6 lines
xen/common/argo.c +2
-0@@ -371,6 +493,418 @@ partner_rings_remove(struct domain *src_d) } } +static int +find_ring_mfn (struct domain *d,gfn_t gfn,mfn_t *mfn)
So you have find_ring_mfn(), find_ring_mfns(), and ring_find_info(). Any chance you could use a consistent ordering of "ring" and "find"? Or is there a reason behind the apparent mismatch?
Quoted 16 lines
xen/common/argo.c +16
-0+{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d,gfn_x (gfn), &p2mt); +#else + *mfn =p2m_lookup (d, gfn, &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 523
Quoted 10 lines
xen/common/argo.c +10
-0+else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type (mfn_to_page (*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d,gfn_x (gfn)); +#endif + +return ret; +}
Please check whether you could leverage check_get_page_from_gfn() here. If you can't, please at least take inspiration as to e.g. the #ifdef-s from that function.
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 529
Quoted 5 lines
xen/common/argo.c +5
-0+static int +find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage, +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd, +uint32_t len)
Noticing it here, but perhaps still an issue elsewhere as well: Didn't we agree on removing unnecessary use of fixed width types? Or was that in the context on an earlier patch of v3?
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 540
Quoted 11 lines
xen/common/argo.c +11
-0+{ +unsigned int i; +int ret =0 ; +mfn_t *mfns; +uint8_t **mfn_mapping; + +/* +* first bounds check on npage here also serves as an overflow check +* before left shifting it +*/ + if ( (unlikely(npage > (XEN_ARGO_MAX_RING_SIZE >> PAGE_SHIFT))) ||
Isn't this redundant with the check in do_argo_p()?
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 548
Quoted 8 lines
xen/common/argo.c +8
-0+ ((npage << PAGE_SHIFT) < len) ) +return -EINVAL ; + +if ( ring_info->mfns ) + { +/* Ring already existed: drop the previous mapping. */ + gprintk(XENLOG_INFO, + "argo: vm%u re-register existing ring (vm%u:%x vm%d) clears mapping\n",
Indentation (also elsewhere).
Quoted 14 lines
xen/common/argo.c +14
-0+ d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id); + +ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; +
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 572
Quoted 10 lines
xen/common/argo.c +10
-0+ mfn_mapping = xzalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ; + } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping;
As the inverse to the cleanup sequence in an earlier patch: Please set ->npage last here even if it doesn't strictly matter.
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 574
Quoted 1 line
xen/common/argo.c +1
-0+ASSERT (ring_info->npage == npage);
What is this trying to make sure, seeing the assignment just a few lines up?
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 577
Quoted 2 lines
xen/common/argo.c +2
-0+if ( ring_info->nmfns == ring_info->npage ) +return 0 ;
Can this happen with the ring_remove_mfns() call above?
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 579
Quoted 1 line
xen/common/argo.c +1
-0+for ( i = ring_info->nmfns; i < ring_info->npage; i++ )
And hence can i start from other than zero here? And why not use the (possibly cheaper to access) function argument "npage" as the loop upper bound? The other similar loop a few lines up is coded that simpler way.
Quoted 24 lines
xen/common/argo.c +24
-0+static long +register_ring (struct domain *currd, +XEN_GUEST_HANDLE_PARAM (xen_argo_register_ring_t) reg_hnd, +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd, +uint32_t npage,bool fail_exist) +{ +xen_argo_register_ring_t reg; +struct argo_ring_id ring_id; +void *map_ringp; +xen_argo_ring_t *ringp; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info =NULL ; +struct domain *dst_d =NULL ; +int ret =0 ; +uint32_t private_tx_ptr; + +if (copy_from_guest (®, reg_hnd,1 ) ) + { + ret = -EFAULT ; + goto out; + } + +/* +* A ring must be large enough to transmit messages, so requires space for:
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 702
Quoted 10 lines
xen/common/argo.c +10
-0+* * 1 message header, plus +* * 1 payload slot (payload is always rounded to a multiple of 16 bytes) +* for the message payload to be written into, plus +* * 1 more slot, so that the ring cannot be filled to capacity with a +* single message -- see the logic in ringbuf_insert -- allowing for this +* ensures that there can be space remaining when a message is present. +* The above determines the minimum acceptable ring size. +*/ + if ( (reg.len < (sizeof(struct xen_argo_ring_message_header) + + ROUNDUP_MESSAGE(1) + ROUNDUP_MESSAGE(1))) ||
These two summands don't look to fulfill the "cannot be filled to capacity" constraint the comment describes, as (aiui) messages can be larger than 16 bytes. What's the deal?
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 715
Quoted 13 lines
xen/common/argo.c +13
-0+ (reg.len > XEN_ARGO_MAX_RING_SIZE) || + (reg.len != ROUNDUP_MESSAGE(reg.len)) || + (reg.pad != 0) ) + { + ret = -EINVAL ; + goto out; + } + + ring_id.partner_id = reg.partner_id; + ring_id.port = reg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock);
From here to ...
Quoted 31 lines
xen/common/argo.c +31
-0+if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +if ( reg.partner_id ==XEN_ARGO_DOMID_ANY ) + { +if ( opt_argo_mac_enforcing ) + { + ret = -EPERM ; + goto out_unlock; + } + } +else + { + dst_d =get_domain_by_id (reg.partner_id); +if ( !dst_d ) + { +argo_dprintk ("!dst_d, ESRCH\n" ); + ret = -ESRCH ; + goto out_unlock; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); + goto out_unlock; + }
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 757
Quoted 10 lines
xen/common/argo.c +10
-0+ + send_info = xzalloc(struct argo_send_info); +if ( !send_info ) + { + ret = -ENOMEM ; +put_domain (dst_d); + goto out_unlock; + } + send_info->id = ring_id; + }
... here, what exactly is it that requires the global read lock to be held ...
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 759
Quoted 1 line
xen/common/argo.c +1
-0+write_lock (&currd->argo->lock);
... prior to this? Holding locks around allocations is not forbidden, but should be avoided whenever possible.
And then further why does the global read lock need continued holding until the end of the function?
Quoted 22 lines
xen/common/argo.c +22
-0+if ( currd->argo->ring_count >=MAX_RINGS_PER_DOMAIN ) + { + ret = -ENOSPC ; + goto out_unlock2; + } + + ring_info =ring_find_info (currd, &ring_id); +if ( !ring_info ) + { + ring_info = xzalloc(struct argo_ring_info); +if ( !ring_info ) + { + ret = -ENOMEM ; + goto out_unlock2; + } + +spin_lock_init (&ring_info->lock); + + ring_info->id = ring_id; +INIT_HLIST_HEAD (&ring_info->pending); + + hlist_add_head(&ring_info->node,
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 792
Quoted 10 lines
xen/common/argo.c +10
-0+ &currd->argo->ring_hash[hash_index(&ring_info->id)]); + +gprintk (XENLOG_DEBUG ,"argo: vm%u registering ring (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } +else + { +if ( ring_info->len ) + {
Please fold into "else if ( )", removing a level of indentation.
Quoted 97 lines
xen/common/argo.c +97
-0+/* +* If the caller specified that the ring must not already exist, +* fail at attempt to add a completed ring which already exists. +*/ +if ( fail_exist ) + { +argo_dprintk ("disallowed reregistration of existing ring\n" ); + ret = -EEXIST ; + goto out_unlock2; + } + +if ( ring_info->len != reg.len ) + { +/* +* Change of ring size could result in entries on the pending +* notifications list that will never trigger. +* Simple blunt solution: disallow ring resize for now. +* TODO: investigate enabling ring resize. +*/ +gprintk (XENLOG_ERR , +"argo: vm%u attempted to change ring size(vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); +/* +* Could return EINVAL here, but if the ring didn't already +* exist then the arguments would have been valid, so: EEXIST. +*/ + ret = -EEXIST ; + goto out_unlock2; + } + +gprintk (XENLOG_DEBUG , +"argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } + } + + ret =find_ring_mfns (currd, ring_info, npage, pg_descr_hnd, reg.len); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to find ring mfns (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + +/* +* The first page of the memory supplied for the ring has the xen_argo_ring +* structure at its head, which is where the ring indexes reside. +*/ + ret =ring_map_page (ring_info,0 , &map_ringp); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to map ring mfn 0 (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + ringp = map_ringp; + + private_tx_ptr =read_atomic (&ringp->tx_ptr); + +if ( (private_tx_ptr >= reg.len) || + (ROUNDUP_MESSAGE (private_tx_ptr) != private_tx_ptr) ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + private_tx_ptr =ROUNDUP_MESSAGE (read_atomic (&ringp->rx_ptr)); + +if ( private_tx_ptr >= reg.len ) + private_tx_ptr =0 ; + +update_tx_ptr (ring_info, private_tx_ptr); + } + + ring_info->tx_ptr = private_tx_ptr; + ring_info->len = reg.len; + currd->argo->ring_count++; + +if ( send_info ) + { +spin_lock (&dst_d->argo->send_lock); + +hlist_add_head (&send_info->node, + &dst_d->argo->send_hash[hash_index (&send_info->id)]); + +spin_unlock (&dst_d->argo->send_lock);
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 899
Quoted 10 lines
xen/common/argo.c +10
-0+ } + + out_unlock2: +if ( !ret && send_info ) +xfree (send_info); + +if ( dst_d ) +put_domain (dst_d); + +write_unlock (&currd->argo->lock);
Surely you can drop the lock before the other two cleanup actions? That would then allow you to add another label to absorb the two separate put_domain() calls on error paths.
xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 32
Quoted 7 lines
xen/include/asm-arm/guest_access.h +1
-0@@ -29,6 +29,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask)))
This is unused throughout the patch afaics.
Quoted 15 lines
xen/include/public/argo.h +9
-0@@ -31,6 +31,26 @@#include "xen.h" +#define XEN_ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define XEN_ARGO_MAX_RING_SIZE (0x1000000ULL) +
xen/include/public/argo.h line 52
Quoted 10 lines
xen/include/public/argo.h +10
-0+/* +* Page descriptor: encoding both page address and size in a 64-bit value. +* Intended to allow ABI to support use of different granularity pages. +* example of how to populate: +* xen_argo_page_descr_t pg_desc = +* (physaddr & PAGE_MASK) | XEN_ARGO_PAGE_DESCR_SIZE_4K; +*/ +typedef uint64_t xen_argo_page_descr_t ; +#define XEN_ARGO_PAGE_DESCR_SIZE_MASK 0x0000000000000fffULL +#define XEN_ARGO_PAGE_DESCR_SIZE_4K 0
Are the _DESCR_ infixes here really useful?
Quoted 31 lines
xen/include/public/argo.h +27
-0@@ -56,4 +76,56 @@ typedef struct xen_argo_ring#endif } xen_argo_ring_t; +typedef struct xen_argo_register_ring +{ +uint32_t port; +domid_t partner_id; +uint16_t pad; +uint32_t len; +}xen_argo_register_ring_t ; + +/* Messages on the ring are padded to a multiple of this size. */ +#define XEN_ARGO_MSG_SLOT_SIZE 0x10 + +struct xen_argo_ring_message_header +{ +uint32_t len; +xen_argo_addr_t source; +uint32_t message_type; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t data[]; +#elif defined(__GNUC__) +uint8_t data[0 ]; +#endif +}; + +/* +* Hypercall operations +*/ +
xen/include/public/argo.h typedef struct xen_argo_ring · line 115
Quoted 10 lines
xen/include/public/argo.h +10
-0+/* +* XEN_ARGO_OP_register_ring +* +* Register a ring using the indicated memory. +* Also used to reregister an existing ring (eg. after resume from hibernate). +* +* arg1: XEN_GUEST_HANDLE(xen_argo_register_ring_t) +* arg2: XEN_GUEST_HANDLE(xen_argo_page_descr_t) +* arg3: unsigned long npages +* arg4: unsigned long flags
The "unsigned long"-s here are not necessarily compatible with compat mode. At the very least flags above bit 31 won't be usable by compat mode guests. Hence I also question ...
Quoted 4 lines
xen/include/public/argo.h +4
-0+*/ +#define XEN_ARGO_OP_register_ring 1 + +/* Register op flags */
xen/include/public/argo.h typedef struct xen_argo_ring · line 129
Quoted 10 lines
xen/include/public/argo.h +10
-0+/* +* Fail exist: +* If set, reject attempts to (re)register an existing established ring. +* If clear, reregistration occurs if the ring exists, with the new ring +* taking the place of the old, preserving tx_ptr if it remains valid. +*/ +#define XEN_ARGO_REGISTER_FLAG_FAIL_EXIST 0x1 + +/* Mask for all defined flags. unsigned long type so ok for both 32/64-bit */ +#define XEN_ARGO_REGISTER_FLAG_MASK 0x1UL
... the UL suffix here. Also this last item should not be exposed (perhaps framed by "#ifdef __XEN__") and would perhaps anyway better be defined in terms of the other XEN_ARGO_REGISTER_FLAG_*.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 2 lines
Quoted 2 lines · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +2
-0@@ -23,16 +23,41 @@#include <xen/event.h> #include <xen/domain_page.h> #include <xen/guest_access.h> +#include <xen/lib.h> +#include <xen/nospec.h> #include <xen/time.h> #include <public/argo.h>
Quoted 6 lines
xen/common/argo.c line 34
Quoted 4 lines
xen/common/argo.c +4
-0+#define MAX_RINGS_PER_DOMAIN 128U + +/* All messages on the ring are padded to a multiple of the slot size. */ +#define ROUNDUP_MESSAGE (a) (ROUNDUP((a), XEN_ARGO_MSG_SLOT_SIZE))Pointless outermost pair of parentheses.
ack, removed
Quoted 37 lines
Quoted 13 lines
xen/common/argo.c +8
-0@@ -198,6 +223,31 @@ static DEFINE_RWLOCK(argo_lock); /* L1 */#define argo_dprintk (format, ... ) ((void)0)#endif +/* +* This hash function is used to distribute rings within the per-domain +* hash tables (d->argo->ring_hash and d->argo_send_hash). The hash table +* will provide a struct if a match is found with a 'argo_ring_id' key: +* ie. the key is a (domain id, port, partner domain id) tuple. +* Since port number varies the most in expected use, and the Linux driver +* allocates at both the high and low ends, incorporate high and low bits to +* help with distribution. xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 243
Quoted 10 lines
xen/common/argo.c +10
-0+* Apply array_index_nospec as a defensive measure since this operates +* on user-supplied input and the array size that it indexes into is known. +*/ +static unsigned int +hash_index (const struct argo_ring_id *id) +{ +unsigned int hash; + + hash = (uint16_t )(id->port >>16 ); + hash ^= (uint16_t )id->port;I may have asked this before, but are the casts really needed with ...
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 246
Quoted 3 lines
xen/common/argo.c +3
-0+ hash ^= id->domain_id; + hash ^= id->partner_id; + hash &= (ARGO_HTABLE_SIZE -1 );... the masking done here?
xen/common/argo.c static DEFINE_RWLOCK(argo_lock); /* L1 */ · line 248
Quoted 1 line
xen/common/argo.c +1
-0+return array_index_nospec (hash,ARGO_HTABLE_SIZE );With the masking above - is this really needed?
Quoted 7 lines
And then the question is whether the quality of the hash is
sufficient: There won't be more set bits in the result than are in any of the three input values, so if they're all small, higher hash table entries won't be used at all. I would assume the goal to be that by the time 32 entities appear, chances be good that at least about 30 of the hash table entries are in use.
ok, I'll replace this function and address the above.
I'm out of time today so have added a FIXME for today's series posting and will get it done tomorrow.
Quoted 5 lines
Quoted 5 lines
xen/common/argo.c +0
-0@@ -219,6 +269,78 @@ ring_unmap(struct argo_ring_info *ring_info) } }
Quoted 10 lines
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 278
Quoted 7 lines
xen/common/argo.c +7
-0+static int +ring_map_page (struct argo_ring_info *ring_info,unsigned int i,void **out_ptr) +{ +if ( i >= ring_info->nmfns ) + { + gprintk(XENLOG_ERR, + "argo: ring (vm%u:%x vm%d) %p attempted to map page %u of %u\n",ring_info->id.{domain,partner}_id look to be of the same type - why once %u and once %d? Same elsewhere.
Fixed across the series to use %u for domid_t output.
Quoted 11 lines
xen/common/argo.c ring_unmap(struct argo_ring_info *ring_info) · line 282
Quoted 5 lines
xen/common/argo.c +4
-0+ ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, i, ring_info->nmfns); +return -ENOMEM ; + }i = array_index_nospec(i, ring_info->nmfns);considering the array indexes here? Of course at this point only zero can be passed in, but I assume this changes in later patches and the index is at least indirectly guest controlled.
Added, thanks.
Quoted 11 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 497
Quoted 7 lines
xen/common/argo.c +2
-0@@ -371,6 +493,418 @@ partner_rings_remove(struct domain *src_d) } } +static int +find_ring_mfn (struct domain *d,gfn_t gfn,mfn_t *mfn)So you have
find_ring_mfn(),find_ring_mfns(), andring_find_info(). Any chance you could use a consistent ordering of "ring" and "find"? Or is there a reason behind the apparent mismatch?
I've renamed them to use 'find_' as the common prefix. Look cleaner to me. thanks.
Quoted 23 lines
Quoted 23 lines
xen/common/argo.c +22
-0+{ +p2m_type_t p2mt; +int ret =0 ; + +#ifdef CONFIG_X86 + *mfn =get_gfn_unshare (d,gfn_x (gfn), &p2mt); +#else + *mfn =p2m_lookup (d, gfn, &p2mt); +#endif + +if ( !mfn_valid (*mfn) ) + ret = -EINVAL ; +#ifdef CONFIG_X86 + else if ( p2m_is_paging(p2mt) || (p2mt == p2m_ram_logdirty) ) + ret = -EAGAIN ; +#endif + else if ( (p2mt != p2m_ram_rw) || + !get_page_and_type(mfn_to_page(*mfn), d, PGT_writable_page) ) + ret = -EINVAL ; + +#ifdef CONFIG_X86 +put_gfn (d,gfn_x (gfn));
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 523
Quoted 4 lines
xen/common/argo.c +4
-0+#endif + +return ret; +}Please check whether you could leverage
check_get_page_from_gfn()here. If you can't, please at least take inspiration as to e.g. the #ifdef-s from that function.
Have added a temporary FIXME for this and will do this tomorrow.
Quoted 10 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 529
Quoted 6 lines
xen/common/argo.c +5
-0+static int +find_ring_mfns (struct domain *d,struct argo_ring_info *ring_info, +uint32_t npage, +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd, +uint32_t len)Noticing it here, but perhaps still an issue elsewhere as well: Didn't we agree on removing unnecessary use of fixed width types? Or was that in the context on an earlier patch of v3?
These are fixed and hopefully all the others that do not belong are also gone in v4.
Quoted 17 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 540
Quoted 12 lines
xen/common/argo.c +11
-0+{ +unsigned int i; +int ret =0 ; +mfn_t *mfns; +uint8_t **mfn_mapping; + +/* +* first bounds check on npage here also serves as an overflow check +* before left shifting it +*/ + if ( (unlikely(npage > (XEN_ARGO_MAX_RING_SIZE >> PAGE_SHIFT))) ||Isn't this redundant with the check in
do_argo_p()?Quoted 2 lines
xen/common/argo.c +2
-0+ ((npage << PAGE_SHIFT) < len) ) +return -EINVAL ;
Answering your point inline above: Yes - do_argo_op does the bounds checking, so I've removed the entire check above.
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 548
Quoted 6 lines
xen/common/argo.c +6
-0+ +if ( ring_info->mfns ) + { +/* Ring already existed: drop the previous mapping. */ + gprintk(XENLOG_INFO, + "argo: vm%u re-register existing ring (vm%u:%x vm%d) clears mapping\n",Indentation (also elsewhere).
Ack, fixed here and elsewhere.
Quoted 20 lines
Quoted 20 lines
xen/common/argo.c +19
-0+ d->domain_id, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id); + +ring_remove_mfns (d, ring_info); +ASSERT (!ring_info->mfns); + } + + mfns =xmalloc_array (mfn_t, npage); +if ( !mfns ) +return -ENOMEM ; + +for ( i =0 ; i < npage; i++ ) + mfns[i] =INVALID_MFN ; + + mfn_mapping = xzalloc_array(uint8_t *, npage); +if ( !mfn_mapping ) + { +xfree (mfns); +return -ENOMEM ;
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 572
Quoted 5 lines
xen/common/argo.c +5
-0+ } + + ring_info->npage = npage; + ring_info->mfns = mfns; + ring_info->mfn_mapping = mfn_mapping;As the inverse to the cleanup sequence in an earlier patch: Please set ->npage last here even if it doesn't strictly matter.
npage is now gone after implementing Roger's feedback to only keep "len".
Quoted 5 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 574
Quoted 2 lines
xen/common/argo.c +1
-0+ASSERT (ring_info->npage == npage);What is this trying to make sure, seeing the assignment just a few lines up?
removed
Quoted 5 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 577
Quoted 3 lines
xen/common/argo.c +2
-0+if ( ring_info->nmfns == ring_info->npage ) +return 0 ;Can this happen with the
ring_remove_mfns()call above?
No, not any more, you're right.
Quoted 7 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 579
Quoted 2 lines
xen/common/argo.c +1
-0+for ( i = ring_info->nmfns; i < ring_info->npage; i++ )And hence can i start from other than zero here? And why not use the (possibly cheaper to access) function argument "npage" as the loop upper bound? The other similar loop a few lines up is coded that simpler way.
Yes, thanks, done.
Quoted 31 lines
Quoted 31 lines
xen/common/argo.c +30
-0+static long +register_ring (struct domain *currd, +XEN_GUEST_HANDLE_PARAM (xen_argo_register_ring_t) reg_hnd, +XEN_GUEST_HANDLE_PARAM (xen_argo_page_descr_t) pg_descr_hnd, +uint32_t npage,bool fail_exist) +{ +xen_argo_register_ring_t reg; +struct argo_ring_id ring_id; +void *map_ringp; +xen_argo_ring_t *ringp; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info =NULL ; +struct domain *dst_d =NULL ; +int ret =0 ; +uint32_t private_tx_ptr; + +if (copy_from_guest (®, reg_hnd,1 ) ) + { + ret = -EFAULT ; + goto out; + } + +/* +* A ring must be large enough to transmit messages, so requires space for: +* * 1 message header, plus +* * 1 payload slot (payload is always rounded to a multiple of 16 bytes) +* for the message payload to be written into, plus +* * 1 more slot, so that the ring cannot be filled to capacity with a +* single message -- see the logic in ringbuf_insert -- allowing for this +* ensures that there can be space remaining when a message is present.
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 702
Quoted 4 lines
xen/common/argo.c +4
-0+* The above determines the minimum acceptable ring size. +*/ + if ( (reg.len < (sizeof(struct xen_argo_ring_message_header) + + ROUNDUP_MESSAGE(1) + ROUNDUP_MESSAGE(1))) ||These two summands don't look to fulfill the "cannot be filled to capacity" constraint the comment describes, as (aiui) messages can be larger than 16 bytes. What's the deal?
This is intended to be about bound checking reg.len against a minimum
size: the smallest ring that you can fit a message onto, as determined by the logic in ringbuf_insert. The smallest message you can send is:sizeof(struct xen_argo_ring_message_header) + ROUNDUP_MESSAGE(1)
and then ringbuf_insert won't accept a message unless there is (at least) ROUNDUP_MESSAGE(1) space remaining, so that, plus the smallest message size, is the smallest viable ring. There's no point accepting registration of a ring smaller than that.
You're right that messages can be larger than 16 bytes, but they can only be sent to rings that are larger than that minimum - on a minimum sized ring, they'll be rejected by sendv.
Quoted 62 lines
Quoted 4 lines
xen/common/argo.c +3
-0+ (reg.len >XEN_ARGO_MAX_RING_SIZE ) || + (reg.len !=ROUNDUP_MESSAGE (reg.len)) || + (reg.pad != 0) )xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 715
Quoted 10 lines
xen/common/argo.c +10
-0+ { + ret = -EINVAL ; + goto out; + } + + ring_id.partner_id = reg.partner_id; + ring_id.port = reg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock);From here to ...
Quoted 31 lines
xen/common/argo.c +31
-0+if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +if ( reg.partner_id ==XEN_ARGO_DOMID_ANY ) + { +if ( opt_argo_mac_enforcing ) + { + ret = -EPERM ; + goto out_unlock; + } + } +else + { + dst_d =get_domain_by_id (reg.partner_id); +if ( !dst_d ) + { +argo_dprintk ("!dst_d, ESRCH\n" ); + ret = -ESRCH ; + goto out_unlock; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; +put_domain (dst_d); + goto out_unlock; + }xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 757
Quoted 10 lines
xen/common/argo.c +10
-0+ + send_info = xzalloc(struct argo_send_info); +if ( !send_info ) + { + ret = -ENOMEM ; +put_domain (dst_d); + goto out_unlock; + } + send_info->id = ring_id; + }... here, what exactly is it that requires the global read lock to be held ...
Quoted 7 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 759
Quoted 1 line
xen/common/argo.c +1
-0+write_lock (&currd->argo->lock);... prior to this? Holding locks around allocations is not forbidden, but should be avoided whenever possible.
And then further why does the global read lock need continued holding until the end of the function?
I've added FIXME to review this tomorrow. I understand the argo-internal locking protocols and this is adhering to what they state, in that accesses to the argo structs of currd and dst_d are protected by the global read lock here, but at the moment I'm less clear on what the expectations are for standard Xen domain locks, references and lifecycle are.
Quoted 27 lines
Quoted 27 lines
xen/common/argo.c +26
-0+if ( currd->argo->ring_count >=MAX_RINGS_PER_DOMAIN ) + { + ret = -ENOSPC ; + goto out_unlock2; + } + + ring_info =ring_find_info (currd, &ring_id); +if ( !ring_info ) + { + ring_info = xzalloc(struct argo_ring_info); +if ( !ring_info ) + { + ret = -ENOMEM ; + goto out_unlock2; + } + +spin_lock_init (&ring_info->lock); + + ring_info->id = ring_id; +INIT_HLIST_HEAD (&ring_info->pending); + +hlist_add_head (&ring_info->node, + &currd->argo->ring_hash[hash_index (&ring_info->id)]); + + gprintk(XENLOG_DEBUG, "argo: vm%u registering ring (vm%u:%x vm%d)\n", + currd->domain_id, ring_id.domain_id, ring_id.port,
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 792
Quoted 6 lines
xen/common/argo.c +6
-0+ ring_id.partner_id); + } +else + { +if ( ring_info->len ) + {Please fold into "else if ( )", removing a level of indentation.
ack, done
Quoted 104 lines
Quoted 104 lines
xen/common/argo.c +103
-0+/* +* If the caller specified that the ring must not already exist, +* fail at attempt to add a completed ring which already exists. +*/ +if ( fail_exist ) + { +argo_dprintk ("disallowed reregistration of existing ring\n" ); + ret = -EEXIST ; + goto out_unlock2; + } + +if ( ring_info->len != reg.len ) + { +/* +* Change of ring size could result in entries on the pending +* notifications list that will never trigger. +* Simple blunt solution: disallow ring resize for now. +* TODO: investigate enabling ring resize. +*/ +gprintk (XENLOG_ERR , +"argo: vm%u attempted to change ring size(vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); +/* +* Could return EINVAL here, but if the ring didn't already +* exist then the arguments would have been valid, so: EEXIST. +*/ + ret = -EEXIST ; + goto out_unlock2; + } + +gprintk (XENLOG_DEBUG , +"argo: vm%u re-registering existing ring (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + } + } + + ret =find_ring_mfns (currd, ring_info, npage, pg_descr_hnd, reg.len); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to find ring mfns (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + +/* +* The first page of the memory supplied for the ring has the xen_argo_ring +* structure at its head, which is where the ring indexes reside. +*/ + ret =ring_map_page (ring_info,0 , &map_ringp); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: vm%u failed to map ring mfn 0 (vm%u:%x vm%d)\n" , + currd->domain_id, ring_id.domain_id, ring_id.port, + ring_id.partner_id); + +ring_remove_info (currd, ring_info); + goto out_unlock2; + } + ringp = map_ringp; + + private_tx_ptr =read_atomic (&ringp->tx_ptr); + +if ( (private_tx_ptr >= reg.len) || + (ROUNDUP_MESSAGE (private_tx_ptr) != private_tx_ptr) ) + { +/* +* Since the ring is a mess, attempt to flush the contents of it +* here by setting the tx_ptr to the next aligned message slot past +* the latest rx_ptr we have observed. Handle ring wrap correctly. +*/ + private_tx_ptr =ROUNDUP_MESSAGE (read_atomic (&ringp->rx_ptr)); + +if ( private_tx_ptr >= reg.len ) + private_tx_ptr =0 ; + +update_tx_ptr (ring_info, private_tx_ptr); + } + + ring_info->tx_ptr = private_tx_ptr; + ring_info->len = reg.len; + currd->argo->ring_count++; + +if ( send_info ) + { +spin_lock (&dst_d->argo->send_lock); + +hlist_add_head (&send_info->node, + &dst_d->argo->send_hash[hash_index (&send_info->id)]); + +spin_unlock (&dst_d->argo->send_lock); + } + + out_unlock2: +if ( !ret && send_info ) +xfree (send_info); +
Quoted 8 lines
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 899
Quoted 4 lines
xen/common/argo.c +4
-0+if ( dst_d ) +put_domain (dst_d); + +write_unlock (&currd->argo->lock);Surely you can drop the lock before the other two cleanup actions? That would then allow you to add another label to absorb the two separate
put_domain()calls on error paths.
That looks correct. Added a FIXME note now and will fix tomorrow. thanks.
Quoted 10 lines
xen/include/asm-arm/guest_access.h int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf, · line 32
Quoted 8 lines
xen/include/asm-arm/guest_access.h +1
-0@@ -29,6 +29,8 @@ int access_guest_memory_by_ipa(struct domain *d, paddr_t ipa, void *buf,/* Is the guest handle a NULL reference? */ #define guest_handle_is_null (hnd) ((hnd).p == NULL) +#define guest_handle_is_aligned (hnd, mask) (!((uintptr_t)(hnd).p & (mask)))This is unused throughout the patch afaics.
Removed.
Quoted 20 lines
Quoted 20 lines
xen/include/public/argo.h +13
-0@@ -31,6 +31,26 @@#include "xen.h" +#define XEN_ARGO_DOMID_ANY DOMID_INVALID + +/* +* The maximum size of an Argo ring is defined to be: 16GB +* -- which is 0x1000000 bytes. +* A byte index into the ring is at most 24 bits. +*/ +#define XEN_ARGO_MAX_RING_SIZE (0x1000000ULL) + +/* +* Page descriptor: encoding both page address and size in a 64-bit value. +* Intended to allow ABI to support use of different granularity pages. +* example of how to populate:
Quoted 8 lines
xen/include/public/argo.h line 52
Quoted 6 lines
xen/include/public/argo.h +6
-0+* xen_argo_page_descr_t pg_desc = +* (physaddr & PAGE_MASK) | XEN_ARGO_PAGE_DESCR_SIZE_4K; +*/ +typedef uint64_t xen_argo_page_descr_t ; +#define XEN_ARGO_PAGE_DESCR_SIZE_MASK 0x0000000000000fffULL +#define XEN_ARGO_PAGE_DESCR_SIZE_4K 0Are the _DESCR_ infixes here really useful?
These are now gone, with Julien's approval for the change back to use the gfn-using interfaces.
Quoted 58 lines
Quoted 32 lines
xen/include/public/argo.h +27
-0@@ -56,4 +76,56 @@ typedef struct xen_argo_ring#endif } xen_argo_ring_t; +typedef struct xen_argo_register_ring +{ +uint32_t port; +domid_t partner_id; +uint16_t pad; +uint32_t len; +}xen_argo_register_ring_t ; + +/* Messages on the ring are padded to a multiple of this size. */ +#define XEN_ARGO_MSG_SLOT_SIZE 0x10 + +struct xen_argo_ring_message_header +{ +uint32_t len; +xen_argo_addr_t source; +uint32_t message_type; +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >=199901L +uint8_t data[]; +#elif defined(__GNUC__) +uint8_t data[0 ]; +#endif +}; + +/* +* Hypercall operations +*/ +xen/include/public/argo.h typedef struct xen_argo_ring · line 115
Quoted 10 lines
xen/include/public/argo.h +10
-0+/* +* XEN_ARGO_OP_register_ring +* +* Register a ring using the indicated memory. +* Also used to reregister an existing ring (eg. after resume from hibernate). +* +* arg1: XEN_GUEST_HANDLE(xen_argo_register_ring_t) +* arg2: XEN_GUEST_HANDLE(xen_argo_page_descr_t) +* arg3: unsigned long npages +* arg4: unsigned long flags The "unsigned long"-s here are not necessarily compatible with compat mode. At the very least flags above bit 31 won't be usable by compat mode guests. Hence I also question ...
Quoted 11 lines
xen/include/public/argo.h +11
-0+*/ +#define XEN_ARGO_OP_register_ring 1 + +/* Register op flags */ +/* +* Fail exist: +* If set, reject attempts to (re)register an existing established ring. +* If clear, reregistration occurs if the ring exists, with the new ring +* taking the place of the old, preserving tx_ptr if it remains valid. +*/ +#define XEN_ARGO_REGISTER_FLAG_FAIL_EXIST 0x1
Quoted 8 lines
xen/include/public/argo.h typedef struct xen_argo_ring · line 129
Quoted 3 lines
xen/include/public/argo.h +3
-0+ +/* Mask for all defined flags. unsigned long type so ok for both 32/64-bit */ +#define XEN_ARGO_REGISTER_FLAG_MASK 0x1UL... the UL suffix here. Also this last item should not be exposed (perhaps framed by "#ifdef
__XEN__") and would perhaps anyway better be defined in terms of the other XEN_ARGO_REGISTER_FLAG_*.
Notes added in place for each of the above; will work on these tomorrow.
thanks
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 15.01.19 at 08:56, <[email protected]> wrote:
Quoted 14 lines · Jan Beulich
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 702
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +12
-0+/* +* A ring must be large enough to transmit messages, so requires space for: +* * 1 message header, plus +* * 1 payload slot (payload is always rounded to a multiple of 16 bytes) +* for the message payload to be written into, plus +* * 1 more slot, so that the ring cannot be filled to capacity with a +* single message -- see the logic in ringbuf_insert -- allowing for this +* ensures that there can be space remaining when a message is present. +* The above determines the minimum acceptable ring size. +*/ + if ( (reg.len < (sizeof(struct xen_argo_ring_message_header) + + ROUNDUP_MESSAGE(1) + ROUNDUP_MESSAGE(1))) ||Quoted 3 lines
These two summands don't look to fulfill the "cannot be filled to capacity" constraint the comment describes, as (aiui) messages can be larger than 16 bytes. What's the deal?
This is intended to be about bound checking reg.len against a minimum
size: the smallest ring that you can fit a message onto, as determined by the logic inringbuf_insert. The smallest message you can send is:sizeof(struct xen_argo_ring_message_header)+ROUNDUP_MESSAGE(1)
Quoted 8 lines
and then
ringbuf_insertwon't accept a message unless there is (at least)ROUNDUP_MESSAGE(1)space remaining, so that, plus the smallest message size, is the smallest viable ring. There's no point accepting registration of a ring smaller than that.You're right that messages can be larger than 16 bytes, but they can only be sent to rings that are larger than that minimum - on a minimum sized ring, they'll be rejected by sendv.
So perhaps the comment wants to say "... cannot be filled to capacity with a single, minimum size message"?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 31 lines · Jan Beulich
Quoted 31 lines
Quoted 2 lines
Quoted 2 lines · On 15.01.19 at 08:56, <[email protected]> wrote:
Quoted 14 lines · Jan Beulich
xen/common/argo.c partner_rings_remove(struct domain *src_d) · line 702
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +12
-0+/* +* A ring must be large enough to transmit messages, so requires space for: +* * 1 message header, plus +* * 1 payload slot (payload is always rounded to a multiple of 16 bytes) +* for the message payload to be written into, plus +* * 1 more slot, so that the ring cannot be filled to capacity with a +* single message -- see the logic in ringbuf_insert -- allowing for this +* ensures that there can be space remaining when a message is present. +* The above determines the minimum acceptable ring size. +*/ + if ( (reg.len < (sizeof(struct xen_argo_ring_message_header) + + ROUNDUP_MESSAGE(1) + ROUNDUP_MESSAGE(1))) ||Quoted 3 lines
These two summands don't look to fulfill the "cannot be filled to capacity" constraint the comment describes, as (aiui) messages can be larger than 16 bytes. What's the deal?
This is intended to be about bound checking reg.len against a minimum
size: the smallest ring that you can fit a message onto, as determined by the logic inringbuf_insert. The smallest message you can send is:sizeof(struct xen_argo_ring_message_header)+ROUNDUP_MESSAGE(1)and then
ringbuf_insertwon't accept a message unless there is (at least)ROUNDUP_MESSAGE(1)space remaining, so that, plus the smallest message size, is the smallest viable ring. There's no point accepting registration of a ring smaller than that.
Quoted 6 lines
Quoted 3 lines
You're right that messages can be larger than 16 bytes, but they can only be sent to rings that are larger than that minimum - on a minimum sized ring, they'll be rejected by sendv.
So perhaps the comment wants to say "... cannot be filled to capacity with a single, minimum size message"?
ack, done.
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
In reply to Christopher Clarkdocs/misc/xen-command-line.pandoc This allows domains access to the Argo hypercall, which supports registration · line 197
Quoted 10 lines · Christopher Clark
docs/misc/xen-command-line.pandoc +2
-0@@ -193,6 +193,21 @@ This allows domains access to the Argo hypercall, which supports registration of memory rings with the hypervisor to receive messages, sending messages to other domains by hypercall and querying the ring status of other domains. +### argo-mac +> `= permissive | enforcing`
Are these command line options already in use in the OpenXT community?
I ask, because we are trying to avoid gaining multiple top level options for related functionatliy.
IMO, this functionality could be covered more succinctly with:
argo = List of [ <bool>, mac = permissive | enforcing ]
which also allows for cleaner addition of future options.
(Unfortunately, to implement this, you need my cmdline_strcmp() fixes, which are still pending an ack.)
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 07/15] argo: implement the register op
Quoted 10 lines · Andrew Cooper
Quoted 8 lines · Christopher Clark
docs/misc/xen-command-line.pandoc +0
-0@@ -193,6 +193,21 @@ This allows domains access to the Argo hypercall, which supports registration of memory rings with the hypervisor to receive messages, sending messages to other domains by hypercall and querying the ring status of other domains.
Quoted 4 lines
docs/misc/xen-command-line.pandoc This allows domains access to the Argo hypercall, which supports registration · line 197
Quoted 2 lines
docs/misc/xen-command-line.pandoc +2
-0+### argo-mac +> `= permissive | enforcing`Are these command line options already in use in the OpenXT community?
No, there's no concern there.
Quoted 11 lines
I ask, because we are trying to avoid gaining multiple top level options for related functionatliy.
IMO, this functionality could be covered more succinctly with:
argo = List of [ <bool>, mac = permissive | enforcing ]
which also allows for cleaner addition of future options.
(Unfortunately, to implement this, you need my
cmdline_strcmp()fixes, which are still pending an ack.)
At Roger's recommendation, the "argo-mac" string option has become the much simpler "argo-mac-permissive" boolean.
For 4.13, I'd like to improve the isolation of wildcard rings to the point where they're able to be enabled whenever argo itself is, and hopefully that will enable retiring this option altogether. If two bools are tolerable for now (as this is still early and an experimental feature), would it be OK to sort this out for 4.13?
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 8/15 argo: implement the unregister op
In reply to Christopher ClarkTakes a single argument: a handle to the ring unregistration struct, which specifies the port and partner domain id or wildcard.
The ring's entry is removed from the hashtable of registered rings; any entries for pending notifications are removed; and the ring is unmapped from Xen's address space.
If the ring had been registered to communicate with a single specified domain (ie. a non-wildcard ring) then the partner domain state is removed from the partner domain's argo send_info hash table.
---
v2 feedback Jan: drop cookie, implement teardown
v2 feedback Jan: drop message from argo_message_op
v2 self: OVERHAUL
v2 self: reorder logic to shorten critical section
v1 #13 feedback Jan: revise use of guest_handle_okay vs __copy ops v1 feedback Roger, Jan: drop argo prefix on static functions v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 #5 (#14) feedback Paul: use currd in do_argo_message_op
v1 #5 (#14) feedback Paul: full use currd in argo_unregister_ring v1 #13 (#14) feedback Paul: replace do/while with goto; reindent v1 self: add blank lines in unregister case in do_argo_message_op
v1: #13 feedback Jan: public namespace: prefix with xen
v1: #13 feedback Jan: blank line after op case in do_argo_message_op
v1: #14 feedback Jan: replace domain id override with validation
v1: #18 feedback Jan: meld the ring count limit into the series
v1: feedback #15 Jan: verify zero in unused hypercall args
xen/common/argo.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 19 ++++++++ xen/include/xlat.lst | 1 + 3 files changed, 135 insertions(+)
xen/common/argo.c +115 -0
@@ -37,6 +37,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_argo_addr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_page_descr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_register_ring_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_unregister_ring_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled;
@@ -666,6 +667,105 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_send_info * +send_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +struct hlist_node *node; +struct argo_send_info *send_info; + +hlist_for_each_entry (send_info, node, &d->argo->send_hash[hash_index (id)], + node) + { +struct argo_ring_id *cmpid = &send_info->id; + +if ( cmpid->port == id->port && + cmpid->domain_id == id->domain_id && + cmpid->partner_id == id->partner_id ) + { +argo_dprintk ("send_info=%p\n" , send_info); +return send_info; + } + } +argo_dprintk ("no send_info found\n" ); + +return NULL ; +} + +static long +unregister_ring (struct domain *currd, +XEN_GUEST_HANDLE_PARAM (xen_argo_unregister_ring_t) unreg_hnd) +{ +xen_argo_unregister_ring_t unreg; +struct argo_ring_id ring_id; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info; +struct domain *dst_d =NULL ; +int ret; + + ret =copy_from_guest (&unreg, unreg_hnd,1 ) ? -EFAULT :0 ; +if ( ret ) + goto out; + + ret = unreg.pad ? -EINVAL :0 ; +if ( ret ) + goto out; + + ring_id.partner_id = unreg.partner_id; + ring_id.port = unreg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock); + +if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +write_lock (&currd->argo->lock); + + ring_info =ring_find_info (currd, &ring_id); +if ( ring_info ) + { +ring_remove_info (currd, ring_info); + currd->argo->ring_count--; + } + + dst_d =get_domain_by_id (ring_id.partner_id); +if ( dst_d ) + { +if ( dst_d->argo ) + { +spin_lock (&dst_d->argo->send_lock); + + send_info =send_find_info (dst_d, &ring_id); +if ( send_info ) + { +hlist_del (&send_info->node); +xfree (send_info); + } + +spin_unlock (&dst_d->argo->send_lock); + } +put_domain (dst_d); + } + +write_unlock (&currd->argo->lock); + +if ( !ring_info ) + { +argo_dprintk ("ENOENT\n" ); + ret = -ENOENT ; + goto out_unlock; + } + + out_unlock: +read_unlock (&argo_lock); + + out: +return ret; +} + static long register_ring(struct domain *currd, XEN_GUEST_HANDLE_PARAM(xen_argo_register_ring_t) reg_hnd,
@@ -958,6 +1058,21 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,break ; } +case XEN_ARGO_OP_unregister_ring: + { +XEN_GUEST_HANDLE_PARAM (xen_argo_unregister_ring_t) unreg_hnd = +guest_handle_cast (arg1, xen_argo_unregister_ring_t); + +if (unlikely ((!guest_handle_is_null (arg2)) || arg3 || arg4) ) + { + rc = -EINVAL ; +break ; + } + + rc =unregister_ring (currd, unreg_hnd); +break ; + } +default : rc = -EOPNOTSUPP ;break ;
xen/include/public/argo.h +19 -0
@@ -84,6 +84,13 @@ typedef struct xen_argo_register_ringuint32_t len; } xen_argo_register_ring_t; +typedef struct xen_argo_unregister_ring +{ +uint32_t port; +domid_t partner_id; +uint16_t pad; +}xen_argo_unregister_ring_t ; +/* Messages on the ring are padded to a multiple of this size. */ #define XEN_ARGO_MSG_SLOT_SIZE 0x10
@@ -128,4 +135,16 @@ struct xen_argo_ring_message_header/* Mask for all defined flags. unsigned long type so ok for both 32/64-bit */ #define XEN_ARGO_REGISTER_FLAG_MASK 0x1UL +/* +* XEN_ARGO_OP_unregister_ring +* +* Unregister a previously-registered ring, ending communication. +* +* arg1: XEN_GUEST_HANDLE(xen_argo_unregister_ring_t) +* arg2: NULL +* arg3: 0 (ZERO) +* arg4: 0 (ZERO) +*/ +#define XEN_ARGO_OP_unregister_ring 2 +#endif
xen/include/xlat.lst +1 -0
@@ -151,3 +151,4 @@ ? argo_addr argo.h ? argo_ring argo.h ? argo_register_ring argo.h +? argo_unregister_ring argo.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 08/15] argo: implement the unregister op
Quoted 52 lines · Christopher Clark
Takes a single argument: a handle to the ring unregistration struct, which specifies the port and partner domain id or wildcard.
The ring's entry is removed from the hashtable of registered rings; any entries for pending notifications are removed; and the ring is unmapped from Xen's address space.
If the ring had been registered to communicate with a single specified domain (ie. a non-wildcard ring) then the partner domain state is removed from the partner domain's argo
send_infohash table.Signed-off-by Christopher Clark <[email protected]>---
v2 feedback Jan: drop cookie, implement teardown
v2 feedback Jan: drop message fromargo_message_op
v2 self: OVERHAUL
v2 self: reorder logic to shorten critical section
v1 #13 feedback Jan: revise use ofguest_handle_okayvs__copyops v1 feedback Roger, Jan: drop argo prefix on static functions v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 #5 (#14) feedback Paul: use currd indo_argo_message_op
v1 #5 (#14) feedback Paul: full use currd inargo_unregister_ringv1 #13 (#14) feedback Paul: replace do/while with goto; reindent v1 self: add blank lines in unregister case indo_argo_message_op
v1: #13 feedback Jan: public namespace: prefix with xen
v1: #13 feedback Jan: blank line after op case indo_argo_message_op
v1: #14 feedback Jan: replace domain id override with validation
v1: #18 feedback Jan: meld the ring count limit into the series
v1: feedback #15 Jan: verify zero in unused hypercall argsxen/common/argo.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 19 ++++++++ xen/include/xlat.lst | 1 + 3 files changed, 135 insertions(+)xen/common/argo.c +1
-0@@ -37,6 +37,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_argo_addr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_page_descr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_register_ring_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_unregister_ring_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled;@@ -666,6 +667,105 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; }
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 679
Quoted 10 lines
xen/common/argo.c +10
-0+static struct argo_send_info * +send_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +struct hlist_node *node; +struct argo_send_info *send_info; + +hlist_for_each_entry (send_info, node, &d->argo->send_hash[hash_index (id)], + node) + { +struct argo_ring_id *cmpid = &send_info->id;
Const.
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 689
Quoted 10 lines
xen/common/argo.c +10
-0+ +if ( cmpid->port == id->port && + cmpid->domain_id == id->domain_id && + cmpid->partner_id == id->partner_id ) + { +argo_dprintk ("send_info=%p\n" , send_info); +return send_info; + } + } +argo_dprintk ("no send_info found\n" );
Is this message actually helpful without printing any of the parameters provided to the function?
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 695
Quoted 6 lines
xen/common/argo.c +6
-0+ +return NULL ; +} + +static long +unregister_ring(struct domain *currd,
Same as the comment made on the other patch, if this parameter is the current domain there's no need to pass it around, or else it should be named d instead of currd.
Quoted 6 lines
xen/common/argo.c +6
-0+ XEN_GUEST_HANDLE_PARAM(xen_argo_unregister_ring_t) unreg_hnd) +{ +xen_argo_unregister_ring_t unreg; +struct argo_ring_id ring_id; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info;
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 711
Quoted 10 lines
xen/common/argo.c +10
-0+struct domain *dst_d =NULL ; +int ret; + + ret =copy_from_guest (&unreg, unreg_hnd,1 ) ? -EFAULT :0 ; +if ( ret ) + goto out; + + ret = unreg.pad ? -EINVAL :0 ; +if ( ret ) + goto out;
I don't see the point in the out label when you could just use 'return -EINVAL' or -EFAULT directly here and above.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 08/15] argo: implement the unregister op
Quoted 55 lines · Roger Pau Monné
Quoted 52 lines · Christopher Clark
Takes a single argument: a handle to the ring unregistration struct, which specifies the port and partner domain id or wildcard.
The ring's entry is removed from the hashtable of registered rings; any entries for pending notifications are removed; and the ring is unmapped from Xen's address space.
If the ring had been registered to communicate with a single specified domain (ie. a non-wildcard ring) then the partner domain state is removed from the partner domain's argo
send_infohash table.Signed-off-by Christopher Clark <[email protected]>---
v2 feedback Jan: drop cookie, implement teardown
v2 feedback Jan: drop message fromargo_message_op
v2 self: OVERHAUL
v2 self: reorder logic to shorten critical section
v1 #13 feedback Jan: revise use ofguest_handle_okayvs__copyops v1 feedback Roger, Jan: drop argo prefix on static functions v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 #5 (#14) feedback Paul: use currd indo_argo_message_op
v1 #5 (#14) feedback Paul: full use currd inargo_unregister_ringv1 #13 (#14) feedback Paul: replace do/while with goto; reindent v1 self: add blank lines in unregister case indo_argo_message_op
v1: #13 feedback Jan: public namespace: prefix with xen
v1: #13 feedback Jan: blank line after op case indo_argo_message_op
v1: #14 feedback Jan: replace domain id override with validation
v1: #18 feedback Jan: meld the ring count limit into the series
v1: feedback #15 Jan: verify zero in unused hypercall argsxen/common/argo.c | 115 ++++++++++++++++++++++++++++++++++++++++++++++ xen/include/public/argo.h | 19 ++++++++ xen/include/xlat.lst | 1 + 3 files changed, 135 insertions(+)xen/common/argo.c +1
-0@@ -37,6 +37,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_argo_addr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_page_descr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_register_ring_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_unregister_ring_t);/* Xen command line option to enable argo */ static bool __read_mostly opt_argo_enabled;@@ -666,6 +667,105 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; }
Quoted 12 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 679
Quoted 10 lines
xen/common/argo.c +10
-0+static struct argo_send_info * +send_find_info (const struct domain *d,const struct argo_ring_id *id) +{ +struct hlist_node *node; +struct argo_send_info *send_info; + +hlist_for_each_entry (send_info, node, &d->argo->send_hash[hash_index (id)], + node) + { +struct argo_ring_id *cmpid = &send_info->id;Const.
ack, done
Quoted 12 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 689
Quoted 11 lines
xen/common/argo.c +10
-0+ +if ( cmpid->port == id->port && + cmpid->domain_id == id->domain_id && + cmpid->partner_id == id->partner_id ) + { +argo_dprintk ("send_info=%p\n" , send_info); +return send_info; + } + } +argo_dprintk ("no send_info found\n" );
Quoted 2 lines
Is this message actually helpful without printing any of the parameters provided to the function?
Good point. I've added the ring data to the output.
Quoted 11 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 695
Quoted 7 lines
xen/common/argo.c +6
-0+ +return NULL ; +} + +static long +unregister_ring(struct domain *currd,Same as the comment made on the other patch, if this parameter is the current domain there's no need to pass it around, or else it should be named d instead of currd.
Response is as per the other for this one: where currd has been retained, it's now ASSERTed as matching current->domain, and otherwise 'd' is used.
Quoted 12 lines
Quoted 12 lines
xen/common/argo.c +11
-0+ XEN_GUEST_HANDLE_PARAM(xen_argo_unregister_ring_t) unreg_hnd) +{ +xen_argo_unregister_ring_t unreg; +struct argo_ring_id ring_id; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info; +struct domain *dst_d =NULL ; +int ret; + + ret =copy_from_guest (&unreg, unreg_hnd,1 ) ? -EFAULT :0 ; +if ( ret )
Quoted 8 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 711
Quoted 5 lines
xen/common/argo.c +5
-0+ goto out; + + ret = unreg.pad ? -EINVAL :0 ; +if ( ret ) + goto out;I don't see the point in the out label when you could just use 'return
-EINVAL' or-EFAULTdirectly here and above.
ack - have done that, much better now.
thanks
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 08/15] argo: implement the unregister op
In reply to Christopher Clarkxen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 671
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +2
-0@@ -666,6 +667,105 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_send_info * +send_find_info (const struct domain *d,const struct argo_ring_id *id)
As per the comment on patch 7, perhaps find_send_info()?
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 673
Quoted 2 lines
xen/common/argo.c +2
-0+{ +struct hlist_node *node;
const?
Quoted 45 lines
xen/common/argo.c +45
-0+static long +unregister_ring (struct domain *currd, +XEN_GUEST_HANDLE_PARAM (xen_argo_unregister_ring_t) unreg_hnd) +{ +xen_argo_unregister_ring_t unreg; +struct argo_ring_id ring_id; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info; +struct domain *dst_d =NULL ; +int ret; + + ret =copy_from_guest (&unreg, unreg_hnd,1 ) ? -EFAULT :0 ; +if ( ret ) + goto out; + + ret = unreg.pad ? -EINVAL :0 ; +if ( ret ) + goto out; + + ring_id.partner_id = unreg.partner_id; + ring_id.port = unreg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock); + +if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +write_lock (&currd->argo->lock); + + ring_info =ring_find_info (currd, &ring_id); +if ( ring_info ) + { +ring_remove_info (currd, ring_info); + currd->argo->ring_count--; + } + + dst_d =get_domain_by_id (ring_id.partner_id); +if ( dst_d ) + { +if ( dst_d->argo ) + {
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 748
Quoted 10 lines
xen/common/argo.c +10
-0+spin_lock (&dst_d->argo->send_lock); + + send_info =send_find_info (dst_d, &ring_id); +if ( send_info ) + { +hlist_del (&send_info->node); +xfree (send_info); + } + +spin_unlock (&dst_d->argo->send_lock);
As per the comment to an earlier patch, if at all possible call allocation (and hence also freeing) functions with as little locks held as possible. Pulling it out of the innermost lock here looks straightforward at least.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 08/15] argo: implement the unregister op
Quoted 10 lines · Jan Beulich
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 671
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines · On 07.01.19 at 08:42, <[email protected]> wrote:
xen/common/argo.c +2
-0@@ -666,6 +667,105 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_send_info * +send_find_info (const struct domain *d,const struct argo_ring_id *id)As per the comment on patch 7, perhaps
find_send_info()?
ack, yes, renamed.
Quoted 5 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 673
Quoted 3 lines
xen/common/argo.c +2
-0+{ +struct hlist_node *node;const?
understood; not got this one done yet, but yes.
Quoted 53 lines
Quoted 53 lines
xen/common/argo.c +52
-0+static long +unregister_ring (struct domain *currd, +XEN_GUEST_HANDLE_PARAM (xen_argo_unregister_ring_t) unreg_hnd) +{ +xen_argo_unregister_ring_t unreg; +struct argo_ring_id ring_id; +struct argo_ring_info *ring_info; +struct argo_send_info *send_info; +struct domain *dst_d =NULL ; +int ret; + + ret =copy_from_guest (&unreg, unreg_hnd,1 ) ? -EFAULT :0 ; +if ( ret ) + goto out; + + ret = unreg.pad ? -EINVAL :0 ; +if ( ret ) + goto out; + + ring_id.partner_id = unreg.partner_id; + ring_id.port = unreg.port; + ring_id.domain_id = currd->domain_id; + +read_lock (&argo_lock); + +if ( !currd->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + +write_lock (&currd->argo->lock); + + ring_info =ring_find_info (currd, &ring_id); +if ( ring_info ) + { +ring_remove_info (currd, ring_info); + currd->argo->ring_count--; + } + + dst_d =get_domain_by_id (ring_id.partner_id); +if ( dst_d ) + { +if ( dst_d->argo ) + { +spin_lock (&dst_d->argo->send_lock); + + send_info =send_find_info (dst_d, &ring_id); +if ( send_info ) + { +hlist_del (&send_info->node); +xfree (send_info);
Quoted 8 lines
xen/common/argo.c ring_find_info(const struct domain *d, const struct argo_ring_id *id) · line 748
Quoted 3 lines
xen/common/argo.c +3
-0+ } + +spin_unlock (&dst_d->argo->send_lock);As per the comment to an earlier patch, if at all possible call allocation (and hence also freeing) functions with as little locks held as possible. Pulling it out of the innermost lock here looks straightforward at least.
ack, have pulled it out of both L2s now which will help.
thanks
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v3 9/15 argo: implement the sendv op; evtchn: expose send_guest_global_virq
In reply to Christopher Clarksendv operation is invoked to perform a synchronous send of buffers contained in iovs to a remote domain's registered ring.
It takes:
* A destination address (domid, port) for the ring to send to. It performs a most-specific match lookup, to allow for wildcard.
* A source address, used to inform the destination of where to reply.
* The address of an array of iovs containing the data to send
* .. and the length of that array of iovs
* and a 32-bit message type, available to communicate message context data (eg. kernel-to-kernel, separate from the application data).
If insufficient space exists in the destination ring, it will return -EAGAIN and Xen will notify the caller when sufficient space becomes available.
Accesses to the ring indices are appropriately atomic. The rings are mapped into Xen's private address space to write as needed and the mappings are retained for later use.
Fixed-size types are used in some areas within this code where caution around avoiding integer overflow is important.
Notifications are sent to guests via VIRQ and send_guest_global_virq is exposed in the change to enable argo to call it. VIRQ_ARGO_MESSAGE is claimed from the VIRQ previously reserved for this purpose (#11).
The VIRQ notification method is used rather than sending events using evtchn functions directly because:
* no current event channel type is an exact fit for the intended behaviour. ECS_IPI is closest, but it disallows migration to other VCPUs which is not necessarily a requirement for Argo.
* at the point of argo_init, allocation of an event channel is complicated by none of the guest VCPUs being initialized yet and the event channel logic expects that a valid event channel has a present VCPU.
* at the point of signalling a notification, the VIRQ logic is already
defensive: if d->vcpu[0] is NULL, the notification is just silently dropped, whereas the evtchn_send logic is not so defensive: vcpu[0] must not be NULL, otherwise a null pointer dereference occurs.
Using a VIRQ removes the need for the guest to query to determine which event channel notifications will be delivered on. This is also likely to simplify establishing future L0/L1 nested hypervisor argo communication.
---
The previous double-read of iovs from guest memory has been removed.
v2 self: use ring_info backpointer in pending_ent to maintain npending v2 feedback Jan: drop cookie, implement teardown
v2 self: pending_queue: reap stale ents when in need of space
v2 self: pending_requeue: reclaim ents for stale domains v2.feedback Jan: only override sender domid if DOMID_ANY
v2 feedback Jan: drop message from argo_message_op
v2 self: check npending vs maximum limit
v2 self: get_sanitized_ring instead of get_rx_ptr
v2 feedback v1#13 Jan: remove double read from ringbuf insert, lower MAX_IOV v2 self: make iov_count const
v2 self: iov_count : return EMSGSIZE for message too big
v2 self: OVERHAUL
v2 self: s/argo_pending_ent/pending_ent/g
v2 feedback v1#13 Roger: use OS-supplied roundup; drop from public header v1,2 feedback Jan/Roger/Paul: drop errno returning guest access functions v1 feedback Roger, Jan: drop argo prefix on static functions
v1 feedback #13 Jan: drop guest_handle_okay when using copy_from_guest
- reorder do_argo_op logic
v2 self: add _hnd suffix to iovs variable name to indicate guest handle type v2 self: replace use of XEN_GUEST_HANDLE_NULL with two existing macros
v1 #15 feedback, Jan: sendv op : s/ECONNREFUSED/ESRCH/
v1 #5 (#15) feedback Paul: sendv: use currd in do_argo_message_op
v1 #13 (#15) feedback Paul: sendv op: do/while reindent only
v1 #13 (#15) feedback Paul: sendv op: do/while: argo_ringbuf_insert to goto style v1 #13 (#15) feedback Paul: sendv op: do/while: reindent only again v1 #13 (#15) feedback Paul: sendv op: do/while : goto
v1 #15 feedback Paul: sendv op: make page var: unsigned
v1 #15 feedback Paul: sendv op: new local var for PAGE_SIZE - offset v1 #8 feedback Jan: XEN_GUEST_HANDLE : C89 compliance
v1 rebase after switching register op from pfns to page descriptors v1 self: move iov DEFINE_XEN_GUEST_HANDLE out of public header into argo.c v1 #13 (#15) feedback Paul: fix loglevel for guest-triggered messages v1 : add compat xlat.lst entries
v1 self: switched notification to send_guest_global_virq instead of event
v1: fix gprintk use for ARM as its defn dislikes split format strings
v1: init len variable to satisfy ARM compiler initialized checking v1 #13 feedback Jan: rename page var
v1:#14 feedback Jan: uint8_t* -> void*
v1: #13 feedback Jan: public namespace: prefix with xen
v1: #13 feedback Jan: blank line after case op in do_argo_message_op
v1: #15 feedback Jan: add comments explaining why the writes don't overrun
v1: self: add ASSERT to support comment that overrun cannot happen
v1: self: fail on short writes where guest manipulated the iov_lens
v1: self: rename ent id to domain_id
v1: self: add moan for iov rewrite
v1. feedback #15 Jan: require the pad bits are zero
v1. feedback #15 Jan: drop NULL check in argo_signal_domain as now using VIRQ v1. self: store domain_cookie in pending ent
v1. feedback #15 Jan: use unsigned where possible
v1. feedback Jan: use handle type for iov_base in public iov interface v1. self: log whenever visible error occurs
v1 feedback #15, Jan: drop unnecessary mb
v1 self: only update internal tx_ptr if able to return success
and update the visible tx_ptr
v1 self: log on failure to map ring to update visible tx_ptr
v1 feedback #15 Jan: add comment re: notification size policy
v1 self/Roger? remove errant space after sizeof
v1. feedback #15 Jan: require iov pad be zero
v1. self: rename iov_base to iov_hnd for handle in public iov interface
v1: feedback #15 Jan: handle upper-halves of hypercall args; changes some
types in function signatures to match.
v1: self: add dprintk to sendv
v1: self: add debug output to argo_iov_count
v1. feedback #14 Jan: blank line before return in argo_iov_count
v1 feedback #15 Jan: verify src id, not override
xen/common/argo.c | 653 +++++++++++++++++++++++++++++++++++++++++++++ xen/common/event_channel.c | 2 +- xen/include/public/argo.h | 60 +++++ xen/include/public/xen.h | 2 +- xen/include/xen/event.h | 7 + xen/include/xlat.lst | 2 + 6 files changed, 724 insertions(+), 2 deletions(-)
xen/common/argo.c +653 -0
@@ -29,14 +29,21 @@#include <public/argo.h> #define MAX_RINGS_PER_DOMAIN 128U +#define MAX_PENDING_PER_RING 32U/* All messages on the ring are padded to a multiple of the slot size. */ #define ROUNDUP_MESSAGE (a) (ROUNDUP((a), XEN_ARGO_MSG_SLOT_SIZE)) +/* The maximum size of a message that may be sent on the largest Argo ring. */ +#define MAX_ARGO_MESSAGE_SIZE ((XEN_ARGO_MAX_RING_SIZE) - \ + (sizeof(struct xen_argo_ring_message_header)) - ROUNDUP_MESSAGE(1)) +DEFINE_XEN_GUEST_HANDLE (xen_argo_addr_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_iov_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_page_descr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_register_ring_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_ring_t); +DEFINE_XEN_GUEST_HANDLE (xen_argo_send_addr_t);DEFINE_XEN_GUEST_HANDLE (xen_argo_unregister_ring_t);/* Xen command line option to enable argo */
@@ -250,6 +257,14 @@ hash_index(const struct argo_ring_id *id) }static void +signal_domain (struct domain *d) +{ +argo_dprintk ("signalling domid:%d\n" , d->domain_id); + +send_guest_global_virq (d,VIRQ_ARGO_MESSAGE ); +} + +static void ring_unmap (struct argo_ring_info *ring_info) {unsigned int i;
@@ -342,6 +357,413 @@ update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr)smp_wmb (); } +static int +memcpy_to_guest_ring (struct argo_ring_info *ring_info,uint32_t offset, +const void *src,XEN_GUEST_HANDLE (uint8_t ) src_hnd, +uint32_t len) +{ +unsigned int mfns_index = offset >>PAGE_SHIFT ; +void *dst; +int ret; +unsigned int src_offset =0 ; + +ASSERT (spin_is_locked (&ring_info->lock)); + + offset &= ~PAGE_MASK ; + +if ( (len >XEN_ARGO_MAX_RING_SIZE ) || (offset >XEN_ARGO_MAX_RING_SIZE ) ) +return -EFAULT ; + +while ( (offset + len) >PAGE_SIZE ) + { +unsigned int head_len =PAGE_SIZE - offset; + + ret =ring_map_page (ring_info, mfns_index, &dst); +if ( ret ) +return ret; + +if ( src ) + { +memcpy (dst + offset, src + src_offset, head_len); + src_offset += head_len; + } +else + { + ret =copy_from_guest (dst + offset, src_hnd, head_len) ? + -EFAULT :0 ; +if ( ret ) +return ret; + +guest_handle_add_offset (src_hnd, head_len); + } + + mfns_index++; + len -= head_len; + offset =0 ; + } + + ret =ring_map_page (ring_info, mfns_index, &dst); +if ( ret ) + { +argo_dprintk ("argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %d of %d\n" , ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, mfns_index, + ring_info->nmfns); +return ret; + } + +if ( src ) +memcpy (dst + offset, src + src_offset, len); +else + ret =copy_from_guest (dst + offset, src_hnd, len) ? -EFAULT :0 ; + +return ret; +} + +/* +* Use this with caution: rx_ptr is under guest control and may be bogus. +* See get_sanitized_ring for a safer alternative. +*/ +static int +get_rx_ptr (struct argo_ring_info *ring_info,uint32_t *rx_ptr) +{ +void *src; +xen_argo_ring_t *ringp; +int ret; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( !ring_info->nmfns || ring_info->nmfns < ring_info->npage ) +return -EINVAL ; + + ret =ring_map_page (ring_info,0 , &src); +if ( ret ) +return ret; + + ringp = (xen_argo_ring_t *)src; + + *rx_ptr =read_atomic (&ringp->rx_ptr); + +return 0 ; +} + +/* +* get_sanitized_ring creates a modified copy of the ring pointers where +* the rx_ptr is rounded up to ensure it is aligned, and then ring +* wrap is handled. Simplifies safe use of the rx_ptr for available +* space calculation. +*/ +static int +get_sanitized_ring (xen_argo_ring_t *ring,struct argo_ring_info *ring_info) +{ +uint32_t rx_ptr; +int ret; + + ret =get_rx_ptr (ring_info, &rx_ptr); +if ( ret ) +return ret; + + ring->tx_ptr = ring_info->tx_ptr; + + rx_ptr =ROUNDUP_MESSAGE (rx_ptr); +if ( rx_ptr >= ring_info->len ) + rx_ptr =0 ; + + ring->rx_ptr = rx_ptr; +return 0 ; +} + +/* +* iov_count returns its count on success via an out variable to avoid +* potential for a negative return value to be used incorrectly +* (eg. coerced into an unsigned variable resulting in a large incorrect value) +*/ +static int +iov_count (const xen_argo_iov_t *piov,unsigned long niov,uint32_t *count) +{ +uint32_t sum_iov_lens =0 ; + +if ( niov >XEN_ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { +/* valid iovs must have the padding field set to zero */ +if ( piov->pad ) + { +argo_dprintk ("invalid iov: padding is not zero\n" ); +return -EINVAL ; + } + +/* check each to protect sum against integer overflow */ +if ( piov->iov_len >XEN_ARGO_MAX_RING_SIZE ) + { +argo_dprintk ("invalid iov_len: too big (%u)>%llu\n" , + piov->iov_len,XEN_ARGO_MAX_RING_SIZE ); +return -EINVAL ; + } + + sum_iov_lens += piov->iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >MAX_ARGO_MESSAGE_SIZE ) + { +argo_dprintk ("invalid iov series: total message too big\n" ); +return -EMSGSIZE ; + } + + piov++; + } + + *count = sum_iov_lens; + +return 0 ; +} + +static int +ringbuf_insert (struct domain *d,struct argo_ring_info *ring_info, +const struct argo_ring_id *src_id, +XEN_GUEST_HANDLE_PARAM (xen_argo_iov_t) iovs_hnd, +unsigned long niov,uint32_t message_type, +unsigned long *out_len) +{ +xen_argo_ring_t ring; +struct xen_argo_ring_message_header mh = {0 }; +int32_t sp; +int32_t ret; +uint32_t len =0 ; +xen_argo_iov_t iovs[XEN_ARGO_MAXIOV ]; +xen_argo_iov_t *piov; +XEN_GUEST_HANDLE (uint8_t ) NULL_hnd = +guest_handle_from_param (guest_handle_from_ptr (NULL , uint8_t), uint8_t); + +ASSERT (spin_is_locked (&ring_info->lock)); + + ret =__copy_from_guest (iovs, iovs_hnd, niov) ? -EFAULT :0 ; +if ( ret ) + goto out; + +/* +* Obtain the total size of data to transmit -- sets the 'len' variable +* -- and sanity check that the iovs conform to size and number limits. +* Enforced below: no more than 'len' bytes of guest data +* (plus the message header) will be sent in this operation. +*/ + ret =iov_count (iovs, niov, &len); +if ( ret ) + goto out; + +/* +* Size bounds check against ring size and static maximum message limit. +* The message must not fill the ring; there must be at least one slot +* remaining so we can distinguish a full ring from an empty one. +*/ +if ( ((ROUNDUP_MESSAGE (len) + +sizeof (struct xen_argo_ring_message_header )) >= ring_info->len) || + (len >MAX_ARGO_MESSAGE_SIZE ) ) + { + ret = -EMSGSIZE ; + goto out; + } + + ret =get_sanitized_ring (&ring, ring_info); +if ( ret ) + goto out; + +argo_dprintk ("ring.tx_ptr=%d ring.rx_ptr=%d ring len=%d" +" ring_info->tx_ptr=%d\n" , + ring.tx_ptr, ring.rx_ptr, ring_info->len, ring_info->tx_ptr); + +if ( ring.rx_ptr == ring.tx_ptr ) + sp = ring_info->len; +else + { + sp = ring.rx_ptr - ring.tx_ptr; +if ( sp <0 ) + sp += ring_info->len; + } + +/* +* Size bounds check against currently available space in the ring. +* Again: the message must not fill the ring leaving no space remaining. +*/ +if ( (ROUNDUP_MESSAGE (len) + +sizeof (struct xen_argo_ring_message_header )) >= sp ) + { +argo_dprintk ("EAGAIN\n" ); + ret = -EAGAIN ; + goto out; + } + + mh.len = len +sizeof (struct xen_argo_ring_message_header ); + mh.source.port = src_id->port; + mh.source.domain_id = src_id->domain_id; + mh.message_type = message_type; + +/* +* For this copy to the guest ring, tx_ptr is always 16-byte aligned +* and the message header is 16 bytes long. +*/ +BUILD_BUG_ON ( +sizeof (struct xen_argo_ring_message_header ) !=ROUNDUP_MESSAGE (1 )); + +/* +* First data write into the destination ring: fixed size, message header. +* This cannot overrun because the available free space (value in 'sp') +* is checked above and must be at least this size. +*/ + ret =memcpy_to_guest_ring (ring_info, ring.tx_ptr +sizeof (xen_argo_ring_t), + &mh, NULL_hnd,sizeof (mh)); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: failed to write message header to ring (vm%u:%x vm%d)\n" , + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id); + + goto out; + } + + ring.tx_ptr +=sizeof (mh); +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + + piov = iovs; + +while ( niov-- ) + { +XEN_GUEST_HANDLE_64 (uint8_t ) buf_hnd = piov->iov_hnd; +uint32_t iov_len = piov->iov_len; + +/* If no data is provided in this iov, moan and skip on to the next */ +if ( !iov_len ) + { +gprintk (XENLOG_ERR , +"argo: no data iov_len=0 iov_hnd=%p ring (vm%u:%x vm%d)\n" , + buf_hnd.p, ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id); + + piov++; +continue ; + } + +if (unlikely (!guest_handle_okay (buf_hnd, iov_len)) ) + { +gprintk (XENLOG_ERR , +"argo: bad iov handle [%p, %" PRIx32"] (vm%u:%x vm%d)\n" , + buf_hnd.p, iov_len, + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id); + + ret = -EFAULT ; + goto out; + } + + sp = ring_info->len - ring.tx_ptr; + +/* Check: iov data size versus free space at the tail of the ring */ +if ( iov_len > sp ) + { +/* +* Second possible data write: ring-tail-wrap-write. +* Populate the ring tail and update the internal tx_ptr to handle +* wrapping at the end of ring. +* Size of data written here: sp +* which is the exact full amount of free space available at the +* tail of the ring, so this cannot overrun. +*/ + ret =memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (xen_argo_ring_t), +NULL , buf_hnd, sp); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: failed to copy {%p, %" PRIx32"} (vm%u:%x vm%d)\n" , + buf_hnd.p, sp, + ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id); + + goto out; + } + + ring.tx_ptr =0 ; + iov_len -= sp; +guest_handle_add_offset (buf_hnd, sp); + +ASSERT (iov_len <= ring_info->len); + } + +/* +* Third possible data write: all data remaining for this iov. +* Size of data written here: iov_len +* +* Case 1: if the ring-tail-wrap-write above was performed, then +* iov_len has been decreased by 'sp' and ring.tx_ptr is zero. +* +* We know from checking the result of iov_count: +* len + sizeof(message_header) <= ring_info->len +* We also know that len is the total of summing all iov_lens, so: +* iov_len <= len +* so by transitivity: +* iov_len <= len <= (ring_info->len - sizeof(msgheader)) +* and therefore: +* (iov_len + sizeof(msgheader) <= ring_info->len) && +* (ring.tx_ptr == 0) +* so this write cannot overrun here. +* +* Case 2: ring-tail-wrap-write above was not performed +* -> so iov_len is the guest-supplied value and: (iov_len <= sp) +* ie. less than available space at the tail of the ring: +* so this write cannot overrun. +*/ + ret =memcpy_to_guest_ring (ring_info, + ring.tx_ptr +sizeof (xen_argo_ring_t), +NULL , buf_hnd, iov_len); +if ( ret ) + { +gprintk (XENLOG_ERR , +"argo: failed to copy [%p, %" PRIx32"] (vm%u:%x vm%d)\n" , + buf_hnd.p, iov_len, ring_info->id.domain_id, + ring_info->id.port, ring_info->id.partner_id); + + goto out; + } + + ring.tx_ptr += iov_len; + +if ( ring.tx_ptr == ring_info->len ) + ring.tx_ptr =0 ; + + piov++; + } + + ring.tx_ptr =ROUNDUP_MESSAGE (ring.tx_ptr); + +if ( ring.tx_ptr >= ring_info->len ) + ring.tx_ptr -= ring_info->len; + +update_tx_ptr (ring_info, ring.tx_ptr); + + out: +/* +* At this point it is possible to unmap the ring_info, ie: +* ring_unmap(ring_info); +* but performance should be improved by not doing so, and retaining +* the mapping. +* An XSM policy control over level of confidentiality required +* versus performance cost could be added to decide that here. +* See the similar comment in ring_map_page re: write-only mappings. +*/ + +if ( !ret ) + *out_len = len; + +return ret; +} +static void wildcard_pending_list_remove (domid_t domain_id,struct pending_ent *ent) {
@@ -359,6 +781,22 @@ wildcard_pending_list_remove(domid_t domain_id, struct pending_ent *ent) }static void +wildcard_pending_list_insert (domid_t domain_id,struct pending_ent *ent) +{ +struct domain *d =get_domain_by_id (domain_id); +if ( !d ) +return ; + +if ( d->argo ) + { +spin_lock (&d->argo->wildcard_lock); +hlist_add_head (&ent->wildcard_node, &d->argo->wildcard_pend_list); +spin_unlock (&d->argo->wildcard_lock); + } +put_domain (d); +} + +static void pending_remove_all (struct argo_ring_info *ring_info) {struct hlist_node *node, *next;
@@ -374,6 +812,67 @@ pending_remove_all(struct argo_ring_info *ring_info) ring_info->npending =0 ; } +static int +pending_queue (struct argo_ring_info *ring_info,domid_t src_id, +unsigned int len) +{ +struct pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +if ( ring_info->npending >=MAX_PENDING_PER_RING ) +return -ENOSPC ; + + ent = xmalloc(struct pending_ent); + +if ( !ent ) +return -ENOMEM ; + + ent->len = len; + ent->domain_id = src_id; + ent->ring_info = ring_info; + +if ( ring_info->id.partner_id ==XEN_ARGO_DOMID_ANY ) +wildcard_pending_list_insert (src_id, ent); +hlist_add_head (&ent->node, &ring_info->pending); + ring_info->npending++; + +return 0 ; +} + +static int +pending_requeue (struct argo_ring_info *ring_info,domid_t src_id, +unsigned int len) +{ +struct hlist_node *node; +struct pending_ent *ent; + +ASSERT (spin_is_locked (&ring_info->lock)); + +hlist_for_each_entry (ent, node, &ring_info->pending, node) + { +if ( ent->domain_id == src_id ) + { +/* +* Reuse an existing queue entry for a notification rather than add +* another. If the existing entry is waiting for a smaller size than +* the current message then adjust the record to wait for the +* current (larger) size to be available before triggering a +* notification. +* This assists the waiting sender by ensuring that whenever a +* notification is triggered, there is sufficient space available +* for (at least) any one of the messages awaiting transmission. +*/ +if ( ent->len < len ) + ent->len = len; + +return 0 ; + } + } + +return pending_queue (ring_info, src_id, len); +} +static void wildcard_rings_pending_remove (struct domain *d) {
@@ -667,6 +1166,28 @@ ring_find_info(const struct domain *d, const struct argo_ring_id *id)return NULL ; } +static struct argo_ring_info * +ring_find_info_by_match (const struct domain *d,uint32_t port, +domid_t partner_id) +{ +struct argo_ring_id id; +struct argo_ring_info *ring_info; + +ASSERT (rw_is_locked (&d->argo->lock)); + + id.port = port; + id.domain_id = d->domain_id; + id.partner_id = partner_id; + + ring_info =ring_find_info (d, &id); +if ( ring_info ) +return ring_info; + + id.partner_id =XEN_ARGO_DOMID_ANY ; + +return ring_find_info (d, &id); +} +static struct argo_send_info *send_find_info (const struct domain *d,const struct argo_ring_id *id) {
@@ -1005,6 +1526,95 @@ register_ring(struct domain *currd,return ret; } +static long +sendv (struct domain *src_d,const xen_argo_addr_t *src_addr, +const xen_argo_addr_t *dst_addr, +XEN_GUEST_HANDLE_PARAM (xen_argo_iov_t) iovs_hnd,unsigned long niov, +uint32_t message_type) +{ +struct domain *dst_d =NULL ; +struct argo_ring_id src_id; +struct argo_ring_info *ring_info; +int ret =0 ; +unsigned long len =0 ; + +ASSERT (src_d->domain_id == src_addr->domain_id); + +argo_dprintk ("sendv: (%d:%x)->(%d:%x) niov:%lu iov:%p type:%u\n" , + src_addr->domain_id, src_addr->port, + dst_addr->domain_id, dst_addr->port, + niov, iovs_hnd.p, message_type); + +read_lock (&argo_lock); + +if ( !src_d->argo ) + { + ret = -ENODEV ; + goto out_unlock; + } + + src_id.port = src_addr->port; + src_id.domain_id = src_d->domain_id; + src_id.partner_id = dst_addr->domain_id; + + dst_d =get_domain_by_id (dst_addr->domain_id); +if ( !dst_d ) + { +argo_dprintk ("!dst_d, ESRCH\n" ); + ret = -ESRCH ; + goto out_unlock; + } + +if ( !dst_d->argo ) + { +argo_dprintk ("!dst_d->argo, ECONNREFUSED\n" ); + ret = -ECONNREFUSED ; + goto out_unlock; + } + +read_lock (&dst_d->argo->lock); + + ring_info =ring_find_info_by_match (dst_d, dst_addr->port, + src_addr->domain_id); +if ( !ring_info ) + { +gprintk (XENLOG_ERR , +"argo: vm%u connection refused, src (vm%u:%x) dst (vm%u:%x)\n" , + current->domain->domain_id, src_id.domain_id, src_id.port, + dst_addr->domain_id, dst_addr->port); + + ret = -ECONNREFUSED ; + goto out_unlock2; + } + +spin_lock (&ring_info->lock); + + ret =ringbuf_insert (dst_d, ring_info, &src_id, iovs_hnd, niov, + message_type, &len); +if ( ret == -EAGAIN ) + { +argo_dprintk ("argo_ringbuf_sendv failed, EAGAIN\n" ); +/* requeue to issue a notification when space is there */ + ret =pending_requeue (ring_info, src_addr->domain_id, len); + } + +spin_unlock (&ring_info->lock); + +if ( ret >=0 ) +signal_domain (dst_d); + + out_unlock2: +read_unlock (&dst_d->argo->lock); + + out_unlock: +if ( dst_d ) +put_domain (dst_d); + +read_unlock (&argo_lock); + +return ( ret <0 ) ? ret : len; +} + long do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, XEN_GUEST_HANDLE_PARAM(void) arg2, unsigned long arg3,
@@ -1073,6 +1683,49 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,break ; } +case XEN_ARGO_OP_sendv: + { +xen_argo_send_addr_t send_addr; + +XEN_GUEST_HANDLE_PARAM (xen_argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, xen_argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (xen_argo_iov_t) iovs_hnd = +guest_handle_cast (arg2, xen_argo_iov_t); +/* arg3 is niov */ +/* arg4 is message_type. Must be a 32-bit value. */ + + rc =copy_from_guest (&send_addr, send_addr_hnd,1 ) ? -EFAULT :0 ; +if ( rc ) +break ; + +if ( send_addr.src.domain_id ==XEN_ARGO_DOMID_ANY ) + send_addr.src.domain_id = currd->domain_id; + +/* No domain is currently authorized to send on behalf of another */ +if (unlikely (send_addr.src.domain_id != currd->domain_id) ) + { + rc = -EPERM ; +break ; + } + +/* Reject niov or message_type values that are outside 32 bit range. */ +if (unlikely ((arg3 >XEN_ARGO_MAXIOV ) || (arg4 & ~0xffffffffUL )) ) + { + rc = -EINVAL ; +break ; + } + +/* +* Check access to the whole array here so we can use the faster __copy +* operations to read each element later. +*/ +if (unlikely (!guest_handle_okay (iovs_hnd, arg3)) ) +break ; + + rc =sendv (currd, &send_addr.src, &send_addr.dst, iovs_hnd, arg3, arg4); +break ; + } +default : rc = -EOPNOTSUPP ;break ;
xen/common/event_channel.c +1 -1
@@ -746,7 +746,7 @@ void send_guest_vcpu_virq(struct vcpu *v, uint32_t virq)spin_unlock_irqrestore (&v->virq_lock, flags); } -static void send_guest_global_virq (struct domain *d,uint32_t virq) +void send_guest_global_virq (struct domain *d,uint32_t virq) {unsigned long flags;int port;
xen/include/public/argo.h +60 -0
76 lines not on this page · raw message
xen/include/public/xen.h +1 -1
@@ -178,7 +178,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t);#define VIRQ_CON_RING 8/* G. (DOM0) Bytes received on console */ #define VIRQ_PCPU_STATE 9/* G. (DOM0) PCPU state changed */ #define VIRQ_MEM_EVENT 10/* G. (DOM0) A memory event has occurred */ -#define VIRQ_XC_RESERVED 11/* G. Reserved for XenClient */ +#define VIRQ_ARGO_MESSAGE 11/* G. Argo interdomain message notification */ #define VIRQ_ENOMEM 12/* G. (DOM0) Low on heap memory */ #define VIRQ_XENPMU 13/* V. PMC interrupt */
xen/include/xen/event.h +7 -0
@@ -29,6 +29,13 @@ void send_guest_vcpu_virq(struct vcpu *v, uint32_t virq);void send_global_virq (uint32_t virq);/* +* send_guest_global_virq: +* @d: Domain to which VIRQ should be sent +* @virq: Virtual IRQ number (VIRQ_*), must be global +*/ +void send_guest_global_virq (struct domain *d,uint32_t virq); + +/* * sent_global_virq_handler: Set a global VIRQ handler. * @d: New target domain for this VIRQ * @virq: Virtual IRQ number (VIRQ_*), must be global
xen/include/xlat.lst +2 -0
@@ -152,3 +152,5 @@ ? argo_ring argo.h ? argo_register_ring argo.h ? argo_unregister_ring argo.h +? argo_iov argo.h +? argo_send_addr argo.h
Signature · 7 lines
2.7.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 09/15] argo: implement the sendv op; evtchn: expose send_guest_global_virq
On Mon, Jan 7, 2019 at 2:43 AM Christopher Clark
<[email protected]> wrote:
<snip>
Quoted 14 lines
xen/common/argo.c +10
-0@@ -342,6 +357,413 @@ update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr)smp_wmb (); } +static int +memcpy_to_guest_ring (struct argo_ring_info *ring_info,uint32_t offset, +const void *src,XEN_GUEST_HANDLE (uint8_t ) src_hnd, +uint32_t len) +{ +unsigned int mfns_index = offset >>PAGE_SHIFT ; +void *dst; +int ret; +unsigned int src_offset =0 ; +
xen/common/argo.c update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 379
Quoted 10 lines
xen/common/argo.c +10
-0+ASSERT (spin_is_locked (&ring_info->lock)); + + offset &= ~PAGE_MASK ; + +if ( (len >XEN_ARGO_MAX_RING_SIZE ) || (offset >XEN_ARGO_MAX_RING_SIZE ) ) +return -EFAULT ; + +while ( (offset + len) >PAGE_SIZE ) + { +unsigned int head_len =PAGE_SIZE - offset;
I think this while loop could be re-written as
while (len) {
head_len = len > PAGE_SIZE ? PAGE_SIZE - offset: len;and then the extra copying below outside the loop could be dropped.
The first loop does a partial copy at offset and then sets offset=0. The next N loops copy exactly PAGE_SIZE.
The Final copy does the remaining len bytes.
Quoted 42 lines
xen/common/argo.c +42
-0+ + ret =ring_map_page (ring_info, mfns_index, &dst); +if ( ret ) +return ret; + +if ( src ) + { +memcpy (dst + offset, src + src_offset, head_len); + src_offset += head_len; + } +else + { + ret =copy_from_guest (dst + offset, src_hnd, head_len) ? + -EFAULT :0 ; +if ( ret ) +return ret; + +guest_handle_add_offset (src_hnd, head_len); + } + + mfns_index++; + len -= head_len; + offset =0 ; + } + + ret =ring_map_page (ring_info, mfns_index, &dst); +if ( ret ) + { +argo_dprintk ("argo: ring (vm%u:%x vm%d) %p attempted to map page" +" %d of %d\n" , ring_info->id.domain_id, ring_info->id.port, + ring_info->id.partner_id, ring_info, mfns_index, + ring_info->nmfns); +return ret; + } + +if ( src ) +memcpy (dst + offset, src + src_offset, len); +else + ret =copy_from_guest (dst + offset, src_hnd, len) ? -EFAULT :0 ; + +return ret; +}
<snip>
Quoted 15 lines
xen/common/argo.c +15
-0+ +/* +* iov_count returns its count on success via an out variable to avoid +* potential for a negative return value to be used incorrectly +* (eg. coerced into an unsigned variable resulting in a large incorrect value) +*/ +static int +iov_count (const xen_argo_iov_t *piov,unsigned long niov,uint32_t *count) +{ +uint32_t sum_iov_lens =0 ; + +if ( niov >XEN_ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- )
xen/common/argo.c update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 499
Quoted 10 lines
xen/common/argo.c +10
-0+ { +/* valid iovs must have the padding field set to zero */ +if ( piov->pad ) + { +argo_dprintk ("invalid iov: padding is not zero\n" ); +return -EINVAL ; + } + +/* check each to protect sum against integer overflow */ +if ( piov->iov_len >XEN_ARGO_MAX_RING_SIZE )
Should this be MAX_ARGO_MESSAGE_SIZE? MAX_ARGO_MESSAGE_SIZE is less than XEN_ARGO_MAX_RING_SIZE, so we can pass this check and then just fail the one below.
Quoted 26 lines
xen/common/argo.c +26
-0+ { +argo_dprintk ("invalid iov_len: too big (%u)>%llu\n" , + piov->iov_len,XEN_ARGO_MAX_RING_SIZE ); +return -EINVAL ; + } + + sum_iov_lens += piov->iov_len; + +/* +* Again protect sum from integer overflow +* and ensure total msg size will be within bounds. +*/ +if ( sum_iov_lens >MAX_ARGO_MESSAGE_SIZE ) + { +argo_dprintk ("invalid iov series: total message too big\n" ); +return -EMSGSIZE ; + } + + piov++; + } + + *count = sum_iov_lens; + +return 0 ; +} +
<snip>
Quoted 25 lines
xen/common/argo.c +21
-0@@ -1073,6 +1683,49 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,break ; } +case XEN_ARGO_OP_sendv: + { +xen_argo_send_addr_t send_addr; + +XEN_GUEST_HANDLE_PARAM (xen_argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, xen_argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (xen_argo_iov_t) iovs_hnd = +guest_handle_cast (arg2, xen_argo_iov_t); +/* arg3 is niov */ +/* arg4 is message_type. Must be a 32-bit value. */ + + rc =copy_from_guest (&send_addr, send_addr_hnd,1 ) ? -EFAULT :0 ; +if ( rc ) +break ; + +if ( send_addr.src.domain_id ==XEN_ARGO_DOMID_ANY ) + send_addr.src.domain_id = currd->domain_id; + +/* No domain is currently authorized to send on behalf of another */ +if (unlikely (send_addr.src.domain_id != currd->domain_id) ) + {
xen/common/argo.c do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1716
Quoted 10 lines
xen/common/argo.c +10
-0+ rc = -EPERM ; +break ; + } + +/* Reject niov or message_type values that are outside 32 bit range. */ +if (unlikely ((arg3 >XEN_ARGO_MAXIOV ) || (arg4 & ~0xffffffffUL )) ) + { + rc = -EINVAL ; +break ; + }
This needs to check send_addr.src.pad and send_addr.dst.pad == 0. sendv() does not check the padding either.
Regards,
Jason
Quoted 5 lines
xen/common/argo.c +5
-0+ +/* +* Check access to the whole array here so we can use the faster __copy +* operations to read each element later. +*/
xen/common/argo.c do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1731
Quoted 10 lines
xen/common/argo.c +7
-0+if (unlikely (!guest_handle_okay (iovs_hnd, arg3)) ) +break ; + + rc =sendv (currd, &send_addr.src, &send_addr.dst, iovs_hnd, arg3, arg4); +break ; + } +default : rc = -EOPNOTSUPP ;break ;
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v3 09/15] argo: implement the sendv op; evtchn: expose send_guest_global_virq
Quoted 32 lines · Jason Andryuk
On Mon, Jan 7, 2019 at 2:43 AM Christopher Clark
<[email protected]> wrote:<snip>
Quoted 14 lines
xen/common/argo.c +10
-0@@ -342,6 +357,413 @@ update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr)smp_wmb (); } +static int +memcpy_to_guest_ring (struct argo_ring_info *ring_info,uint32_t offset, +const void *src,XEN_GUEST_HANDLE (uint8_t ) src_hnd, +uint32_t len) +{ +unsigned int mfns_index = offset >>PAGE_SHIFT ; +void *dst; +int ret; +unsigned int src_offset =0 ; +xen/common/argo.c update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 379
Quoted 10 lines
xen/common/argo.c +10
-0+ASSERT (spin_is_locked (&ring_info->lock)); + + offset &= ~PAGE_MASK ; + +if ( (len >XEN_ARGO_MAX_RING_SIZE ) || (offset >XEN_ARGO_MAX_RING_SIZE ) ) +return -EFAULT ; + +while ( (offset + len) >PAGE_SIZE ) + { +unsigned int head_len =PAGE_SIZE - offset;I think this while loop could be re-written as
Quoted 8 lines
while (len) { head_len = len > PAGE_SIZE ? PAGE_SIZE - offset: len;and then the extra copying below outside the loop could be dropped.
The first loop does a partial copy at offset and then sets offset=0. The next N loops copy exactly
PAGE_SIZE.
The Final copy does the remaining len bytes.
That looks right to me and makes a nice simplification to that function -- thanks.
Quoted 23 lines
<snip>
Quoted 21 lines
xen/common/argo.c +21
-0+ +/* +* iov_count returns its count on success via an out variable to avoid +* potential for a negative return value to be used incorrectly +* (eg. coerced into an unsigned variable resulting in a large incorrect value) +*/ +static int +iov_count (const xen_argo_iov_t *piov,unsigned long niov,uint32_t *count) +{ +uint32_t sum_iov_lens =0 ; + +if ( niov >XEN_ARGO_MAXIOV ) +return -EINVAL ; + +while ( niov-- ) + { +/* valid iovs must have the padding field set to zero */ +if ( piov->pad ) + { +argo_dprintk ("invalid iov: padding is not zero\n" ); +return -EINVAL ;
Quoted 8 lines
xen/common/argo.c update_tx_ptr(struct argo_ring_info *ring_info, uint32_t tx_ptr) · line 499
Quoted 4 lines
xen/common/argo.c +4
-0+ } + +/* check each to protect sum against integer overflow */ +if ( piov->iov_len >XEN_ARGO_MAX_RING_SIZE )Should this be
MAX_ARGO_MESSAGE_SIZE?MAX_ARGO_MESSAGE_SIZEis less thanXEN_ARGO_MAX_RING_SIZE, so we can pass this check and then just fail the one below.
ack - I'll switch it to MAX_ARGO_MESSAGE_SIZE.
Quoted 32 lines
<snip>
Quoted 30 lines
xen/common/argo.c +26
-0@@ -1073,6 +1683,49 @@ do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1,break ; } +case XEN_ARGO_OP_sendv: + { +xen_argo_send_addr_t send_addr; + +XEN_GUEST_HANDLE_PARAM (xen_argo_send_addr_t) send_addr_hnd = +guest_handle_cast (arg1, xen_argo_send_addr_t); +XEN_GUEST_HANDLE_PARAM (xen_argo_iov_t) iovs_hnd = +guest_handle_cast (arg2, xen_argo_iov_t); +/* arg3 is niov */ +/* arg4 is message_type. Must be a 32-bit value. */ + + rc =copy_from_guest (&send_addr, send_addr_hnd,1 ) ? -EFAULT :0 ; +if ( rc ) +break ; + +if ( send_addr.src.domain_id ==XEN_ARGO_DOMID_ANY ) + send_addr.src.domain_id = currd->domain_id; + +/* No domain is currently authorized to send on behalf of another */ +if (unlikely (send_addr.src.domain_id != currd->domain_id) ) + { + rc = -EPERM ; +break ; + } + +/* Reject niov or message_type values that are outside 32 bit range. */
Quoted 8 lines
xen/common/argo.c do_argo_op(unsigned int cmd, XEN_GUEST_HANDLE_PARAM(void) arg1, · line 1716
Quoted 5 lines
xen/common/argo.c +5
-0+if (unlikely ((arg3 >XEN_ARGO_MAXIOV ) || (arg4 & ~0xffffffffUL )) ) + { + rc = -EINVAL ; +break ; + }This needs to check send_addr.src.pad and send_addr.dst.pad == 0.
sendv()does not check the padding either.
ack - will fix.
thanks
Christopher
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel