xen: optimize xenbus performance
Subsystems (6)
Juergen Gross
maintainerv4 cover xen: optimize xenbus performance
The xenbus driver used for communication with Xenstore (all kernel accesses to Xenstore and in case of Xenstore living in another domain all accesses of the local domain to Xenstore) is rather simple especially regarding multiple concurrent accesses: they are just being serialized in spite of Xenstore being capable to handle multiple parallel accesses.
Clean up the external interface(s) of xenbus and optimize its performance by allowing multiple concurrent accesses to Xenstore.
V4:
- patch 3: correct xs_request_enter()
- patch 3: correct xs_request_exit() as requested by Boris Ostrovsky
V3:
- patch 3: simplify coding as requested by Boris Ostrovsky
V2:
- patch 1: update commit message, re-add lost copyright
- patch 3: address comments of Boris Ostrovsky:
- guard xs_request_id by lock
- move state struct definitions to the place where they are being used
- rate limit some warnings
- use barrier() instead of cpu_relax()
- add/remove some comments
- minor changes like naming of variables
Juergen Gross (3):
xen: clean up xenbus internal headers
xen: modify xenstore watch event interface
xen: optimize xenbus driver for multiple concurrent xenstore accesses
drivers/block/xen-blkback/xenbus.c | 6 +- drivers/net/xen-netback/xenbus.c | 8 +- drivers/xen/cpu_hotplug.c | 5 +- drivers/xen/manage.c | 6 +- drivers/xen/xen-balloon.c | 2 +- drivers/xen/xen-pciback/xenbus.c | 2 +- drivers/xen/xenbus/xenbus.h | 135 +++++++ drivers/xen/xenbus/xenbus_client.c | 6 +- drivers/xen/xenbus/xenbus_comms.c | 315 +++++++++++++++-- drivers/xen/xenbus/xenbus_comms.h | 51 --- drivers/xen/xenbus/xenbus_dev_backend.c | 2 +- drivers/xen/xenbus/xenbus_dev_frontend.c | 213 ++++++----- drivers/xen/xenbus/xenbus_probe.c | 14 +- drivers/xen/xenbus/xenbus_probe.h | 88 ----- drivers/xen/xenbus/xenbus_probe_backend.c | 11 +- drivers/xen/xenbus/xenbus_probe_frontend.c | 17 +- drivers/xen/xenbus/xenbus_xs.c | 544 +++++++++++++---------------- drivers/xen/xenfs/super.c | 2 +- drivers/xen/xenfs/xenstored.c | 2 +- include/xen/xenbus.h | 18 +- 20 files changed, 835 insertions(+), 612 deletions(-) create mode 100644 drivers/xen/xenbus/xenbus.h delete mode 100644 drivers/xen/xenbus/xenbus_comms.h delete mode 100644 drivers/xen/xenbus/xenbus_probe.h
Signature · 7 lines
2.10.2 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Juergen Gross
maintainerPATCH v4 1/3 xen: clean up xenbus internal headers
The xenbus driver has an awful mixture of internally and globally visible headers: some of the internally used only stuff is defined in the global header include/xen/xenbus.h while some stuff defined in internal headers is used by other drivers, too.
Clean this up by moving the externally used symbols to include/xen/xenbus.h and the symbols used internally only to a new header drivers/xen/xenbus/xenbus.h replacing xenbus_comms.h and xenbus_probe.h
---
v2: update commit message, re-add lost copyright
---
drivers/xen/xenbus/{xenbus_probe.h => xenbus.h} | 63 ++++++++++++++-----------
drivers/xen/xenbus/xenbus_client.c | 2 +-
drivers/xen/xenbus/xenbus_comms.c | 2 +-
drivers/xen/xenbus/xenbus_comms.h | 51 --------------------
drivers/xen/xenbus/xenbus_dev_backend.c | 2 +-
drivers/xen/xenbus/xenbus_dev_frontend.c | 4 +-
drivers/xen/xenbus/xenbus_probe.c | 3 +-
drivers/xen/xenbus/xenbus_probe_backend.c | 3 +-
drivers/xen/xenbus/xenbus_probe_frontend.c | 3 +-
drivers/xen/xenbus/xenbus_xs.c | 3 +-
drivers/xen/xenfs/super.c | 2 +-
drivers/xen/xenfs/xenstored.c | 2 +-
include/xen/xenbus.h | 12 ++---
13 files changed, 52 insertions(+), 100 deletions(-)
rename drivers/xen/xenbus/{xenbus_probe.h => xenbus.h} (59%)
delete mode 100644 drivers/xen/xenbus/xenbus_comms.hdrivers/xen/xenbus/xenbus.h renamed from drivers/xen/xenbus/xenbus_probe.h +36 -27
@@ -1,7 +1,5 @@ -/****************************************************************************** -* xenbus_probe.h -* -* Talks to Xen Store to figure out what devices we have. +/* +* Private include for xenbus communications. * * Copyright (C) 2005 Rusty Russell, IBM Corporation * Copyright (C) 2005 XenSource Ltd.
@@ -31,8 +29,8 @@* IN THE SOFTWARE. */ -#ifndef _XENBUS_PROBE_H -#define _XENBUS_PROBE_H +#ifndef _XENBUS_XENBUS_H +#define _XENBUS_XENBUS_H#define XEN_BUS_ID_SIZE 20
@@ -54,35 +52,46 @@ enum xenstore_init { XS_LOCAL, }; +extern enum xenstore_init xen_store_domain_type;extern const struct attribute_group *xenbus_dev_groups[]; -extern int xenbus_match (struct device *_dev,struct device_driver *_drv); -extern int xenbus_dev_probe (struct device *_dev); -extern int xenbus_dev_remove (struct device *_dev); -extern int xenbus_register_driver_common (struct xenbus_driver *drv, -struct xen_bus_type *bus, -struct module *owner, -const char *mod_name); -extern int xenbus_probe_node (struct xen_bus_type *bus, -const char *type, -const char *nodename); -extern int xenbus_probe_devices (struct xen_bus_type *bus); +int xs_init (void ); +int xb_init_comms (void ); +void xb_deinit_comms (void ); +int xb_write (const void *data,unsigned int len); +int xb_read (void *data,unsigned int len); +int xb_data_to_read (void ); +int xb_wait_for_data_to_read (void ); -extern void xenbus_dev_changed (const char *node,struct xen_bus_type *bus); +int xenbus_match (struct device *_dev,struct device_driver *_drv); +int xenbus_dev_probe (struct device *_dev); +int xenbus_dev_remove (struct device *_dev); +int xenbus_register_driver_common (struct xenbus_driver *drv, +struct xen_bus_type *bus, +struct module *owner, +const char *mod_name); +int xenbus_probe_node (struct xen_bus_type *bus, +const char *type, +const char *nodename); +int xenbus_probe_devices (struct xen_bus_type *bus); -extern void xenbus_dev_shutdown (struct device *_dev); +void xenbus_dev_changed (const char *node,struct xen_bus_type *bus); -extern int xenbus_dev_suspend (struct device *dev); -extern int xenbus_dev_resume (struct device *dev); -extern int xenbus_dev_cancel (struct device *dev); +void xenbus_dev_shutdown (struct device *_dev); -extern void xenbus_otherend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len, -int ignore_on_shutdown); +int xenbus_dev_suspend (struct device *dev); +int xenbus_dev_resume (struct device *dev); +int xenbus_dev_cancel (struct device *dev); -extern int xenbus_read_otherend_details (struct xenbus_device *xendev, -char *id_node,char *path_node); +void xenbus_otherend_changed (struct xenbus_watch *watch, +const char **vec,unsigned int len, +int ignore_on_shutdown); + +int xenbus_read_otherend_details (struct xenbus_device *xendev, +char *id_node,char *path_node);void xenbus_ring_ops_init (void ); +void *xenbus_dev_request_and_reply (struct xsd_sockmsg *msg); +#endif
drivers/xen/xenbus/xenbus_client.c +1 -1
@@ -47,7 +47,7 @@#include <xen/xen.h> #include <xen/features.h> -#include "xenbus_probe.h" +#include "xenbus.h" #define XENBUS_PAGES (_grants) (DIV_ROUND_UP(_grants, XEN_PFN_PER_PAGE))
drivers/xen/xenbus/xenbus_comms.c +1 -1
@@ -40,7 +40,7 @@#include <asm/xen/hypervisor.h> #include <xen/events.h> #include <xen/page.h> -#include "xenbus_comms.h" +#include "xenbus.h" static int xenbus_irq;
drivers/xen/xenbus/xenbus_comms.h deleted +0 -51
51 lines not on this page · raw message
drivers/xen/xenbus/xenbus_dev_backend.c +1 -1
@@ -16,7 +16,7 @@#include <xen/events.h> #include <asm/xen/hypervisor.h> -#include "xenbus_comms.h" +#include "xenbus.h" static int xenbus_backend_open (struct inode *inode,struct file *filp) {
drivers/xen/xenbus/xenbus_dev_frontend.c +2 -2
@@ -57,12 +57,12 @@#include <linux/miscdevice.h> #include <linux/init.h> -#include "xenbus_comms.h" -#include <xen/xenbus.h> #include <xen/xen.h> #include <asm/xen/hypervisor.h> +#include "xenbus.h" +/* * An element of a list of outstanding transactions, for which we're * still waiting a reply.
drivers/xen/xenbus/xenbus_probe.c +1 -2
@@ -62,8 +62,7 @@#include <xen/hvm.h> -#include "xenbus_comms.h" -#include "xenbus_probe.h" +#include "xenbus.h" int xen_store_evtchn;
drivers/xen/xenbus/xenbus_probe_backend.c +1 -2
@@ -53,8 +53,7 @@#include <xen/xenbus.h> #include <xen/features.h> -#include "xenbus_comms.h" -#include "xenbus_probe.h" +#include "xenbus.h" /* backend/<type>/<fe-uuid>/<id> => <type>-<fe-domid>-<id> */ static int backend_bus_id (char bus_id[XEN_BUS_ID_SIZE ],const char *nodename)
drivers/xen/xenbus/xenbus_probe_frontend.c +1 -2
@@ -27,8 +27,7 @@#include <xen/platform_pci.h> -#include "xenbus_comms.h" -#include "xenbus_probe.h" +#include "xenbus.h"
drivers/xen/xenbus/xenbus_xs.c +1 -2
@@ -48,8 +48,7 @@#include <asm/xen/hypervisor.h> #include <xen/xenbus.h> #include <xen/xen.h> -#include "xenbus_comms.h" -#include "xenbus_probe.h" +#include "xenbus.h" struct xs_stored_msg {struct list_head list;
drivers/xen/xenfs/super.c +1 -1
@@ -16,10 +16,10 @@#include <linux/magic.h> #include <xen/xen.h> +#include <xen/xenbus.h> #include "xenfs.h" #include "../privcmd.h" -#include "../xenbus/xenbus_comms.h" #include <asm/xen/hypervisor.h>
drivers/xen/xenfs/xenstored.c +1 -1
@@ -4,9 +4,9 @@#include <linux/fs.h> #include <xen/page.h> +#include <xen/xenbus.h> #include "xenfs.h" -#include "../xenbus/xenbus_comms.h" static ssize_t xsd_read (struct file *file,char __user *buf,size_t size,loff_t *off)
include/xen/xenbus.h +5 -7
@@ -38,6 +38,7 @@#include <linux/notifier.h> #include <linux/mutex.h> #include <linux/export.h> +#include <linux/fs.h> #include <linux/completion.h> #include <linux/init.h> #include <linux/slab.h>
@@ -175,16 +176,9 @@ void xs_suspend(void);void xs_resume (void );void xs_suspend_cancel (void ); -/* Used by xenbus_dev to borrow kernel's store connection. */ -void *xenbus_dev_request_and_reply (struct xsd_sockmsg *msg); -struct work_struct ; -/* Prepare for domain suspend: then resume or cancel the suspend. */ -void xenbus_suspend (void ); -void xenbus_resume (void );void xenbus_probe (struct work_struct *); -void xenbus_suspend_cancel (void );#define XENBUS_IS_ERR_READ (str) ({ \ if (!IS_ERR(str) && strlen(str) == 0) { \
@@ -235,4 +229,8 @@ const char *xenbus_strstate(enum xenbus_state state);int xenbus_dev_is_online (struct xenbus_device *dev);int xenbus_frontend_closed (struct xenbus_device *dev); +extern const struct file_operations xen_xenbus_fops; +extern struct xenstore_domain_interface *xen_store_interface; +extern int xen_store_evtchn; +#endif /* _XEN_XENBUS_H */
Signature · 7 lines
2.10.2 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Juergen Gross
maintainerPATCH v4 2/3 xen: modify xenstore watch event interface
In reply to Juergen GrossToday a Xenstore watch event is delivered via a callback function declared as:
void (*callback)(struct xenbus_watch *,
const char **vec, unsigned int len);
As all watch events only ever come with two parameters (path and token) changing the prototype to:
void (*callback)(struct xenbus_watch *,
const char *path, const char *token);
is the natural thing to do.
Apply this change and adapt all users.
--- drivers/block/xen-blkback/xenbus.c | 6 +++--- drivers/net/xen-netback/xenbus.c | 8 ++++---- drivers/xen/cpu_hotplug.c | 5 ++--- drivers/xen/manage.c | 6 +++--- drivers/xen/xen-balloon.c | 2 +- drivers/xen/xen-pciback/xenbus.c | 2 +- drivers/xen/xenbus/xenbus.h | 6 +++--- drivers/xen/xenbus/xenbus_client.c | 4 ++-- drivers/xen/xenbus/xenbus_dev_frontend.c | 21 ++++++++------------- drivers/xen/xenbus/xenbus_probe.c | 11 ++++------- drivers/xen/xenbus/xenbus_probe_backend.c | 8 ++++---- drivers/xen/xenbus/xenbus_probe_frontend.c | 14 +++++++------- drivers/xen/xenbus/xenbus_xs.c | 29 ++++++++++++++--------------- include/xen/xenbus.h | 6 +++--- 14 files changed, 59 insertions(+), 69 deletions(-)
drivers/block/xen-blkback/xenbus.c +3 -3
@@ -38,8 +38,8 @@ struct backend_info {static struct kmem_cache *xen_blkif_cachep;static void connect (struct backend_info *);static int connect_ring (struct backend_info *); -static void backend_changed (struct xenbus_watch *,const char **, -unsigned int ); +static void backend_changed (struct xenbus_watch *,const char *, +const char *);static void xen_blkif_free (struct xen_blkif *blkif);static void xen_vbd_free (struct xen_vbd *vbd);
@@ -661,7 +661,7 @@ static int xen_blkbk_probe(struct xenbus_device *dev,* ready, connect. */ static void backend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {int err;unsigned major;
drivers/net/xen-netback/xenbus.c +4 -4
@@ -734,7 +734,7 @@ static int xen_net_read_mac(struct xenbus_device *dev, u8 mac[]) }static void xen_net_rate_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) { struct xenvif *vif = container_of(watch, struct xenvif, credit_watch);struct xenbus_device *dev =xenvif_to_xenbus_device (vif);
@@ -791,7 +791,7 @@ static void xen_unregister_credit_watch(struct xenvif *vif) }static void xen_mcast_ctrl_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) { struct xenvif *vif = container_of(watch, struct xenvif, mcast_ctrl_watch);
@@ -866,8 +866,8 @@ static void unregister_hotplug_status_watch(struct backend_info *be) }static void hotplug_status_changed (struct xenbus_watch *watch, -const char **vec, -unsigned int vec_size) +const char *path, +const char *token) { struct backend_info *be = container_of(watch, struct backend_info,
drivers/xen/cpu_hotplug.c +2 -3
@@ -68,13 +68,12 @@ static void vcpu_hotplug(unsigned int cpu) }static void handle_vcpu_hotplug_event (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {unsigned int cpu;char *cpustr; -const char *node = vec[XS_WATCH_PATH ]; - cpustr =strstr (node,"cpu/" ); + cpustr =strstr (path,"cpu/" );if (cpustr !=NULL ) {sscanf (cpustr,"cpu/%u" , &cpu);vcpu_hotplug (cpu);
drivers/xen/manage.c +3 -3
@@ -218,7 +218,7 @@ static struct shutdown_handler shutdown_handlers[] = { };static void shutdown_handler (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {char *str;struct xenbus_transaction xbt;
@@ -266,8 +266,8 @@ static void shutdown_handler(struct xenbus_watch *watch, }#ifdef CONFIG_MAGIC_SYSRQ -static void sysrq_handler (struct xenbus_watch *watch,const char **vec, -unsigned int len) +static void sysrq_handler (struct xenbus_watch *watch,const char *path, +const char *token) {char sysrq_key ='\0' ;struct xenbus_transaction xbt;
drivers/xen/xen-balloon.c +1 -1
@@ -55,7 +55,7 @@ static int register_balloon(struct device *dev);/* React to a change in the target key */ static void watch_target (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {unsigned long long new_target;int err;
drivers/xen/xen-pciback/xenbus.c +1 -1
@@ -652,7 +652,7 @@ static int xen_pcibk_setup_backend(struct xen_pcibk_device *pdev) }static void xen_pcibk_be_watch (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {struct xen_pcibk_device *pdev = container_of(watch, struct xen_pcibk_device, be_watch);
drivers/xen/xenbus/xenbus.h +3 -3
@@ -40,8 +40,8 @@ struct xen_bus_type {int (*get_bus_id)(char bus_id[XEN_BUS_ID_SIZE ],const char *nodename);int (*probe)(struct xen_bus_type *bus,const char *type,const char *dir); -void (*otherend_changed)(struct xenbus_watch *watch,const char **vec, -unsigned int len); +void (*otherend_changed)(struct xenbus_watch *watch,const char *path, +const char *token);struct bus_type bus; };
@@ -84,7 +84,7 @@ int xenbus_dev_resume(struct device *dev);int xenbus_dev_cancel (struct device *dev);void xenbus_otherend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len, +const char *path,const char *token,int ignore_on_shutdown); int xenbus_read_otherend_details(struct xenbus_device *xendev,
drivers/xen/xenbus/xenbus_client.c +2 -2
@@ -115,7 +115,7 @@ EXPORT_SYMBOL_GPL(xenbus_strstate);int xenbus_watch_path (struct xenbus_device *dev,const char *path,struct xenbus_watch *watch,void (*callback)(struct xenbus_watch *, -const char **,unsigned int )) +const char *,const char *)) {int err;
@@ -153,7 +153,7 @@ EXPORT_SYMBOL_GPL(xenbus_watch_path);int xenbus_watch_pathfmt (struct xenbus_device *dev,struct xenbus_watch *watch,void (*callback)(struct xenbus_watch *, -const char **,unsigned int ), +const char *,const char *),const char *pathfmt, ...) {int err;
drivers/xen/xenbus/xenbus_dev_frontend.c +8 -13
@@ -258,26 +258,23 @@ static struct watch_adapter *alloc_watch_adapter(const char *path, }static void watch_fired (struct xenbus_watch *watch, -const char **vec, -unsigned int len) +const char *path, +const char *token) {struct watch_adapter *adap;struct xsd_sockmsg hdr; -const char *path, *token; -int path_len, tok_len, body_len, data_len =0 ; +const char *token_caller; +int path_len, tok_len, body_len;int ret;LIST_HEAD (staging_q); adap = container_of(watch, struct watch_adapter, watch); - path = vec[XS_WATCH_PATH ]; - token = adap->token; + token_caller = adap->token; path_len =strlen (path) +1 ; - tok_len =strlen (token) +1 ; -if (len >2 ) - data_len = vec[len] - vec[2 ] +1 ; - body_len = path_len + tok_len + data_len; + tok_len =strlen (token_caller) +1 ; + body_len = path_len + tok_len; hdr.type =XS_WATCH_EVENT ; hdr.len = body_len;
@@ -288,9 +285,7 @@ static void watch_fired(struct xenbus_watch *watch,if (!ret) ret =queue_reply (&staging_q, path, path_len);if (!ret) - ret =queue_reply (&staging_q, token, tok_len); -if (!ret && len >2 ) - ret =queue_reply (&staging_q, vec[2 ], data_len); + ret =queue_reply (&staging_q, token_caller, tok_len);if (!ret) {/* success: pass reply list onto watcher */
drivers/xen/xenbus/xenbus_probe.c +4 -7
@@ -169,7 +169,7 @@ int xenbus_read_otherend_details(struct xenbus_device *xendev,EXPORT_SYMBOL_GPL (xenbus_read_otherend_details);void xenbus_otherend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len, +const char *path,const char *token,int ignore_on_shutdown) { struct xenbus_device *dev =
@@ -180,18 +180,15 @@ void xenbus_otherend_changed(struct xenbus_watch *watch,/* Protect us against watches firing on old details when the otherend details change, say immediately after a resume. */ if (!dev->otherend || -strncmp (dev->otherend, vec[XS_WATCH_PATH ], -strlen (dev->otherend))) { -dev_dbg (&dev->dev,"Ignoring watch at %s\n" , - vec[XS_WATCH_PATH ]); +strncmp (dev->otherend, path,strlen (dev->otherend))) { +dev_dbg (&dev->dev,"Ignoring watch at %s\n" , path);return ; } state =xenbus_read_driver_state (dev->otherend);dev_dbg (&dev->dev,"state is %d, (%s), %s, %s\n" , - state,xenbus_strstate (state), dev->otherend_watch.node, - vec[XS_WATCH_PATH ]); + state,xenbus_strstate (state), dev->otherend_watch.node, path);/* * Ignore xenbus transitions during shutdown. This prevents us doing
drivers/xen/xenbus/xenbus_probe_backend.c +4 -4
@@ -181,9 +181,9 @@ static int xenbus_probe_backend(struct xen_bus_type *bus, const char *type, }static void frontend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) { -xenbus_otherend_changed (watch, vec, len,0 ); +xenbus_otherend_changed (watch, path, token,0 ); }static struct xen_bus_type xenbus_backend = {
@@ -204,11 +204,11 @@ static struct xen_bus_type xenbus_backend = { };static void backend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {DPRINTK ("" ); -xenbus_dev_changed (vec[XS_WATCH_PATH ], &xenbus_backend); +xenbus_dev_changed (path, &xenbus_backend); }static struct xenbus_watch be_watch = {
drivers/xen/xenbus/xenbus_probe_frontend.c +7 -7
@@ -86,9 +86,9 @@ static int xenbus_uevent_frontend(struct device *_dev,static void backend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) { -xenbus_otherend_changed (watch, vec, len,1 ); +xenbus_otherend_changed (watch, path, token,1 ); }static void xenbus_frontend_delayed_resume (struct work_struct *w)
@@ -153,11 +153,11 @@ static struct xen_bus_type xenbus_frontend = { };static void frontend_changed (struct xenbus_watch *watch, -const char **vec,unsigned int len) +const char *path,const char *token) {DPRINTK ("" ); -xenbus_dev_changed (vec[XS_WATCH_PATH ], &xenbus_frontend); +xenbus_dev_changed (path, &xenbus_frontend); }
@@ -332,13 +332,13 @@ static DECLARE_WAIT_QUEUE_HEAD(backend_state_wq);static int backend_state;static void xenbus_reset_backend_state_changed (struct xenbus_watch *w, -const char **v,unsigned int l) +const char *path,const char *token) { -if (xenbus_scanf (XBT_NIL , v[XS_WATCH_PATH ],"" ,"%i" , +if (xenbus_scanf (XBT_NIL , path,"" ,"%i" , &backend_state) !=1 ) backend_state = XenbusStateUnknown;printk (KERN_DEBUG "XENBUS: backend %s %s\n" , - v[XS_WATCH_PATH ],xenbus_strstate (backend_state)); + path,xenbus_strstate (backend_state));wake_up (&backend_state_wq); }
drivers/xen/xenbus/xenbus_xs.c +14 -15
@@ -64,8 +64,8 @@ struct xs_stored_msg {/* Queued watch events. */ struct {struct xenbus_watch *handle; -char **vec; -unsigned int vec_size; +const char *path; +const char *token; } watch; } u; };
@@ -765,7 +765,7 @@ void unregister_xenbus_watch(struct xenbus_watch *watch)if (msg->u.watch.handle != watch)continue ;list_del (&msg->list); -kfree (msg->u.watch.vec); +kfree (msg->u.watch.path);kfree (msg); }spin_unlock (&watch_events_lock);
@@ -833,11 +833,10 @@ static int xenwatch_thread(void *unused)if (ent != &watch_events) { msg = list_entry(ent, struct xs_stored_msg, list); - msg->u.watch.handle->callback ( - msg->u.watch.handle, - (const char **)msg->u.watch.vec, - msg->u.watch.vec_size); -kfree (msg->u.watch.vec); + msg->u.watch.handle->callback (msg->u.watch.handle, + msg->u.watch.path, + msg->u.watch.token); +kfree (msg->u.watch.path);kfree (msg); }
@@ -903,24 +902,24 @@ static int process_msg(void) body[msg->hdr.len] ='\0' ;if (msg->hdr.type ==XS_WATCH_EVENT ) { - msg->u.watch.vec =split (body, msg->hdr.len, - &msg->u.watch.vec_size); -if (IS_ERR (msg->u.watch.vec)) { - err =PTR_ERR (msg->u.watch.vec); +if (count_strings (body, msg->hdr.len) !=2 ) { + err = -EINVAL ;kfree (msg); +kfree (body); goto out; } + msg->u.watch.path = (const char *)body; + msg->u.watch.token = (const char *)strchr (body,'\0' ) +1 ;spin_lock (&watches_lock); - msg->u.watch.handle =find_watch ( - msg->u.watch.vec[XS_WATCH_TOKEN ]); + msg->u.watch.handle =find_watch (msg->u.watch.token);if (msg->u.watch.handle !=NULL ) {spin_lock (&watch_events_lock);list_add_tail (&msg->list, &watch_events);wake_up (&watch_events_waitq);spin_unlock (&watch_events_lock); }else { -kfree (msg->u.watch.vec); +kfree (body);kfree (msg); }spin_unlock (&watches_lock);
include/xen/xenbus.h +3 -3
@@ -61,7 +61,7 @@ struct xenbus_watch/* Callback (executed in a process context with no locks held). */ void (*callback)(struct xenbus_watch *, -const char **vec,unsigned int len); +const char *path,const char *token); };
@@ -193,11 +193,11 @@ void xenbus_probe(struct work_struct *);int xenbus_watch_path (struct xenbus_device *dev,const char *path,struct xenbus_watch *watch,void (*callback)(struct xenbus_watch *, -const char **,unsigned int )); +const char *,const char *)); __printf(4, 5)int xenbus_watch_pathfmt (struct xenbus_device *dev,struct xenbus_watch *watch,void (*callback)(struct xenbus_watch *, -const char **,unsigned int ), +const char *,const char *),const char *pathfmt, ...);int xenbus_switch_state (struct xenbus_device *dev,enum xenbus_state new_state);
Signature · 7 lines
2.10.2 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Juergen Gross
maintainerPATCH v4 3/3 xen: optimize xenbus driver for multiple concurrent xenstore accesses
In reply to Juergen GrossHandling of multiple concurrent Xenstore accesses through xenbus driver either from the kernel or user land is rather lame today: xenbus is capable to have one access active only at one point of time.
Rewrite xenbus to handle multiple requests concurrently by making use of the request id of the Xenstore protocol. This requires to:
- Instead of blocking inside xb_read() when trying to read data from the xenstore ring buffer do so only in the main loop of xenbus_thread().
- Instead of doing writes to the xenstore ring buffer in the context of the caller just queue the request and do the write in the dedicated xenbus thread.
- Instead of just forwarding the request id specified by the caller of xenbus to xenstore use a xenbus internal unique request id. This will allow multiple outstanding requests.
- Modify the locking scheme in order to allow multiple requests being active in parallel.
- Instead of waiting for the reply of a user's xenstore request after writing the request to the xenstore ring buffer return directly to the caller and do the waiting in the read path.
Additionally signal handling was optimized by avoiding waking up the xenbus thread or sending an event to Xenstore in case the addressed entity is known to be running already.
As a result communication with Xenstore is sped up by a factor of up to 5: depending on the request type (read or write) and the amount of data transferred the gain was at least 20% (small reads) and went up to a factor of 5 for large writes.
In the end some more rough edges of xenbus have been smoothed:
- Handling of memory shortage when reading from xenstore ring buffer in the xenbus driver was not optimal: it was busy looping and issuing a warning in each loop.
- In case of xenstore not running in dom0 but in a stubdom we end up with two xenbus threads running as the initialization of xenbus in dom0 expecting a local xenstored will be redone later when connecting to the xenstore domain. Up to now this was no problem as locking would prevent the two xenbus threads interfering with each other, but this was just a waste of kernel resources.
- An out of memory situation while writing to or reading from the xenstore ring buffer no longer will lead to a possible loss of synchronization with xenstore.
- The user read and write part are now interruptible by signals.
---
V4: - correct xs_request_enter() to avoid deadlocks in case of a
connection being used for transactions and non-transactional accesses in parallel
- correct xs_request_exit() to account for failed transactions as requested by Boris Ostrovsky
V3: simplify xs_request_enter() as requested by Boris Ostrovsky
V2: address comments of Boris Ostrovsky:
- guard xs_request_id by lock
- move state struct definitions to the place where they are being used
- rate limit some warnings
- use barrier() instead of cpu_relax()
- add/remove some comments
- minor changes like naming of variables
--- drivers/xen/xenbus/xenbus.h | 48 ++- drivers/xen/xenbus/xenbus_comms.c | 313 ++++++++++++++++-- drivers/xen/xenbus/xenbus_dev_frontend.c | 188 +++++++---- drivers/xen/xenbus/xenbus_xs.c | 538 ++++++++++++++----------------- 4 files changed, 684 insertions(+), 403 deletions(-)
drivers/xen/xenbus/xenbus.h +43 -5
@@ -32,6 +32,10 @@#ifndef _XENBUS_XENBUS_H#define _XENBUS_XENBUS_H +#include <linux/mutex.h> +#include <linux/uio.h> +#include <xen/xenbus.h> +#define XEN_BUS_ID_SIZE 20 struct xen_bus_type {
@@ -52,16 +56,49 @@ enum xenstore_init { XS_LOCAL, }; +struct xs_watch_event { +struct list_head list; +unsigned int len; +struct xenbus_watch *handle; +const char *path; +const char *token; +char body[]; +}; + +enum xb_req_state { + xb_req_state_queued, + xb_req_state_wait_reply, + xb_req_state_got_reply, + xb_req_state_aborted +}; + +struct xb_req_data { +struct list_head list; +wait_queue_head_t wq; +struct xsd_sockmsg msg; +enum xsd_sockmsg_type type; +char *body; +const struct kvec *vec; +int num_vecs; +int err; +enum xb_req_state state; +void (*cb)(struct xb_req_data *); +void *par; +}; +extern enum xenstore_init xen_store_domain_type;extern const struct attribute_group *xenbus_dev_groups[]; +extern struct mutex xs_response_mutex; +extern struct list_head xs_reply_list; +extern struct list_head xb_write_list; +extern wait_queue_head_t xb_waitq; +extern struct mutex xb_write_mutex;int xs_init (void );int xb_init_comms (void );void xb_deinit_comms (void ); -int xb_write (const void *data,unsigned int len); -int xb_read (void *data,unsigned int len); -int xb_data_to_read (void ); -int xb_wait_for_data_to_read (void ); +int xs_watch_msg (struct xs_watch_event *event); +void xs_request_exit (struct xb_req_data *req);int xenbus_match (struct device *_dev,struct device_driver *_drv);int xenbus_dev_probe (struct device *_dev);
@@ -92,6 +129,7 @@ int xenbus_read_otherend_details(struct xenbus_device *xendev,void xenbus_ring_ops_init (void ); -void *xenbus_dev_request_and_reply (struct xsd_sockmsg *msg); +int xenbus_dev_request_and_reply (struct xsd_sockmsg *msg,void *par); +void xenbus_dev_queue_reply (struct xb_req_data *req);#endif
drivers/xen/xenbus/xenbus_comms.c +279 -34
414 lines not on this page · raw message
drivers/xen/xenbus/xenbus_dev_frontend.c +120 -68
247 lines not on this page · raw message
drivers/xen/xenbus/xenbus_xs.c +242 -296
728 lines not on this page · raw message
Signature · 7 lines
2.10.2 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Boris Ostrovsky
AppliedQuoted 10 lines · Juergen Gross
The xenbus driver used for communication with Xenstore (all kernel accesses to Xenstore and in case of Xenstore living in another domain all accesses of the local domain to Xenstore) is rather simple especially regarding multiple concurrent accesses: they are just being serialized in spite of Xenstore being capable to handle multiple parallel accesses.
Clean up the external interface(s) of xenbus and optimize its performance by allowing multiple concurrent accesses to Xenstore.
Applied to for-linus-4.11.
-boris
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Boris Ostrovsky
AppliedQuoted 10 lines · Juergen Gross
The xenbus driver used for communication with Xenstore (all kernel accesses to Xenstore and in case of Xenstore living in another domain all accesses of the local domain to Xenstore) is rather simple especially regarding multiple concurrent accesses: they are just being serialized in spite of Xenstore being capable to handle multiple parallel accesses.
Clean up the external interface(s) of xenbus and optimize its performance by allowing multiple concurrent accesses to Xenstore.
Applied to for-linus-4.11.
-boris