Skip to contentSkip to conversation

add support for libvirt-like channels

PATCH New David Scott 3 participants 11 messages Sep 15, 2014 – Sep 26, 2014

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 add support for libvirt-like channels

Hi,

I've incorporated IanC's feedback from the v4. IanJ: I've tried to incorporate your feedback from the earlier v3 -- let me know what you think.

The blurb:

Several libvirt applications (e.g. oVirt, CloudStack) make use of 'channels':
low-bandwidth private host <-> guest communication links which resemble serial ports. Typical uses include:

* initial VM configuration without using the network: read the config data directly from the channel on boot

* controlling a guest agent: signal shutdown reqests, exchange clipboard data, trigger resolution changes

This patch set re-uses the existing PV console protocol implemented by qemu to provide this service.

If you declare a channel in your .xl file as follows:

channel = [ "connection=socket,path=/tmp/mysock,name=org.mydomain.guest-agent" ]

then an extra PV console will be added to your guest. This console has the extra key in the frontend

name = guest-agent

which allows udev scripts in the VM to create a named device in a well-known location (e.g. /dev/xen-channels/guest-agent, similar to the KVM /dev/vports). The qemu process in the backend domain will proxy the data to/from the named Unix domain socket (in this case /tmp/mysock).

Note this mechanism is intended for low-bandwidth communication only. If an application requires a high-bandwith connection then it should use a direct vchan connection (and not proxy it via a qemu).

Changes since v4:
* doc: highlight that the 'output' key in the console configuration should be considered an internal implementation artifact
* return EINVAL if a primary console has invalid configuration
* Use LOG(ERROR,... rather than LIBXL__LOG(CTX, LIBXL__LOG_ERROR
* Use an abort() when implementation code is missing
* move READ_BACKEND to the top of the file (since it's generally useful)
* Remove stray tab

Changes since v3:
* docs: coalesce the docs patch into the libxl patch
* docs: in channels.txt give high-level usage information, pitfalls and channgle registry
* docs: move the xenstore paths into the existing console.txt, which is referenced from xenstore-paths.markdown
* idl: rename 'kind' to 'connection'
* xl: add parser functions for comma-separated lists of pairs

Changes since v2:
* trim down the 'kinds' of channels to PTY and SOCKET -- these seem the most useful and we can add more later
* add a channelinfo (queryable by 'channel-list') to check the state of each channel, and for a kind=PTY discover the slave device path
* IDL: switched to KeyedUnion for both the channel and channelinfo since each 'kind' will have different parameters (e.g. only SOCKET has PATH)
* write all the backend configuration parameters to xenstore -- where we were using qemu -chardev some crucial information was only on the command-line.
* add LIBXL_HAVE_DEVICE_CHANNEL
* docs: replace 'should' with 'will' e.g. the backend will be connected to a Unix domain socket
* squash all the libxl patches together into a coherent whole
* explain that there is no registry of channel names and so people should use unique information to create them (e.g. include domain name and interface version)

Changes since v1:
* extend xl.cfg(5)
* add docs/misc/channel.txt
* libxl_types.idl: omit unnecessary init_val = 0
* xl_cmdimpl.c: fixed over-long lines
* xl_cmdimpl.c: use xrealloc (via ARRAY_EXTEND_INIT)
* xl_cmdimpl.c: exit() on parse failure rather than ignore configuration
* libxl_create.c: use libxl__device_console_init instead of memset
* libxl_create.c: use libxl__strdup(NOGC rather than raw strdup
* libxl.c: add name=<name> to console frontend
* libxl.c: resolve the backend_domid from backend_domname
* libxl_dm.c: channels[i].devid rather than i
* libxl_dm.c: fix indentation
* libxl_dm.c: use GCSPRINTF convenience macro

#1 lore raw

PATCH v5 1/2 libxl: add support for 'channels'

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

The following 'connection' types are defined:

* PTY: the I/O surfaces as a pty in the backend domain
* SOCKET: a listening Unix domain socket accepts a connection in the backend domain and proxies

Channels may be listed but don't currently support hotplug/unplug.

---
 docs/misc/channel.txt                |   95 ++++++++++++
 docs/misc/console.txt                |   61 +++++---
 tools/libxl/libxl.c                  |  268 +++++++++++++++++++++++++++++++---
 tools/libxl/libxl.h                  |   17 +++
 tools/libxl/libxl_create.c           |   38 +++--
 tools/libxl/libxl_dm.c               |   46 +++++-
 tools/libxl/libxl_internal.h         |   10 +-
 tools/libxl/libxl_types.idl          |   35 +++++
 tools/libxl/libxl_types_internal.idl |    4 +
 9 files changed, 524 insertions(+), 50 deletions(-)
 create mode 100644 docs/misc/channel.txt
docs/misc/channel.txt new file +95 -0
new file mode 100644
index 0000000..1b8e405
@@ -0,0 +1,95 @@
+Xen PV Channels
+===============
+
+A channel is a low-bandwidth private byte stream similar to a serial
+link. Typical uses of channels are
+
+  1. to provide initial configuration information to a VM on boot
+     (example use: CloudStack's cloud-early-config service)
+  2. to signal/query an in-guest agent
+     (example use: oVirt's guest agent)
+
+Channels are similar to virtio-serial devices and emulated serial links.
+Channels are intended to be used in the implementation of libvirt <channel>s
+when running on Xen.
+
+Note: if an application requires a high-bandwidth link then it should use
+vchan instead.
+
+How to use channels: an example
+-------------------------------
+
+Consider a cloud deployment where VMs are cloned from pre-made templates,
+and customised on first boot by an in-guest agent which sets the IP address,
+hostname, ssh keys etc. To install the system the cloud administrator would
+first:
+
+  1. Install a guest as normal (no channel configuration necessary)
+  2. Install the in-guest agent specific to the cloud software. This will
+     prepare the guest to communicate over the channel, and also prepare
+     the guest to be cloned safely (sometimes known as "sysprepping")
+  3. Shutdown the guest
+  4. Register the guest as a template with the cloud orchestration software
+  5. Install the cloud orchestration agent in dom0
+
+At runtime, when a cloud tenant requests that a VM is created from the template,
+the sequence of events would be:
+
+  1. A VM is "cloned" from the template
+  2. A unique Unix domain socket path in dom0 is allocated
+     (e.g. /my/cloud/software/talk/to/domain/<vm uuid>)
+  3. Domain configuration is created for the VM, listing the channel
+     name expected by the in-guest agent. In xl syntax this would be:
+
+     channel = [ "connection=socket, name=org.my.cloud.software.agent.version1,
+                  path = /my/cloud/software/talk/to/domain/<vm uuid>" ]
+
+  4. The VM is started
+  5. In dom0 the cloud orchestration agent connects to the Unix domain
+     socket, writes a handshake message and waits for a reply
+  6. In the guest, a udev rule (part of the guest agent package) is activated
+     by the hotplug event, and it starts the in-guest agent.
+  7. The in-guest agent completes a handshake with the dom0 agent
+  8. The dom0 agent transmits the unique VM configuration: hostname, IP
+     address, ssh keys etc etc
+  9. The in-guest agent receives the configuration and applies it.
+
+Using channels avoids having to use a temporary disk device or network
+connection.
+
+Design recommendations and pitfalls
+-----------------------------------
+
+It's necessary to install channel-specific software (an "agent") into the guest
+before you can use a channel. By default a channel will appear as a device
+which could be mistaken for a serial port or regular console. It is known
+that some software will proactively seek out serial ports and issue AT commands
+at them; make sure such software is disabled!
+
+Since channels are identified by names, application authors must ensure their
+channel names are unique to avoid clashes. We recommend that channel names
+include parts unique to the application such as a domain names. To assist
+prevent clashes we recommend authors add their names to our global channel
+registry at the end of this document.
+
+Limitations
+-----------
+
+Hotplug and unplug of channels is not currently implemented.
+
+Channel name registry
+---------------------
+
+It is important that channel names are globally unique. To help ensure
+that no-one's name clashes with yours, please add yours to this list.
+
+Key:
+N: Name
+C: Contact
+D: Short description of use, possibly including a URL to your software
+   or API
+
+N: org.xenproject.guest.clipboard.0.1
+C: David Scott <[email protected]>
+D: Share clipboard data via an in-guest agent. See:
+   http://wiki.xenproject.org/wiki/Clipboard_sharing_protocol
docs/misc/console.txt +44 -17

89 lines not on this page · raw message

tools/libxl/libxl.c +250 -18

324 lines not on this page · raw message

tools/libxl/libxl.h +17 -0
index 460207b..0d19aa7 100644
@@ -568,6 +568,15 @@ typedef struct libxl__ctx libxl_ctx;
  */
 #define LIBXL_HAVE_BUILDINFO_KERNEL 1
 
+/*
+ * LIBXL_HAVE_DEVICE_CHANNEL
+ *
+ * If this is defined, then the libxl_device_channel struct exists
+ * and channels can be attached to a domain both at build-time and at
+ * run-time by hotplug.
+ */
+#define LIBXL_HAVE_DEVICE_CHANNEL 1
+
 /* Functions annotated with LIBXL_EXTERNAL_CALLERS_ONLY may not be
  * called from within libxl itself. Callers outside libxl, who
  * do not #include libxl_internal.h, are fine. */
@@ -1101,6 +1110,14 @@ libxl_device_nic *libxl_device_nic_list(libxl_ctx *ctx, uint32_t domid, int *num
 int libxl_device_nic_getinfo(libxl_ctx *ctx, uint32_t domid,
                               libxl_device_nic *nic, libxl_nicinfo *nicinfo);
 
+/* Virtual Channels */
+libxl_device_channel *libxl_device_channel_list(libxl_ctx *ctx,
+                                                uint32_t domid,
+                                                int *num);
+int libxl_device_channel_getinfo(libxl_ctx *ctx, uint32_t domid,
+                                 libxl_device_channel *channel,
+                                 libxl_channelinfo *channelinfo);
+
 /* Virtual TPMs */
 int libxl_device_vtpm_add(libxl_ctx *ctx, uint32_t domid, libxl_device_vtpm *vtpm,
                           const libxl_asyncop_how *ao_how)
tools/libxl/libxl_create.c +28 -10
index ee328e9..c3f1c25 100644
@@ -387,14 +387,16 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
     return 0;
 }
 
-static int init_console_info(libxl__device_console *console, int dev_num)
+static int init_console_info(libxl__gc *gc,
+                             libxl__device_console *console,
+                             int dev_num)
 {
-    memset(console, 0x00, sizeof(libxl__device_console));
+    libxl__device_console_init(console);
     console->devid = dev_num;
     console->consback = LIBXL__CONSOLE_BACKEND_XENCONSOLED;
-    console->output = strdup("pty");
-    if (!console->output)
-        return ERROR_NOMEM;
+    console->output = libxl__strdup(NOGC, "pty");
+    /* console->{name,connection,path} are NULL on normal consoles.
+       Only 'channels' when mapped to consoles have a string name. */
     return 0;
 }
 
@@ -1194,17 +1196,31 @@ static void domcreate_launch_dm(libxl__egc *egc, libxl__multidev *multidev,
         }
     }
 
+    /* For both HVM and PV the 0th console is a regular console. We
+       map channels to IOEMU consoles starting at 1 */
+    for (i = 0; i < d_config->num_channels; i++) {
+        libxl__device_console console;
+        libxl__device device;
+        ret = libxl__init_console_from_channel(gc, &console, i + 1,
+                                               &d_config->channels[i]);
+        if ( ret )
+            goto error_out;
+        libxl__device_console_add(gc, domid, &console, NULL, &device);
+        libxl__device_console_dispose(&console);
+    }
+
     switch (d_config->c_info.type) {
     case LIBXL_DOMAIN_TYPE_HVM:
     {
         libxl__device_console console;
+        libxl__device device;
         libxl_device_vkb vkb;
 
-        ret = init_console_info(&console, 0);
+        ret = init_console_info(gc, &console, 0);
         if ( ret )
             goto error_out;
         console.backend_domid = state->console_domid;
-        libxl__device_console_add(gc, domid, &console, state);
+        libxl__device_console_add(gc, domid, &console, state, &device);
         libxl__device_console_dispose(&console);
 
         libxl_device_vkb_init(&vkb);
@@ -1222,22 +1238,24 @@ static void domcreate_launch_dm(libxl__egc *egc, libxl__multidev *multidev,
     {
         int need_qemu = 0;
         libxl__device_console console;
+        libxl__device device;
 
         for (i = 0; i < d_config->num_vfbs; i++) {
             libxl__device_vfb_add(gc, domid, &d_config->vfbs[i]);
             libxl__device_vkb_add(gc, domid, &d_config->vkbs[i]);
         }
 
-        ret = init_console_info(&console, 0);
+        ret = init_console_info(gc, &console, 0);
         if ( ret )
             goto error_out;
 
         need_qemu = libxl__need_xenpv_qemu(gc, 1, &console,
                 d_config->num_vfbs, d_config->vfbs,
-                d_config->num_disks, &d_config->disks[0]);
+                d_config->num_disks, &d_config->disks[0],
+                d_config->num_channels, &d_config->channels[0]);
 
         console.backend_domid = state->console_domid;
-        libxl__device_console_add(gc, domid, &console, state);
+        libxl__device_console_add(gc, domid, &console, state, &device);
         libxl__device_console_dispose(&console);
 
         if (need_qemu) {
tools/libxl/libxl_dm.c +43 -3

83 lines not on this page · raw message

tools/libxl/libxl_internal.h +8 -2
index 04c9378..656bf7a 100644
@@ -1017,7 +1017,8 @@ _hidden int libxl__device_disk_dev_number(const char *virtpath,
 
 _hidden int libxl__device_console_add(libxl__gc *gc, uint32_t domid,
                                       libxl__device_console *console,
-                                      libxl__domain_build_state *state);
+                                      libxl__domain_build_state *state,
+                                      libxl__device *device);
 
 _hidden int libxl__device_generic_add(libxl__gc *gc, xs_transaction_t t,
         libxl__device *device, char **bents, char **fents, char **ro_fents);
@@ -1030,6 +1031,10 @@ _hidden int libxl__wait_for_backend(libxl__gc *gc, const char *be_path,
                                     const char *state);
 _hidden int libxl__nic_type(libxl__gc *gc, libxl__device *dev,
                             libxl_nic_type *nictype);
+_hidden int libxl__init_console_from_channel(libxl__gc *gc,
+                                             libxl__device_console *console,
+                                             int dev_num,
+                                             libxl_device_channel *channel);
 
 /*
  * For each aggregate type which can be used as an input we provide:
@@ -1449,7 +1454,8 @@ _hidden const char *libxl__domain_device_model(libxl__gc *gc,
 _hidden int libxl__need_xenpv_qemu(libxl__gc *gc,
         int nr_consoles, libxl__device_console *consoles,
         int nr_vfbs, libxl_device_vfb *vfbs,
-        int nr_disks, libxl_device_disk *disks);
+        int nr_disks, libxl_device_disk *disks,
+        int nr_channels, libxl_device_channel *channels);
 
 /*
  * This function will cause the whole libxl process to hang
tools/libxl/libxl_types.idl +35 -0
index 931c9e9..dab65ee 100644
@@ -66,6 +66,12 @@ libxl_domain_type = Enumeration("domain_type", [
     (2, "PV"),
     ], init_val = "LIBXL_DOMAIN_TYPE_INVALID")
 
+libxl_channel_connection = Enumeration("channel_connection", [
+    (0, "UNKNOWN"),
+    (1, "PTY"),
+    (2, "SOCKET"), # a listening Unix domain socket
+    ])
+
 libxl_device_model_version = Enumeration("device_model_version", [
     (0, "UNKNOWN"),
     (1, "QEMU_XEN_TRADITIONAL"), # Historical qemu-xen device model (qemu-dm)
@@ -266,6 +272,22 @@ libxl_cpupoolinfo = Struct("cpupoolinfo", [
     ("cpumap",      libxl_bitmap)
     ], dir=DIR_OUT)
 
+libxl_channelinfo = Struct("channelinfo", [
+    ("backend", string),
+    ("backend_id", uint32),
+    ("frontend", string),
+    ("frontend_id", uint32),
+    ("devid", libxl_devid),
+    ("state", integer),
+    ("evtch", integer),
+    ("rref", integer),
+    ("u", KeyedUnion(None, libxl_channel_connection, "connection",
+           [("unknown", None),
+            ("pty", Struct(None, [("path", string),])),
+            ("socket", None),
+           ])),
+    ], dir=DIR_OUT)
+
 libxl_vminfo = Struct("vminfo", [
     ("uuid", libxl_uuid),
     ("domid", libxl_domid),
@@ -488,6 +510,18 @@ libxl_device_vtpm = Struct("device_vtpm", [
     ("uuid",             libxl_uuid),
 ])
 
+libxl_device_channel = Struct("device_channel", [
+    ("backend_domid", libxl_domid),
+    ("backend_domname", string),
+    ("devid", libxl_devid),
+    ("name", string),
+    ("u", KeyedUnion(None, libxl_channel_connection, "connection",
+           [("unknown", None),
+            ("pty", None),
+            ("socket", Struct(None, [("path", string)])),
+           ])),
+])
+
 libxl_domain_config = Struct("domain_config", [
     ("c_info", libxl_domain_create_info),
     ("b_info", libxl_domain_build_info),
@@ -498,6 +532,7 @@ libxl_domain_config = Struct("domain_config", [
     ("vfbs", Array(libxl_device_vfb, "num_vfbs")),
     ("vkbs", Array(libxl_device_vkb, "num_vkbs")),
     ("vtpms", Array(libxl_device_vtpm, "num_vtpms")),
+    ("channels", Array(libxl_device_channel, "num_channels")),
 
     ("on_poweroff", libxl_action_on_shutdown),
     ("on_reboot", libxl_action_on_shutdown),
tools/libxl/libxl_types_internal.idl +4 -0
index 800361b..a62167c 100644
@@ -34,6 +34,10 @@ libxl__device_console = Struct("device_console", [
     ("devid", integer),
     ("consback", libxl__console_backend),
     ("output", string),
+    # These are specific to the 'channels', built on top of consoles:
+    ("name", string),
+    ("connection", string),
+    ("path", string),
     ])
 
 libxl__device_action = Enumeration("device_action", [
Signature · 1 line
1.7.10.4
#2 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I have a harder one:

tools/libxl/libxl.c out: · line 3362

Quoted 4 lines
tools/libxl/libxl.c +4 -0
+int libxl__init_console_from_channel(libxl__gc *gc,
+                                     libxl__device_console *console,
+                                     int dev_num,
+                                     libxl_device_channel *channel)

AFAICT this function is used to recreate the domain's channel configuration info for the benefit of libxl's caller (making enquiries) and setting up the qemu arguments.

But nowadays, following Wei's save/restore/JSON patches, I think we would expect libxl to retrieve this information from the JSON configuration. That is, the console information should be in the stored JSON config and can be retrieved from there.

(There are also unfortunate security implications to reading the backend directory like that - if we have a driver domain, the qemu might get untrustworthy paths.)

Wei, am I right ?

Thanks,
Ian.

#3 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

Quoted 32 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I have a harder one:

tools/libxl/libxl.c out: · line 3362

Quoted 4 lines
tools/libxl/libxl.c +4 -0
+int libxl__init_console_from_channel(libxl__gc *gc,
+                                     libxl__device_console *console,
+                                     int dev_num,
+                                     libxl_device_channel *channel)
Quoted 9 lines

AFAICT this function is used to recreate the domain's channel configuration info for the benefit of libxl's caller (making enquiries) and setting up the qemu arguments.

But nowadays, following Wei's save/restore/JSON patches, I think we would expect libxl to retrieve this information from the JSON configuration. That is, the console information should be in the stored JSON config and can be retrieved from there.

The design Dave proposed has dedicated libxl_device_channel structure, and that's the thing being saved in JSON.

AIUI this channel device happens to be backed by a QEMU console device, which doesn't precludes it from being backed by dedicated backend or other devices. In this case I think generating a console device internally in libxl is actually the right thing to do.

Dave, am I right about the design?

Wei.

Quoted 9 lines

(There are also unfortunate security implications to reading the backend directory like that - if we have a driver domain, the qemu might get untrustworthy paths.)

Wei, am I right ?

Thanks,
Ian.

author
#4 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

Quoted 46 lines · Wei Liu
Quoted 32 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I have a harder one:

tools/libxl/libxl.c out: · line 3362

Quoted 4 lines
tools/libxl/libxl.c +4 -0
+int libxl__init_console_from_channel(libxl__gc *gc,
+                                     libxl__device_console *console,
+                                     int dev_num,
+                                     libxl_device_channel *channel)
Quoted 9 lines

AFAICT this function is used to recreate the domain's channel configuration info for the benefit of libxl's caller (making enquiries) and setting up the qemu arguments.

But nowadays, following Wei's save/restore/JSON patches, I think we would expect libxl to retrieve this information from the JSON configuration. That is, the console information should be in the stored JSON config and can be retrieved from there.

The design Dave proposed has dedicated libxl_device_channel structure, and that's the thing being saved in JSON.

Quoted 6 lines

AIUI this channel device happens to be backed by a QEMU console device, which doesn't precludes it from being backed by dedicated backend or other devices. In this case I think generating a console device internally in libxl is actually the right thing to do.

Dave, am I right about the design?

That’s right. I’ve added a libxl_device_channel to the domain config, which should be being saved in the JSON (is that automatic? I’ve not looked at the code for it yet.) The libxl__device_console remains an internal type. The existing domain create codepath synthesises an instance of libxl__device_console to represent the primary console. My patches extend this to create secondary consoles to implement the channels.

So if a domain is saved then the libxl_device_channels should end up in the JSON. On restore, these libxl_device_channels should get reloaded from the JSON and internally mapped onto libxl__device_consoles — does that sound right?

Thanks,
Dave

Quoted 12 lines

Wei.

Quoted 9 lines

(There are also unfortunate security implications to reading the backend directory like that - if we have a driver domain, the qemu might get untrustworthy paths.)

Wei, am I right ?

Thanks,
Ian.

#5 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

Quoted 54 lines · Dave Scott
Quoted 51 lines · Wei Liu
Quoted 32 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I have a harder one:

tools/libxl/libxl.c out: · line 3362

Quoted 4 lines
tools/libxl/libxl.c +4 -0
+int libxl__init_console_from_channel(libxl__gc *gc,
+                                     libxl__device_console *console,
+                                     int dev_num,
+                                     libxl_device_channel *channel)
Quoted 9 lines

AFAICT this function is used to recreate the domain's channel configuration info for the benefit of libxl's caller (making enquiries) and setting up the qemu arguments.

But nowadays, following Wei's save/restore/JSON patches, I think we would expect libxl to retrieve this information from the JSON configuration. That is, the console information should be in the stored JSON config and can be retrieved from there.

The design Dave proposed has dedicated libxl_device_channel structure, and that's the thing being saved in JSON.

AIUI this channel device happens to be backed by a QEMU console device, which doesn't precludes it from being backed by dedicated backend or other devices. In this case I think generating a console device internally in libxl is actually the right thing to do.

Quoted 5 lines
Quoted 1 line

Dave, am I right about the design?

That’s right. I’ve added a libxl_device_channel to the domain config, which should be being saved in the JSON (is that automatic? I’ve not looked at the code for it yet.) The libxl__device_console remains an

Yes, it will be saved in JSON provided xl feeds it to libxl in the first place.

But if you plan to add in channel hotplug / remove functionality, you need to look at how other devices do that.

Quoted 10 lines

internal type. The existing domain create codepath synthesises an instance of libxl__device_console to represent the primary console. My patches extend this to create secondary consoles to implement the channels.

So if a domain is saved then the libxl_device_channels should end up in the JSON. On restore, these libxl_device_channels should get reloaded from the JSON and internally mapped onto libxl__device_consoles — does that sound right?

That sounds right to me.

Wei.

Quoted 14 lines

Thanks,
Dave

Quoted 12 lines

Wei.

Quoted 9 lines

(There are also unfortunate security implications to reading the backend directory like that - if we have a driver domain, the qemu might get untrustworthy paths.)

Wei, am I right ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

author
#6 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

In reply to Ian Jackson
Quoted 10 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

Quoted 8 lines

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

Good spot. I think I’m blind to some of these documentation problems.

Quoted 4 lines

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I’ve added some comments to the domain configuration definition in the IDL and to libxl.h. Interestingly the console device is still an internal type in libxl_types_internal.idl and not user-configurable via the libxl_domain_config, so that removes one source of confusion (hopefully)

Cheers,
Dave

author
#7 lore raw

Re: [PATCH v5 for-4.5 1/2] libxl: add support for 'channels'

In reply to Ian Jackson
Quoted 36 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 1/2] libxl: add support for 'channels'"):

Quoted 8 lines

A 'channel':
- is a low-bandwidth private communication channel that resembles a physical serial port.
- is implemented as a PV console with a well-known string name which is used to hook the channel to the appropriate software in the guest (i.e. some kind of guest agent).
- has a backend 'connection' which describes what should happen to the data.

This looks good in principle.

I have a couple of easy quibbles:

You have forgotten to document that you transport the channel name in xenstore in the subkey name'. This should be in your console.txt' patch I think, given that xenstore-paths.markdown refers to that for all the nodes in .../console.

And the interaction between consoles in the console part of the domain configuration and the channels listed in the channels part, should be documented. (Even if it's just that consoles always have no name and channels always have one.)

I have a harder one:

tools/libxl/libxl.c out: · line 3362

Quoted 4 lines
tools/libxl/libxl.c +4 -0
+int libxl__init_console_from_channel(libxl__gc *gc,
+                                     libxl__device_console *console,
+                                     int dev_num,
+                                     libxl_device_channel *channel)

AFAICT this function is used to recreate the domain's channel configuration info for the benefit of libxl's caller (making enquiries) and setting up the qemu arguments.

Quoted 8 lines

But nowadays, following Wei's save/restore/JSON patches, I think we would expect libxl to retrieve this information from the JSON configuration. That is, the console information should be in the stored JSON config and can be retrieved from there.

(There are also unfortunate security implications to reading the backend directory like that - if we have a driver domain, the qemu might get untrustworthy paths.)

Sorry I missed this bit, thanks Wei for highlighting it.

I agree that we must be careful not to start a driver domain and tell it ‘talk to the socket at path x’ when in fact ‘x’ is in a different domain. At the moment paths are in the driver domain and the person who writes the domain config must consider this when they choose their driver domain.

Thinking about the future, although we could arrange it such that the data gets routed to other places (e.g. run qemu/xenconsoled in the driver domain, have it proxy the data over a vchan, have a thing in dom0 proxy to the right socket) I suspect that we’ll want to add additional ‘connection’ behaviours which reference resources via global names rather than local ones. I quite like the idea of adding some kind of URI to name a network endpoint and have the driver domain proxy all the data to/from there.

I’m not totally sure that’s what you were getting at — sorry if I’ve missed the point!

Cheers,
Dave

Quoted 10 lines

Wei, am I right ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#8 lore raw

PATCH v5 2/2 xl: add support for 'channels'

In reply to David Scott

This adds support for channel declarations of the form:

channel = [ "name=...,connection=...[,path=...][,backend=...]" ]

where 'name' is a label to identify the channel to the frontend.

If 'connection = pty' then the channel is connected to a pty in the
backend domain
If 'connection = socket' then the channel is connected to a Unix domain
socket given by 'path = ...' in the backend domain.

This patch also adds the command:

xl channel-list <domain>

which allows the state of channels to be queried. In particular if 'connection=pty' this will show the path of the pty slave device.

---
 docs/man/xl.cfg.pod.5     |   51 +++++++++++
 docs/man/xl.pod.1         |   10 +++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |  207 ++++++++++++++++++++++++++++++++++++++++++---
 tools/libxl/xl_cmdtable.c |    5 ++
 5 files changed, 264 insertions(+), 10 deletions(-)
docs/man/xl.cfg.pod.5 +51 -0
index 517ae2f..cf7b790 100644
@@ -530,6 +530,57 @@ L<qemu(1)> manpage. The default is B<en-us>.
 
 =back
 
+=item B<channel=[ "CHANNEL_SPEC_STRING", "CHANNEL_SPEC_STRING", ...]>
+
+Specifies the virtual channels to be provided to the guest. A
+channel is a low-bandwidth, bidirectional byte stream, which resembles
+a serial link. Typical uses for channels include transmitting VM
+configuration after boot and signalling to in-guest agents. Please see
+F<docs/misc/channels.txt> for more details.
+
+Each B<CHANNEL_SPEC_STRING> is a comma-separated list of C<KEY=VALUE>
+settings, from the following list:
+
+=over 4
+
+=item C<backend=DOMAIN>
+
+Specify the backend domain name or id. This parameter is optional. If
+this parameter is omitted then the toolstack domain will be assumed.
+
+=item C<name=NAME>
+
+Specify the string name for this device. This parameter is mandatory.
+This should be a well-known name for the specific application (e.g.
+guest agent) and should be used by the frontend to connect the
+application to the right channel device. There is no formal registry
+of channel names, so application authors are encouraged to make their
+names unique by including domain name and version number in the string
+(e.g. org.mydomain.guestagent.1).
+
+=item C<connection=CONNECTION>
+
+Specify how the backend will be implemented. This following options are
+available:
+
+=over 4
+
+=item B<connection=SOCKET>:
+
+The backend will bind a Unix domain socket (at the path given by
+B<path=PATH>), call listen and accept connections. The backend will proxy
+data between the channel and the connected socket.
+
+=item B<connection=PTY>:
+
+The backend will create a pty and proxy data between the channel and the
+master device. The command B<xl channel-list> can be used to discover the
+assigned slave device.
+
+=back
+
+=back
+
 =item B<pci=[ "PCI_SPEC_STRING", "PCI_SPEC_STRING", ... ]>
 
 Specifies the host PCI devices to passthrough to this guest. Each B<PCI_SPEC_STRING>
docs/man/xl.pod.1 +10 -0
index 9d1c2a5..b8a3d0e 100644
@@ -1210,6 +1210,16 @@ List virtual network interfaces for a domain.
 
 =back
 
+=head2 CHANNEL DEVICES
+
+=over 4
+
+=item B<channel-list> I<domain-id>
+
+List virtual channel interfaces for a domain.
+
+=back
+
 =head2 VTPM DEVICES
 
 =over 4
tools/libxl/xl.h +1 -0
index 10a2e66..afc5f8b 100644
@@ -78,6 +78,7 @@ int main_top(int argc, char **argv);
 int main_networkattach(int argc, char **argv);
 int main_networklist(int argc, char **argv);
 int main_networkdetach(int argc, char **argv);
+int main_channellist(int argc, char **argv);
 int main_blockattach(int argc, char **argv);
 int main_blocklist(int argc, char **argv);
 int main_blockdetach(int argc, char **argv);
tools/libxl/xl_cmdimpl.c +197 -10

261 lines not on this page · raw message

tools/libxl/xl_cmdtable.c +5 -0
index 7b7fa92..da8435e 100644
@@ -335,6 +335,11 @@ struct cmd_spec cmd_table[] = {
       "Destroy a domain's virtual network device",
       "<Domain> <DevId|mac>",
     },
+    { "channel-list",
+      &main_channellist, 0, 0,
+      "List virtual channel devices for a domain",
+      "<Domain(s)>",
+    },
     { "block-attach",
       &main_blockattach, 1, 1,
       "Create a new virtual block device",
Signature · 1 line
1.7.10.4
#9 lore raw

Re: [PATCH v5 for-4.5 2/2] xl: add support for 'channels'

David Scott writes ("[PATCH v5 for-4.5 2/2] xl: add support for 'channels'"):

Quoted 1 line

This adds support for channel declarations of the form:

This looks pretty good to me. I have some small complaints:

docs/man/xl.cfg.pod.5 L<qemu(1)> manpage. The default is B<en-us>. · line 542

Quoted 2 lines
docs/man/xl.cfg.pod.5 +2 -0
+Each B<CHANNEL_SPEC_STRING> is a comma-separated list of C<KEY=VALUE>
+settings, from the following list:

You should specify that spaces are stripped, and where.

You should also probably mention that there is no way to include a comma, or AFAICT leading/trailing whitespace, in any of the values of these fields.

tools/libxl/xl_cmdimpl.c static void *xrealloc(void *ptr, size_t sz) { · line 304

Quoted 4 lines
tools/libxl/xl_cmdimpl.c +4 -0
+static char *xstrdup(const char *x)
+{
+    char *r;
+    r = strdup(x);

Breaking this kind of thing out into a separate patch would make review a bit easier. This applies the movement of replace_string and arguably to the introduction of the trim and split functions.

Quoted 1 line

+typedef int (*char_predicate_t)(const int c);

You need a comment here about the bizarre interface to ctype.h macros. See the comment by the CTYPE macro in libxl_internal.h.

Quoted 1 line

+static void trim(char_predicate_t predicate, const char *input, char **output)

...

Quoted 1 line

+ while ((*p != '\000') && (predicate(*p)))

And these calls to predicate() are wrong because they don't take account of the ctype API bug. (Please don't ask why they left this ridiculous landmine here...)

I haven't reviewed your string fiddling in detail. It would probaby be good for someone to do so because this kind of thing can contain security bugs (which are relevant if anyone constructs an xl domain config out of laundered pieces some of which are supplied by an attacker).

tools/libxl/xl_cmdimpl.c int main_networkdetach(int argc, char **argv) · line 6078

Quoted 2 lines
tools/libxl/xl_cmdimpl.c +2 -0
+int main_channellist(int argc, char **argv)
+{

What is the way to get the channel list in machine-readable JSON format ?

Thanks,
Ian.

author
#10 lore raw

Re: [PATCH v5 for-4.5 2/2] xl: add support for 'channels'

Quoted 5 lines · Ian Jackson

David Scott writes ("[PATCH v5 for-4.5 2/2] xl: add support for 'channels'"):

Quoted 1 line

This adds support for channel declarations of the form:

This looks pretty good to me. I have some small complaints:

Quoted 8 lines

docs/man/xl.cfg.pod.5 L<qemu(1)> manpage. The default is B<en-us>. · line 542

Quoted 2 lines
docs/man/xl.cfg.pod.5 +2 -0
+Each B<CHANNEL_SPEC_STRING> is a comma-separated list of C<KEY=VALUE>
+settings, from the following list:

You should specify that spaces are stripped, and where.

You should also probably mention that there is no way to include a comma, or AFAICT leading/trailing whitespace, in any of the values of these fields.

Good point.

Quoted 9 lines

tools/libxl/xl_cmdimpl.c static void *xrealloc(void *ptr, size_t sz) { · line 304

Quoted 5 lines
tools/libxl/xl_cmdimpl.c +4 -0
+static char *xstrdup(const char *x)
+{
+    char *r;
+    r = strdup(x);

Breaking this kind of thing out into a separate patch would make review a bit easier. This applies the movement of replace_string and arguably to the introduction of the trim and split functions.

OK. I’ll split these out for review and make it easy to squash them back together (if that’s desired).

Quoted 6 lines
Quoted 2 lines

+typedef int (*char_predicate_t)(const int c);

You need a comment here about the bizarre interface to ctype.h macros. See the comment by the CTYPE macro in libxl_internal.h.

Quoted 7 lines
Quoted 1 line

+static void trim(char_predicate_t predicate, const char *input, char **output)

...

Quoted 1 line

+ while ((*p != '\000') && (predicate(*p)))

And these calls to predicate() are wrong because they don't take account of the ctype API bug. (Please don't ask why they left this ridiculous landmine here…)

Thanks, I’d never noticed that before!

Quoted 7 lines

I haven't reviewed your string fiddling in detail. It would probaby be good for someone to do so because this kind of thing can contain security bugs (which are relevant if anyone constructs an xl domain config out of laundered pieces some of which are supplied by an attacker).

Quoted 5 lines

tools/libxl/xl_cmdimpl.c int main_networkdetach(int argc, char **argv) · line 6078

Quoted 2 lines
tools/libxl/xl_cmdimpl.c +2 -0
+int main_channellist(int argc, char **argv)
+{

What is the way to get the channel list in machine-readable JSON format ?

‘xl list —long’ (via list_domains_details) can display the json:

$ sudo xl list --long
[
    {
        "domid": 30,
...
            "channels": [
                {
                    "devid": 0,
                    "name": "foo",
                    "connection.socket": {
                        "path": "/var/lib/libvirt/qemu/s-4-VM.agent"
                    }
                }
            ],

...

There’s no option to ‘xl channel-list’ to show them in JSON. I notice the xl hotplug commands have a ‘dryrun’ option which dumps the JSON of the device they would have added, but I’ve not implemented channel hotplug (yet)

Cheers,
Dave