ivshmem: update documentation, add client/server tools

configure +0 -0

 fi
 if test "$softmmu" = yes ; then
   if test "$virtfs" != no ; then

contrib/ivshmem-client/ivshmem-client.h new file +34 -0

new file mode 100644
index 0000000..284c4a3
@@ -0,0 +1,212 @@
+/*
+ * Copyright 6WIND S.A., 2014
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * (at your option) any later version.  See the COPYING file in the
+ * top-level directory.
+ */
+
+#ifndef _IVSHMEM_CLIENT_H_
+#define _IVSHMEM_CLIENT_H_
+
+/**
+ * This file provides helper to implement an ivshmem client. It is used
+ * on the host to ask QEMU to send an interrupt to an ivshmem PCI device in a
+ * guest. QEMU also implements an ivshmem client similar to this one, they both
+ * connect to an ivshmem server.
+ *
+ * A standalone ivshmem client based on this file is provided for debug/test
+ * purposes.
+ */
+
+#include <limits.h>
+#include <sys/select.h>
+
+#include "qemu/queue.h"
+
+/**
+ * Maximum number of notification vectors supported by the client
+ */
+#define IVSHMEM_CLIENT_MAX_VECTORS 64
+
+/**
+ * Structure storing a peer
+ *

contrib/ivshmem-client/ivshmem-client.h new file +5 -0

+QTAILQ_HEAD(IvshmemClientPeerList, IvshmemClientPeer);
+
+typedef struct IvshmemClientPeerList IvshmemClientPeerList;
+typedef struct IvshmemClient IvshmemClient;
+

Add some notes on the parts needed to use ivshmem devices: more specifically, explain the purpose of an ivshmem server and the basic concept to use the ivshmem devices in guests.
Move some parts of the documentation and re-organise it.

Signed-off-by David Marchand <[email protected]>

Reviewed-by Claudio Fontana <[email protected]>

Reviewed-by Stefan Hajnoczi <[email protected]>

---
 docs/specs/ivshmem_device_spec.txt |  124 +++++++++++++++++++++++++++---------
 1 file changed, 93 insertions(+), 31 deletions(-)

docs/specs/ivshmem_device_spec.txt +11 -4

index 667a862..12f338e 100644
@@ -2,30 +2,103 @@
 Device Specification for Inter-VM shared memory device
 ------------------------------------------------------
 
-The Inter-VM shared memory device is designed to share a region of memory to
-userspace in multiple virtual guests.  The memory region does not belong to any
-guest, but is a POSIX memory object on the host.  Optionally, the device may
-support sending interrupts to other guests sharing the same memory region.
+The Inter-VM shared memory device is designed to share a memory region (created
+on the host via the POSIX shared memory API) between multiple QEMU processes
+running different guests. In order for all guests to be able to pick up the
+shared memory area, it is modeled by QEMU as a PCI device exposing said memory
+to the guest as a PCI BAR.
+The memory region does not belong to any guest, but is a POSIX memory object on
+the host. The host can access this shared memory if needed.
+
+The device also provides an optional communication mechanism between guests
+sharing the same memory object. More details about that in the section 'Guest to
+guest communication' section.
 
 
 The Inter-VM PCI device

docs/specs/ivshmem_device_spec.txt +12 -0

+
+- basically, if you only need the shared memory part, you will map BAR2.
+  This way, you have access to the shared memory in guest and can use it as you
+  see fit (memnic, for example, uses it in userland
+  http://dpdk.org/browse/memnic).
+
+- BAR0 and BAR1 are used to implement an optional communication mechanism
+  through interrupts in the guests. If you need an event mechanism between the
+  guests accessing the shared memory, you will most likely want to write a
+  kernel driver that will handle interrupts. See details in the section 'Guest
+  to guest communication' section.
+

docs/specs/ivshmem_device_spec.txt +46 -26

-The device supports three BARs.  BAR0 is a 1 Kbyte MMIO region to support
-registers.  BAR1 is used for MSI-X when it is enabled in the device.  BAR2 is
-used to map the shared memory object from the host.  The size of BAR2 is
-specified when the guest is started and must be a power of 2 in size.
+*ivshmem server*
 
-*Registers*
+This server code is available in qemu.git/contrib/ivshmem-server.
 
-The device currently supports 4 registers of 32-bits each.  Registers
-are used for synchronization between guests sharing the same memory object when
-interrupts are supported (this requires using the shared memory server).
+The server must be started on the host before any guest.
+It creates a shared memory object then waits for clients to connect on a unix
+socket.
 
-The server assigns each VM an ID number and sends this ID number to the QEMU
-process when the guest starts.
+For each client (QEMU process) that connects to the server:
+- the server assigns an ID for this client and sends this ID to him as the first
+  message,
+- the server sends a fd to the shared memory object to this client,
+- the server creates a new set of host eventfds associated to the new client and
+  sends this set to all already connected clients,
+- finally, the server sends all the eventfds sets for all clients to the new
+  client.
+
+The server signals all clients when one of them disconnects.
+
+The client IDs are limited to 16 bits because of the current implementation (see
+Doorbell register in 'PCI device registers' subsection). Hence only 65536
+clients are supported.
+
+All the file descriptors (fd to the shared memory, eventfds for each client)
+are passed to clients using SCM_RIGHTS over the server unix socket.
+
+Apart from the current ivshmem implementation in QEMU, an ivshmem client has
+been provided in qemu.git/contrib/ivshmem-client for debug.
+
+*QEMU as an ivshmem client*
+
+At initialisation, when creating the ivshmem device, QEMU gets its ID from the
+server then makes it available through BAR0 IVPosition register for the VM to
+use (see 'PCI device registers' subsection).
+QEMU then uses the fd to the shared memory to map it to BAR2.
+eventfds for all other clients received from the server are stored to implement
+BAR0 Doorbell register (see 'PCI device registers' subsection).
+Finally, eventfds assigned to this QEMU process are used to send interrupts in
+this VM.
+
+*PCI device registers*
+
+From the VM point of view, the ivshmem PCI device supports 4 registers of
+32-bits each.
 
 enum ivshmem_registers {
     IntrMask = 0,

@@ -49,8 +122,8 @@ bit to 0 and unmasked by setting the first bit to 1.
 IVPosition Register: The IVPosition register is read-only and reports the
 guest's ID number.  The guest IDs are non-negative integers.  When using the
 server, since the server is a separate process, the VM ID will only be set when
-the device is ready (shared memory is received from the server and accessible via
-the device).  If the device is not ready, the IVPosition will return -1.
+the device is ready (shared memory is received from the server and accessible
+via the device).  If the device is not ready, the IVPosition will return -1.
 Applications should ensure that they have a valid VM ID before accessing the
 shared memory.
 

@@ -59,8 +132,8 @@ Doorbell register.  The doorbell register is 32-bits, logically divided into
 two 16-bit fields.  The high 16-bits are the guest ID to interrupt and the low
 16-bits are the interrupt vector to trigger.  The semantics of the value
 written to the doorbell depends on whether the device is using MSI or a regular
-pin-based interrupt.  In short, MSI uses vectors while regular interrupts set the
-status register.
+pin-based interrupt.  In short, MSI uses vectors while regular interrupts set
+the status register.
 
 Regular Interrupts
 

@@ -71,7 +144,7 @@ interrupt in the destination guest.
 
 Message Signalled Interrupts
 
-A ivshmem device may support multiple MSI vectors.  If so, the lower 16-bits
+An ivshmem device may support multiple MSI vectors.  If so, the lower 16-bits
 written to the Doorbell register must be between 0 and the maximum number of
 vectors the guest supports.  The lower 16 bits written to the doorbell is the
 MSI vector that will be raised in the destination guest.  The number of MSI

@@ -83,14 +156,3 @@ interrupt itself should be communicated via the shared memory region.  Devices
 supporting multiple MSI vectors can use different vectors to indicate different
 events have occurred.  The semantics of interrupt vectors are left to the
 user's discretion.
-
-
-Usage in the Guest
-------------------
-
-The shared memory device is intended to be used with the provided UIO driver.
-Very little configuration is needed.  The guest should map BAR0 to access the
-registers (an array of 32-bit ints allows simple writing) and map BAR2 to
-access the shared memory region itself.  The size of the shared memory region
-is specified when the guest (or shared memory server) is started.  A guest may
-map the whole shared memory region or only part of it.

Signature · 1 line

1.7.10.4

Send a protocol version as the first message from server, clients must close communication if they don't support this protocol version.

Older QEMUs should be fine with this change in the protocol since they overrides

their own vm_id on reception of an id associated to no eventfd.

Signed-off-by David Marchand <[email protected]>

---
 contrib/ivshmem-client/ivshmem-client.c |   13 ++++++++++---
 contrib/ivshmem-client/ivshmem-client.h |    1 +
 contrib/ivshmem-server/ivshmem-server.c |    9 +++++++++
 contrib/ivshmem-server/ivshmem-server.h |    1 +
 docs/specs/ivshmem_device_spec.txt      |    9 ++++++---
 hw/misc/ivshmem.c                       |   27 +++++++++++++++++++++++++--
 include/hw/misc/ivshmem.h               |   17 +++++++++++++++++
 7 files changed, 69 insertions(+), 8 deletions(-)
 create mode 100644 include/hw/misc/ivshmem.h

contrib/ivshmem-client/ivshmem-client.c +10 -3

index 11c805c..4413e26 100644
@@ -201,10 +201,17 @@ ivshmem_client_connect(IvshmemClient *client)
         goto err_close;
     }
 
-    /* first, we expect our index + a fd == -1 */
+    /* first, we expect a protocol version */
+    if (ivshmem_client_read_one_msg(client, &tmp, &fd) < 0 ||
+        (tmp != IVSHMEM_PROTOCOL_VERSION) || fd != -1) {
+        IVSHMEM_CLIENT_DEBUG(client, "cannot read from server\n");
+        goto err_close;
+    }
+
+    /* then, we expect our index + a fd == -1 */
     if (ivshmem_client_read_one_msg(client, &client->local.id, &fd) < 0 ||
         client->local.id < 0 || fd != -1) {
-        IVSHMEM_CLIENT_DEBUG(client, "cannot read from server\n");
+        IVSHMEM_CLIENT_DEBUG(client, "cannot read from server (2)\n");
         goto err_close;
     }
     IVSHMEM_CLIENT_DEBUG(client, "our_id=%ld\n", client->local.id);

@@ -216,7 +223,7 @@ ivshmem_client_connect(IvshmemClient *client)
         if (fd >= 0) {
             close(fd);
         }
-        IVSHMEM_CLIENT_DEBUG(client, "cannot read from server (2)\n");
+        IVSHMEM_CLIENT_DEBUG(client, "cannot read from server (3)\n");
         goto err_close;
     }
     client->shm_fd = fd;

contrib/ivshmem-client/ivshmem-client.h +1 -0

index 284c4a3..9215f34 100644
@@ -23,6 +23,7 @@
 #include <sys/select.h>
 
 #include "qemu/queue.h"
+#include "hw/misc/ivshmem.h"
 
 /**
  * Maximum number of notification vectors supported by the client

contrib/ivshmem-server/ivshmem-server.c +9 -0

index 91dc208..ea290f6 100644
@@ -99,6 +99,15 @@ ivshmem_server_send_initial_info(IvshmemServer *server, IvshmemServerPeer *peer)
 {
     int ret;
 
+    /* send our protocol version first */
+    ret = ivshmem_server_send_one_msg(peer->sock_fd, IVSHMEM_PROTOCOL_VERSION,
+                                      -1);
+    if (ret < 0) {
+        IVSHMEM_SERVER_DEBUG(server, "cannot send version: %s\n",
+                             strerror(errno));
+        return -1;
+    }
+
     /* send the peer id to the client */
     ret = ivshmem_server_send_one_msg(peer->sock_fd, peer->id, -1);
     if (ret < 0) {

contrib/ivshmem-server/ivshmem-server.h +1 -0

index f85dcd2..38df7bb 100644
@@ -30,6 +30,7 @@
 #include <sys/select.h>
 
 #include "qemu/queue.h"
+#include "hw/misc/ivshmem.h"
 
 /**
  * Maximum number of notification vectors supported by the server

docs/specs/ivshmem_device_spec.txt +6 -3

index 12f338e..3435116 100644
@@ -64,6 +64,8 @@ It creates a shared memory object then waits for clients to connect on a unix
 socket.
 
 For each client (QEMU process) that connects to the server:
+- the server sends a protocol version, if client does not support it, the client
+  closes the communication,
 - the server assigns an ID for this client and sends this ID to him as the first
   message,
 - the server sends a fd to the shared memory object to this client,

@@ -86,9 +88,10 @@ been provided in qemu.git/contrib/ivshmem-client for debug.
 
 *QEMU as an ivshmem client*
 
-At initialisation, when creating the ivshmem device, QEMU gets its ID from the
-server then makes it available through BAR0 IVPosition register for the VM to
-use (see 'PCI device registers' subsection).
+At initialisation, when creating the ivshmem device, QEMU first receives a
+protocol version and closes communication with server if it does not match.
+Then, QEMU gets its ID from the server then makes it available through BAR0
+IVPosition register for the VM to use (see 'PCI device registers' subsection).
 QEMU then uses the fd to the shared memory to map it to BAR2.
 eventfds for all other clients received from the server are stored to implement
 BAR0 Doorbell register (see 'PCI device registers' subsection).

hw/misc/ivshmem.c +25 -2

index bd9d718..e08632b 100644
@@ -26,6 +26,8 @@
 #include "qemu/event_notifier.h"
 #include "sysemu/char.h"
 
+#include "hw/misc/ivshmem.h"
+
 #include <sys/mman.h>
 #include <sys/types.h>
 

@@ -528,6 +530,27 @@ static void ivshmem_read(void *opaque, const uint8_t * buf, int flags)
     }
 }
 
+static void ivshmem_check_version(void *opaque, const uint8_t * buf, int flags)
+{
+    IVShmemState *s = opaque;
+    int tmp;
+    long version;
+
+    memcpy(&version, buf, sizeof(long));
+    tmp = qemu_chr_fe_get_msgfd(s->server_chr);
+    if (tmp != -1 || version != IVSHMEM_PROTOCOL_VERSION) {
+        fprintf(stderr, "incompatible version, you are connecting to a ivshmem-"
+                "server using a different protocol please check your setup\n");
+        qemu_chr_delete(s->server_chr);
+        s->server_chr = NULL;
+        return;
+    }
+
+    IVSHMEM_DPRINTF("version check ok, switch to real chardev handler\n");
+    qemu_chr_add_handlers(s->server_chr, ivshmem_can_receive, ivshmem_read,
+                          ivshmem_event, s);
+}
+
 /* Select the MSI-X vectors used by device.
  * ivshmem maps events to vectors statically, so
  * we just enable all vectors on init and after reset. */

@@ -742,8 +765,8 @@ static int pci_ivshmem_init(PCIDevice *dev)
 
         s->eventfd_chr = g_malloc0(s->vectors * sizeof(CharDriverState *));
 
-        qemu_chr_add_handlers(s->server_chr, ivshmem_can_receive, ivshmem_read,
-                     ivshmem_event, s);
+        qemu_chr_add_handlers(s->server_chr, ivshmem_can_receive,
+                              ivshmem_check_version, ivshmem_event, s);
     } else {
         /* just map the file immediately, we're not using a server */
         int fd;

include/hw/misc/ivshmem.h new file +17 -0

new file mode 100644
index 0000000..22c9f1f
@@ -0,0 +1,17 @@
+/*
+ * Copyright 6WIND S.A., 2014
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * (at your option) any later version.  See the COPYING file in the
+ * top-level directory.
+ */
+
+#ifndef _IVSHMEM_H_
+#define _IVSHMEM_H_
+
+/**
+ * Protocol version
+ */
+#define IVSHMEM_PROTOCOL_VERSION 0x0
+
+#endif

Signature · 1 line

1.7.10.4

Quoted 2 lines · David Marchand

Send a protocol version as the first message from server, clients must close communication if they don't support this protocol version.

What's the motivation here?
This is at best a way to break all clients if an incompatible change in the server is made.
Would not it be better to send a bitmap, or a list of supported versions, so it's possible to write servers compatible with multiple clients?

I'm not sure a full-fledged feature negotiation system is needed. The ivshmem protocol is local to the host and all participants are under control of the administrator.

I suggested a protocol version to protect against misconfiguration. For example, building QEMU from source but talking to an outdated ivhsmem server that is still running from before.

Remember that ivshmem-server and QEMU are shipped together by the distro. So in 99% of the cases they will have the same version anyway. But we want to protect against rare misconfiguration that break things (user mixing and matching incompatible software).