xl - Remus network buffering support

The functions contained in xl_netbuf.c are built on top of libnl3 API. They setup the necessary infrastructure required to for network output buffering in Remus. In xend, the Remus python code performed this setup by invoking shell commands. This code is built purely on top of APIs supplied by libnl3.0.

There are two public helper functions:
1) remus_install_netbuf_on_dev to install a network buffer on a vif. Its basically equivalent to:
a)find free ifb (say ifb0)
b)ip link set ifb0 up
c)tc qdisc add dev vif1.0 ingress
d)tc filter add dev vif1.0 parent ffff: proto ip pref 10 u32 match u32 0 0 \
action mirred egress redirect dev ifb0
e)tc qdisc add dev ifb0 root plug
f)get handle to plug qdisc and control it programmatically after suspending
a VM/receiving checkpoint ack from remote

2) remus_uninstall_netbufs to remove all
network buffers and ingress qdiscs from the supplied interface list.

I have managed to code up 5 of the 6 steps mentioned above, purely in C, by building on top of libnl3. There is currently no support in libnl3 to implement step (d) (redirection) and it is pretty complicated to implement. So I am currently resorting to system() lib call to get the job done.

N.B. This implementation is just xl (cmdline utility) specific.

Other toolstacks may choose
to reuse this code or do this setup in their own way (scripts/shell commands, etc.).

Signed-off-by Shriram Rajagopalan <[email protected]>

Quoted 27 lines · Shriram Rajagopalan

The functions contained in xl_netbuf.c are built on top of libnl3 API. They setup the necessary infrastructure required to for network output buffering in Remus. In xend, the Remus python code performed this setup by invoking shell commands. This code is built purely on top of APIs supplied by libnl3.0.

There are two public helper functions:
1) remus_install_netbuf_on_dev to install a network buffer on a vif. Its basically equivalent to:
a)find free ifb (say ifb0)
b)ip link set ifb0 up
c)tc qdisc add dev vif1.0 ingress
d)tc filter add dev vif1.0 parent ffff: proto ip pref 10 u32 match u32 0 0 \
action mirred egress redirect dev ifb0
e)tc qdisc add dev ifb0 root plug
f)get handle to plug qdisc and control it programmatically after suspending
a VM/receiving checkpoint ack from remote

2) remus_uninstall_netbufs to remove all
network buffers and ingress qdiscs from the supplied interface list.

I have managed to code up 5 of the 6 steps mentioned above, purely in C, by building on top of libnl3. There is currently no support in libnl3 to implement step (d) (redirection) and it is pretty complicated to implement. So I am currently resorting to system() lib call to get the job done.

Quoted 1 line

N.B. This implementation is just xl (cmdline utility) specific.

You haven't included a Makefile change so I can't easily tell if this is really part of the xl binary, or a separate helper or indeed part of libxl.

Quoted 2 lines

Other toolstacks may choose
to reuse this code or do this setup in their own way (scripts/shell commands, etc.).

It's not clear to me why this code wouldn't be part of libxl to be called by xl and other toolstacks as needed. Why would a toolstack want to do this differently?

The point of libxl is precisely to relieve toolstacks of the burden of doing these sorts of complex domain setup activities themselves by putting them in a common library, I can't see why Remus setup would be an exception to this.

I think we need to answer this question before reviewing this code, since doing it libxl would change the requirements of the code (from the API exposed, to logging, to memory managment, to not using system directly).

Quoted 30 lines · Ian Campbell

Quoted 28 lines · Shriram Rajagopalan

The functions contained in xl_netbuf.c are built on top of libnl3 API. They setup the necessary infrastructure required to for network output buffering in Remus. In xend, the Remus python code performed this setup by invoking shell commands. This code is built purely on top of APIs supplied by libnl3.0.

There are two public helper functions:
1) remus_install_netbuf_on_dev to install a network buffer on a vif. Its basically equivalent to:
a)find free ifb (say ifb0)
b)ip link set ifb0 up
c)tc qdisc add dev vif1.0 ingress
d)tc filter add dev vif1.0 parent ffff: proto ip pref 10 u32 match u32 0 0 \
action mirred egress redirect dev ifb0
e)tc qdisc add dev ifb0 root plug
f)get handle to plug qdisc and control it programmatically after suspending
a VM/receiving checkpoint ack from remote

2) remus_uninstall_netbufs to remove all
network buffers and ingress qdiscs from the supplied interface list.

I have managed to code up 5 of the 6 steps mentioned above, purely in C, by building on top of libnl3. There is currently no support in libnl3 to implement step (d) (redirection) and it is pretty complicated to implement. So I am currently resorting to system() lib call to get the job done.

N.B. This implementation is just xl (cmdline utility) specific.

Quoted 4 lines

You haven't included a Makefile change so I can't easily tell if this is really part of the xl binary, or a separate helper or indeed part of libxl.

The Makefile changes were clumped together in patch 4, along with changes to configure.ac.

Quoted 12 lines

Quoted 2 lines

Other toolstacks may choose
to reuse this code or do this setup in their own way (scripts/shell commands, etc.).

It's not clear to me why this code wouldn't be part of libxl to be called by xl and other toolstacks as needed. Why would a toolstack want to do this differently?

The point of libxl is precisely to relieve toolstacks of the burden of doing these sorts of complex domain setup activities themselves by putting them in a common library, I can't see why Remus setup would be an exception to this.

Quoted 5 lines

I think we need to answer this question before reviewing this code, since doing it libxl would change the requirements of the code (from the API exposed, to logging, to memory managment, to not using system directly).

If every one had to do this in the same way, then I agree that this code should be moved into libxl.

The setup can be done using a a few lines of shell script or in a 100-200 line C code (like this patch). Given that there is a choice, I didnt see any point
in forcing all toolstacks to go the libxl way (resorting to its internal c code for setting
up the buffers). But if you think that we might as well make a sane choice and abstract
this complexity from other toolstacks, I would be happy to push this into libxl itself.

Add appropriate code to xl_cmdline.c to setup network buffers for each vif belonging to the guest. Also provide a command line switch to explicitly "enable" network buffering.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/libxl_types.idl

tools/libxl/libxl_types.idl +1 -0

@@ -521,6 +521,7 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf_iflist", libxl_string_list),
     ])
 
 libxl_event_type = Enumeration("event_type", [

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/xl_cmdimpl.c

tools/libxl/xl_cmdimpl.c +9 -0

@@ -7039,10 +7039,109 @@ done:
     return ret;
 }
 
+static char **get_guest_vifnames(uint32_t domid, int *num_vifs)
+{
+    char **viflist;
+    libxl_device_nic *nics;
+    libxl_nicinfo nicinfo;
+    int nb, i;
+
+    nics = libxl_device_nic_list(ctx, domid, &nb);
+    if (!nics) { *num_vifs = 0; return NULL;}

Quoted 37 lines · Shriram Rajagopalan

Add appropriate code to xl_cmdline.c to setup network buffers for each vif belonging to the guest. Also provide a command line switch to explicitly "enable" network buffering.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/libxl_types.idl

tools/libxl/libxl_types.idl Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/libxl_types.idl Wed Jul 24 22:55:00 2013 -0700 +1 -0

@@ -521,6 +521,7 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf_iflist", libxl_string_list),
     ])
 
 libxl_event_type = Enumeration("event_type", [

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/xl_cmdimpl.c

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/xl_cmdimpl.c Wed Jul 24 22:55:00 2013 -0700 +13 -0

@@ -7039,10 +7039,109 @@ done:
     return ret;
 }
 
+static char **get_guest_vifnames(uint32_t domid, int *num_vifs)
+{
+    char **viflist;
+    libxl_device_nic *nics;
+    libxl_nicinfo nicinfo;
+    int nb, i;
+
+    nics = libxl_device_nic_list(ctx, domid, &nb);
+    if (!nics) { *num_vifs = 0; return NULL;}
+
+    viflist = calloc((nb + 1), sizeof(char *));
+    if (!viflist) {
+        perror("failed to allocate memory to hold vif names!");

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 done: · line 7060

Quoted 6 lines

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/xl_cmdimpl.c Wed Jul 24 22:55:00 2013 -0700 +6 -0

+        exit(-1);
+    }
+
+    for (i = 0; i < nb; ++i) {
+        if (!libxl_device_nic_getinfo(ctx, domid, &nics[i], &nicinfo))  {
+            if (asprintf(&viflist[i], "vif%u.%d", domid, nicinfo.devid) < 0) {

This doesn't account for the ifname field of the vif.

Also, I'm not sure how this is supposed to work when driver domains are in use either,

if this were done via libxl then it would naturally get incorporated into Roger's work to make hotplug scripts etc work properly with stub domains.

Most of the other comments I had would become invalid/irrelevant when this was moved to libxl, since you'd naturally end up doing things differently anyway.

Quoted 38 lines · Ian Campbell

Quoted 37 lines · Shriram Rajagopalan

Add appropriate code to xl_cmdline.c to setup network buffers for each vif belonging to the guest. Also provide a command line switch to explicitly "enable" network buffering.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/libxl_types.idl

tools/libxl/libxl_types.idl Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/libxl_types.idl Wed Jul 24 22:55:00 2013 -0700 +1 -0

@@ -521,6 +521,7 @@ libxl_domain_remus_info = Struct("domain
     ("interval",     integer),
     ("blackhole",    bool),
     ("compression",  bool),
+    ("netbuf_iflist", libxl_string_list),
     ])
 
 libxl_event_type = Enumeration("event_type", [

diff -r 3ae38cbe535c -r 3cd67f6ff63a tools/libxl/xl_cmdimpl.c

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/xl_cmdimpl.c Wed Jul 24 22:55:00 2013 -0700 +13 -0

@@ -7039,10 +7039,109 @@ done:
     return ret;
 }
 
+static char **get_guest_vifnames(uint32_t domid, int *num_vifs)
+{
+    char **viflist;
+    libxl_device_nic *nics;
+    libxl_nicinfo nicinfo;
+    int nb, i;
+
+    nics = libxl_device_nic_list(ctx, domid, &nb);
+    if (!nics) { *num_vifs = 0; return NULL;}
+
+    viflist = calloc((nb + 1), sizeof(char *));
+    if (!viflist) {
+        perror("failed to allocate memory to hold vif names!");

Quoted 9 lines

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 done: · line 7060

Quoted 6 lines

tools/libxl/xl_cmdimpl.c Thu Jul 25 00:02:19 2013 -0700 renamed from tools/libxl/xl_cmdimpl.c Wed Jul 24 22:55:00 2013 -0700 +6 -0

+        exit(-1);
+    }
+
+    for (i = 0; i < nb; ++i) {
+        if (!libxl_device_nic_getinfo(ctx, domid, &nics[i], &nicinfo))  {
+            if (asprintf(&viflist[i], "vif%u.%d", domid, nicinfo.devid) < 0) {

This doesn't account for the ifname field of the vif.

Ah thanks. good catch.. Will add a check for that.

Quoted 2 lines

Also, I'm not sure how this is supposed to work when driver domains are in use either,

Remus wont work with driver domains, unless there are agents in each of the driver domains, that are co-ordinated by the memory checkpoint code in dom0.

Irrespective of the hotplug scripts, Remus needs to control the IFB network interface
attached to the Guest's vifs/tap devices. Given that all these interfaces will be inside
a driver domain, which does not have a fast (not xenstore) communication channel to dom0,
there is no way the memory checkpointing code can co-ordinate with the driver domain in order
to buffer/release its network packets after each checkpoint.

One alternative would be to have a network agent running inside each of these driver domains,
assuming that the driver domains would have network access (practical ? ). The memory checkpoint code would have to control the IFB devices via the network agents.

All this is in the long run.. The immediate goal is to get network/disk buffering to work with xl.

Quoted 10 lines

if this were done via libxl then it would naturally get incorporated into Roger's work to make hotplug scripts etc work properly with stub domains.

Most of the other comments I had would become invalid/irrelevant when this was moved to libxl, since you'd naturally end up doing things differently anyway.

Even if the code moved to libxl, most of the setup would remain the same, i.e. setup on demand
using C code or something. It does not make sense to setup an IFB device for every domain
right from the boot. You will eventually run out of ifb devices in the system.

If you are suggesting that we invoke a hotplug script when "xl remus" command is issued,
I dont mind doing so either. The code in libxl (to control the plug qdisc) is not going to go away
in either case.

Quoted 6 lines

Quoted 3 lines

Also, I'm not sure how this is supposed to work when driver domains are in use either,

Remus wont work with driver domains, unless there are agents in each of the driver domains, that are co-ordinated by the memory checkpoint code in dom0.

That doesn't seem to be to hard to arrange. We are already thinking of a "xenbackendd" daemon which runs hotplug scripts in both dom0 and domU in order to reduce our dependency on udev (in dom0 the daemon may actually be libxl directly).

Quoted 9 lines

Irrespective of the hotplug scripts, Remus needs to control the IFB network interface
attached to the Guest's vifs/tap devices. Given that all these interfaces will be inside
a driver domain, which does not have a fast (not xenstore) communication channel to dom0,
there is no way the memory checkpointing code can co-ordinate with the driver domain in order
to buffer/release its network packets after each checkpoint.

It doesn't seem implausible that the toolstack and that daemon might not also negotiate a memory-checkpoint evtchn or something like that for a given domain.

Quoted 6 lines

One alternative would be to have a network agent running inside each of these driver domains,
assuming that the driver domains would have network access (practical ? ). The memory checkpoint code would have to control the IFB devices via the network agents.

Quoted 2 lines

All this is in the long run.. The immediate goal is to get network/disk buffering to work with xl.

We still need to consider the future when designing the libxl API since that is stable and we avoid changing it once committed (there are mechanisms to evolve the interface, see libxl.h, but they are best avoided).

Quoted 5 lines

If you are suggesting that we invoke a hotplug script when "xl remus" command is issued,
I dont mind doing so either. The code in libxl (to control the plug qdisc) is not going to go away
in either case.

Would doing the setup in a shell script be easier/cleaner etc?

I'm still not sure I understand why finding an ifb and binding it to a vif cannot be done by the vif hotplug script and the ifb communicated back to the toolstack via xenstore.

Quoted 32 lines

Quoted 6 lines

Quoted 3 lines

Also, I'm not sure how this is supposed to work when driver domains are in use either,

Remus wont work with driver domains, unless there are agents in each of the driver domains, that are co-ordinated by the memory checkpoint code in dom0.

That doesn't seem to be to hard to arrange. We are already thinking of a "xenbackendd" daemon which runs hotplug scripts in both dom0 and domU in order to reduce our dependency on udev (in dom0 the daemon may actually be libxl directly).

Quoted 9 lines

Irrespective of the hotplug scripts, Remus needs to control the IFB network interface
attached to the Guest's vifs/tap devices. Given that all these interfaces will be inside
a driver domain, which does not have a fast (not xenstore) communication channel to dom0,
there is no way the memory checkpointing code can co-ordinate with the driver domain in order
to buffer/release its network packets after each checkpoint.

It doesn't seem implausible that the toolstack and that daemon might not also negotiate a memory-checkpoint evtchn or something like that for a given domain.

Quoted 6 lines

One alternative would be to have a network agent running inside each of these driver domains,
assuming that the driver domains would have network access (practical ? ). The memory checkpoint code would have to control the IFB devices via the network agents.

Quoted 8 lines

Quoted 2 lines

All this is in the long run.. The immediate goal is to get network/disk buffering to work with xl.

We still need to consider the future when designing the libxl API since that is stable and we avoid changing it once committed (there are mechanisms to evolve the interface, see libxl.h, but they are best avoided).

Assuming that we push all of the remus setup into libxl, to ease the pain off other toolstacks, then the external API to activate/deactivate remus will not change.
(Also assuming that libxl.h is the libxl API you are talking about).

The API is already there. libxl_domain_remus_start with the remus_info structure. So, even if we add driver domain support later, external toolstacks would
not be impacted. I will certainly add stub code to handle driver domains. However,
adding any piece of mechanism in there would be premature unless we know how xenbackendd is going to work and the nature of communication channels.

Quoted 8 lines

Quoted 5 lines

If you are suggesting that we invoke a hotplug script when "xl remus" command is issued,
I dont mind doing so either. The code in libxl (to control the plug qdisc) is not going to go away
in either case.

Would doing the setup in a shell script be easier/cleaner etc?

Neither. The current C-code (xl_netbuf) can do everything that a shell script does.
However, if we were to invoke an external "hotplug" style script, it would give us
more flexibility.
For e.g., people could tweak the script to add other traffic shaping stuff onto the VM's traffic. Or, with openvswitch/SDN, other fun stuff could be done.

Quoted 4 lines

I'm still not sure I understand why finding an ifb and binding it to a vif cannot be done by the vif hotplug script and the ifb communicated back to the toolstack via xenstore.

To make sure we are on the same page:
I am under the assumption that the vif hotplug script is invoked
during the domain boot.
I am also assuming that you are talking about existing vif hotplug
scripts (vif-setup, vif-bridge, etc).

So, if we lock onto an IFB and bind it to the vif (i.e. redirect all
egress traffic via the IFB),
then what we are basically doing is to acquire an IFB device for
every vif in the VM, for every VM,
that may be run under Remus sometime in future.
This basically means we would be setting up an IFB device for every
domain right from boot
and not when Remus is started for the domain.

In case you mean something like a "vif-remus" hotplug script that is invoked when one invokes
libxl_domain_remus_start API,

then, you are right. all the setup can
be done in this hotplug script
and the chosen ifbs can be communicated via xenstore back to libxl.

This patch constitutes the core network buffering logic. Libxl would receive a list of IFB devices that collectively act as network buffering devices, for a given guest. Also, libxl expects that every ifb device in the list supplied by a toolstack (netbuf_iflist) should have a plug qdisc installed.

This patch does the following:
a) establish a dedicated remus context containing libnl related state (netlink sockets, qdisc caches, etc.,)
b) Obtain handles to plug qdiscs installed on the supplied list of IFB devices
c) add a new network buffer (i.e., create a new one) when the domain is suspended (remus_domain_suspend_callback)
d) release the network buffer pertaining to the acknowledged checkpoint in remus_domain_checkpoint_dm_saved, which is invoked for both PV & HVM.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r 3cd67f6ff63a -r bef729fc4336 tools/libxl/libxl_dom.c

tools/libxl/libxl_dom.c +128 -8

@@ -20,6 +20,7 @@
 #include "libxl_internal.h"
 #include "libxl_arch.h"
 
+#include <netlink/route/qdisc/plug.h>
 #include <xc_dom.h>
 #include <xen/hvm/hvm_info_table.h>
 #include <xen/hvm/hvm_xs_strings.h>

@@ -1212,12 +1213,36 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */
 static int libxl__remus_domain_suspend_callback(void *data)
 {
-    /* REMUS TODO: Issue disk and network checkpoint reqs. */
-    return libxl__domain_suspend_common_callback(data);
+    libxl__save_helper_state *shs = data;
+    libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+    STATE_AO_GC(dss->ao);
+
+    int is_suspended = 0, i, ret;
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    is_suspended = libxl__domain_suspend_common_callback(data);
+    if (!remus_ctx->num_netbufs) return is_suspended;
+
+    if (is_suspended) {
+        for (i = 0; i < remus_ctx->num_netbufs; ++i) {
+            ret = rtnl_qdisc_plug_buffer(remus_ctx->netbuf_qdisc_list[i]);
+            if (!ret)
+                ret = rtnl_qdisc_add(remus_ctx->nlsock, remus_ctx->netbuf_qdisc_list[i],
+                                     NLM_F_REQUEST);
+            if (ret) {
+                LOG(ERROR, "Cannot create new buffer on %s:%s",
+                    dss->remus->netbuf_iflist[i], nl_geterror(ret));
+                return 0;
+            }
+        }
+    }
+
+    return is_suspended;
 }
 
+/* REMUS TODO: Deal with disk. */
 static int libxl__remus_domain_resume_callback(void *data)
 {
     libxl__save_helper_state *shs = data;

@@ -1228,7 +1253,6 @@ static int libxl__remus_domain_resume_ca
     if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
         return 0;
 
-    /* REMUS TODO: Deal with disk. Start a new network output buffer */
     return 1;
 }
 

@@ -1255,13 +1279,104 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */
-    usleep(dss->interval * 1000);
+    /* REMUS TODO: Wait for disk and explicit memory ack (through restore
+       callback from remote) before release network buffer. */
+    int i, ret;
+    libxl__remus_ctx *remus_ctx = dss->remus_ctx;
+    STATE_AO_GC(dss->ao);
+
+    assert(!rc);
+
+    if (remus_ctx->num_netbufs  > 0) {
+        for (i = 0; i < remus_ctx->num_netbufs; ++i) {
+            ret = rtnl_qdisc_plug_release_one(remus_ctx->netbuf_qdisc_list[i]);
+            if (!ret)
+                ret = rtnl_qdisc_add(remus_ctx->nlsock, remus_ctx->netbuf_qdisc_list[i],
+                                     NLM_F_REQUEST);
+            if (ret) {
+                LOG(ERROR, "Cannot release buffer from %s:%s",
+                    dss->remus->netbuf_iflist[i], nl_geterror(ret));
+                ret= 0;
+                break;
+            }
+        }
+    }
+
+    usleep(dss->remus_ctx->interval * 1000);
     libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
 }
 
+static libxl__remus_ctx *setup_remus_ctx(libxl__gc *gc,
+                                         const libxl_domain_remus_info *r_info)
+{
+    libxl__remus_ctx *remus_ctx = NULL;
+    int num_netbufs = 0, i;
+    struct nl_cache *qdisc_cache = NULL;
+    struct rtnl_link *ifb = NULL;
+    struct nl_sock *nlsock = NULL;
+    struct rtnl_qdisc *qdisc = NULL;
+    libxl_string_list l;
+    int ifindex;
+
+    remus_ctx = libxl__zalloc(gc, sizeof(libxl__remus_ctx));
+    remus_ctx->interval = r_info->interval;
+
+    l = r_info->netbuf_iflist;
+    num_netbufs = libxl_string_list_length(&l);
+    if (!num_netbufs) return remus_ctx;
+
+    nlsock = nl_socket_alloc();
+    if (!nlsock) {
+        LOG(ERROR, "setup_remus_ctx: cannot allocate nl socket");
+        return NULL;
+    }
+
+    nl_connect(nlsock, NETLINK_ROUTE);
+
+    if (rtnl_qdisc_alloc_cache(nlsock, &qdisc_cache) < 0) {
+        LOG(ERROR, "setup_remus_ctx: failed to allocate qdisc cache");
+        goto end;
+    }
+
+    remus_ctx->netbuf_qdisc_list = libxl__calloc(gc, num_netbufs + 1, 
+                                                 sizeof(struct rtnl_qdisc *));
+    remus_ctx->num_netbufs = num_netbufs;
+    remus_ctx->nlsock = nlsock;
+    remus_ctx->qdisc_cache = qdisc_cache;
+    ifb = rtnl_link_alloc();
+
+    for (i = 0; i < num_netbufs; ++i) {
+
+        if (rtnl_link_get_kernel(nlsock, 0, l[i], &ifb) < 0) {
+            LOG(ERROR, "setup_remus_ctx: cannot obtain handle for %s", l[i]);
+            goto end;
+        }
+
+        ifindex = rtnl_link_get_ifindex(ifb);
+        if (!ifindex) {
+            LOG(ERROR, "setup_remus_ctx: invalid interface %s", l[i]);
+            goto end;
+        }
+
+        qdisc = rtnl_qdisc_get_by_parent(qdisc_cache, ifindex, TC_H_ROOT);
+        if (!qdisc || strcmp(rtnl_tc_get_kind(TC_CAST(qdisc)), "plug")) {
+            LOG(ERROR, "setup_remus_ctx: plug qdisc is not installed on %s", l[i]);
+            goto end;
+        }
+
+        remus_ctx->netbuf_qdisc_list[i] = qdisc;
+    }
+
+    rtnl_link_put(ifb);
+    return remus_ctx;
+
+ end:
+    if (ifb) rtnl_link_put(ifb);
+    if (qdisc_cache) nl_cache_free(qdisc_cache);
+    if (nlsock) nl_close(nlsock);
+    return NULL;
+}
+
 /*----- main code for suspending, in order of execution -----*/
 
 void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)

@@ -1312,7 +1427,12 @@ void libxl__domain_suspend(libxl__egc *e
     dss->dm_savefile = libxl__device_model_savefile(gc, domid);
 
     if (r_info != NULL) {
-        dss->interval = r_info->interval;
+        /* This suspend is for Remus. We need to get a handle on
+         * the network output buffers and setup the remus_ctx;
+         */
+        dss->remus_ctx = setup_remus_ctx(gc, r_info);
+        if (!dss->remus_ctx)
+            goto out;
         if (r_info->compression)
             dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
     }

diff -r 3cd67f6ff63a -r bef729fc4336 tools/libxl/libxl_internal.h

tools/libxl/libxl_internal.h +20 -1

@@ -44,6 +44,13 @@
 #include <sys/wait.h>
 #include <sys/socket.h>
 
+#include <netlink/cache.h>
+#include <netlink/socket.h>
+#include <netlink/attr.h>
+#include <netlink/route/link.h>
+#include <netlink/route/route.h>
+#include <netlink/route/qdisc.h>
+
 #include <xenstore.h>
 #include <xenctrl.h>
 #include <xenguest.h>

@@ -2242,6 +2249,18 @@ typedef struct libxl__logdirty_switch {
     libxl__ev_time timeout;
 } libxl__logdirty_switch;
 
+typedef struct libxl__remus_ctx {
+    /* checkpoint interval */
+    int interval;
+    /* array of plug qdisc pointers, that hold
+     * network output from the guest's vifs.
+     */
+    int num_netbufs;
+    struct rtnl_qdisc **netbuf_qdisc_list;
+    struct nl_sock *nlsock;
+    struct nl_cache *qdisc_cache;
+} libxl__remus_ctx;
+
 struct libxl__domain_suspend_state {
     /* set by caller of libxl__domain_suspend */
     libxl__ao *ao;

@@ -2260,7 +2279,7 @@ struct libxl__domain_suspend_state {
     int xcflags;
     int guest_responded;
     const char *dm_savefile;
-    int interval; /* checkpoint interval (for Remus) */
+    libxl__remus_ctx *remus_ctx;
     libxl__save_helper_state shs;
     libxl__logdirty_switch logdirty;
     /* private for libxl__domain_save_device_model */

This really needs Ian's review WRT the lifetime of the remus context object vs. the ao machinery. IOW I'm not sure if this is the appropriate gc to be using or if one associated with the ao context isn't required.

Quoted 1 line

void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)

Quoted 1 line

+ dss->remus_ctx = setup_remus_ctx(gc, r_info);

This patch constitutes the core network buffering logic. Libxl would receive a list of IFB devices that collectively act as network buffering devices, for a given guest. Also, libxl expects that every ifb device in the list supplied by a toolstack (netbuf_iflist) should have a plug qdisc installed.

tools/libxl/libxl_dom.c +1 -0

 #include <xc_dom.h>
 #include <xen/hvm/hvm_info_table.h>
 #include <xen/hvm/hvm_xs_strings.h>

@@ -1212,12 +1213,36 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */
 static int libxl__remus_domain_suspend_callback(void *data)

tools/libxl/libxl_dom.c +0 -3

@@ -1255,13 +1279,104 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss, int rc)
 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer */
-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */

+ assert(!rc);

+ usleep(dss->remus_ctx->interval * 1000);

tools/libxl/libxl_dom.c +3 -0

+static libxl__remus_ctx *setup_remus_ctx(libxl__gc *gc,
+                                         const libxl_domain_remus_info *r_info)
+{

+ libxl_string_list l;

+ l = r_info->netbuf_iflist;

+ nl_connect(nlsock, NETLINK_ROUTE);

tools/libxl/libxl_dom.c +4 -0

+            LOG(ERROR, "setup_remus_ctx: plug qdisc is not installed on %s", l[i]);
+            goto end;
+        }
+

Quoted 2 lines

+#include <netlink/route/qdisc/plug.h>

We need a configure option to disable this, in case the host doesn't have it.

#include <xc_dom.h>

Quoted 8 lines

#include <xen/hvm/hvm_info_table.h>
#include <xen/hvm/hvm_xs_strings.h>

tools/libxl/libxl_dom.c +1 -0

@@ -1212,12 +1213,36 @@ int libxl__toolstack_save(uint32_t domid
 
 /*----- remus callbacks -----*/
 
+/* REMUS TODO: Issue disk checkpoint reqs. */
 static int libxl__remus_domain_suspend_callback(void *data)

...

tools/libxl/libxl_dom.c int libxl__toolstack_save(uint32_t domid

Quoted 5 lines

tools/libxl/libxl_dom.c +5 -0

+    if (!remus_ctx->num_netbufs) return is_suspended;
+
+    if (is_suspended) {
+        for (i = 0; i < remus_ctx->num_netbufs; ++i) {
+            ret =

rtnl_qdisc_plug_buffer(remus_ctx->netbuf_qdisc_list[i]);

Review

Quoted 2 lines

(quoted diff) +2 -0

+            if (!ret)
+                ret = rtnl_qdisc_add(remus_ctx->nlsock,

remus_ctx->netbuf_qdisc_list[i],

Needs the line length reducing to 70-75 characters, maximum. (Multiple occcurrences of this problem in the patch.)

Review static void libxl__remus_domain_checkpoi · line 1280

Quoted 3 lines

(quoted diff) +0 -0

@@ -1255,13 +1279,104 @@ static void libxl__remus_domain_checkpoi
 static void remus_checkpoint_dm_saved(libxl__egc *egc,
                                       libxl__domain_suspend_state *dss,

int rc)

Review static void libxl__remus_domain_checkpoi · line 1258

Quoted 2 lines

(quoted diff) +0 -1

 {
-    /* REMUS TODO: Wait for disk and memory ack, release network buffer

*/

Quoted 2 lines

tools/libxl/libxl_dom.c +0 -2

-    /* REMUS TODO: make this asynchronous */
-    assert(!rc); /* REMUS TODO handle this error properly */

...

Quoted 1 line

+ assert(!rc);

Does this TODO not need to remain ?

Review

Quoted 2 lines

(quoted diff) +2 -0

+            if (!ret)
+                ret = rtnl_qdisc_add(remus_ctx->nlsock,

remus_ctx->netbuf_qdisc_list[i],

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1299

Quoted 5 lines

tools/libxl/libxl_dom.c +5 -0

+                                     NLM_F_REQUEST);
+            if (ret) {
+                LOG(ERROR, "Cannot release buffer from %s:%s",
+                    dss->remus->netbuf_iflist[i], nl_geterror(ret));
+                ret= 0;

Missing space after "=".

...

Quoted 1 line

+ usleep(dss->remus_ctx->interval * 1000);

This is still pretty bad, surely ?

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1376

Quoted 7 lines

tools/libxl/libxl_dom.c +7 -0

+    rtnl_link_put(ifb);
+    return remus_ctx;
+
+ end:
+    if (ifb) rtnl_link_put(ifb);
+    if (qdisc_cache) nl_cache_free(qdisc_cache);
+    if (nlsock) nl_close(nlsock);

I think this can perhaps leak remus_ctx and qdisc.

Ian.

Quoted 2 lines

+ usleep(dss->remus_ctx->interval * 1000);

This is still pretty bad, surely ?

Was thinking of updating this in a separate patch.. But I guess since this line is changing, might as well do it in this patch.

tools/libxl/libxl_dom.c static void libxl__remus_domain_checkpoi · line 1376

Quoted 7 lines

tools/libxl/libxl_dom.c +7 -0

+    rtnl_link_put(ifb);
+    return remus_ctx;
+
+ end:
+    if (ifb) rtnl_link_put(ifb);
+    if (qdisc_cache) nl_cache_free(qdisc_cache);
+    if (nlsock) nl_close(nlsock);

I think this can perhaps leak remus_ctx and qdisc.

There is no issue of leaking qdisc. Because it simply obtains references from qdisc_cache, which is freed if things fail.

WRT remus_ctx, I had intentionally skipped the free call.
Its allocated in the GC context. [the same one used by domain_suspend, etc].
And if this call fails, then the parent call libxl_domain_remus_start will fail.

I assumed that when libxl_free_all was called at this point, remus_ctx would also be
freed. Am I missing something in the overall control flow, that would cause this leak ?

Add dependency on libnl3 version 3.2.17 or higher to autoconf. Add include flags and link to relevant libraries in tools/libxl/Makefile.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r bef729fc4336 -r c4c05e4e4e02 tools/configure.ac

tools/configure.ac +8 -0

@@ -171,4 +171,12 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+# Checks for libnl3 libraries and headers.
+PKG_CHECK_MODULES(LIBNL3, libnl-3.0 >= 3.2.17 libnl-route-3.0 >= 3.2.17, [have_libnl3=yes], [have_libnl3=no])
+if (test "${have_libnl3}" = "yes"); then
+        CFLAGS+="-I$LIBNL3_CFLAGS"
+else
+	AC_MSG_ERROR([Need libnl version 3.2.17 or higher])
+fi
+
 AC_OUTPUT()

diff -r bef729fc4336 -r c4c05e4e4e02 tools/libxl/Makefile

tools/libxl/Makefile +4 -3

 ifeq ($(CONFIG_Linux),y)
 LIBUUID_LIBS += -luuid

@@ -29,7 +29,7 @@ CFLAGS_LIBXL += $(CFLAGS_libblktapctl)
 CFLAGS_LIBXL += -Wshadow
 
 CFLAGS += $(PTHREAD_CFLAGS)
-LDFLAGS += $(PTHREAD_LDFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS) -L/usr/local/lib/libnl3/
 LIBXL_LIBS += $(PTHREAD_LIBS)
 
 LIBXLU_LIBS =

@@ -68,6 +68,7 @@ ifeq ($(BISON),)
 endif
 
 LIBXL_LIBS += -lyajl
+LIBXL_LIBS += -lnl-3 -lnl-route-3
 
 LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \

@@ -92,7 +93,7 @@ CLIENTS = xl testidl libxl-save-helper
 CFLAGS_XL += $(CFLAGS_libxenlight)
 CFLAGS_XL += -Wshadow
 
-XL_OBJS = xl.o xl_cmdimpl.o xl_cmdtable.o xl_sxp.o
+XL_OBJS = xl.o xl_cmdimpl.o xl_cmdtable.o xl_sxp.o xl_netbuf.o
 $(XL_OBJS) _libxl.api-for-check: \
             CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
 $(XL_OBJS): CFLAGS += $(CFLAGS_XL)

@@ -189,7 +190,7 @@ libxlutil.a: $(LIBXLU_OBJS)
 	$(AR) rcs libxlutil.a $^
 
 xl: $(XL_OBJS) libxlutil.so libxenlight.so
-	$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) -lyajl $(APPEND_LDFLAGS)
+	$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight) $(LDLIBS_libxenctrl) -lyajl -lnl-3 -lnl-route-3 $(APPEND_LDFLAGS)
 
 libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so
 	$(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

At 07/25/2013 03:09 PM, Shriram Rajagopalan Wrote:

Quoted 2 lines

Add dependency on libnl3 version 3.2.17 or higher to autoconf. Add include flags and link to relevant libraries in tools/libxl/Makefile.

Is it OK to use libnl, not libnl3?
Some systems(For example: RHEL6) don't have libnl3.

Quoted 24 lines

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r bef729fc4336 -r c4c05e4e4e02 tools/configure.ac

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +8 -0

@@ -171,4 +171,12 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 
+# Checks for libnl3 libraries and headers.
+PKG_CHECK_MODULES(LIBNL3, libnl-3.0 >= 3.2.17 libnl-route-3.0 >= 3.2.17, [have_libnl3=yes], [have_libnl3=no])
+if (test "${have_libnl3}" = "yes"); then
+        CFLAGS+="-I$LIBNL3_CFLAGS"
+else
+     AC_MSG_ERROR([Need libnl version 3.2.17 or higher])
+fi
+
 AC_OUTPUT()

diff -r bef729fc4336 -r c4c05e4e4e02 tools/libxl/Makefile

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 renamed from tools/libxl/Makefile Thu Jul 25 00:02:22 2013 -0700 +0 -0

@@ -13,7 +13,7 @@ XLUMINOR = 0
 

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 XLUMINOR = 0 · line 16

Quoted 4 lines

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 renamed from tools/libxl/Makefile Thu Jul 25 00:02:22 2013 -0700 +1 -1

 CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
      -Wno-declaration-after-statement -Wformat-nonliteral
-CFLAGS += -I. -fPIC
+CFLAGS += -I. -fPIC -I /usr/local/include/libnl3/

Hmm, why use /usr/local?

Add dependency on libnl3 version 3.2.17 or higher to autoconf. Add include flags and link to relevant libraries in tools/libxl/Makefile.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r bef729fc4336 -r c4c05e4e4e02 tools/configure.ac

Quoted 12 lines · Shriram Rajagopalan

Add dependency on libnl3 version 3.2.17 or higher to autoconf. Add include flags and link to relevant libraries in tools/libxl/Makefile.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r bef729fc4336 -r c4c05e4e4e02 tools/configure.ac

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +0 -0

@@ -171,4 +171,12 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 AC_SUBST(libiconv) · line 180

Quoted 7 lines

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +7 -0

+# Checks for libnl3 libraries and headers.
+PKG_CHECK_MODULES(LIBNL3, libnl-3.0 >= 3.2.17 libnl-route-3.0 >= 3.2.17, [have_libnl3=yes], [have_libnl3=no])
+if (test "${have_libnl3}" = "yes"); then
+        CFLAGS+="-I$LIBNL3_CFLAGS"
+else
+     AC_MSG_ERROR([Need libnl version 3.2.17 or higher])
+fi

Using pkg-config here...

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 XLUMINOR = 0 · line 16

Quoted 10 lines

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +0 -0

 AC_OUTPUT()

diff -r bef729fc4336 -r c4c05e4e4e02 tools/libxl/Makefile

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 renamed from tools/libxl/Makefile Thu Jul 25 00:02:22 2013 -0700 +1 -1

@@ -13,7 +13,7 @@ XLUMINOR = 0
 
 CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
      -Wno-declaration-after-statement -Wformat-nonliteral
-CFLAGS += -I. -fPIC
+CFLAGS += -I. -fPIC -I /usr/local/include/libnl3/

Means you should need to include the -I option here.

David

Quoted 13 lines · Shriram Rajagopalan

Quoted 12 lines

Add dependency on libnl3 version 3.2.17 or higher to autoconf. Add include flags and link to relevant libraries in tools/libxl/Makefile.

Signed-off-by Shriram Rajagopalan <[email protected]>

diff -r bef729fc4336 -r c4c05e4e4e02 tools/configure.ac

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +0 -0

@@ -171,4 +171,12 @@ AC_SUBST(libiconv)
 # Checks for header files.
 AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
 

Quoted 10 lines

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 AC_SUBST(libiconv) · line 180

Quoted 7 lines

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +7 -0

+# Checks for libnl3 libraries and headers.
+PKG_CHECK_MODULES(LIBNL3, libnl-3.0 >= 3.2.17 libnl-route-3.0 >= 3.2.17, [have_libnl3=yes], [have_libnl3=no])
+if (test "${have_libnl3}" = "yes"); then
+        CFLAGS+="-I$LIBNL3_CFLAGS"
+else
+     AC_MSG_ERROR([Need libnl version 3.2.17 or higher])
+fi

Using pkg-config here...

The pkg-config idea was based on the suggestion here
http://www.carisma.slowglass.com/~tgr/libnl/doc/core.html#_linking_to_this_library

If I use AC_CHECK_LIB and AC_CHECK_HEADERS, would it automatically add the include path to CFLAGS, such that I wouldnt have to manually specify -I flags in libxl/Makefile ?

Quoted 7 lines

Quoted 7 lines

tools/configure.ac Thu Jul 25 00:02:33 2013 -0700 renamed from tools/configure.ac Thu Jul 25 00:02:22 2013 -0700 +0 -0

 AC_OUTPUT()

diff -r bef729fc4336 -r c4c05e4e4e02 tools/libxl/Makefile

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 renamed from tools/libxl/Makefile Thu Jul 25 00:02:22 2013 -0700 +0 -0

@@ -13,7 +13,7 @@ XLUMINOR = 0
 

Quoted 9 lines

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 XLUMINOR = 0 · line 16

Quoted 4 lines

tools/libxl/Makefile Thu Jul 25 00:02:33 2013 -0700 renamed from tools/libxl/Makefile Thu Jul 25 00:02:22 2013 -0700 +1 -1

 CFLAGS += -Werror -Wno-format-zero-length -Wmissing-declarations \
      -Wno-declaration-after-statement -Wformat-nonliteral
-CFLAGS += -I. -fPIC
+CFLAGS += -I. -fPIC -I /usr/local/include/libnl3/

Means you should need to include the -I option here.

David

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Quoted 3 lines

If I use AC_CHECK_LIB and AC_CHECK_HEADERS, would it automatically add the include path to CFLAGS, such that I wouldnt have to manually specify -I flags in libxl/Makefile ?

No. AC_CHECK_HEADERS can check if the header file is under the default directory. If the header file is under the directory /usr/include, you can use it. But why your header file is under the directory /usr/local/include?

Quoted 8 lines · Wen Congyang

Quoted 3 lines

If I use AC_CHECK_LIB and AC_CHECK_HEADERS, would it automatically add the include path to CFLAGS, such that I wouldnt have to manually specify -I flags in libxl/Makefile ?

No. AC_CHECK_HEADERS can check if the header file is under the default directory. If the header file is under the directory /usr/include, you can use it. But why your header file is under the directory /usr/local/include?

I think its because I installed libnl3 in /usr/local, from the sources. If there is a proper way to specify custom install directories for certain libraries,
then I dont think we need that hard coded -I /usr/local/include/libnl3

This patch series adds support for network buffering in the Remus codebase in xl/libxl. In previous emails, I had proposed for a script invocation to setup network buffering. After digging through libnl API, I managed to find most of what I needed (except for one command, which right now is executed through system() call).

The patch series in its current state would allow xl to dynamically setup and teardown buffering devices, qdiscs, etc associated with the guest, instead of resorting to clunky one time configurations.

The series is organized as follows:
1/4 - Network buffering setup functions - abstractions built on top of libnl3 API
to implement functionality such as add/delete qdisc, interface up/down, search for free ifb devices, etc.

2/4 - xl cmdline utility uses these abstractions to setup network buffers and
provides libxl with a list of ifb devices where packets would be buffered

3/4 - Libxl interaction with network buffer module in the kernel via libnl3.

4/4 - adds libnl3 (>= v3.2.17) dependency to autoconf scripts and linker flags
in tools/libxl/Makefile.

Functionality tested on debian squeeze (kernel 3.4) + openvswitch + 64-bit PV domU (kernel 3.4).

I've just skimmed quickly through the patches, but it seems like they add a bunch of Linux specific code to general parts of libxl, which would prevent building or using libxl in any OS different than Linux. If you have to add Linux specific code there's libxl_linux.c, but please remember to also add the necessary stubs to libxl_netbsd.c in order to not break libxl for OSes different than Linux.