Using UML in libguestfs

Today I wrote some patches (for libguestfs) which add an experimental UML backend:

https://www.redhat.com/archives/libguestfs/2013-August/msg00005.html

There are some observations from this.

* 'ubd<N>=cow,original' is not parsed correctly. This was reported a while back and I'm seeing the same thing with the latest kernel from git:

http://thread.gmane.org/gmane.linux.uml.devel/13556

If I have some time I'll have a go at fixing this since it seems like a simple regression.

* I'm trying to use a serial port to communicate between guest and host, using ssl3=fd:<N>. This causes the Linux process to crash soon (but not immediately) after the guest has opened /dev/ttyS3:

spurious interrupt in ubd_handler, err = -4

I've put the full call trace at the end of the email because it's quite long[1]. This was also reported before, although the suggested patch did not fix it:

http://marc.info/?l=linux-kernel&m=123242364211797&w=2

I wonder if anyone has used the fd:<N> option in anger? I will try using a Unix domain socket and see if it makes a difference.

* UML is pretty fast! It's certainly faster (by a factor of > 5) than spinning up a lightweight KVM VM.

Rich.

[1] Call trace

spurious interrupt in ubd_handler, err = -4

Modules linked in:
Pid: 1614169860, comm:  Not tainted 3.11.0-rc4-00103-g6c2580c
RIP: 0033:[<000000006001db74>]
RSP: 0000000080ca9f98  EFLAGS: 00010206
RAX: 0000000000000008 RBX: 0000000000000008 RCX: 0000003f2cae6970
RDX: 0000000000000000 RSI: 0000000080ca9fb0 RDI: 0000000000000005
RBP: 0000000080ca9fe8 R08: 0000000080ca9ed8 R09: 0000000080ca9e40
R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000001000
R13: 0000000080e867c4 R14: 0000000000000001 R15: 68c00000000080ea
Call Trace: 
80ca9538:  [<600151ac>]------------[ cut here ]------------
WARNING: CPU: 0 PID: 6 at block/blk.h:210 generic_make_request_checks+0x1aa/0x234()
BUG: recent printk recursion!
80ca9648:  [<60022207>] sig_handler_common+0x8c/0xa2
80ca96d0:  [<6001db74>] io_thread+0x63/0x243
80ca9988:  [<600222c2>] sig_handler+0x2e/0x39
80ca99a8:  [<60021f33>] hard_handler+0x83/0xab
80ca9aa8:  [<6001db74>] io_thread+0x63/0x243
80ca9d88:  [<60021eb0>] hard_handler+0x0/0xab
80ca9f70:  [<60020308>] os_seek_file+0x11/0x27
80ca9f80:  [<6002032a>] os_read_file+0xc/0x1f
80ca9f90:  [<6001db3b>] io_thread+0x2a/0x243

Kernel panic - not syncing: Segfault with no mm
CPU: 0 PID: 1614169860 Comm:  Not tainted 3.11.0-rc4-00103-g6c2580c #3

Modules linked in:
Pid: 1614169860, comm:  Not tainted 3.11.0-rc4-00103-g6c2580c
RIP: 0033:[<0000000060026715>]
RSP: 0000000080ca9328  EFLAGS: 00010246
RAX: 0000000080ca9398 RBX: 00000000602d2460 RCX: 0000000080ca8000
RDX: 0000000000000008 RSI: 636f6c6e6f6e5f88 RDI: 0000000080ca9398
RBP: 0000000080ca9350 R08: 636f6c6e6f6e5f70 R09: 636f6c6e6f6e5f70
R10: 0000003f2ca892da R11: 0000003f2ca892da R12: 000000006026b27c
R13: 000000006001db74 R14: 0000000000000004 R15: 0000000000000000
Call Trace: 
80ca88b8:  [<600151ac>] segv+0x66/0x241
80ca8988:  [<600153e9>] segv_handler+0x62/0x6c
80ca89c8:  [<60022207>] sig_handler_common+0x8c/0xa2
80ca89e0:  [<6001db74>] io_thread+0x63/0x243
80ca8a50:  [<60026715>] memcpy+0xb5/0x140
80ca8d08:  [<600222c2>] sig_handler+0x2e/0x39
80ca8d28:  [<60021f33>] hard_handler+0x83/0xab
80ca8d58:  [<6001db74>] io_thread+0x63/0x243
80ca8dd0:  [<6001db74>] io_thread+0x63/0x243
80ca8e28:  [<60026715>] memcpy+0xb5/0x140
80ca90e8:  [<60017226>] generic_write+0xf/0x38
80ca9108:  [<600172f9>] generic_console_write+0x3e/0x165
80ca9288:  [<6004c396>] vprintk_emit+0x2dd/0x2ee
80ca92d8:  [<6001db74>] io_thread+0x63/0x243
80ca92f8:  [<601edf38>] printk+0x97/0x99
80ca9308:  [<60015eff>] copy_from_user+0x36/0x8a
80ca9328:  [<60015eff>] copy_from_user+0x36/0x8a
80ca9358:  [<600699d4>] __probe_kernel_read+0x3c/0x6d
80ca9368:  [<6003f2ac>] probe_kthread_data+0x29/0x2f
80ca9388:  [<6003becc>] print_worker_info+0x66/0x124
80ca93f8:  [<6004d204>] dump_stack_print_info+0xa6/0xaf
80ca9428:  [<601efbeb>] dump_stack+0xe/0x19
80ca9438:  [<601ed8b7>] panic+0xf7/0x1e4
80ca9490:  [<60056982>] __module_text_address+0xd/0x4d
80ca94a8:  [<6005a007>] is_module_text_address+0x9/0x11
80ca94b8:  [<6003cd40>] __kernel_text_address+0x26/0x49
80ca94d8:  [<60014146>] show_trace+0x8e/0x95
80ca9508:  [<60026491>] show_regs+0x27/0x2a
80ca9520:  [<6001db74>] io_thread+0x63/0x243
80ca9538:  [<6001524f>] segv+0x109/0x241
80ca9608:  [<600153e9>] segv_handler+0x62/0x6c
80ca9628:  [<60022169>] unblock_signals+0x4a/0x5c
80ca9648:  [<60022207>] sig_handler_common+0x8c/0xa2
80ca96d0:  [<6001db74>] io_thread+0x63/0x243
80ca9988:  [<600222c2>] sig_handler+0x2e/0x39
80ca99a8:  [<60021f33>] hard_handler+0x83/0xab
80ca9aa8:  [<6001db74>] io_thread+0x63/0x243
80ca9d88:  [<60021eb0>] hard_handler+0x0/0xab
80ca9f70:  [<60020308>] os_seek_file+0x11/0x27
80ca9f80:  [<6002032a>] os_read_file+0xc/0x1f
80ca9f90:  [<6001db3b>] io_thread+0x2a/0x243

Kernel panic - not syncing: Segfault with no mm

Modules linked in:
Pid: 1614169860, comm:  Not tainted 3.11.0-rc4-00103-g6c2580c
RIP: 0000:[<000001a400000004>]
RSP: 0000000000000000  EFLAGS: 6002e720
RAX: 000000006002e770 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000602706ae
RBP: 0000000000000000 R08: 0000000000000000 R09: 000001a400000004
R10: 00000000602d04ac R11: 000000006027069b R12: 0000000000000000
R13: 000000006002e720 R14: 0000000000000000 R15: 000001a400000004
Call Trace: 
80ca8748:  [<6001564c>] panic_exit+0x2b/0x41
80ca8758:  [<6015a3ac>] vsnprintf+0xd0/0x431
80ca8768:  [<600427c7>] notifier_call_chain+0x39/0x5c
80ca8790:  [<60026715>] memcpy+0xb5/0x140
80ca87a8:  [<60042804>] atomic_notifier_call_chain+0xf/0x11
80ca87b8:  [<601ed8d2>] panic+0x112/0x1e4
80ca8810:  [<60056982>] __module_text_address+0xd/0x4d
80ca8828:  [<6005a007>] is_module_text_address+0x9/0x11
80ca8838:  [<6003cd40>] __kernel_text_address+0x26/0x49
80ca8858:  [<60014146>] show_trace+0x8e/0x95
80ca8888:  [<60026491>] show_regs+0x27/0x2a
80ca88a0:  [<60026715>] memcpy+0xb5/0x140
80ca88b8:  [<6001524f>] segv+0x109/0x241
80ca8988:  [<600153e9>] segv_handler+0x62/0x6c
80ca89c8:  [<60022207>] sig_handler_common+0x8c/0xa2
80ca89e0:  [<6001db74>] io_thread+0x63/0x243

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Patches are very welcome!

Hmmm. I *should* work.
Which fd are you using?
Can you also try pts?

Wow!
Does it only boot faster or is it also faster with reading data from a filesystem?

Maybe it's time to exhume the libLinux project (UML as a normal userspace library).
http://user-mode-linux.sourceforge.net/old/projects.html

Thanks,
//richard

Can you please decode that address using addr2line?

Ditto.

Thanks,
//richard

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

I've only got it booting so far, so I haven't been able to test file transfer speeds.

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

just for the record, rootstrap in debian (script to create a debian FS) uses that option
umlargs.extend(['con0=fd:0,fd:1',

                'con=pts',                                                                                                                
                                'root=/dev/root',                                                                                         
                                'rootflags=/',                                                                                            
                                'rootfstype=hostfs',                                                                                      
                'ubd1=' + imagefile,                                                                                                      
                'init=/usr/lib/rootstrap/builder',                                                                                        
                'devfs=nomount',                                                                                                          
                'rsworkdir=' + os.getcwd()])                                                                                              

Last time I checked it worked.

mattia
:wq!

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

So I realized this is caused by a coding mistake by me. I had set to O_CLOEXEC flag on the file descriptor, so it was being closed across the exec of vmlinux.

It still doesn't quite work, but the error above has gone away.

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Am 09.08.2013 23:15, schrieb Richard W.M. Jones:

So you are still facing the crash are reported in your first mail?

Thanks,
//richard

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

No, the crash has gone away. There's now a protocol error which I'm trying to get to the bottom of. Perhaps /dev/ttyS<N> isn't 8 bit clean..?

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

This is interesting. Typically people say you have to use KVM or XEN for good virtualization performance. But for that you need sufficient privileges. With UML, all you need is to be able to compile and run your own executables.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

I think this is true of Xen, but to be fair to KVM it doesn't require special privileges, with one exception. If you want KVM to use a tunX network interface then you have to start it as root (or use a setuid helper). I think this is probably true of UML too although I've not explored UML's networking yet. Basically it's a restriction in Linux itself.

With libguestfs we specifically don't want people to use root, so UML not requiring root was one reason why it's possible to write a UML-based libguestfs backend.

I now have a UML backend for libguestfs working, with a significant part of the testsuite passing too.

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

There you name it...

With UML, I can use slirp-fullbolt.

So if I want to run e.g. wireshark on the development machine at work (no root for me), I just launch a UML with a Debian root fs that has wireshark and run my app.

Gr{oetje,eeting}s,

Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds

I performed some benchmarks against KVM:

http://rwmj.wordpress.com/2013/08/14/performance-of-user-mode-linux-as-a-libguestfs-backend/#content

Notes for those that don't want to read all that, there are at least two bugs I found:

(1) fsync /dev/ubda (in the guest) is not honoured by the user-mode linux process. It should do an fsync on the corresponding backing file.

(2) The ssl<N>=... serial port is really slow, at least 10 times slower than virtio-serial which is the equivalent KVM feature. libguestfs is chatty over the serial port, and so this has a significant knock-on impact all over the place.

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

What exactly does libguestfs do?

So, libguestfs transfers all data over a serial line?! We could hack a faster transport into UML.
As UML is only a process we can very easily access the host.
Have looked at hostfs?

Thanks,
//richard

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

http://libguestfs.org/ ...

Well, it requires an 8-bit clean, reliable, bi-directional connection into the guest. We used to use TCP/IP but it's pretty slow compared to virtio-serial and has a bunch of other problems.

For the UML backend I used the serial port. There's no problem with it -- it works and it's reliable, it's just slow.

Not sure how hostfs would help?

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Meant regarding fsync()...

hostfs show how you can directly access the host...

Thanks,
//richard

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Ah right, see what you mean :-)

It calls sync(2). Then it opens each /dev/ubdX device and calls fsync on the file descriptor:

https://github.com/libguestfs/libguestfs/blob/master/daemon/sync.c#L54

The reason for this is a bit complicated, but has to do with write barriers. I believe this has been fixed since then.

https://github.com/libguestfs/libguestfs/commit/c0a3c9ce70b98171e737e49e6dccc4457963f2ec

In any case, we're calling sync & fsync in the guest, and that ain't causing the host cache to be flushed.

It's pretty easy to show this with libguestfs:

LIBGUESTFS_BACKEND=uml LIBGUESTFS_HV=~/d/linux-um/linux \
time ./run ./fish/guestfish -N fs:ext2:1G -m /dev/sda1 fallocate64 /data 800M time sync

The first command finishes in 8 seconds, with barely any disk activity. The sync afterwards takes 9 seconds with the disk light on the whole time, while it actually writes the whole 800 MB of data.

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Hmm, AFAIK a fsync() after sync() is pointless.

I think you don't see the desired effect because UML does not open the backing file with O_SYNC.
But you can tell UML to do so.
e.g. write ubdas=/foo instead of ubda=/foo.

See:
ubd<n><flags>=<filename>[(:|,)<filename2>]
This is used to associate a device with a file in the underlying filesystem. When specifying two filenames, the first one is the COW name and the second is the backing file name. As separator you can use either a ':' or a ',': the first one allows writing things like;
ubd0=~/Uml/root_cow:~/Uml/root_backing_file
while with a ',' the shell would not expand the 2nd '~'. When using only one filename, UML will detect whether to treat it like a COW file or a backing file. To override this detection, add the 'd' flag:
ubd0d=BackingFile
Usually, there is a filesystem in the file, but
that's not required. Swap devices containing swap files can be specified like this. Also, a file which doesn't contain a filesystem can have its contents read in the virtual
machine by running 'dd' on the device. <n> must be in the range 0 to 7. Appending an 'r' to the number will cause that device to be mounted read-only. For example ubd1r=./ext_fs. Appending an 's' will cause data to be written to disk on the host immediately. 'c' will cause the device to be treated as being shared between multiple UMLs and file locking will be turned off - this is appropriate for a cluster filesystem and inappropriate at almost all other times.

Please let me know if that helps. :-)

Thanks,
//richard

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

That's definitely what you'd expect, but Linux was buggy until last
year: it only synched mounted filesystems, not devices in general.

This was fixed:

commit a8c7176b6ded413d5044a00f1d05477b95a6d7ad
Author: Jan Kara <[email protected]>
Date: Tue Jul 3 16:45:32 2012 +0200

vfs: Make sys_sync writeout also block device inodes

but in libguestfs we can't rely on people using a new enough kernel so we do the safe thing.

I didn't know about these flags.

I modified libguestfs to use the 's' flag, and reran the test above:

UML (with O_SYNC): 598 seconds
UML (unsafe): 9 seconds
UML + explicit host sync: 20 seconds
KVM: 20 seconds

However using O_SYNC is not the same as properly handling write barriers. UML is buggy in this respect because a user in the guest who does 'sync' (and the sync returns to the prompt) wouldn't expect that if the power failed immediately afterwards they would lose all their data. If they are using a journaled filesystem they'd also risk unrecoverable data corruption during general usage.

KVM and LVM have both been here a few years ago and have fixed these issues (mind you, KVM only does the right thing if you use the right cache=... option).

Rich.

Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead. 
Download for free and get started troubleshooting in minutes. 
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel