libxen-3.0 (libxc rewrite)

Hi all,

I've been doing a lot of work on libxc. I've got it to the point that I'm ready to share. Below are the major changes. Feedback is greatly appreciated, especially with respect to things that would be required for it to be integrated into the xen-unstable tree.

o Rename libxc => libxen
o Use pkg-config to control versioning and parallel installs
o Use autoconf to detect dependencies, provide separate build directory, cross-compile
o Use doxygen to autogenerate HTML documentation
o Organize hypercalls by groups in separate headers (dom.h, evtchn.h, etc.).
o Provide consistent error semantics for all functions (-errno is returned on error).
o Use pyrex to autogenerate python bindings
o Provide high-level python interface (Xen() and Domain() classes instead of direct hypercalls)

Below is a URL for a tarball and another URL for the doxygen generate documentation.

http://www.cs.utexas.edu/users/aliguori/libxen-3.0.0.tar.gz

http://www.cs.utexas.edu/users/aliguori/libxen-3.0.0/files.html

Regards,
Anthony Liguori

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

Is there any good reason for this? Will this include renaming all the xc_* symbols as well? This will break a lot of code in a lot of places, so I think the motivation needs to be more than just feel-good value.

Don't know what that is or why I need it, but I suppose that means yet another dependency added. I am against adding dependencies. Used to be I could just untar xen and type make, now I need to install most of f*cking Gnome, that horrible Twisted-library and I don't know what, I think we are headed in the wrong direction with all this. This is a kernel-level project, not Freshmeat Greatest.

I like having a separate build-directory, I still think at non-broken build tool (i.e. not make) could have done the job and done it better. The whole .d or .deps approach (pollution of source or build-tree with a static version of information that could and should be determined at run-time) is a gross hack, even MS Visual Studio can do better.

Here is my (a little out of date) Jamfile for libxc btw:

-----------------
SubDir TOP tools libxc ;
SubDirHdrs $(TOP) tools libxutil ;

Library libxc :
xc_atropos.c
xc_bvtsched.c
xc_domain.c
xc_evtchn.c
xc_io.c
xc_linux_build.c
xc_linux_restore.c
xc_linux_save.c
xc_misc.c
xc_physdev.c
xc_plan9_build.c
xc_private.c
xc_rrobin.c
;
------------------

Will this be optional, or part of the standard build process? Will the comments still be human-readable? If find the code and comments in libxc fairly easy to read and understand inline, isn't doxygen overkill for this small amount of code? Will I be able to build xen and libxc without installing doxygen on my system?

Fine with me.

If it reduces the code size I guess it makes sense, hopefully I will still be able to compile the raw library without compiling the bindings, and without having pyrex installed? Please also consider that sometimes I and others need to build (not run, obviously) this stuff on boxes where we do not have root-access, and the more stuff that needs to be installed, the less likely this is to work. This is not a contrived example, when I visited Cambridge (yes, the home of Xen) last year, I was doing Xen-development from a regular user account, without having root-access.

I still like vm-tools though :-)

Best regards,
Jacob

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

The API is completely different. This interface is a thin wrapper around the hypercalls. All the code needs to be rewritten anyway. I'm a big believer that things should be self-documenting. xc seems like a bad prefix because it's pretty much meaningless (I guess it's supposed to be xen-control or something).

You may be aware that many libraries distribution custom -config scripts. pkg-config is a unified interface for -config scripts. If your system doesn't have pkg-config, no harm is done. It's not checked for in configure because it's not required to build. If there really is a strong desire to have a -config script instead of using pkg-config, that's fine. That seems a bit odd though.

Using pkg-config just means that you put a file in a known location (${prefix}/lib/pkgconfig) that describes your libraries CFLAGS and LDFLAGS. Otherwise, you have to install your own -config script in ${prefix}/bin.

Some sort of -config mechanism is necessary for libraries. Otherwise, a developer has no chance of being able to link against your software unless you always install in /usr/include and /usr/lib (which is not a very good thing to require).

I feel your pain, I agree with you completely :-) If you don't believe me, take a busy box chroot and ./configure libxen. Everything will work just fine :-)

Yes, yes, no, yes. If you don't have doxygen, it's not used. The comments are as human-readable as they could possibly be. libxc currently has almost no interface level comments. Documentation of all public interfaces is a requirement for EAL certification and doing it now with doxygen is going to save a lot of headache in the future when someone has to do this.

Not to mention the fact that trying to write a robust management tool without interface-level documentation is painful. It basically requires sifting through the Xen code to find which errors can be returned in what circumstances.

We basically have three options:

1) Use a standard format for documenting APIs and use a tool to extract that documentation
2) Maintain a separate document documenting APIs
3) Sparsely document header files

1 & 2 seem to be the only reasonable things to do, and 1 is going to be a lot easier to deal with in the long run.

Yes. If you don't have pyrex or python the bindings won't be built. Moreover, pyrex generates C code and the build process is completely python free so the generated C files could be included as part of the source tarballs (much like configure is autogenerated and yet still usually included in source tarballs).

Yes, that's one of the reasons to use autoconf. It lets you specify non-standard locations for header files so you can do just what you describe.

Take a look at this library, I think you'll find that it provides more support for the things you're concerned about than libxc currently does.

Regards,
Anthony Liguori

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

runtime? how do you mean? The .d is source file deps.

ps: autoconf itself doesn't give a separate build dir, so don't naively assume so.

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

Jam and MS Visual Studio (for example, not that I am advocating the use of the latter in the scope of this project) both know how to deduce header-dependencies from the C-source, without littering the tree with .d files. Since this can be done in virtually zero time, there is no reason to keep all the .d-files around. Basically that means you no longer strictly need a separate configure-step, because your source tree no longer contains any build-specific persistent state, and all your object files and executables go somewhere if you want them to.

A good build tool will be quick enough that you can just always run it 'just in case', every time you run your program, e.g.

$ jam && ./run

appears to be just as quick as

$ ./run

Sorry if I was being naive. There is even less reason to choose autoconf over Jam then.

Jacob

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

Why -errno? What's wrong with regular errno, like used everywhere else in userspace?

christian

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

errno's a global variable and it's way to easy to lose it's value. For instance:

if (read(fd, buffer, sizeof(buffer)) == -1) {
close(fd);
perror("read");
}

Doesn't do what you'd expect. It gets particularly hairy when you're not sure what functions modify errno and which functions don't.

Regards,
Anthony Liguori

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

I think it does what I expect. And it seems to work for a lot of libraries just fine. By not using the global errno, you're preventing people from using perror, warn, err and the likes. Also some of the interfaces in your library are slightly awkward because you're wasting the return parameter to return the failure reason.

Even if we don't use the global errno, I'm still wondering why you're returning -errno and not errno?

christian

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

You're right. Some of the interfaces are a little awkward (especially the memory mapping ones). It seemed like a reasonable trade-off to make though.

Good question. I guess since we never returned > 0 it would be reasonable to return errno instead of -errno. -errno is the convention used in the Linux kernel. That's what I was modelling.

Regards,
Anthony Liguori

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

What about perror, warn, err and the likes, I really like to use those. It seems very illogical having to stick the returned value into errno to be able to use those...

Again, it seems illogical having to negate the returned value to be able to use it, when there's no reason to return a negative value in the first place.

christian

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

You're not going to like this answer but I don't think I would use any of those functions in a real management application. Management tools should be using standard logging suites (like syslog).

strerror() still works with these return codes btw. My test code usually likes like this:

domid_t domid;
int ret = dom_create_domain(3000, 0, &domid);

if (ret < 0) error("dom_create_domain: failed %s", strerror(-ret));

This would be even easier if the library returned errno instead of -errno ;-)

Regards,
Anthony Liguori

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

You mean like using `%m' in your format string to syslog(3)?

While most other people's test code would look like this:

domid = dom_create_domain(3000, 0);
if (domid < 0)
err(1, "dom_create_domain");

I claim that this is what more people are used to since most libraries support this interface, i.e. return _value_ with documented specific error values, failure condition in errno and error reporting using one of the various standard functions.

christian

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

'%m' is a GNU extension.

There seems to be wide consensus on this one though and I don't seem to have the popular opinion. So perhaps we can comprimise. What about:

1) Have most functions return an integer. An error is indicated if the return is < 0. In the event of an error, the function will return -errno but make sure to set errno properly.

2) For functions that want to return addresses, they can do so and return NULL on error. Since all functions set errno properly, errno will have to be used for extended error conditions on these functions.

This should make everyone happy. Your code will work just fine and an application developer can avoid using errno. Thoughts?

Regards,
Anthony Liguori

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

I'll complicate the argument some more by suggesting that errno, and the standard errno values, aren't the best things to be returning from Xen across the hypercall interface anyway. Many errno values are not applicable to a hypervisor; also, if we defined our own then we could return a richer set of error notifications to guests and to control tools. Currently EINVAL, EPERM, EFAULT and ESRCH are approx the limit of our 'richness'.

The reason I used errno's originally in Xen was laziness and time constraints. :-)

I guess we'd need strong motivation (some cases where better error returns would be really useful) to be bothered to change the status quo though.

-- Keir

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

works for me.

sure.

ron

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

It exists in the BSD's implementations and it is documented in SUS V3. In any case, the point was that while you can argue that management applications might not use perror, warn or err, you can't really argue that there won't be at least one library usable for developing management applications which supports printing error messages depending on the value of errno.

It's inconsistent. Why would anybody examine errno for the 2nd type of functions and then not do so for the 1st type?

So far your only point for not using errno has been that the programmer can inadvertently overwrite it by calling a function which resets it before using its value. But even that now seems a moot point since it can still happen with the 2nd type of functions. What's your problem with errno?

christian

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

It's too easy to overwrite. Error paths are rarely invoked so if you're overwriting errno in your error path the chances of you realizing you're doing that is low.

This is a real problem. If you don't believe me, look at libxc. In fact, the first file I opened, xc_private.c, has that problem exact problem in the first function (xc_map_foreign_batch()). If the function fails for any reason, it does cleanup (while also calling perror). The cleanup function (munmap) will squash errno.

This function would have to save errno after the ioctl, then restore it after the cleanup was done. That's ugly and most programmers don't think to do it.

Regards,
Anthony Liguori

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click

Is there a particular reason for opening /proc/xen/privcmd on every invocation, rather than keeping a handle continuously open? Also I rather liked the xc_ name tags: the tag was a nice indication that I had reached the bottom of a tools call path and was about to hit Xen. The -errno return values are bizarre: user space has the errno variable for conveying this information.

Overall the library seems to be a greater modification of libxc than I would have thought necessary. It basically looks like a rewrite to pretty much the same interface but with names and copyrights changed.

-- Keir

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click

It's not opened on every invocation. It uses lazy evaluation to only open it once for the library. I should have perhaps documented that a little better.

I don't mind adding a common prefix back. Do other agree with this? What about using xen_ instead of xc_?

I can't speak for every library, but I think it's pretty rare to actually return errno in userspace. It just makes life so much easier.

The goals were to provide a interface tightly tied to the actual hypercalls. I know it's probably a lot more than you expected but there were a lot of problems with libxc. I can attest to this from trying to develop tools off of them.

Here were some of the problems with libxc:

1) Inconsistent errors
- some calls returned errno from the hypercall, some returned errno
from the call that failed, some set errno themselves, some never touched errno and made calls that would squash the hypercall's errno.
- some calls just printf()'d an error message and squashed errno

2) Inconsistent mlock()'ing
- some functions required arguments to be mlock()'d and some did it
for you.

3) Poor typing
- some functions used the wrong types in the interface (like passing
an int instead of a domid_t for xc_domain_create()).

4) Lack of documentation

5) Poor developer support
- Assumes headers and libs are in /usr/{include, lib}

So, making all of these changes touched every function. Especially documenting every error condition. So it was significantly easier to just start with fresh code.

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click