Skip to contentSkip to conversation

An alternative http protocol

PATCH In review Nolan 3 participants 3 messages Apr 10, 2009 – Apr 13, 2009 Kbuild

lore
Share & tools
lore

Nolan

author
#0 lore raw

PATCH 1/1 An alternative http protocol

Alexander Graf's posting of his HTTP protocol for block devices prompted me to finally cleanup and submit my implementation of the same thing, so that both may be considered and compared.

The most obvious difference is that this implementation uses libneon instead of libcurl.

It supports HTTPS, proxies, and HTTP basic auth, though be aware that if you pass a URL with credentials on the command line, anyone who can run "ps" on the host can see your credentials!

It also supports rounding range request sizes up and a simple 1 entry cache, defaulting to 256KB. This _greatly_ reduces the load on the server when using a HTTP cdrom .iso, as well as when using an HTTP image as a source for "qemu-img convert". Without this, booting off an HTTP livecd iso results in tens of thousands to hundreds of thousands of 1500 byte HTTP range requests.

It has been heavily tested in production patched into kvm-84, and lightly tested on qemu trunk.

It does, however, use bdrv_read, so Anthony's complaint about AIO applies to this version as well.

Signed-off-by Nolan Leake <nolan <at> sigbus.net>
 Makefile        |    7 
 Makefile.target |    5 
 block-http.c    |  463 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 block.c         |    3 
 block.h         |    1 
 configure       |   24 ++
 6 files changed, 503 insertions(+)
Makefile.target +5 -0
Index: Makefile.target
===================================================================
@@ -557,6 +557,11 @@
 LIBS += $(CONFIG_BLUEZ_LIBS)
 endif
 
+ifdef CONFIG_HTTP_BLOCK
+LIBS += $(CONFIG_HTTP_BLOCK_LIBS)
+CPPFLAGS += $(CONFIG_HTTP_BLOCK_CFLAGS)
+endif
+
 # SCSI layer
 OBJS+= lsi53c895a.o esp.o
 
Makefile +7 -0
Index: Makefile
===================================================================
@@ -54,6 +54,10 @@
 BLOCK_OBJS+=block-dmg.o block-bochs.o block-vpc.o block-vvfat.o
 BLOCK_OBJS+=block-qcow2.o block-parallels.o block-nbd.o
 BLOCK_OBJS+=nbd.o block.o aio.o
+ifdef CONFIG_HTTP_BLOCK
+BLOCK_OBJS+=block-http.o
+LIBS+=$(CONFIG_HTTP_BLOCK_LIBS)
+endif
 
 ifdef CONFIG_WIN32
 BLOCK_OBJS += block-raw-win32.o
@@ -192,6 +196,9 @@
 
 bt-host.o: CFLAGS += $(CONFIG_BLUEZ_CFLAGS)
 
+block-http.o: block-http.c
+	$(CC) $(CFLAGS) $(CPPFLAGS) $(CONFIG_HTTP_BLOCK_CFLAGS) -c -o $@ $<
+
 libqemu_common.a: $(OBJS)
 
 #######################################################################
block.c +3 -0
Index: block.c
===================================================================
@@ -1488,6 +1488,9 @@
     bdrv_register(&bdrv_qcow2);
     bdrv_register(&bdrv_parallels);
     bdrv_register(&bdrv_nbd);
+#ifdef CONFIG_HTTP_BLOCK
+    bdrv_register(&bdrv_http);
+#endif
 }
 
 void aio_pool_init(AIOPool *pool, int aiocb_size,
block.h +1 -0
Index: block.h
===================================================================
@@ -20,6 +20,7 @@
 extern BlockDriver bdrv_qcow2;
 extern BlockDriver bdrv_parallels;
 extern BlockDriver bdrv_nbd;
+extern BlockDriver bdrv_http;
 
 typedef struct BlockDriverInfo {
     /* in bytes, 0 if irrelevant */
configure +24 -0
Index: configure
===================================================================
@@ -191,6 +191,7 @@
 fdt="yes"
 sdl_x11="no"
 pkgversion=""
+http_block="yes"
 
 # OS specific
 if check_define __linux__ ; then
@@ -473,6 +474,8 @@
   ;;
   --with-pkgversion=*) pkgversion=" ($optarg)"
   ;;
+  --disable-http-block) http_block="no"
+  ;;
   *) echo "ERROR: unknown option $opt"; show_help="yes"
   ;;
   esac
@@ -597,6 +600,7 @@
 echo "  --disable-aio            disable AIO support"
 echo "  --disable-blobs          disable installing provided firmware blobs"
 echo "  --kerneldir=PATH         look for kernel includes in PATH"
+echo "  --disable-http-block     disable support for http block devices"
 echo ""
 echo "NOTE: The object files are built at the place where configure is launched"
 exit 1
@@ -1137,6 +1141,19 @@
   fi
 fi
 
+##########################################
+# libneon probe
+if test "$http_block" = "yes" ; then
+  cat > $TMPC << EOF
+#include <ne_socket.h>
+int main(void) { return ne_sock_init(); }
+EOF
+  if ! $cc $ARCH_CFLAGS `neon-config --cflags` `neon-config --libs` \
+      -o $TMPE $TMPC 2> /dev/null ; then
+    http_block=no
+  fi
+fi
+
 # Check if tools are available to build documentation.
 if [ -x "`which texi2html 2>/dev/null`" ] && \
    [ -x "`which pod2man 2>/dev/null`" ]; then
@@ -1240,6 +1257,7 @@
 echo "KVM support       $kvm"
 echo "fdt support       $fdt"
 echo "preadv support    $preadv"
+echo "HTTP block        $http_block"
 
 if test $sdl_too_old = "yes"; then
 echo "-> Your SDL version is too old - please upgrade to have SDL support"
@@ -1550,6 +1568,12 @@
   echo "#define HAVE_FDT 1" >> $config_h
   echo "FDT_LIBS=-lfdt" >> $config_mak
 fi
+if test "$http_block" = "yes" ; then
+  echo "#define CONFIG_HTTP_BLOCK 1" >> $config_h
+  echo "CONFIG_HTTP_BLOCK=yes" >> $config_mak
+  echo "CONFIG_HTTP_BLOCK_CFLAGS=`neon-config --cflags`" >> $config_mak
+  echo "CONFIG_HTTP_BLOCK_LIBS=`neon-config --libs`" >> $config_mak
+fi 
 
 # XXX: suppress that
 if [ "$bsd" = "yes" ] ; then
block-http.c +463 -0

463 lines not on this page · raw message

#1 lore raw
Quoted 3 lines · Nolan

Alexander Graf's posting of his HTTP protocol for block devices prompted me to finally cleanup and submit my implementation of the same thing, so that both may be considered and compared.

Phew - I didn't know HTTP backends were this popular :-).

Quoted 2 lines

The most obvious difference is that this implementation uses libneon instead of libcurl.

Oh - I didn't even know about libneon before :-o.

Quoted 3 lines

It supports HTTPS, proxies, and HTTP basic auth, though be aware that if you pass a URL with credentials on the command line, anyone who can run "ps" on the host can see your credentials!

FWIW libcurl supports these too plus FTP and TFTP, though I haven't tried it myself. The good thing about the FTP/TFTP support is that it should bascially work exactly the same, so libcurl interprets the "range" parameter and ends up doing "the right thing" for the respective protocols.

Quoted 6 lines

It also supports rounding range request sizes up and a simple 1 entry cache, defaulting to 256KB. This _greatly_ reduces the load on the server when using a HTTP cdrom .iso, as well as when using an HTTP image as a source for "qemu-img convert". Without this, booting off an HTTP livecd iso results in tens of thousands to hundreds of thousands of 1500 byte HTTP range requests.

My ultimate goal was to have a background downloader integrated, so you could basically run an image on-the-fly, save it to disk and have it there once it's done downloading.

This way there's a pretty good chance the "download all" background downloading thread might have gotten to the part you actually want to read already. Also it might be clever to implement some basic read-ahead based on line speed, so if you request 2 sectors at offset n, you'd basically download say 20 blocks, giving the AIO return as soon as 2 are downloaded.

I don't know how much of this is possible with libcurl, but I was hoping it's feasible :-).

So in case you're planning on developing this on, please keep my design goals in mind as well. In the end I'd like to get this functionally en par with the VMware on-demand execution solution.

Alex

#2 lore raw
Quoted 5 lines · Alexander Graf <agraf <at> suse.de> writes:

FWIW libcurl supports these too plus FTP and TFTP, though I haven't tried it myself. The good thing about the FTP/TFTP support is that it should bascially work exactly the same, so libcurl interprets the "range" parameter and ends up doing "the right thing" for the respective protocols.

I've poked around a bit, and it looks like libneon makes doing AIO difficult. Getting FTP for free is pretty compelling, and there might be a use-case for TFTP out there somewhere.

So I think the smart thing to do is to add a simple cache to your libcurl based implementation. Either the extremely simple 1 entry cache I used, or a more clever one. I'm happy to add it after you convert to AIO.

For the use case of booting over HTTP from an .iso file, the cache is not really optional. A cacheless install of debian etch from an Apache server resulted in ~60MB of log entries, and took hours. Doing this to someone else's server is extremely anti-social.