non-blocking disk IO

Non-blocking disk IO now works for any type of disk image, not just "raw" format. There is no longer any format-specific code in the patch:

http://people.brandeis.edu/~jcoiner/qemu_idedma/qemu_dma_patch.html

You might want this patch if:
* you run a multitasking guest OS,
* you access a disk sometimes, and
* you wouldn't mind if QEMU ran a little faster.

Why I have not got feedback in droves I do not understand ;)

-John

John,

I'd love to try out your patch, but I have too little free time as always. I have very high hopes for this patch and I do think that this need to be done somehow. One thing that I'd really like to see is if this patch makes the w2k install hack obsolete...

Keeep up the good work,

/ magnus

With which pthreads library have you compiled it under windows?

I've tried it with the one here:

ftp://sources.redhat.com/pub/pthreads-win32/

but it didn't compile.

teris.

I managed to make it work (qemu+non blocking IO on windows host), with a rough estimation of 10% speed increase at the early stage of windows setup. I expect more once windows installs itself in true multitasking mode. :)

What you need to do is:
- download pthreads-w32-2-6-0-release.tar.gz and compile it from scratch (make clean GC)
- you also need to add the following lines inside block.c (usleep is not present in win32/mingw32)

#ifdef WIN32
void usleep(unsigned long usec)

{
    Sleep(usec/1000);
}

#endif

- you need to update the link to include "-lpthreadGC2 -lz", and keep a copy of the pthreadGC2.dll where qemu.exe will be (to pu it just in from of "-lz" is just my personal suggestion)

HIH,
Christian

Whoops. I had the mistaken belief that Qemu already used pthreads.

So, the makefiles and configure script need to handle pthreads somehow (especially on windows.) Pthreads-w32 is refusing to 'make install' itself into standard lib and include directories. Unlike SDL, there is no 'pthread-config' to tell us where it lives.

Suppose that on Windows, we tell people to unpack the pthread distribution within the qemu tree. Qemu's configure script could find it, build a static pthread library, and link that into Qemu.

This removes the need for the user to:
* read pthreads-w32 documents,
* issue a separate build command for pthreads-w32,
* get confused before realizing that pthreads-w32 doesn't really
support 'make install',
* hack at qemu until it points to the uninstalled pthreads dir,
* copy the pthreads DLL into c:\windows

I'll look at adding this to the configure script.

-john

I did not manage a static link on win32 yesterday. I needed the pthread dll in the same directory as qemu.exe.

I'll look more into it.

Thanks for the info.

The division "usec/1000" is an expensive operation on any processor.

Maybe it would be better just to use "_sleep(1)" which is analogous to usleep(1000) ?

This is what i did to have it compiled and it worked like a charm.

teris.

indeed, this was a quick and dirty hack :) I just managed to compile pthreadGC2 statically... pthread.h has to be modified to include extra PTW32_STATIC_LIB info, like this:

#define PTW32_VERSION 2,6,0,0
#define PTW32_VERSION_STRING "2, 6, 0, 0\0"
#define PTW32_STATIC_LIB 1

I'll test it later today and will report realistic benchmarks on windows hosts.

--
Christian

hmmm....

the static binary crashes, the one with ptreadGC2.dll is ok.

What I tried to measure, on windows host, kqemu-0.7.2 loaded, was:
(1) tar -jxf linux-2.6.10.tar.bz2
(2) make allnoconfig; make

I confirm ~18% speed improvement in case (1), but no change more or less for case (2). I believe maybe someone with a windows host SMP could get different figures. If so, please update these accordingly and let the list know.

Indeed, threading on a busy single CPU doesn't make it faster. A kernel compilation with kqemu makes full use of the CPU. :(

I think we'll benefit mostly when copying files or setup OSes when the CPU is quite in IDLE state. I've no figures without kqemu yet.

At least, it's a good start for speed improvement. Thanks for the patch! :)

--
Christian

That's not quite true. The io thread basically takes no real CPU, but it often has to wait for io to become available. So even (especially!) for a single cpu system, you definitely want to make sure that blocking io doesn't eat away cpu cycles from a cpu bound case like the compile.

If you did the kernel compile with a hot disk cache, I'm not surprised you're not seeing a performance benefit of the non-blocking io patch. Even for a cold cache compile it will generally be cpu bound.

Jens Axboe

(1) is a parallel task -- 'tar' and 'bunzip2' run concurrently. You see a benefit here when one process can use the CPU while the other is blocked waiting for IO.

(2) is a serial task. It can't benefit from non-blocking IO. 'make -j 2' should benefit, in cases where the host file cache is cold.

-- john

Then all is good and I should not use (2) as a benchmark. Thanks for clarifying :)

--
Christian

Why not use aio for this instead, seems like a better fit than spawning a thread per block device? That would still require a thread for handling completions, but you could easily just use a single completion thread for all devices for this as it would not need to do any real work.

Jens Axboe

Pthreads vs. AIO is an open question.

The pthreads implementation works with all disk image formats, without modifying any format-specific code. To use AIO, each image format driver would have to be modified.

QEMU already uses pthreads, therefore using pthreads does not add a new dependency on the host platform. (Or so I thought. The build on Windows broke anyway. Haha... my bad...)

My questions are:
* Which QEMU host platforms support Posix AIO?
* Which host platform has the slowest pthreads library, and how many
cycles will it waste on pthreads overhead?
* What is the likely performance benefit of using AIO?

When I have time, I'll hack AIO into the "raw" image driver, and take some benchmarks on a linux host.

Thanks.

-John

That is true, I appreciate why you did it the way you did. Using libaio would be more invasive, indeed.

Using aio would result in cleaner design in the end, but it would require extra host specific code for the abstraction. It's also non-buffered, which may or may not be a win for qemu (it should be, would be trivial to test this with the existing code already).

For the normal case of a single or two block devices, performance of the threaded vs libaio model will likely be the same. For many block devices, libaio would definitely be a big improvement.

At least on Linux, the posix aio model is not worth spending time on, as it just relies on threading as well on the glibc side. Then you might as well just keep the design you have now.

BTW, thanks for starting this work, it's sorely needed! A future experiment would be to write a sata/scsi qemu driver so you could do sane command queueing. Since file layout isn't always perfect, that should further increase the io performance.

Jens Axboe

Why is there a posix async IO 'standard', but a different linux AIO standard? Having something that wraps both Linux-native AIO and posix async I/O might make supporting it a little easier to swallow on cross-platform applications.

Because the Posix AIO sucks quite badly both to implement and use.

But all Linuxes I know of have a Posix AIO library. Some even have a Posix AIO library using Linux AIO.

If you anyway plan on having Posix AIO support then go for the Posix AIO interface. The performance reasons why Linux AIO exists is very unlikely to be an issue to qemu as you need to be quite I/O intensive to see any performance difference.

Regards
Henrik

Ideally, we should be able to use a Posix AIO support libary that uses native Linux AIO underneath.

What we want is to be able to have the guest OS request some DMA I/O operation, and have qemu be able to use AIO so that the actual disk hardware can dump the data directly in the pages the userspace process on the guest OS ends up wanting it in, avoiding several expensive memcopy and context switch operations.

That should be easy enough to do already, with or without the nonblocking patch. Just make sure to open the files O_DIRECT and align the io buffers and lengths. With a 2.6 host, you can usually get away with 512-b aligment, on 2.4 you may have to ensure 1k/4k alignment.

Jens Axboe

Linux has a Posix AIO library that sits on top of linux's native AIO:

http://www.bullopensource.org/posix/

It's pretty new, version 0.6. We might not want qemu to depend on it, because most distros probably do not bundle it yet, so it would be an extra dependency for all new users to chase down.

-john

Thank you for the patch. I'll look at it ASAP. If there is no regression (in terms of bugs and performance) I will include it soon: I consider that non blocking I/O is now a critical feature in QEMU in order to get better performances.

Fabrice.

I have 3 or 4 images here (of different flavours of windows from 95 -> XP) that I have base images about 4G and then I use qcow files to do my testing and other damaging stuff to. With this patch qemu will not boot the qcow files but will boot the base images (I copy them and then try to boot to test). As soon as I patch -R && make, qemu will boot the qcow files no problems..

Let me clarify that..
qemu-img create -f qcow xp.img 4G
qemu -cdrom xp.iso -hda xp.img -boot d (Install XP from scratch) qemu-img create -b xp.img -f qcow xp2.img
qemu -hda xp2.img

This gives me an invalid hard disk error. If I revert the patch I can boot from xp2.img with no issues. With or without the patch I can boot from xp.img.

I have tried installing XP with and without the patch and that works fine..

Regards,
Brad

"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams

Thanks for the report. I was able to reproduce this.

The problem was due to missing Makefile dependencies. The patch modifies 'block_int.h', which is #included in several '.c' files. The file 'block.c' recompiled at 'make' time, but 'block-qcow.c' didn't. The resulting '.o' files had different opinions about what a BlockDriverState struct looked like, which broke the qcow code.

Unaware of this, I added a printf to 'block-qcow.c', and the printf fixed the problem! The man page for printf doesn't mention this... and then I was enlightened. :P

To be safe, try a 'make clean ; make install' and you should be all good. Let me know if not.

-- John

Having had it running for a while I did some performance tests and found no performance gains _at all_, but then I'm running my entire system over the network on an NFS filesystem (this machine has no disks in it).

*NOW* having said that, one of the tricks I do to keep qemu from "going to sleep" when running a long process like a big software install (sometimes it appears to stall until I move the mouse around a bit) is to ping the machine constantly from the host. Prior to this patch I would get pings that varied from 1ms to 250ms. With this patch I get a spread from about 1ms to 15ms tops. So it has smoothed out the operation of the host noticably (also noticable in screen redraws and UI responsiveness while under a heavy disk load).

No performance for me, but it just "feels" better to use all round.

So, no functional regressions, better usability but no performance here (probably due to running the whole thing over nfs) One day I'll get a local disk and test it out.

As a lark, I also did a fresh compile of the BOCHS latest CVS bios with the apm and dma patches included. It appears to give me a slight speed improvement on booting my XP guest, but given the nature of my system that could be statistical noise.. If anyone wants to give it a whirl I have put my newly compiled bios here http://fnarfbargle.dyndns.org:81/bios

Regards,
Brad

"Human beings, who are almost unique in having the ability
to learn from the experience of others, are also remarkable
for their apparent disinclination to do so." -- Douglas Adams

What network do you have between the client and nfs server? Also, what throughput do you see to the NFS server? (and what are the nfs 'wsize' and 'rsize' settings)

Does the stall still happen if you don't ping the guest?

I've been using the non-blocking IO / DMA patch with a FreeBSD host and Windows 98 guest. It seems to work well -- I haven't done any benchmarking, but subjectively it seems to make a difference.

However, I tried enabling DMA in the Windows 98 system settings for the C: drive, and that doesn't work -- when I reboot the emulator, it stalls fairly early in the boot process.

Have others been able to get DMA working (with the patch) with Windows 98? Is there something I could to try to track down the problem (assuming it is supposed to work).