linux-aio usable?
Hi,
sorry for this pretty generic question, I did not find any real pros and cons on the net anywhere, but I might just have missed them.
In a pure x86_64 environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm), is enabling linux-aio in KVM a good idea? What are the advantages/disadvantages? Are there any potential pitfalls?
The reason I'm asking is that there has been some traffic on the list about it, so it seems to be something people want to get working. qemu-kvm in Ubuntu Lucid is currently not compiled with that option. I've made a local version with aio and it seems to work fine (and performs a bit better at first glance).
Is there any reason one should not compile that feature by default? Does it do anything if not explicitly run with aio=native?
Best Regards,
Bernhard
Quoted 7 lines · Bernhard Schmidt
Hi,
sorry for this pretty generic question, I did not find any real pros and cons on the net anywhere, but I might just have missed them.
In a pure
x86_64environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm), is enabling linux-aio in KVM a good idea?
Yes.
Quoted 2 lines
What are the
advantages/disadvantages?
It's faster.
Quoted 2 lines
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
Quoted 8 lines
The reason I'm asking is that there has been some traffic on the list about it, so it seems to be something people want to get working. qemu-kvm in Ubuntu Lucid is currently not compiled with that option. I've made a local version with aio and it seems to work fine (and performs a bit better at first glance).
Is there any reason one should not compile that feature by default?
Not to my knowledge.
Quoted 2 lines
Does it do anything if not explicitly run with aio=native?
IIUC, no.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 2 lines · Avi Kivity
Quoted 1 line
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
What does "work well" mean in this context? Potential dataloss?
Quoted 2 lines
Quoted 1 line
Is there any reason one should not compile that feature by default?
Not to my knowledge.
Thanks, I've filed a bug with Ubuntu to get it enabled.
Bernhard
Quoted 10 lines · Bernhard Schmidt
Quoted 4 lines · Avi Kivity
Quoted 2 lines
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
What does "work well" mean in this context? Potential dataloss?
No, it becomes synchronous (=extra slow).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Bernhard Schmidt
Quoted 3 lines · Avi Kivity
Quoted 1 line
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
What does "work well" mean in this context? Potential dataloss?
No, it becomes synchronous (=extra slow).
But for this to happen, the user would have had to consciously enter into the situation by creating/using a non block device, non-pre-allocated backing disk AND specify the aio=native option, correct?
:-Dustin
Quoted 10 lines · Dustin Kirkland
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Bernhard Schmidt
Quoted 4 lines · Avi Kivity
Quoted 2 lines
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
What does "work well" mean in this context? Potential dataloss?
No, it becomes synchronous (=extra slow).
But for this to happen, the user would have had to consciously enter into the situation by creating/using a non block device, non-pre-allocated backing disk AND specify the aio=native option, correct?
I thought there was some autodetection involved, but perhaps I just imagined it.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Dustin Kirkland
Quoted 5 lines · Avi Kivity
Quoted 4 lines · Bernhard Schmidt
Quoted 2 lines · Avi Kivity
Quoted 1 line
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
What does "work well" mean in this context? Potential dataloss?
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
No, it becomes synchronous (=extra slow).
But for this to happen, the user would have had to consciously enter into the situation by creating/using a non block device, non-pre-allocated backing disk AND specify the aio=native option, correct?
I thought there was some autodetection involved, but perhaps I just imagined it.
There's no autodetection.
linux-aio support in the kernel downgrades to synchronous IO if the underlying storage does not support linux-aio. There is no indication to userspace that this has happened.
If this happens, besides having a slow guest, the guest VCPU will be starved during the I/O requests potentially resulting in things like soft lockups and time drift.
Generally, speaking, linux-aio will work well under the following circumstances:
- cache=off is specified
- the underlying file system is XFS or you are using a block device
We cannot detect this reliably though so it's really up to the user to decide whether to use it. We're working on improving the linux-aio kernel interface though to eliminate this detectability problem after which, we can enable it in a more automatic fashion.
Regards,
Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 1 line
I thought there was some autodetection involved, but perhaps I just imagined it.
There's no autodetection.
linux-aio support in the kernel downgrades to synchronous IO if the underlying storage does not support linux-aio. There is no indication to userspace that this has happened.
Quoted 8 lines
If this happens, besides having a slow guest, the guest VCPU will be starved during the I/O requests potentially resulting in things like soft lockups and time drift.
Generally, speaking, linux-aio will work well under the following circumstances:
- cache=off is specified
- the underlying file system is XFS or you are using a block deviceWe cannot detect this reliably though so it's really up to the user to decide whether to use it. We're working on improving the linux-aio kernel interface though to eliminate this detectability problem after which, we can enable it in a more automatic fashion.
Well, the common case of cache=none on a block device certainly can be autodetected.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 2 lines
Are there any potential pitfalls?
It won't work well unless running on a block device (partition or LVM).
It will actually work well on pre-allocated filesystem images, at least on XFS and NFS.
The real pitfal is that cache=none is required for kernel support as it only supports O_DIRECT.
Quoted 1 line
Quoted 1 line
Is there any reason one should not compile that feature by default?
It's compiled by default if libaio and it's development headers are found.
Quoted 1 line
Quoted 1 line
Does it do anything if not explicitly run with aio=native?
No.
Quoted 10 lines · Avi Kivity
Quoted 7 lines · Bernhard Schmidt
Hi,
sorry for this pretty generic question, I did not find any real pros and cons on the net anywhere, but I might just have missed them.
In a pure
x86_64environment (~2.6.32 vanilla kernel, 0.12.3 qemu-kvm), is enabling linux-aio in KVM a good idea?Yes.
Apparently that does not quite work. I just re-compiled kvm with --enable-linux-aio (actually I just installed libaio-dev on debian and qemu-kvm's configure picked it up automatically), and tried a guest. But any I/O fails.
kvm-0.12.3 ... -drive file=/dev/sda10,if=virtio,cache=none,aio=native
(/dev/sda10 is a (spare) partition on my hard drive I use for testing). Here's the resulting dmesg in the guest (2.6.32):
vdb:end_request: I/O error, dev vdb, sector 0 Buffer I/O error on device vdb, logical block 0 Buffer I/O error on device vdb, logical block 1 Buffer I/O error on device vdb, logical block 2 Buffer I/O error on device vdb, logical block 3 Buffer I/O error on device vdb, logical block 4 Buffer I/O error on device vdb, logical block 5 Buffer I/O error on device vdb, logical block 6 Buffer I/O error on device vdb, logical block 7end_request: I/O error, dev vdb, sector 0 Buffer I/O error on device vdb, logical block 0 Buffer I/O error on device vdb, logical block 1end_request: I/O error, dev vdb, sector 0
unable to read partition table
And any I/O - be it reads of writes - fails.
I see some aio_submit() etc are happening in strace, but no errors. Unfortunately my strace does not decode io_*() routines.
# fgrep io_ trc
...
1227 io_submit(4152147968, 1, {...}) = 1
1226 io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
1227 io_submit(4152147968, 1, {...}) = 1
1226 io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
1227 io_submit(4152147968, 1, {...}) = 1
1226 io_getevents(-142819328, 1, 128, {...}{0, 0}) = 1
...Oh well.... ;)
/mjt
Michael Tokarev wrote:
[]
Quoted 4 lines
Apparently that does not quite work. I just re-compiled kvm with --enable-linux-aio (actually I just installed libaio-dev on debian and qemu-kvm's configure picked it up automatically), and tried a guest. But any I/O fails.
It has nothing to do with kvm. It is compat_ioctl32 in the kernel wrt aio calls. Historically I've a 64bit kernel with 32bit userland, and tried 32bit kvm too, and that does not work. But 64bit kvm works just fine with aio, and the performance numbers are indeed better.
Thanks!
/mjt
Quoted 8 lines · Michael Tokarev
Michael Tokarev wrote:
[]Quoted 5 lines
Apparently that does not quite work. I just re-compiled kvm with --enable-linux-aio (actually I just installed libaio-dev on debian and qemu-kvm's configure picked it up automatically), and tried a guest. But any I/O fails.
Quoted 6 lines
It has nothing to do with kvm. It is
compat_ioctl32in the kernel wrt aio calls. Historically I've a 64bit kernel with 32bit userland, and tried 32bit kvm too, and that does not work. But 64bit kvm works just fine with aio, and the performance numbers are indeed better.
Can you elaborate? This sounds like a bug that wants to be fixed.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Michael Tokarev
Michael Tokarev wrote:
[]Quoted 5 lines
Apparently that does not quite work. I just re-compiled kvm with --enable-linux-aio (actually I just installed libaio-dev on debian and qemu-kvm's configure picked it up automatically), and tried a guest. But any I/O fails.
Quoted 6 lines
Quoted 4 lines
It has nothing to do with kvm. It is
compat_ioctl32in the kernel wrt aio calls. Historically I've a 64bit kernel with 32bit userland, and tried 32bit kvm too, and that does not work. But 64bit kvm works just fine with aio, and the performance numbers are indeed better.Can you elaborate? This sounds like a bug that wants to be fixed.
http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
It's missing compat_ioctl for some of the aio opcodes, namely it's PREADV and PWRITE - the only ones used by kvm and the only ones missing in kernel.
As far as i can see, current code converts the iocb array just fine, but does not touch iovec array used with p{read,write}v.
/mjt
Quoted 12 lines · Michael Tokarev
Quoted 3 lines
Can you elaborate? This sounds like a bug that wants to be fixed.
http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
It's missing
compat_ioctlfor some of the aio opcodes, namely it's PREADV and PWRITE - the only ones used by kvm and the only ones missing in kernel.As far as i can see, current code converts the iocb array just fine, but does not touch iovec array used with p{read,write}v.
Yikes, looks pretty bad.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 9 lines
http://thread.gmane.org/gmane.linux.kernel.aio.general/2891
It's missing
compat_ioctlfor some of the aio opcodes, namely it's PREADV and PWRITE - the only ones used by kvm and the only ones missing in kernel.As far as i can see, current code converts the iocb array just fine, but does not touch iovec array used with p{read,write}v.
Yikes, looks pretty bad.
No need for compat_ioctl handlers as there are no ioctls involved in aio, we just need better compat versions of io_submit, which by itself is a bit of an ioctl-like multiplexer.
Adding the vector compat might be a bit messy, but shouldn't be too hard given that we already deal with this for native (p)readv/(p)writev.
Quoted 1 line
It's faster.
Hi Avi,
Could You give some rough estimate on how much faster?
I'm stuck with glibc-2.5 now, but I'm always eager to improve performance, so I wonder if it would make sense to either port eventfd + aio stuff, or switch to glibc-2.8 for me...
Signature · 14 lines
------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 7 lines · Nikola Ciprich
Quoted 1 line
It's faster.
Hi Avi,
Could You give some rough estimate on how much faster?
I'm stuck with glibc-2.5 now, but I'm always eager to improve performance, so I wonder if it would make sense to either port eventfd + aio stuff, or switch to glibc-2.8 for me...
I saw approx. 10% improvement in sequential i/o. Random i/o was only marginally faster in our setup. We generally have problems with random i/o here... Something to do with our setup.
Quoted 5 lines · Nikola Ciprich
Quoted 2 lines
It's faster.
Hi Avi,
Could You give some rough estimate on how much faster?
The standard "it depends on the workload".
Quoted 4 lines
I'm stuck with glibc-2.5 now, but I'm always eager to improve performance, so I wonder if it would make sense to either port eventfd + aio stuff, or switch to glibc-2.8 for me...
Switching to a modern setup should be much easier and safer. Esp. a modern kernel.
Signature · 1 line
error compiling committee.c: too many arguments to function