Huge memory leak in virtio, see kvm-Bugs-2989366
Hi,
this is a follow-up to bug 2989366:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
after extensive debugging with the guys on #kvm it turns out that the leak is in the qemu-kvm userland process, in virtio-blk.
A summary of my setup is described in the bug report above.
The affected guests have a common load profile: frequent sequential I/O on large (~ 2 GB) files.
I tried switching off or changing almost all options in my qemu command line and the only option that makes a difference is -drive if=virtio.
When an affected guest is run with virtio drives the qemu-kvm process starts leaking immediately after startup and grows (for the most heavily leaking guests) by ~ 1 GB RSS for every ten hours (and keeps growing until OOM).
With -drive if=ide or scsi, it doesn't leak at all.
A diff of /proc/<pid>/maps of an affected qemu-kvm at startup and after 1.5 hrs:
-039b9000-5ccd0000 rw-p 00000000 00:00 0
+039b9000-65803000 rw-p 00000000 00:00 0
(a heap leak?)
I'm willing to debug further. The problem is 100% reproducible.
Signature · 4 lines
Leszek "Tygrys" Urbanski, SCSA, SCNA
"Unix-to-Unix Copy Program;" said PDP-1. "You will never find a more
wretched hive of bugs and flamers. We must be cautious." -- DECWARS
http://cygnus.moo.pl/ -- Cygnus High Altitude BalloonQuoted 26 lines · Leszek Urbanski
Hi,
this is a follow-up to bug 2989366:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
after extensive debugging with the guys on #kvm it turns out that the leak is in the qemu-kvm userland process, in virtio-blk.
A summary of my setup is described in the bug report above.
The affected guests have a common load profile: frequent sequential I/O on large (~ 2 GB) files.
I tried switching off or changing almost all options in my qemu command line and the only option that makes a difference is -drive if=virtio.
When an affected guest is run with virtio drives the qemu-kvm process starts leaking immediately after startup and grows (for the most heavily leaking guests) by ~ 1 GB RSS for every ten hours (and keeps growing until OOM).
With -drive if=ide or scsi, it doesn't leak at all.
A diff of /proc/<pid>/maps of an affected qemu-kvm at startup and after 1.5 hrs:
Quoted 6 lines
-039b9000-5ccd0000 rw-p 00000000 00:00 0
+039b9000-65803000 rw-p 00000000 00:00 0(a heap leak?)
I'm willing to debug further. The problem is 100% reproducible.
Certainly something malloc()'ed. It'd be great to send this through valgrind. Thanks to KVM the guest still runs natively, so the slowdown isn't _that_ bug through it. I'm also not a valgrind expert, but IIRC there was a separate memory allocation module.
Alex
<[email protected]>; from Alexander Graf on Wed, Apr 21, 2010 at 00:51:21 +0200
Quoted 8 lines
Quoted 6 lines
-039b9000-5ccd0000 rw-p 00000000 00:00 0
+039b9000-65803000 rw-p 00000000 00:00 0(a heap leak?)
I'm willing to debug further. The problem is 100% reproducible.
Certainly something malloc()'ed. It'd be great to send this through valgrind. Thanks to KVM the guest still runs natively, so the slowdown isn't _that_ bug through it. I'm also not a valgrind expert, but IIRC there was a separate memory allocation module.
Unfortunately, KVM guests with virtio won't even finish booting with valgrind. The guest's kernel complains about I/O errors in the first sectors of the virtio root device.
Signature · 4 lines
Leszek "Tygrys" Urbanski, SCSA, SCNA
"Unix-to-Unix Copy Program;" said PDP-1. "You will never find a more
wretched hive of bugs and flamers. We must be cautious." -- DECWARS
http://cygnus.moo.pl/ -- Cygnus High Altitude BalloonQuoted 8 lines · Leszek Urbanski
<[email protected]>; from Alexander Graf on Wed, Apr 21, 2010 at 00:51:21 +0200
Quoted 5 lines
Quoted 5 lines
-039b9000-5ccd0000 rw-p 00000000 00:00 0
+039b9000-65803000 rw-p 00000000 00:00 0(a heap leak?)
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
I'm willing to debug further. The problem is 100% reproducible.
Certainly something malloc()'ed. It'd be great to send this through valgrind. Thanks to KVM the guest still runs natively, so the slowdown isn't _that_ bug through it. I'm also not a valgrind expert, but IIRC there was a separate memory allocation module.
Unfortunately, KVM guests with virtio won't even finish booting with valgrind. The guest's kernel complains about I/O errors in the first sectors of the virtio root device.
You can try glibc's malloc tracing, see libc's mtrace().
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 26 lines · Leszek Urbanski
Hi,
this is a follow-up to bug 2989366:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
after extensive debugging with the guys on #kvm it turns out that the leak is in the qemu-kvm userland process, in virtio-blk.
A summary of my setup is described in the bug report above.
The affected guests have a common load profile: frequent sequential I/O on large (~ 2 GB) files.
I tried switching off or changing almost all options in my qemu command line and the only option that makes a difference is -drive if=virtio.
When an affected guest is run with virtio drives the qemu-kvm process starts leaking immediately after startup and grows (for the most heavily leaking guests) by ~ 1 GB RSS for every ten hours (and keeps growing until OOM).
With -drive if=ide or scsi, it doesn't leak at all.
A diff of /proc/<pid>/maps of an affected qemu-kvm at startup and after 1.5 hrs:
Quoted 6 lines
-039b9000-5ccd0000 rw-p 00000000 00:00 0
+039b9000-65803000 rw-p 00000000 00:00 0(a heap leak?)
I'm willing to debug further. The problem is 100% reproducible.
Is that qemu-kvm 0.12.3 compiled from source? or using the distro package?
If you drop the -smp 4 part, you could also try plain qemu to eliminate if there was a qemu-kvm merge issue.
Also, if you switch to a different guest do you still see the same leak? This should help determine if the virtio-blk front end is part of the issue.
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Quoted 9 lines · Ryan Harper
Quoted 8 lines · Leszek Urbanski
Hi,
this is a follow-up to bug 2989366:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
after extensive debugging with the guys on #kvm it turns out that the leak is in the qemu-kvm userland process, in virtio-blk.
[]
Quoted 2 lines
Is that qemu-kvm 0.12.3 compiled from source? or using the distro package?
(i'm not the OP, but we talked with him on irc about the issue)
It's a debian package of qemu-kvm. There are a couple of cosmetic and unrelated patches applied to it, with one important to fix the large iovecs issue. See
for the complete set of patches in there (it's debian/patches directory in http://git.debian.org/?p=collab-maint/qemu-kvm.git , in v0.12.3+dfsg-4 branch). The only interesting patch in there is the avoid_creating_too_large_iovecs_in_multiwrite_merge.patch one, the rest are not relevant.
Quoted 2 lines
If you drop the -smp 4 part, you could also try plain qemu to eliminate if there was a qemu-kvm merge issue.
So basically, upstream qemu now works as good
as qemu-kvm for non-smp guests?
Quoted 3 lines
Also, if you switch to a different guest do you still see the same leak? This should help determine if the virtio-blk front end is part of the issue.
There are only a few guests which are affected. So far it is not really clear what differs them from others: a reinstall of a new guest with the same components and doing same functions will not necessary show the leak.
Thanks!
/mjt
<[email protected]>; from Michael Tokarev on Wed, Apr 21, 2010 at 10:08:20 +0400
Quoted 5 lines
There are only a few guests which are affected. So far it is not really clear what differs them from others: a reinstall of a new guest with the same components and doing same functions will not necessary show the leak.
I checked further and I can reproduce it on every guest with a simple:
while true ; do dd if=/dev/urandom of=file bs=10M count=200 ; cat file > /dev/null ; done
It will leak slowly.
Signature · 4 lines
Leszek "Tygrys" Urbanski, SCSA, SCNA
"Unix-to-Unix Copy Program;" said PDP-1. "You will never find a more
wretched hive of bugs and flamers. We must be cautious." -- DECWARS
http://cygnus.moo.pl/ -- Cygnus High Altitude BalloonLeszek,
Please try the qemu-kvm.git patch I have sent called "block: Free iovec arrays allocated by multiwrite_merge()" to confirm that it fixes the leak.
Thanks,
Stefan
Quoted 29 lines · Michael Tokarev
Quoted 10 lines · Ryan Harper
Quoted 9 lines · Leszek Urbanski
Hi,
this is a follow-up to bug 2989366:
https://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
after extensive debugging with the guys on #kvm it turns out that the leak is
in the qemu-kvm userland process, in virtio-blk.[]
Quoted 2 lines
Is that qemu-kvm 0.12.3 compiled from source? or using the distro package?
(i'm not the OP, but we talked with him on irc about the issue)
It's a debian package of qemu-kvm. There are a couple of cosmetic and unrelated patches applied to it, with one important to fix the large iovecs issue. See
for the complete set of patches in there (it's debian/patches directory in http://git.debian.org/?p=collab-maint/qemu-kvm.git , in v0.12.3+dfsg-4 branch). The only interesting patch in there is the avoid_creating_too_large_iovecs_in_multiwrite_merge.patch one, the rest are not relevant.
Quoted 5 lines
Quoted 2 lines
If you drop the -smp 4 part, you could also try plain qemu to eliminate if there was a qemu-kvm merge issue.
So basically, upstream qemu now works as good
as qemu-kvm for non-smp guests?
Not all of qemu-kvm features are upstream in qemu, but upstream qemu supports kvm enough that it's worth testing to see if can't reproduce there should provide some insight as to where the bug might be.
Quoted 9 lines
Quoted 4 lines
Also, if you switch to a different guest do you still see the same leak? This should help determine if the virtio-blk front end is part of the issue.
There are only a few guests which are affected. So far it is not really clear what differs them from others: a reinstall of a new guest with the same components and doing same functions will not necessary show the leak.
That is troublesome. Are you saying if I install lenny guest and do the dd; cat I might not reproduce it?
Quoted 8 lines
Thanks!
/mjt
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
PATCH 1/1 block: Free iovec arrays allocated by multiwrite_merge()
A new iovec array is allocated when creating a merged write request. This patch ensures that the iovec array is deleted in addition to its qiov owner.
--- block.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
block.c +3 -0
@@ -1731,6 +1731,9 @@ static void multiwrite_user_cb(MultiwriteCB *mcb)for (i =0 ; i < mcb->num_callbacks; i++) { mcb->callbacks[i].cb (mcb->callbacks[i].opaque, mcb->error); +if (mcb->callbacks[i].free_qiov) { +qemu_iovec_destroy (mcb->callbacks[i].free_qiov); + }qemu_free (mcb->callbacks[i].free_qiov);qemu_vfree (mcb->callbacks[i].free_buf); }
Signature · 1 line
1.7.0
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
Quoted 3 lines · Stefan Hajnoczi
A new iovec array is allocated when creating a merged write request. This patch ensures that the iovec array is deleted in addition to its qiov owner.
Nice catch. Send this to qemu-devel and Avi and merge into qemu-kvm once it's commited there.
Quoted 27 lines
Signed-off-by Stefan Hajnoczi <[email protected]>--- block.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)block.c +3
-0@@ -1731,6 +1731,9 @@ static void multiwrite_user_cb(MultiwriteCB *mcb)for (i =0 ; i < mcb->num_callbacks; i++) { mcb->callbacks[i].cb (mcb->callbacks[i].opaque, mcb->error); +if (mcb->callbacks[i].free_qiov) { +qemu_iovec_destroy (mcb->callbacks[i].free_qiov); + }qemu_free (mcb->callbacks[i].free_qiov);qemu_vfree (mcb->callbacks[i].free_buf); }Signature · 6 lines
1.7.0 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
Quoted 7 lines · On Wednesday 21 April 2010 13:35:36 Ryan Harper wrote:
Quoted 3 lines · Stefan Hajnoczi
A new iovec array is allocated when creating a merged write request. This patch ensures that the iovec array is deleted in addition to its qiov owner.
Nice catch. Send this to qemu-devel and Avi and merge into qemu-kvm once it's commited there.
And tag for the stable release that should be coming soon?
Quoted 21 lines
Quoted 20 lines
Signed-off-by Stefan Hajnoczi <[email protected]>--- block.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)block.c +3
-0@@ -1731,6 +1731,9 @@ static void multiwrite_user_cb(MultiwriteCB *mcb)for (i =0 ; i < mcb->num_callbacks; i++) { mcb->callbacks[i].cb (mcb->callbacks[i].opaque, mcb->error); +if (mcb->callbacks[i].free_qiov) { +qemu_iovec_destroy (mcb->callbacks[i].free_qiov); + }qemu_free (mcb->callbacks[i].free_qiov);qemu_vfree (mcb->callbacks[i].free_buf); }
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
In reply to Ryan Harper<[email protected]>; from Ryan Harper on Wed, Apr 21, 2010 at 13:35:36 -0500
Quoted 7 lines · Stefan Hajnoczi
Quoted 3 lines
A new iovec array is allocated when creating a merged write request. This patch ensures that the iovec array is deleted in addition to its qiov owner.
Nice catch. Send this to qemu-devel and Avi and merge into qemu-kvm once it's commited there.
Nice catch indeed.
Debugging with mtrace() also pointed to the iovec code as the culprit.
It's been running for 45 minutes with this patch applied - no leaks. Thanks!
Signature · 4 lines
Leszek "Tygrys" Urbanski, SCSA, SCNA
"Unix-to-Unix Copy Program;" said PDP-1. "You will never find a more
wretched hive of bugs and flamers. We must be cautious." -- DECWARS
http://cygnus.moo.pl/ -- Cygnus High Altitude BalloonRe: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
Quoted 6 lines · Leszek Urbanski
<[email protected]>; from Ryan Harper on Wed, Apr 21, 2010 at 13:35:36 -0500
Quoted 5 lines · Stefan Hajnoczi
Quoted 3 lines
A new iovec array is allocated when creating a merged write request. This patch ensures that the iovec array is deleted in addition to its qiov owner.
Quoted 6 lines
Quoted 2 lines
Nice catch. Send this to qemu-devel and Avi and merge into qemu-kvm once it's commited there.
Nice catch indeed.
Debugging with
mtrace()also pointed to the iovec code as the culprit.
I've not used mtrace before, could you dump your command invocation for the list? I know other's would be glad to see an example with kvm
Signature · 4 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx [email protected]
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
<[email protected]>; from Ryan Harper on Wed, Apr 21, 2010 at 15:03:58 -0500
Quoted 4 lines
Quoted 1 line
Debugging with
mtrace()also pointed to the iovec code as the culprit.I've not used mtrace before, could you dump your command invocation for the list? I know other's would be glad to see an example with kvm
#include <mcheck.c>, put mtrace() and muntrace() around the code in main() in vl.c
export MALLOC_TRACE=/some/file
then run qemu with your usual options. After powering off the guest, run "mtrace /path/to/qemu-binary /some/file" - it's a perl script that makes the output more human readable.
It's a little tricky, though - remember that it will see most allocations occurring in qemu-malloc.c - it only traces explicit glibc malloc() calls.
Signature · 4 lines
Leszek "Tygrys" Urbanski, SCSA, SCNA
"Unix-to-Unix Copy Program;" said PDP-1. "You will never find a more
wretched hive of bugs and flamers. We must be cautious." -- DECWARS
http://cygnus.moo.pl/ -- Cygnus High Altitude BalloonRe: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
Quoted 11 lines · Leszek Urbanski
<[email protected]>; from Ryan Harper on Wed, Apr 21, 2010 at 15:03:58 -0500
Quoted 5 lines
Quoted 2 lines
Debugging with
mtrace()also pointed to the iovec code as the culprit.I've not used mtrace before, could you dump your command invocation for the list? I know other's would be glad to see an example with kvm
#include<mcheck.c>, put
mtrace()andmuntrace()around the code inmain()invl.c
Quoted 9 lines
export MALLOC_TRACE=/some/file
then run qemu with your usual options. After powering off the guest, run "mtrace /path/to/qemu-binary /some/file" - it's a perl script that makes the output more human readable.
It's a little tricky, though - remember that it will see most allocations occurring in
qemu-malloc.c- it only traces explicit glibcmalloc()calls.
Does it show only the immediate caller, without a complete stack trace?
If so, that sucks.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
From: Stefan Hajnoczi <[email protected]>
The MALLOC_TRACE output didn't look useful when I tried it either.
Instead I used the following to find origin of the leak. Still very basic but works better with qemu_malloc() and friends.
This is just a hack but I wanted to share it in case someone finds it useful in the future.
--- Makefile.objs | 2 +- leakcheck.c | 17 +++++++++++++++ leakcheck.py | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ osdep.c | 7 +++++- qemu-malloc.c | 26 +++++++++++++++++++---- 5 files changed, 108 insertions(+), 7 deletions(-) create mode 100644 leakcheck.c create mode 100755 leakcheck.py
Makefile.objs +1 -1
@@ -7,7 +7,7 @@ qobject-obj-y += qerror.o ####################################################################### # block-obj-y is code used by both qemu system emulation and qemu-img -block-obj-y = cutils.o cache-utils.o qemu-malloc.o qemu-option.o module.o +block-obj-y = cutils.o cache-utils.o qemu-malloc.o qemu-option.o module.o leakcheck.o block-obj-y += nbd.o block.o aio.o aes.o osdep.o block-obj-$(CONFIG_POSIX) += posix-aio-compat.o block-obj-$(CONFIG_LINUX_AIO) += linux-aio.o
leakcheck.c new file +17 -0
@@ -0,0 +1,17 @@ +#include <stdio.h> + +static FILE *fp; + +extern void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1); + +void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1) +{ +if (!fp) { + fp =fopen ("/tmp/leakcheck.log" ,"w" ); +if (!fp) { +return ; + } + } + +fprintf (fp,"%c %p %p %zd %p\n" , action, old_addr, addr, size, ret1); +}
leakcheck.py new file +63 -0
63 lines not on this page · raw message
osdep.c +6 -1
@@ -95,6 +95,8 @@ void qemu_vfree(void *ptr)#else +extern void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1); +void *qemu_memalign (size_t alignment,size_t size) {#if defined(_POSIX_C_SOURCE) && !defined(__sun__)
@@ -110,7 +112,9 @@ void *qemu_memalign(size_t alignment, size_t size)#elif defined(CONFIG_BSD )return oom_check (valloc (size));#else -return oom_check (memalign (alignment, size)); +void *p =oom_check (memalign (alignment, size)); +leakcheck_log ('a' ,NULL , p, size,__builtin_return_address (0 )); +return p;#endif }
@@ -126,6 +130,7 @@ void *qemu_vmalloc(size_t size)void qemu_vfree (void *ptr) { +leakcheck_log ('v' ,NULL , ptr,0 ,__builtin_return_address (0 ));free (ptr); }
qemu-malloc.c +21 -5
@@ -24,6 +24,8 @@#include "qemu-common.h" #include <stdlib.h> +extern void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1); +static void *oom_check (void *ptr) {if (ptr ==NULL ) {
@@ -39,6 +41,7 @@ void *get_mmap_addr(unsigned long size)void qemu_free (void *ptr) { +leakcheck_log ('f' ,NULL , ptr,0 ,__builtin_return_address (0 ));free (ptr); }
@@ -51,7 +54,7 @@ static int allow_zero_malloc(void)#endif } -void *qemu_malloc (size_t size) +static void *qemu_malloc_common (size_t size) {if (!size && !allow_zero_malloc ()) {abort ();
@@ -59,19 +62,30 @@ void *qemu_malloc(size_t size)return oom_check (malloc (size ? size :1 )); } +void *qemu_malloc (size_t size) +{ +void *p =qemu_malloc_common (size); +leakcheck_log ('m' ,NULL , p, size,__builtin_return_address (0 )); +return p; +} +void *qemu_realloc (void *ptr,size_t size) {if (!size && !allow_zero_malloc ()) {abort (); } -return oom_check (realloc (ptr, size ? size :1 )); + size = size ? size :1 ; +void *p =oom_check (realloc (ptr, size)); +leakcheck_log ('r' , ptr, p, size,__builtin_return_address (0 )); +return p; }void *qemu_mallocz (size_t size) {void *ptr; - ptr =qemu_malloc (size); + ptr =qemu_malloc_common (size);memset (ptr,0 , size); +leakcheck_log ('z' ,NULL , ptr, size,__builtin_return_address (0 ));return ptr; }
@@ -79,8 +93,9 @@ char *qemu_strdup(const char *str) {char *ptr;size_t len =strlen (str); - ptr =qemu_malloc (len +1 ); + ptr =qemu_malloc_common (len +1 );memcpy (ptr, str, len +1 ); +leakcheck_log ('s' ,NULL , ptr, len +1 ,__builtin_return_address (0 ));return ptr; }
@@ -93,8 +108,9 @@ char *qemu_strndup(const char *str, size_t size) size = end - str; } - new =qemu_malloc (size +1 ); + new =qemu_malloc_common (size +1 ); new[size] =0 ; +leakcheck_log ('s' ,NULL , new, size +1 ,__builtin_return_address (0 ));return memcpy (new, str, size); }
Signature · 1 line
1.7.0
Re: [PATCH] block: Free iovec arrays allocated by multiwrite_merge()
Quoted 20 lines · Stefan Hajnoczi
From: Stefan Hajnoczi<[email protected]>
The
MALLOC_TRACEoutput didn't look useful when I tried it either.Instead I used the following to find origin of the leak. Still very basic but works better with
qemu_malloc()and friends.This is just a hack but I wanted to share it in case someone finds it useful in the future.
new file mode 100644
index 0000000..a5fa51aleakcheck.c new file +4
-0@@ -0,0 +1,17 @@ +#include <stdio.h> + +static FILE *fp; +
leakcheck.c line 14
Quoted 10 lines
leakcheck.c new file +10
-0+extern void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1); + +void leakcheck_log (char action,void *old_addr,void *addr,size_t size,void *ret1) +{ +if (!fp) { + fp =fopen ("/tmp/leakcheck.log" ,"w" ); +if (!fp) { +return ; + } + }
use backtrace(3) or backtrace_symbols(3), or even better, an external tool?
Signature · 1 line
error compiling committee.c: too many arguments to function