Fix memory leak in xend

PATCH In review Jim Fehlig 3 participants 7 messages Jul 10, 2008 – Aug 29, 2008

Share & tools

Jim Fehlig

author

2008-07-10 22:28 UTC #0 lore raw

PATCH 1/1 Fix memory leak in xend

We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.

Cheers,
Jim

Signed-off-by Jim Fehlig <[email protected]>

tools/python/xen/xend/XendDomainInfo.py +1 -0

diff -r bd97e45e073a tools/python/xen/xend/XendDomainInfo.py
@@ -2607,6 +2607,7 @@ class XendDomainInfo:
     def _cleanupVm(self):
         """Cleanup VM resources.  Idempotent.  Nothrow guarantee."""
 
+        self.metrics.destroy()
         self._unwatchVm()
 
         try:

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Stefan de Konink

2008-08-28 21:52 UTC #1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Jim Fehlig schreef:

Quoted 3 lines

We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.

Did you see problems after this patch? I have applied it on 3.2.1 which borks about missing vm_metrics.

Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEAREKAAYFAki3HhkACgkQYH1+F2Rqwn049gCeIYOm5MSf/NQfgdb9TC009wjS 9l8An0PeV5xp/jYamySsSACJO1OI2Jp7
=bJwS
-----END PGP SIGNATURE-----

Zhigang Wang

2008-08-29 01:14 UTC #2 lore raw

I also encountered this many times. I come up with a patch.

Jim, please review.

thanks,

zhigang

Quoted 22 lines · Stefan de Konink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jim Fehlig schreef:
Quoted 3 lines
We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.
Did you see problems after this patch? I have applied it on 3.2.1 which borks about missing vm_metrics.
Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAki3HhkACgkQYH1+F2Rqwn049gCeIYOm5MSf/NQfgdb9TC009wjS 9l8An0PeV5xp/jYamySsSACJO1OI2Jp7
=bJwS
-----END PGP SIGNATURE-----

Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

This patch prevent VM metrics destroy from raising exceptions.

Everytime an error accured during VM start, VM metrics destroy will raise an exception and override the right exception which should be raised to the upper invocation.

Signed-off-by Zhigang Wang <[email protected]>

tools/python/xen/xend/XendDomainInfo.py +5 -1

@@ -2656,7 +2656,11 @@ class XendDomainInfo:
 
         from xen.xend import XendDomain
         if not XendDomain.instance().is_domain_managed(self):
-            self.metrics.destroy()
+            try:
+                self.metrics.destroy()
+            except:
+                log.exception("Destroy VM metrics failed.")
+
         self._unwatchVm()
 
         try:

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Stefan de Konink

2008-08-29 01:42 UTC #3 lore raw

Quoted 36 lines · Zhigang Wang

I also encountered this many times. I come up with a patch.
Jim, please review.
thanks,
zhigang
Quoted 26 lines · Stefan de Konink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jim Fehlig schreef:
Quoted 3 lines
We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.
Did you see problems after this patch? I have applied it on 3.2.1 which borks about missing vm_metrics.
Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAki3HhkACgkQYH1+F2Rqwn049gCeIYOm5MSf/NQfgdb9TC009wjS 9l8An0PeV5xp/jYamySsSACJO1OI2Jp7
=bJwS
-----END PGP SIGNATURE-----
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

The patch you provide seems to be for the 3.3 version right? The is_managed attribute/function, is it available for 3.2? Can anyone backport this?

(I think giving me a reference to a kernel that is older than 2.6.18 and supports x86_64 + Xen 3.3 is more work ;)

Stefan

Zhigang Wang

2008-08-29 01:54 UTC #4 lore raw

Quoted 32 lines · Stefan de Konink

Quoted 30 lines · Zhigang Wang
I also encountered this many times. I come up with a patch.
Jim, please review.
thanks,
zhigang
Quoted 21 lines · Stefan de Konink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jim Fehlig schreef:
Quoted 3 lines
We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.
Did you see problems after this patch? I have applied it on 3.2.1 which borks about missing vm_metrics.
Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEAREKAAYFAki3HhkACgkQYH1+F2Rqwn049gCeIYOm5MSf/NQfgdb9TC009wjS 9l8An0PeV5xp/jYamySsSACJO1OI2Jp7
=bJwS
-----END PGP SIGNATURE-----

Quoted 9 lines

Quoted 4 lines
Quoted 4 lines
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
The patch you provide seems to be for the 3.3 version right? The is_managed attribute/function, is it available for 3.2? Can anyone backport this?

I don't think this patch can solve your VM start problem. It only gives your more accurate description of the problem you are encountering. you should check /var/log/xen/xend-debug.log to see which is the right causing of this problem.

Quoted 2 lines

(I think giving me a reference to a kernel that is older than 2.6.18 and supports x86_64 + Xen 3.3 is more work ;)

Xen 3.3 has the same ABI of 3.0+. you can try the latest fedora kernels.

zhigang

Quoted 4 lines

Stefan

Stefan de Konink

2008-08-29 01:58 UTC #5 lore raw

Quoted 8 lines · Zhigang Wang

Quoted 4 lines
The patch you provide seems to be for the 3.3 version right? The is_managed attribute/function, is it available for 3.2? Can anyone backport this?
I don't think this patch can solve your VM start problem. It only gives your more accurate description of the problem you are encountering. you should check /var/log/xen/xend-debug.log to see which is the right causing of this problem.

The problem that I encounter like Jim are massive python processes... that will eventually OOM. So only in that perspective I want a solution.

Quoted 3 lines

Quoted 2 lines
(I think giving me a reference to a kernel that is older than 2.6.18 and supports x86_64 + Xen 3.3 is more work ;)
Xen 3.3 has the same ABI of 3.0+. you can try the latest fedora kernels.

I'm running on a Gentoo 2.6.21 Xen patched kernel now, so if I can use that within 3.3 I'll give it a shot. (And apply your patch)

Stefan

Jim Fehlig

author

2008-08-29 05:28 UTC #6 lore raw

In reply to Stefan de Konink

Quoted 5 lines · Stefan de Konink

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Jim Fehlig schreef:memory leak

Quoted 8 lines

Quoted 4 lines
We observed OOM kills of xend after ~20 days in openSUSE build service workers. xend reached around 2.5 G virt, 1.5G res, and 1.8G shared memory usage. Attached patch provides a plug.
Did you see problems after this patch? I have applied it on 3.2.1 which borks about missing vm_metrics.

Yeah :-(. The patch got committed as c/s 18030. You'll also want 18064 - and maybe even 18140.

See http://lists.xensource.com/archives/html/xen-devel/2008-07/msg01230.html

Jim