Swap usage with KVM

DISCUSSION Daniel Bareiro 5 participants 15 messages Jul 11, 2010 – Aug 27, 2010

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

Daniel Bareiro

author

2010-07-11 15:13 UTC #0 lore raw

Hi all!

I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:

Hostname | RAM
===============+===============

Aps4           |    7 GiB
Leela          |    7 GiB

===============+===============
TOTAL | 14 GiB

Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?

I copy the Nagios Service Log Entries for the VMHost:

Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
99% free (939 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
79% free (754 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 07:08:17 0d 7h 8m 17s SERVICE OK (HARD) SWAP OK -
51% free (482 MB out of 956 MB)
11-07-2010 07:08:17 11-07-2010 10:41:07 0d 3h 32m 50s SERVICE WARNING (HARD) SWAP WARNING
- 29% free (272 MB out of 956 MB)
11-07-2010 10:41:07 11-07-2010 10:45:57 0d 0h 4m 50s SERVICE CRITICAL (HARD) SWAP
CRITICAL - 9% free (83 MB out of 956 MB)
---------------------------------------------------------------------------------------------------

I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical volume that maintains the logical volumes that are used for VM's disks.

The Nagios Service Log Entries for VMs shows no excessive use of swap in the window of time when the problem occurred in the VMHost:

Aps4:

Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (2850 MB out of 2863 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2797 MB out of 2863 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
99% free (2812 MB out of 2863 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2784 MB out of 2863 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (2754 MB out of 2863 MB)
11-07-2010 00:00:00 11-07-2010 11:53:38 0d 11h 53m 38s+ SERVICE OK (HARD) SWAP OK -
100% free (2839 MB out of 2863 MB)
---------------------------------------------------------------------------------------------------

Leela:

Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (919 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 17:01:36 08-07-2010 17:04:16 0d 0h 2m 40s SERVICE CRITICAL (HARD) Connection
refused by host
08-07-2010 17:04:16 09-07-2010 00:00:00 0d 6h 55m 44s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 11:58:29 0d 11h 58m 29s+ SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
---------------------------------------------------------------------------------------------------

Unfortunately I could not take much more data because we had to act quickly, but if you need any additional information, please feel free to ask.

Thanks in advance for your replies.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Daniel Bareiro

author

2010-07-11 19:12 UTC #1 lore raw

Quoted 94 lines · Daniel Bareiro

I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Hostname | RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL | 14 GiB
Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?
I copy the Nagios Service Log Entries for the VMHost:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
100% free (956 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
99% free (939 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
79% free (754 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 07:08:17 0d 7h 8m 17s SERVICE OK (HARD) SWAP OK -
51% free (482 MB out of 956 MB)
11-07-2010 07:08:17 11-07-2010 10:41:07 0d 3h 32m 50s SERVICE WARNING (HARD) SWAP WARNING
- 29% free (272 MB out of 956 MB)
11-07-2010 10:41:07 11-07-2010 10:45:57 0d 0h 4m 50s SERVICE CRITICAL (HARD) SWAP
CRITICAL - 9% free (83 MB out of 956 MB)
---------------------------------------------------------------------------------------------------
I'm not using qcow2 files. The /dev/cciss/c0d0p3 partition is a physical volume that maintains the logical volumes that are used for VM's disks.
The Nagios Service Log Entries for VMs shows no excessive use of swap in the window of time when the problem occurred in the VMHost:
Aps4:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
100% free (2850 MB out of 2863 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2797 MB out of 2863 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
99% free (2812 MB out of 2863 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
98% free (2784 MB out of 2863 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (2754 MB out of 2863 MB)
11-07-2010 00:00:00 11-07-2010 11:53:38 0d 11h 53m 38s+ SERVICE OK (HARD) SWAP OK -
100% free (2839 MB out of 2863 MB)
---------------------------------------------------------------------------------------------------
Leela:
Event Start Time Event End Time Event Duration Event/State Type Event/State
Information
---------------------------------------------------------------------------------------------------
06-07-2010 00:00:00 07-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (919 MB out of 956 MB)
07-07-2010 00:00:00 08-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 00:00:00 08-07-2010 16:41:43 0d 16h 41m 43s SERVICE OK (HARD) SWAP OK -
97% free (920 MB out of 956 MB)
08-07-2010 17:01:36 08-07-2010 17:04:16 0d 0h 2m 40s SERVICE CRITICAL (HARD) Connection
refused by host
08-07-2010 17:04:16 09-07-2010 00:00:00 0d 6h 55m 44s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
09-07-2010 00:00:00 10-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
10-07-2010 00:00:00 11-07-2010 00:00:00 1d 0h 0m 0s SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
11-07-2010 00:00:00 11-07-2010 11:58:29 0d 11h 58m 29s+ SERVICE OK (HARD) SWAP OK -
97% free (921 MB out of 956 MB)
---------------------------------------------------------------------------------------------------

Quoted 3 lines

Unfortunately I could not take much more data because we had to act quickly, but if you need any additional information, please feel free to ask.

Has anyone experienced something like this? Avi? I remember late last year there was a regression in Linux swapping and Rik and Hugh were working on it. Are you aware of any?

Thanks in advance for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Rik van Riel

2010-07-11 21:49 UTC #2 lore raw

Quoted 17 lines · Daniel Bareiro

Quoted 14 lines · Daniel Bareiro
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Hostname | RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL | 14 GiB

Quoted 5 lines

Quoted 5 lines
Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?

That depends on what is going on in the host.

Did you notice any performance issues in the guest when you started using swap?

Quoted 3 lines

Has anyone experienced something like this? Avi? I remember late last year there was a regression in Linux swapping and Rik and Hugh were working on it. Are you aware of any?

That one should have been fixed.

Signature · 1 line

All rights reversed

Daniel Bareiro

2010-08-02 18:57 UTC #3 lore raw

Hi, Rik.

Quoted 20 lines · Rik van Riel

Quoted 14 lines
Quoted 14 lines
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Hostname | RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL | 14 GiB
Quoted 5 lines
Quoted 5 lines
Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?

Quoted 4 lines

That depends on what is going on in the host.
Did you notice any performance issues in the guest when you started using swap?

After the logical volume of 1 GB that I added when I found this problem (being the operating system with 2 GB), I added other 3 GB to have a little more margin, but today I got again a new alert of Nagios:

Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)

I don't see performance issue in the VMs.

Marcelo Tosatti recommended me to apply the EPT patch (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in production? Are there plans that this patch is applied on some version of Linux stable?

Also it can be advisable to update to qemu-kvm 0.12.4 considering what David Weber said:

http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599

Thanks for your reply.

Regards,
Daniel

Signature · 8 lines

Mi frase del día:
Collaboration, n.:
	A literary partnership based on the false assumption that the
	other fellow can spell.

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
15:28:17 up 23 days, 20:20, 12 users,  load average: 0.16, 0.14, 0.09

Rik van Riel

2010-08-02 19:31 UTC #4 lore raw

Quoted 38 lines · Daniel Bareiro

Hi, Rik.
Quoted 20 lines · Rik van Riel
Quoted 14 lines
Quoted 14 lines
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Hostname | RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL | 14 GiB
Quoted 5 lines
Quoted 5 lines
Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?
Quoted 4 lines
That depends on what is going on in the host.
Did you notice any performance issues in the guest when you started using swap?
After the logical volume of 1 GB that I added when I found this problem (being the operating system with 2 GB), I added other 3 GB to have a little more margin, but today I got again a new alert of Nagios:
Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)

Quoted 6 lines

I don't see performance issue in the VMs.
Marcelo Tosatti recommended me to apply the EPT patch (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in production? Are there plans that this patch is applied on some version of Linux stable?

Yes, that patch is safe and definitely recommended.

Signature · 1 line

All rights reversed

Daniel Bareiro

2010-08-02 19:53 UTC #5 lore raw

Hi, Rik.

Quoted 34 lines · Rik van Riel

Quoted 34 lines
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Hostname | RAM
===============+===============
Aps4           |    7 GiB
Leela          |    7 GiB
===============+===============
TOTAL | 14 GiB
Quoted 5 lines
Quoted 5 lines
Initially the host was created with a swap partition of 1 GiB, but today we found that the use of swap quickly began to grow increasingly. Therefore, as a contingency, we had to hot-add a logical volume of 1 GB of swap on the VMHost. Is 'normal' this use of memory?
Quoted 4 lines
That depends on what is going on in the host.
Did you notice any performance issues in the guest when you started using swap?
After the logical volume of 1 GB that I added when I found this problem (being the operating system with 2 GB), I added other 3 GB to have a little more margin, but today I got again a new alert of Nagios:
Swap usage WARNING [...] SWAP WARNING - 30% free (1490 MB out of 5052 MB)

Quoted 8 lines

Quoted 6 lines
I don't see performance issue in the VMs.
Marcelo Tosatti recommended me to apply the EPT patch (6316e1c8c6af6ccb55ff8564231710660608f46c). Should it be safe for use in production? Are there plans that this patch is applied on some version of Linux stable?
Yes, that patch is safe and definitely recommended.

Perfect. Thanks for so fast answer.

And there are some estimates of when this patch is in Linux stable?

Regards,
Daniel

Signature · 9 lines

Mi frase del día:
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
16:44:53 up 23 days, 21:37, 11 users,  load average: 0.02, 0.06, 0.08

Rik van Riel

2010-08-02 20:22 UTC #6 lore raw

Quoted 1 line · Daniel Bareiro

And there are some estimates of when this patch is in Linux stable?

It should be there already in 2.6.33-stable and 2.6.34-stable.

Signature · 1 line

All rights reversed

Daniel Bareiro

2010-08-02 21:43 UTC #7 lore raw

Quoted 3 lines · Rik van Riel

Quoted 1 line
And there are some estimates of when this patch is in Linux stable?
It should be there already in 2.6.33-stable and 2.6.34-stable.

Great! Thanks for your reply.

Regards,
Daniel

Signature · 8 lines

Mi frase del día:
i dont even know if it makes sense at all :) This is an experimental patch
for an experimental kernel :))
		-- Ingo Molnar on linux-kernel

Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
18:38:51 up 23 days, 23:31, 11 users,  load average: 0.17, 0.16, 0.10

Daniel Bareiro

2010-08-27 10:04 UTC #8 lore raw

Re: Swap usage with KVM (and KSM)

In reply to Rik van Riel

Hi, Rik.

Quoted 3 lines · Rik van Riel

Quoted 1 line
And there are some estimates of when this patch is in Linux stable?
It should be there already in 2.6.33-stable and 2.6.34-stable.

In a test environment I have Linux 2.6.34.5 compiled with the source code from kernel.org and qemu-kvm 0.12.4 installed from the Backports repository of Debian GNU/Linux.

In this environment I have running the following virtual machines:

* 2 x OpenBSD (2 x 512 MB)            |
* 2 x Debian GNU/Linux (2 x 256 MB)   |
* 2 x Debian GNU/Linux (2 x 512 MB)   +--= 11008 MB
* 1 x Centos 5.5 (1 x 256 MB)         |
* 8 x Debian GNU/Linux (8 x 1024 MB)  |

I'm just starting to test KSM, so I was using as reference the read of the tests made by RedHat about run as many as 52 Windows XP VMs with 1 GB of RAM each on a server with just 16 GB of RAM.

In the previous case the ratio would be 52/16 = 3.25. In my case the VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH tests do not talk about the amount of swap used in that case, so I wonder if a distribution of VMs as I have, it is normal to use 290 MB of swap.

Regards,
Daniel

Signature · 9 lines

Mi frase del día:
BOFH excuse #335:

the AA battery in the wallclock sends magnetic interference


Daniel Bareiro - GNU/Linux registered user #188.598
Proudly running Debian GNU/Linux with uptime:
05:57:46 up 48 days, 10:50, 11 users,  load average: 0.01, 0.04, 0.06

Rik van Riel

2010-08-27 13:42 UTC #9 lore raw

Re: Swap usage with KVM (and KSM)

Quoted 5 lines · Daniel Bareiro

In the previous case the ratio would be 52/16 = 3.25. In my case the VMHost has 4 GB of RAM, so the ratio would be 10.75/4 = 2.6875. In RH tests do not talk about the amount of swap used in that case, so I wonder if a distribution of VMs as I have, it is normal to use 290 MB of swap.

Given that you have 10.75GB worth of virtual machine on a system with 4GB of RAM, I'd say that 290MB of swap is not out of the ordinary.

Signature · 1 line

All rights reversed

Freddie Cash

2010-07-11 21:05 UTC #10 lore raw

In reply to Daniel Bareiro

Quoted 6 lines · Daniel Bareiro

I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:

Are you using virtio drivers in the VMs?

There was an issue with KVM-72 and virtio that leaks memory in the host until all RAM and swap is used (inside the VMs, no swap is used).
It was supposed to be fixed in KVM-80-something, though.

Perhaps something similar is happening again? If you switch the disks to scsi instead of virtio, does the problem go away?

We are running KVM-72 on Debian 5.0 and have run into this issue. We'll be upgrading our hosts this month to fix this.

Signature · 2 lines

Freddie Cash
[email protected]

Daniel Bareiro

author

2010-07-11 22:09 UTC #11 lore raw

Hi, Freddie.

Quoted 13 lines · Freddie Cash

Quoted 6 lines
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Are you using virtio drivers in the VMs?
There was an issue with KVM-72 and virtio that leaks memory in the host until all RAM and swap is used (inside the VMs, no swap is used). It was supposed to be fixed in KVM-80-something, though.

Quoted 5 lines

Perhaps something similar is happening again? If you switch the disks to scsi instead of virtio, does the problem go away?
We are running KVM-72 on Debian 5.0 and have run into this issue. We'll be upgrading our hosts this month to fix this.

Yes, we are using Virtio drivers for networking and storage in both VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from Lenny Backports repositories. For VMHost, we are using a stable version of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM.

This is the syntax I'm using to boot the virtual machines:

8587 ? Sl 6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio -drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait

9769 ? Rl 11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio -drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio -drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio -drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait

To make the switch from Virtio to SCSI I would have to shut down the hosts, which would not be a good idea whereas are two productive systems. At least, before doing so I would be sure of what might be the problem.

Taking a current measurement in VMHost with free, I got the following:

ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16406504      55084          0       2920      21504
-/+ buffers/cache:   16382080      79508
Swap:      2028492     983140    1045352

It draws attention to me that thinking about initially leaving a margin of 2 GB of RAM for the VMHost, already it has used almost half of swap.

Thanks for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

Daniel Bareiro

author

2010-07-20 15:04 UTC #12 lore raw

Quoted 63 lines · Daniel Bareiro

Quoted 13 lines
Quoted 6 lines
I have an installation with Debian GNU/Linux 5.0.4 amd64 with qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM and Linux 2.6.32.12 compiled from source code of kernel.org. All this is installed on an HP Proliant DL380 G6 with two Xeon E5530 quadcore processors and 16 GiB of RAM which has two VMs with the following configuration of memory:
Are you using virtio drivers in the VMs?
There was an issue with KVM-72 and virtio that leaks memory in the host until all RAM and swap is used (inside the VMs, no swap is used). It was supposed to be fixed in KVM-80-something, though.
Quoted 5 lines
Perhaps something similar is happening again? If you switch the disks to scsi instead of virtio, does the problem go away?
We are running KVM-72 on Debian 5.0 and have run into this issue. We'll be upgrading our hosts this month to fix this.
Yes, we are using Virtio drivers for networking and storage in both VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from Lenny Backports repositories. For VMHost, we are using a stable version of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM.
This is the syntax I'm using to boot the virtual machines:
8587 ? Sl 6515:25 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/aps4-raiz,cache=none,if=virtio,boot=on -drive file=/dev/vm/aps4-cache,cache=none,if=virtio -drive file=/dev/vm/aps4-index,cache=none,if=virtio -drive file=/dev/vm/aps4-space,cache=none,if=virtio -m 7168 -smp 4 -net nic,model=virtio,macaddr=00:16:3e:00:00:95 -net tap -daemonize -vnc :3 -k es -localtime -monitor telnet:localhost:4003,server,nowait -serial telnet:localhost:4043,server,nowait
9769 ? Rl 11968:47 /usr/local/qemu-kvm/bin/qemu-system-x86_64 -drive
file=/dev/vm/leela-raiz,cache=none,if=virtio,boot=on -drive file=/dev/vm/leela-u01,cache=none,if=virtio -drive file=/dev/vm/leela-u02,cache=none,if=virtio -drive file=/dev/vm/leela-u03,cache=none,if=virtio -drive file=/dev/vm/leela-u04,cache=none,if=virtio -drive file=/dev/vm/leela-u05,cache=none,if=virtio -drive file=/dev/vm/leela-u06,cache=none,if=virtio -drive file=/dev/vm/leela-u07,cache=none,if=virtio -drive file=/dev/vm/leela-u08,cache=none,if=virtio -drive file=/dev/vm/leela-u09,cache=none,if=virtio -drive file=/dev/vm/leela-space,cache=none,if=virtio -m 7168 -smp 8 -net nic,model=virtio,macaddr=00:16:3e:00:00:96 -net tap -daemonize -vnc :4 -k es -localtime -monitor telnet:localhost:4004,server,nowait -serial telnet:localhost:4044,server,nowait
To make the switch from Virtio to SCSI I would have to shut down the hosts, which would not be a good idea whereas are two productive systems. At least, before doing so I would be sure of what might be the problem.
Taking a current measurement in VMHost with free, I got the following:
ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16406504      55084          0       2920      21504
-/+ buffers/cache:   16382080      79508
Swap:      2028492     983140    1045352

Quoted 2 lines

It draws attention to me that thinking about initially leaving a margin of 2 GB of RAM for the VMHost, already it has used almost half of swap.

This is a current measurement I've taken in both the VMs and in VMHost:

* VMHost:

ss04:~# free
             total       used       free     shared    buffers     cached
Mem:      16461588   16405140      56448          0       3496      18604
-/+ buffers/cache:   16383040      78548
Swap:      5174220    2401552    2772668

* Aps4:

aps4:~# free
             total       used       free     shared    buffers     cached
Mem:       7164300    7120192      44108          0      23108     239076
-/+ buffers/cache:    6858008     306292
Swap:      2931820      14084    2917736

* Leela:

leela:~# free
             total       used       free     shared    buffers     cached
Mem:       7163836    6905224     258612          0     123380    6282816
-/+ buffers/cache:     499028    6664808
Swap:       979924      35640     944284

As you can see, I added more swap in VMHost for more margin, but currently only 54% is free.

Thanks in advance for your replies.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

David Weber

2010-07-20 19:18 UTC #13 lore raw

In reply to Daniel Bareiro

Quoted 8 lines

Yes, we are using Virtio drivers for networking and storage in both VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from Lenny Backports repositories. For VMHost, we are using a stable version of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM.

Afaik this should be this bug
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599

try upgrading to 0.12.4 or backport this commit
http://git.kernel.org/?p=virt/kvm/qemu-
kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960

David

Daniel Bareiro

author

2010-07-22 10:09 UTC #14 lore raw

Hi, David.

Quoted 7 lines · David Weber

Quoted 6 lines
Yes, we are using Virtio drivers for networking and storage in both VMs with cache=none. Both VMs are running Linux 2.6.32-bpo.5-amd64 from Lenny Backports repositories. For VMHost, we are using a stable version of KVM with Linux 2.6.32.12 compiled from source code of kernel.org and qemu-kvm 0.12.3 compiled with the source code obtained from the official site of KVM.

Quoted 6 lines

Afaik this should be this bug
http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2989366&group_id=180599
try upgrading to 0.12.4 or backport this commit
http://git.kernel.org/?p=virt/kvm/qemu-
kvm.git;a=commit;h=012d4869c1eb195e83f159ed7b2bced33f37f960

Interesting... I'll try upgrading to qemu-kvm 0.12.4.

Marcelo Tosatti also recommended me to apply this patch I'm attaching. Someone could confirm whether this is already included in Linux stable?

Thanks for your reply.

Regards,
Daniel

Signature · 2 lines

Fingerprint: BFB3 08D6 B4D1 31B2 72B9  29CE 6696 BF1B 14E6 1D37
Powered by Debian GNU/Linux Lenny - Linux user #188.598

commit 6316e1c8c6af6ccb55ff8564231710660608f46c

Author: Rik van Riel <[email protected]>
Date:   Wed Feb 3 16:11:03 2010 -0500

KVM: VMX: emulate accessed bit for EPT

Currently KVM pretends that pages with EPT mappings never got accessed. This has some side effects in the VM, like swapping out actively used guest pages and needlessly breaking up actively used hugepages.

We can avoid those very costly side effects by emulating the accessed bit for EPT PTEs, which should only be slightly costly because pages pass through page_referenced infrequently.

TLB flushing is taken care of by kvm_mmu_notifier_clear_flush_young().

This seems to help prevent KVM guests from being swapped out when they should not on my system.

Signed-off-by Rik van Riel <[email protected]>

Signed-off-by Avi Kivity <[email protected]>

arch/x86/kvm/mmu.c +8 -2

index 913ef4b..b8da671 100644
@@ -838,9 +838,15 @@ static int kvm_age_rmapp(struct kvm *kvm, unsigned long *rmapp,
 	u64 *spte;
 	int young = 0;
 
-	/* always return old for EPT */
+	/*
+	 * Emulate the accessed bit for EPT, by checking if this page has
+	 * an EPT mapping, and clearing it if it does. On the next access,
+	 * a new EPT mapping will be established.
+	 * This has some overhead, but not as much as the cost of swapping
+	 * out actively used pages or breaking up actively used hugepages.
+	 */
 	if (!shadow_accessed_mask)
-		return 0;
+		return kvm_unmap_rmapp(kvm, rmapp, data);
 
 	spte = rmap_next(kvm, rmapp, NULL);
 	while (spte) {