maybe a virtio-balloon-device issue ?

DISCUSSION Mario Smarduch 2 participants 4 messages Feb 6, 2015 – Feb 20, 2015

Subsystems (4)

KVM virtio core KVM arm64 virtio balloon

lore

Share & tools

lore

Mario Smarduch

author

2015-02-06 01:19 UTC #0 lore raw

Hi,

I'm looking into qemu/balloon driver VM overcommit. I noticed virtio-balloon driver will take any setting from virtio-balloon-device to the point Guest dies.

For a 1G guest

$ sudo echo balloon 100 | socat - tcp4-connect:127.0.0.1:4444

you get (same with libvirt setmem)

root@localhost:~# free
-bash: fork: Cannot allocate memory
root@localhost:~# ps
-bash: fork: Cannot allocate memory

$ sudo info balloon | socat ... - confirms setting

The balloon driver has been there for a while, not sure what I'm missing?

virtio-balloon-device provide free memory, i.e., - externally accessible to host. But this appears more like a hint for an inflate request, snmp mibs
provide more detailed resource info then that.

I'm wondering if the driver should not have some heuristic check for an inflate request so it doesn't over inflate? Similar to kernel overcommit.

Thanks.

Paolo Bonzini

maintainer

2015-02-06 15:57 UTC #1 lore raw

Quoted 20 lines · Mario Smarduch

Hi,
I'm looking into qemu/balloon driver VM overcommit. I noticed virtio-balloon driver will take any setting from virtio-balloon-device to the point Guest dies.
For a 1G guest
$ sudo echo balloon 100 | socat - tcp4-connect:127.0.0.1:4444
you get (same with libvirt setmem)
root@localhost:~# free
-bash: fork: Cannot allocate memory
root@localhost:~# ps
-bash: fork: Cannot allocate memory
$ sudo info balloon | socat ... - confirms setting
The balloon driver has been there for a while, not sure what I'm missing?

Quoted 8 lines

virtio-balloon-device provide free memory, i.e., - externally accessible to host. But this appears more like a hint for an inflate request, snmp mibs
provide more detailed resource info then that.
I'm wondering if the driver should not have some heuristic check for an inflate request so it doesn't over inflate? Similar to kernel overcommit.

See this patch:

commit 5a10b7dbf904bfe01bb9fcc6298f7df09eed77d5

Author: Raushaniya Maksudova <[email protected]>
Date:   Mon Nov 10 09:36:29 2014 +1030

virtio_balloon: free some memory from balloon on OOM

Looks like the QEMU part was never posted though.

Paolo

Mario Smarduch

author

2015-02-09 17:38 UTC #2 lore raw

Thanks for the pointer I'll try it out.

- mario

Quoted 44 lines · Paolo Bonzini

Quoted 20 lines · Mario Smarduch
Hi,
I'm looking into qemu/balloon driver VM overcommit. I noticed virtio-balloon driver will take any setting from virtio-balloon-device to the point Guest dies.
For a 1G guest
$ sudo echo balloon 100 | socat - tcp4-connect:127.0.0.1:4444
you get (same with libvirt setmem)
root@localhost:~# free
-bash: fork: Cannot allocate memory
root@localhost:~# ps
-bash: fork: Cannot allocate memory
$ sudo info balloon | socat ... - confirms setting
The balloon driver has been there for a while, not sure what I'm missing?
Quoted 8 lines
virtio-balloon-device provide free memory, i.e., - externally accessible to host. But this appears more like a hint for an inflate request, snmp mibs
provide more detailed resource info then that.
I'm wondering if the driver should not have some heuristic check for an inflate request so it doesn't over inflate? Similar to kernel overcommit.
See this patch:
commit 5a10b7dbf904bfe01bb9fcc6298f7df09eed77d5
Author: Raushaniya Maksudova <[email protected]>
Date:   Mon Nov 10 09:36:29 2014 +1030
virtio_balloon: free some memory from balloon on OOM
Looks like the QEMU part was never posted though.
Paolo

Mario Smarduch

author

2015-02-20 02:46 UTC #3 lore raw

In reply to Paolo Bonzini

I've been testing the patch few days, the bolloon OOM notifier patch seems to
do a good jobs freeing memory before OOM killer is about to kill a process. Although on few ocassions it wasn't enough.

But if you're memory target is anywhere within some range of freeram, and it's hard to say how much but 3 to 6% or less, guest locks up, with stack backtraces from vballoon thread, or other processes and page reclaim code prints messages. Also I get bad page state errors. Admin looses access to guest.

I tried to use the 'admin_reserve_kbytes' approach, create a buffer range - and
sanity check the target value. I have not seen any issues with loads I'm running but it's still may be susceptible to a rapid free memory drop while balloon is inflating. Of course making the gap larger (min_reserve) makes it less susceptible or removes it altogether. I included sample patch.

Would appreciate suggestions, on how to make inflate a safe operation, so in the worst case aggressive page reclaim, or OOM killer kick in.
I tested on ARM & x86.

- Mario

drivers/virtio/virtio_balloon.c +0 -0

@@ -284,12 +284,22 @@ static void virtballoon_changed(struct

virtio_device *vdev)
static inline s64 towards_target(struct virtio_balloon *vb)

 {
        __le32 v;
-       s64 target;
+       s64 diff;
+       unsigned long min_reserve;
+       struct sysinfo si;

        virtio_cread(vb->vdev, struct virtio_balloon_config, num_pages, &v);

-       target = le32_to_cpu(v);
-       return target - vb->num_pages;
+       diff = (s64) le32_to_cpu(v) - vb->num_pages;
+       si_meminfo(&si);

+
+ min_reserve = min((unsigned long) (si.freeram - (3 *
si.freeram/100)),
+ (unsigned long) (0x800000 >> PAGE_SHIFT));
+
+ if(diff == 0 || diff >= (si.freeram - min_reserve))
+ return 0;
+
+ return diff;
}

Quoted 44 lines · Paolo Bonzini

Quoted 20 lines · Mario Smarduch
Hi,
I'm looking into qemu/balloon driver VM overcommit. I noticed virtio-balloon driver will take any setting from virtio-balloon-device to the point Guest dies.
For a 1G guest
$ sudo echo balloon 100 | socat - tcp4-connect:127.0.0.1:4444
you get (same with libvirt setmem)
root@localhost:~# free
-bash: fork: Cannot allocate memory
root@localhost:~# ps
-bash: fork: Cannot allocate memory
$ sudo info balloon | socat ... - confirms setting
The balloon driver has been there for a while, not sure what I'm missing?
Quoted 8 lines
virtio-balloon-device provide free memory, i.e., - externally accessible to host. But this appears more like a hint for an inflate request, snmp mibs
provide more detailed resource info then that.
I'm wondering if the driver should not have some heuristic check for an inflate request so it doesn't over inflate? Similar to kernel overcommit.
See this patch:
commit 5a10b7dbf904bfe01bb9fcc6298f7df09eed77d5
Author: Raushaniya Maksudova <[email protected]>
Date:   Mon Nov 10 09:36:29 2014 +1030
virtio_balloon: free some memory from balloon on OOM
Looks like the QEMU part was never posted though.
Paolo