Balloon malfunctions with memory hotplug
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)
3. Hotplug some memory:
(qemu) object_add memory-backend-ram,id=mem1,size=1G (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
4. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)
BUG: The guest now has 2GB of memory, but the balloon thinks
the guest has 1GBOne may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the ram_size global variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotplugged ram_size is _not_ updated and the balloon device breaks.
I see two possible solutions for this problem:
1. In addition to reading ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.
This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
It works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.
Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of ram_size in the balloon device
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...
Quoted 62 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GBOne may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
It works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.
Quoted 5 lines
Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Yes, it is really a problem, balloon can't work well with memory block online/offline now. virtio-balloon can't be notified when memory block online/offline now, actually, we can add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. ( just a simple register_memory_notifier().)
Quoted 11 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon deviceI'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon, why not the memory 'target' size ? Is there any special reason ?
For linux guest, it can always know exactly its current real memory size, but QEMU may not, because guest can do online/offline memory block by themselves.
If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size that should be adjuested. and also can do corresponding action (fill or leak balloon) when there is online/offline memory block occurred.
Quoted 12 lines
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests....
Quoted 65 lines · On Fri, 27 Feb 2015 12:09:20 +0800 zhanghailiang…
Quoted 64 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GBOne may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
It works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.Can we live with that? I guess not, but I'm open for discussion.
Quoted 8 lines
Quoted 3 lines
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Yes, it is really a problem, balloon can't work well with memory block online/offline now. virtio-balloon can't be notified when memory block online/offline now, actually, we can add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. ( just a simple
register_memory_notifier().)
I'm leaning towards applying your series now and do anything that involves the guest on top if needed.
Quoted 7 lines
Quoted 7 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon device
Quoted 7 lines
Quoted 4 lines
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon, why not the memory 'target' size ? Is there any special reason ?
I don't know either. I guess it's just how the balloon feature was designed.
Quoted 6 lines
For linux guest, it can always know exactly its current real memory size, but QEMU may not, because guest can do online/offline memory block by themselves.
If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size that should be adjuested. and also can do corresponding action (fill or leak balloon) when there is online/offline memory block occurred.
I'm not sure this would work as all sorts of races are possible with memory allocations that may occur during or after the calculation is done.
Your series makes the best case work, which is memory is inserted and the guest uses all of it. Today not even the best case works.
Quoted 15 lines
Quoted 13 lines
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests....
Quoted 65 lines · zhanghailiang
Quoted 64 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GBOne may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.htmlIt works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.Can we live with that? I guess not, but I'm open for discussion.
Quoted 8 lines
Quoted 3 lines
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Yes, it is really a problem, balloon can't work well with memory block online/offline now. virtio-balloon can't be notified when memory block online/offline now, actually, we can add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. ( just a simple
register_memory_notifier().)
The Linux driver can come to know, but it can't tell the host out-of-band about it. A new feature / config option can be added so that a guest can update the host on what the current available RAM is.
Quoted 7 lines
Quoted 7 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon device
Quoted 7 lines
Quoted 4 lines
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon, why not the memory 'target' size ? Is there any special reason ?
This is just how the design / code was done for balloon. I've proposed we move to a target-based solution rather than the current way. While drafting the new virtio spec, this was considered, but I lost track of it. The proposal was to just ditch the current balloon, and come up with a new one with a saner design. Don't know who's keeping track of that, though.
BTW another problem for Luiz's option 2 here is we don't want to wait for the guest to reply before making decisions. E.g. the guest could be in S3 mode, and we may wait indefinitely for a reply, blocking everything (the situation is slightly better with more threads, but in older days, blocking for the guest to reply for balloon stats meant the entire qemu froze till the guest replied. That's the reason the feature was disabled).
Quoted 16 lines
For linux guest, it can always know exactly its current real memory size, but QEMU may not, because guest can do online/offline memory block by themselves.
If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size that should be adjuested. and also can do corresponding action (fill or leak balloon) when there is online/offline memory block occurred.
Quoted 9 lines
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...
Not just 32-bit guests; even 64-bit guests restricted with mem= on the cmdline. I know we've discussed this in the past, and I recall virtio-balloon v2 was going to address this all; sadly I've not kept uptodate with it.
Amit
Quoted 9 lines · On Mon, 2 Mar 2015 11:52:34 +0530 Amit Shah <[email protected]> wrote:
Quoted 6 lines
Quoted 6 lines
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...Not just 32-bit guests; even 64-bit guests restricted with mem= on the cmdline.
You're right. So, it's an already existing issue that becomes very apparent with memory hotplug.
Quoted 3 lines
I know we've discussed this in the past, and I recall virtio-balloon v2 was going to address this all; sadly I've not kept uptodate with it.
Me neither :(
Quoted 70 lines · Amit Shah
Quoted 69 lines · zhanghailiang
Quoted 67 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GBOne may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
It works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Quoted 9 lines
Quoted 4 lines
Yes, it is really a problem, balloon can't work well with memory block online/offline now. virtio-balloon can't be notified when memory block online/offline now, actually, we can add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. ( just a simple
register_memory_notifier().)The Linux driver can come to know, but it can't tell the host out-of-band about it. A new feature / config option can be added so that a guest can update the host on what the current available RAM is.
This is a feasible scenario. Maybe Luiz could consider this. He is working on it now.
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon deviceI'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Quoted 6 lines
Quoted 2 lines
Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon, why not the memory 'target' size ? Is there any special reason ?
This is just how the design / code was done for balloon. I've proposed we move to a target-based solution rather than the current way. While drafting the new virtio spec, this was considered, but I
OK, it is really a good idea to redesign virtio-balloon which is target-based.
Quoted 4 lines
lost track of it. The proposal was to just ditch the current balloon, and come up with a new one with a saner design. Don't know who's keeping track of that, though.
:(
Quoted 8 lines
BTW another problem for Luiz's option 2 here is we don't want to wait for the guest to reply before making decisions. E.g. the guest could be in S3 mode, and we may wait indefinitely for a reply, blocking everything (the situation is slightly better with more threads, but in older days, blocking for the guest to reply for balloon stats meant the entire qemu froze till the guest replied. That's the reason the feature was disabled).
Got it. thanks.
Quoted 26 lines
Quoted 16 lines
For linux guest, it can always know exactly its current real memory size, but QEMU may not, because guest can do online/offline memory block by themselves.
If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size that should be adjuested. and also can do corresponding action (fill or leak balloon) when there is online/offline memory block occurred.
Quoted 9 lines
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...Not just 32-bit guests; even 64-bit guests restricted with mem= on the cmdline. I know we've discussed this in the past, and I recall virtio-balloon v2 was going to address this all; sadly I've not kept uptodate with it.
Amit
.
Quoted 23 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
Quoted 8 lines
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GB
Impact other than "info balloon"?
Quoted 35 lines
One may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.htmlIt works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Quoted 6 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon device
What happens when the guest lies?
Quoted 7 lines
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Besides, this solution is more complex than solution 1 and won't address older guests.
Quoted 6 lines
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...
Fun...
Quoted 27 lines · On Fri, 27 Feb 2015 08:27:00 +0100 Markus Armbruster <[email protected]> wrote:
Quoted 25 lines · Luiz Capitulino
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)3. Hotplug some memory:
(qemu)
object_addmemory-backend-ram,id=mem1,size=1G (qemu)device_addpc-dimm,id=dimm1,memdev=mem14. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
Quoted 8 lines
Quoted 6 lines
(qemu) info balloon
balloon: actual=1024
(qemu)BUG: The guest now has 2GB of memory, but the balloon thinks the guest has 1GBImpact other than "info balloon"?
You can only balloon what's reported by "info balloon". If the guest was booted with 1GB but you hot added another 6GB, then you'll only be able to balloon 1GB.
Quoted 35 lines
Quoted 35 lines
One may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the
ram_sizeglobal variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotpluggedram_sizeis _not_ updated and the balloon device breaks.I see two possible solutions for this problem:
1. In addition to reading
ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.htmlIt works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
Quoted 8 lines
Quoted 6 lines
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of
ram_sizein the balloon deviceWhat happens when the guest lies?
There are two kinds of guests that would lie: a broken guest or a malicious guest. For a malicious guest, the worst case I can think of is that we get the same problem we have today. Not a big deal for the host, I guess. A broken guest has to be fixed :)
However, I'm getting to the conclusion that this solution will complicate things even more and may add a bunch of new problems.
I'm leaning towards applying zhanghailiang's series for now. This series fixes the best case: memory is inserted and the guests uses all of it right away.
The worst case is: memory is inserted and the guest doesn't use it. In this case QEMU will allow you to balloon more memory than the guest is using, which can crash the guest. For example, the guest is booted with 1GB you hot add 6GB but the guest doesn't use it. info balloon will report 6GB and will allow you to balloon the guest down to 2GB, which will crash the guest.
In theory I think this case has always been broken, but in practice it's very hard (almost impossible?) to reproduce in a Linux 64-bit guest as you'd have to be able to start the guest with more memory than it can recognize.
Quoted 16 lines
Quoted 7 lines
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Besides, this solution is more complex than solution 1 and won't address older guests.
Quoted 6 lines
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...Fun...
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
I think they was already reported some month ago,
and a patch was submitted to the mailing list (but waiting that memory unplug was merged before apply it)
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
----- Mail original -----
De: "Luiz Capitulino" <[email protected]>
À: "qemu-devel" <[email protected]>
Cc: "kvm" <[email protected]>, "Igor Mammedov" <[email protected]>, "zhang zhanghailiang" <[email protected]>, [email protected], "Eric Blake" <[email protected]>, "Michael S. Tsirkin" <[email protected]>, "amit shah" <[email protected]> Envoyé: Jeudi 26 Février 2015 20:26:29
Objet: [BUG] Balloon malfunctions with memory hotplug
Hello,
Reproducer:
1. Start QEMU with balloon and memory hotplug support:
# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
2. Check balloon size:
(qemu) info balloon
balloon: actual=1024
(qemu)
3. Hotplug some memory:
(qemu) object_add memory-backend-ram,id=mem1,size=1G (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
4. This is step is _not_ needed to reproduce the problem, but you may need to online memory manually on Linux so that it becomes available in the guest
5. Check balloon size again:
(qemu) info balloon
balloon: actual=1024
(qemu)
BUG: The guest now has 2GB of memory, but the balloon thinks
the guest has 1GB
One may think that the problem is that the balloon driver is ignoring hotplugged memory. This is not what's happening. If you do balloon your guest, there's nothing stopping the balloon driver in the guest from ballooning hotplugged memory.
The problem is that the balloon device in QEMU needs to know the current amount of memory available to the guest.
Before memory hotplug this information was easy to obtain: the current amount of memory available to the guest is the memory the guest was booted with. This value is stored in the ram_size global variable in QEMU and this is what the balloon device emulation code uses today. However, when memory is hotplugged ram_size is _not_ updated and the balloon device breaks.
I see two possible solutions for this problem:
1. In addition to reading ram_size, the balloon device in QEMU could scan pc-dimm devices to account for hotplugged memory.
This solution was already implemented by zhanghailiang:
http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
It works, except that on Linux memory hotplug is a two-step
procedure: first memory is inserted then it has to be onlined from user-space. So, if memory is inserted but not onlined this solution gives the opposite problem: the balloon device will report a larger memory amount than the guest actually has.
Can we live with that? I guess not, but I'm open for discussion.
If QEMU could be notified when Linux makes memory online, then the problem would be gone. But I guess this can't be done.
2. Modify the balloon driver in the guest to inform the balloon device on the host about the current memory available to the guest. This way, whenever the balloon device in QEMU needs to know the current amount of memory in the guest, it asks the guest. This drops any usage of ram_size in the balloon device
I'm not completely sure this is feasible though. For example, what happens if the guest reports a memory amount to QEMU and right after this more memory is plugged?
Besides, this solution is more complex than solution 1 and won't address older guests.
Another important detail is that, I *suspect* that a very similar bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE support? I think the same problem we're seeing with memory hotplug will happen and solution 1 won't fix this, although no one seems to care about 32-bit guests...
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html