Skip to contentSkip to conversation

libxl: memory management patches

PATCH Superseded Daniel Kiper 4 participants 29 messages Apr 4, 2013 – Apr 16, 2013

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover libxl: memory management patches

Hi,

Here are three small but important libxl/xl memory management patches:
- libxl: xl mem-max et consortes must update static-max in xenstore too,
- xl: Allow user to configure xl mem-set behavior,
- xl: Improve xl documentation in regards to guest memory management.

Daniel

#1 lore raw

PATCH v3 1/3 libxl: xl mem-max et consortes must update static-max in xenstore too

xl mem-max et consortes must update static-max in xenstore too. Without this patch there is no chance to increase memory reservation for given domain above memory limit defined in config file. It means that memory hotplug is practicaly unusable without this patch.

---
 tools/libxl/libxl.c |    3 +++
 1 file changed, 3 insertions(+)
tools/libxl/libxl.c +3 -0
index 572c2c6..6f4cf09 100644
@@ -3484,6 +3484,9 @@ int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t max_memkb)
         goto out;
     }
 
+    libxl__xs_write(gc, XBT_NULL, libxl__sprintf(gc, "%s/memory/static-max",
+			    dompath), "%"PRIu32, max_memkb);
+
     rc = 0;
 out:
     GC_FREE;
Signature · 1 line
1.7.10.4
#2 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Daniel Kiper writes ("[Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 4 lines

xl mem-max et consortes must update static-max in xenstore too. Without this patch there is no chance to increase memory reservation for given domain above memory limit defined in config file. It means that memory hotplug is practicaly unusable without this patch.

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

I thought that the static-max xenstore key was used for the former.

Ian.

#3 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 6 lines · Ian Jackson

Daniel Kiper writes ("[Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 4 lines

xl mem-max et consortes must update static-max in xenstore too. Without this patch there is no chance to increase memory reservation for given domain above memory limit defined in config file. It means that memory hotplug is practicaly unusable without this patch.

Quoted 6 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

Daniel

#4 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

Quoted 7 lines
Quoted 1 line

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

So that's not "enforcing a limit on guests".

Ian.

#5 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 15 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

Quoted 7 lines
Quoted 1 line

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

Quoted 3 lines

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not. That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Quoted 1 line

So that's not "enforcing a limit on guests".

OK, it should be phrased that xl checks that itself does not exceeds limits.

Daniel

#6 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 7 lines · Ian Jackson
Quoted 3 lines

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

Right.

Quoted 3 lines

That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Right. I don't see what any of that has to do with "xl mem-max" though. The purpose of xl mem-max is to update the guest's enforcement limit.

I think it would be great for the guest to write something to xenstore saying it supports memory hotplug. How does the protocol work ? The guest writes "I support memory hotplug", and the tools then ignore static-max and simply update the memory target with a new value ? Do the tools need to do anything else ?

Ian.

#7 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 11 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 7 lines · Ian Jackson
Quoted 3 lines

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

Right.

Quoted 7 lines
Quoted 3 lines

That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Right. I don't see what any of that has to do with "xl mem-max" though. The purpose of xl mem-max is to update the guest's enforcement limit.

...and static-max if you would like to use memory hotplug in real. If not, then xl mem-set does not allow you to allocate more memory to a guest than static-max.

Quoted 3 lines

I think it would be great for the guest to write something to xenstore saying it supports memory hotplug. How does the protocol work ? The guest writes "I support memory hotplug", and the tools then ignore

More or less...

Quoted 2 lines

static-max and simply update the memory target with a new value ? Do the tools need to do anything else ?

I think that tools should not ignore static-max in any casy. Memory target should not be never ever above static-max.

Daniel

#8 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 3 lines

That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Quoted 7 lines
Quoted 3 lines

Right. I don't see what any of that has to do with "xl mem-max" though. The purpose of xl mem-max is to update the guest's enforcement limit.

...and static-max if you would like to use memory hotplug in real. If not, then xl mem-set does not allow you to allocate more memory to a guest than static-max.

You seem to be losing sight of the purpose of the static-max key in xenstore. It is for the benefit of the toolstack - and, indirectly, the user - so that we know the maximum amount of memory we can ask the guest to us.

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 1 line

No, memory hotplug should not change "static maximum" in any case.

Why shouldn't it ?

Quoted 3 lines

"static maximum" should define memory limit for a given guest. If "static maximum" is above "build maximum" then memory hotplug in the guest should be involved (if it is available).

If a guest supports memory hotplug, does it simply hotplug more memory response to changes in target ? Is there not a limit on how much memory hotplug a guest can cope with ?

Perhaps the guest should advertise its actual maximum ?

Quoted 5 lines
Quoted 2 lines

No, because a guest which is being asked to shrink will experience memory allocation failures if it doesn't shrink monotonically.

IIRC if you shrink memory for a given guest and it does not have memory balloon driver installed nothing happens.

Yes.

Quoted 2 lines

I did not tested it but it would be crazy when host could steal
random page from the guest without its acceptance.

It can't steal memory from the guest. But if the guest does any operation which allocates memory, when it is above the limit, that operation can fail.

So for example consider a guest with a balloon driver (but without memory hotplug) which was booted with 1G and subsequently ballooned down to 768M.

If the user says "xl mem-set blah 512M" then the limit and target are both set to 512M. If the guest's balloon driver, instead of shrinking the guest as it is supposed to, tries to allieviate some memory pressure inside the guest by allocating _more_ memory from Xen, that allocation will fail.

Ian.

#9 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 4 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 1 line

No, memory hotplug should not change "static maximum" in any case.

Why shouldn't it ?

FWIW, at least XenServer uses "static max" to mean "cannot be changed except by rebooting". It *also* uses *that* for HVM guests to say "how much memory to advertize to the guest". That is, if you boot a VM with memory = X and static_max = Y, then it will put Y amount of memory in the virtual e820 map, allocate X memory for the PoD, and then rely on the balloon driver to balloon from Y down to X during boot.

This is two slightly different meanings -- "how much to put in the guest e820" and "how much maximum a guest is ever allowed to have". The plain meaning of "static max" should be the second (i.e., can't be changed dynamically at run time). If we want a "max in the e820 at build time" we should probably come up with a new term; and if "how much the guest is ever allowed to have without rebooting" is not a useful concept, we should probably just discard it.

-George

#10 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 14 lines · George Dunlap
Quoted 4 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 1 line

No, memory hotplug should not change "static maximum" in any case.

Why shouldn't it ?

FWIW, at least XenServer uses "static max" to mean "cannot be changed except by rebooting". It *also* uses *that* for HVM guests to say "how much memory to advertize to the guest". That is, if you boot a VM with memory = X and static_max = Y, then it will put Y amount of memory in the virtual e820 map, allocate X memory for the PoD, and then rely on the balloon driver to balloon from Y down to X during boot.

Quoted 7 lines

This is two slightly different meanings -- "how much to put in the guest e820" and "how much maximum a guest is ever allowed to have". The plain meaning of "static max" should be the second (i.e., can't be changed dynamically at run time). If we want a "max in the e820 at build time" we should probably come up with a new term; and if "how much the guest is ever allowed to have without rebooting" is not a useful concept, we should probably just discard it.

George, thanks for this clarification.

Daniel

#11 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

In reply to Ian Jackson
Quoted 11 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 10 lines · Ian Jackson
Quoted 8 lines

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 3 lines

That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Right. I don't see what any of that has to do with "xl mem-max" though. The purpose of xl mem-max is to update the guest's enforcement limit.

Quoted 8 lines
Quoted 3 lines

...and static-max if you would like to use memory hotplug in real. If not, then xl mem-set does not allow you to allocate more memory to a guest than static-max.

You seem to be losing sight of the purpose of the static-max key in xenstore. It is for the benefit of the toolstack - and, indirectly, the user - so that we know the maximum amount of memory we can ask the guest to us.

OK, I understand that. However, static-max equal to maxmem makes sens for a guest without memory hotplug. If you have memory hotplug and you could not change static-max then this is an obstacle to use memory hotplug in real. You could not raise target above static-max using xl mem-set and you could not change target from guest because of "xen maximum" too.

Now we have two options:
- we could allow user to change static-max for a given guest by calling xl mem-max (my current solution); this way we change a bit meaning of static-max from "maximum amount of memory allowed for the guest (usualy all guest OS structures were prepared for this amount of memory but they do not need to be filled at boot time)" to "maximum amount of memory allowed for the guest at a given moment",
- we could leave static-max as is and use "xen maximum" as "maximum amount of memory allowed for the guest at a given moment"; However, in this case comparison with static-max in libxl_set_memory_target() should be changed to comparison with "xen maximum".

[...]

Quoted 19 lines
Quoted 5 lines
Quoted 2 lines

No, because a guest which is being asked to shrink will experience memory allocation failures if it doesn't shrink monotonically.

IIRC if you shrink memory for a given guest and it does not have memory balloon driver installed nothing happens.

Yes.

Quoted 2 lines

I did not tested it but it would be crazy when host could steal
random page from the guest without its acceptance.

It can't steal memory from the guest. But if the guest does any operation which allocates memory, when it is above the limit, that operation can fail.

So for example consider a guest with a balloon driver (but without memory hotplug) which was booted with 1G and subsequently ballooned down to 768M.

Quoted 5 lines

If the user says "xl mem-set blah 512M" then the limit and target are both set to 512M. If the guest's balloon driver, instead of shrinking the guest as it is supposed to, tries to allieviate some memory pressure inside the guest by allocating _more_ memory from Xen, that allocation will fail.

I suppose it could be phrased like that:

Guests which attempts to exceed or exceeds the target may experience memory allocation failures.

It takes in to account your case and my case when a guest attempts to raise its allocation above target.

Daniel

#12 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 28 lines · Daniel Kiper
Quoted 11 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 10 lines · Ian Jackson
Quoted 8 lines

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 3 lines

That is why I think that static-max should be unconditionally changed or the guest should write something in xenstore to inform that it supports memory hotplug and relevant check should be waived.

Right. I don't see what any of that has to do with "xl mem-max" though. The purpose of xl mem-max is to update the guest's enforcement limit.

Quoted 8 lines
Quoted 3 lines

...and static-max if you would like to use memory hotplug in real. If not, then xl mem-set does not allow you to allocate more memory to a guest than static-max.

You seem to be losing sight of the purpose of the static-max key in xenstore. It is for the benefit of the toolstack - and, indirectly, the user - so that we know the maximum amount of memory we can ask the guest to us.

OK, I understand that. However, static-max equal to maxmem makes sens for a guest without memory hotplug. If you have memory hotplug and you could not change static-max then this is an obstacle to use memory hotplug in real. You could not raise target above static-max using xl mem-set and you could not change target from guest because of "xen maximum" too.

Quoted 11 lines

Now we have two options:
- we could allow user to change static-max for a given guest by calling xl mem-max (my current solution); this way we change a bit meaning of static-max from "maximum amount of memory allowed for the guest (usualy all guest OS structures were prepared for this amount of memory but they do not need to be filled at boot time)" to "maximum amount of memory allowed for the guest at a given moment",
- we could leave static-max as is and use "xen maximum" as "maximum amount of memory allowed for the guest at a given moment"; However, in this case comparison with static-max in libxl_set_memory_target() should be changed to comparison with "xen maximum".

How does this stuff work for physical memory hotplug? Understanding that might help us decide what admins expect (and is also directly relevant to HVM memory hotplug).

In the e820 of a physical system memory which is actually present at boot is obviously represented as E820_MEMORY, but how are the holes in the memory map where DIMMs could subsequently be physically inserted represented? Are they just "reserved" or is there a special "unpopulated memory" type?

And on the PV kernel side how does this appear to the guest? If you boot a "massively ballooned" guest (e.g. 1G out of max 1TB) does it automatically switch to making the bulk of the difference a hotpluggable region rather than a balloon region? What does "pre-ballooned" even mean to a guest which supports memory hotplug?

Do the kernels support memory unplug?

Ian.

#13 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 1 line · Ian Campbell

On Thu, 2013-04-11 at 13:24 +0100, Daniel Kiper wrote:

[...]

Quoted 12 lines
Quoted 11 lines

Now we have two options:
- we could allow user to change static-max for a given guest by calling xl mem-max (my current solution); this way we change a bit meaning of static-max from "maximum amount of memory allowed for the guest (usualy all guest OS structures were prepared for this amount of memory but they do not need to be filled at boot time)" to "maximum amount of memory allowed for the guest at a given moment",
- we could leave static-max as is and use "xen maximum" as "maximum amount of memory allowed for the guest at a given moment"; However, in this case comparison with static-max in libxl_set_memory_target() should be changed to comparison with "xen maximum".

Quoted 3 lines

How does this stuff work for physical memory hotplug? Understanding that might help us decide what admins expect (and is also directly relevant to HVM memory hotplug).

Memory hotplug works in the same way in PV and HVM guest.

Quoted 5 lines

In the e820 of a physical system memory which is actually present at boot is obviously represented as E820_MEMORY, but how are the holes in the memory map where DIMMs could subsequently be physically inserted represented? Are they just "reserved" or is there a special "unpopulated memory" type?

Region for hotplugged memory is not reserved in any way. Only memory available at boot have e820 entries. After hotplugging memory hardware places it at relevant address and informs system about new memory and its config usualy via ACPI. System admin must online new memory via sysfs interface.

Current memory hotplug implementation in balloon driver does not use ACPI and establishes placement for new memory itself (simply above max_pfn; algorithm is not perfect and it fails in some cases; I am going to fix it). Other things work like in physical case.

Quoted 5 lines

And on the PV kernel side how does this appear to the guest? If you boot a "massively ballooned" guest (e.g. 1G out of max 1TB) does it automatically switch to making the bulk of the difference a hotpluggable region rather than a balloon region? What does "pre-ballooned" even mean to a guest which supports memory hotplug?

Guest is "massively ballooned". There is no automatic change from balloon to memory hotplug.

Quoted 1 line

Do the kernels support memory unplug?

Linux Kernel supports memory unplug (hot remove) on baremetal. In Xen guest case memory is ballooned down after memory hotplug.

Daniel

#14 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 47 lines · Daniel Kiper
Quoted 1 line · Ian Campbell

On Thu, 2013-04-11 at 13:24 +0100, Daniel Kiper wrote:

[...]

Quoted 12 lines
Quoted 11 lines

Now we have two options:
- we could allow user to change static-max for a given guest by calling xl mem-max (my current solution); this way we change a bit meaning of static-max from "maximum amount of memory allowed for the guest (usualy all guest OS structures were prepared for this amount of memory but they do not need to be filled at boot time)" to "maximum amount of memory allowed for the guest at a given moment",
- we could leave static-max as is and use "xen maximum" as "maximum amount of memory allowed for the guest at a given moment"; However, in this case comparison with static-max in libxl_set_memory_target() should be changed to comparison with "xen maximum".

Quoted 3 lines

How does this stuff work for physical memory hotplug? Understanding that might help us decide what admins expect (and is also directly relevant to HVM memory hotplug).

Memory hotplug works in the same way in PV and HVM guest.

Quoted 5 lines

In the e820 of a physical system memory which is actually present at boot is obviously represented as E820_MEMORY, but how are the holes in the memory map where DIMMs could subsequently be physically inserted represented? Are they just "reserved" or is there a special "unpopulated memory" type?

Region for hotplugged memory is not reserved in any way. Only memory available at boot have e820 entries. After hotplugging memory hardware places it at relevant address and informs system about new memory and its config usualy via ACPI. System admin must online new memory via sysfs interface.

Current memory hotplug implementation in balloon driver does not use ACPI and establishes placement for new memory itself (simply above max_pfn; algorithm is not perfect and it fails in some cases; I am going to fix it). Other things work like in physical case.

Quoted 5 lines

And on the PV kernel side how does this appear to the guest? If you boot a "massively ballooned" guest (e.g. 1G out of max 1TB) does it automatically switch to making the bulk of the difference a hotpluggable region rather than a balloon region? What does "pre-ballooned" even mean to a guest which supports memory hotplug?

Quoted 7 lines

Guest is "massively ballooned". There is no automatic change from balloon to memory hotplug.

Quoted 1 line

Do the kernels support memory unplug?

Linux Kernel supports memory unplug (hot remove) on baremetal. In Xen guest case memory is ballooned down after memory hotplug.

Ian, any additional thoughts, comments, ...? I would like to prepare next version of patch ASAP. I hope it will be included in Xen 4.3.

Daniel

#15 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 1 line · Daniel Kiper

Ian, any additional thoughts, comments, ...?

Do you mean me or IanJ?

I think I'm still of the opinion that operations which may involve memory hotplug on the guest side rather than just ballooning should be explicit, at least at the libxl API layer. TBH I'm not sure what this would look like since the current API doesn't really seem to suggest a natural way to extend it like this.

At the xl layer I'm not so sure what the answer is but xl is easier to change if we aren't happy with it.

Ian.

#16 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Quoted 4 lines · Ian Campbell
Quoted 1 line · Daniel Kiper

Ian, any additional thoughts, comments, ...?

Do you mean me or IanJ?

You, but if IanJ has comments I am happy to hear them too...

Quoted 8 lines

I think I'm still of the opinion that operations which may involve memory hotplug on the guest side rather than just ballooning should be explicit, at least at the libxl API layer. TBH I'm not sure what this would look like since the current API doesn't really seem to suggest a natural way to extend it like this.

At the xl layer I'm not so sure what the answer is but xl is easier to change if we aren't happy with it.

I think that we should not care (from a host point of view) which mechanism is used in a guest to increase its memory size. From the host point of view in balloon case and in memory hotplug case we just allocate pages for the guest. We do not care where they will be placed and when they will be used. That is it.

However, we should create a mechanism which allows the guest to allocate memory above limit set at boot and not eat all host memory in uncontrolled manner (e.g. using memory hotplug mechanism). Last is provided by xl now but former is not. Currently there is no way to increase memory limit above memory limit set at boot (maxmem). Now I think we should just decide which value should be used as a cap reference: static-max or "xen maximum". Both are not perfect and some recalculation is always needed to establish real cap internaly in the guest.

Daniel

#17 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too [and 1 more messages]"):

Quoted 5 lines

I think that we should not care (from a host point of view) which mechanism is used in a guest to increase its memory size. From the host point of view in balloon case and in memory hotplug case we just allocate pages for the guest. We do not care where they will be placed and when they will be used. That is it.

Yes, exactly.

Quoted 4 lines

However, we should create a mechanism which allows the guest to allocate memory above limit set at boot and not eat all host memory in uncontrolled manner (e.g. using memory hotplug mechanism). Last is provided by xl now but former is not.

Yes. That limit mechanism is the Xen memory limit, which by default is set by xl mem-set and with your changes can be independently set with xl mem-max.

Quoted 2 lines

Currently there is no way to increase memory limit above memory limit set at boot (maxmem).

That's the wrong way to think about it. Currently xl will allow neither the target nor the xen maximum to be increased beyond the boot maximum, because they think that won't work.

Quoted 4 lines

Now I think we should just decide which value should be used as a cap reference: static-max or "xen maximum". Both are not perfect and some recalculation is always needed to establish real cap internaly in the guest.

I don't know what "cap reference" is.

It seems to me that if the guest could in principle do memory hotplug without assistance from the toolstack, all that's needed is:

* Guest advertises memory hotplug capability in xenstore (presumably it advertises a maximum figure, rather than just promising to be able to use any amount of memory, since most guests will be able to hotplug only a certain amount?)

* When memory hotplug limit is advertised, libxl ignores static-max in xenstore in favour of the guest's advertised capability. xl mem-set (and if different, xl mem-max) can be told to set the guest's memory target (in xenstore), and the xen memory limit, to more than the boot time memory. When this happens the guest's balloon/hotplug driver can hotplug memory appropriately to bring the guest's memory usage to the requested target.

Ian.

#18 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

In reply to Daniel Kiper
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 13 lines · Ian Jackson
Quoted 6 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

Quoted 6 lines
Quoted 3 lines

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be a conscious admin decision.

#19 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 21 lines · Ian Campbell
Quoted 20 lines · Daniel Kiper
Quoted 18 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

Quoted 7 lines
Quoted 1 line

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

Quoted 5 lines
Quoted 2 lines

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be a conscious admin decision.

Sure but I do not understand how this is related to above. Addtitionally, which admin do you mean? Guest or host?

Daniel

#20 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 22 lines · Daniel Kiper
Quoted 21 lines · Ian Campbell
Quoted 20 lines · Daniel Kiper
Quoted 18 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

Quoted 7 lines
Quoted 1 line

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be a conscious admin decision.

Sure but I do not understand how this is related to above.

Because an admin might know whether hotplug is (potentially) an option for the domain.

Quoted 1 line

Addtitionally, which admin do you mean? Guest or host?

Host.

Ian.

#21 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 29 lines · Ian Campbell
Quoted 28 lines · Daniel Kiper
Quoted 26 lines · Ian Campbell
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 13 lines · Ian Jackson
Quoted 6 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

Quoted 6 lines
Quoted 3 lines

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be a conscious admin decision.

Quoted 8 lines
Quoted 1 line

Sure but I do not understand how this is related to above.

Because an admin might know whether hotplug is (potentially) an option for the domain.

Quoted 1 line

Addtitionally, which admin do you mean? Guest or host?

Host.

I thought once that balloon driver with memory hotplug support might store something in xenstore. Maybe it will be sufficient.

Daniel

#22 lore raw

Re: [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 35 lines · Daniel Kiper
Quoted 34 lines · Ian Campbell
Quoted 22 lines · Daniel Kiper
Quoted 21 lines · Ian Campbell
Quoted 20 lines · Daniel Kiper
Quoted 18 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 1/3] libxl: xl mem-max et consortes must update static-max in xenstore too"):

Quoted 6 lines · Ian Jackson
Quoted 5 lines

Doesn't this, together with your previous patch, conflate the "static maximum" (ie, boot-time memory size which in the absence of memory hotplug can never be exceeded), with the "xen maximum" (ie the enforced memory limit) ?

Quoted 7 lines
Quoted 1 line

I thought that the static-max xenstore key was used for the former.

To some extent. However, static-max has always a bit larger value than "xen maximum". xl uses static-max to enforce limits on guests but it is just an info for guest itself. "xen maximum" is a kind of hard limit which could not be exceeded and is enforced on guest by Xen hypervisor.

The reason for xl using static-max is that in the absence of memory hotplug, attempting to raise a guest above static-max will not work. And this check takes effect in xl.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

OK but now it is quiet difficult (or close to impossible) to know in advance that a given guest supports memory hotplug or not.

This is why I previously queried whether memory hotplug ought not to be a conscious admin decision.

Sure but I do not understand how this is related to above.

Because an admin might know whether hotplug is (potentially) an option for the domain.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Addtitionally, which admin do you mean? Guest or host?

Host.

I thought once that balloon driver with memory hotplug support might store something in xenstore. Maybe it will be sufficient.

Same as in previous email...

Daniel

#23 lore raw

PATCH v3 2/3 xl: Allow user to configure xl mem-set behavior

In reply to Daniel Kiper

Add mem_set_enforce_limit option to xl.conf file and equivalent xl command line option. Those two things give a chance to align xl mem-set behavior to xm mem-set behavior. Default xl mem-set behavior is not changed.

v3 - suggestions/fixes:
- add xl command line option
(suggested by Ian Jackson).

---
 tools/libxl/xl.c          |    4 ++++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |   16 ++++++++++------
 tools/libxl/xl_cmdtable.c |    3 ++-
 4 files changed, 17 insertions(+), 7 deletions(-)
tools/libxl/xl.c +4 -0
index 4c598db..5c445ca 100644
@@ -38,6 +38,7 @@ xentoollog_logger_stdiostream *logger;
 int dryrun_only;
 int force_execution;
 int autoballoon = 1;
+int mem_set_enforce_limit = 1;
 char *blkdev_start;
 int run_hotplug_scripts = 1;
 char *lockfile;
@@ -72,6 +73,9 @@ static void parse_global_config(const char *configfile,
     if (!xlu_cfg_get_long (config, "autoballoon", &l, 0))
         autoballoon = l;
 
+    if (!xlu_cfg_get_long (config, "mem_set_enforce_limit", &l, 0))
+        mem_set_enforce_limit = l;
+
     if (!xlu_cfg_get_long (config, "run_hotplug_scripts", &l, 0))
         run_hotplug_scripts = l;
 
tools/libxl/xl.h +1 -0
index b881f92..321a0d0 100644
@@ -143,6 +143,7 @@ int xl_child_pid(xlchildnum); /* returns 0 if child struct is not in use */
 
 /* global options */
 extern int autoballoon;
+extern int mem_set_enforce_limit;
 extern int run_hotplug_scripts;
 extern int dryrun_only;
 extern char *lockfile;
tools/libxl/xl_cmdimpl.c +10 -6
index 2d40f8f..a0ff3e5 100644
@@ -2495,7 +2495,7 @@ int main_memmax(int argc, char **argv)
     return 0;
 }
 
-static void set_memory_target(uint32_t domid, const char *mem)
+static void set_memory_target(uint32_t domid, const char *mem, int enforce_limit)
 {
     long long int memorykb;
 
@@ -2505,23 +2505,27 @@ static void set_memory_target(uint32_t domid, const char *mem)
         exit(3);
     }
 
-    libxl_set_memory_target(ctx, domid, memorykb, 0, /* enforce */ 1);
+    libxl_set_memory_target(ctx, domid, memorykb, 0, enforce_limit);
 }
 
 int main_memset(int argc, char **argv)
 {
     uint32_t domid;
-    int opt = 0;
+    int enforce_limit, opt = 0;
     const char *mem;
 
-    SWITCH_FOREACH_OPT(opt, "", NULL, "mem-set", 2) {
-        /* No options */
+    enforce_limit = mem_set_enforce_limit;
+
+    SWITCH_FOREACH_OPT(opt, "e:", NULL, "mem-set", 2) {
+    case 'e':
+        enforce_limit = atoi(optarg) ? 1 : 0;
+        break;
     }
 
     domid = find_domain(argv[optind]);
     mem = argv[optind + 1];
 
-    set_memory_target(domid, mem);
+    set_memory_target(domid, mem, enforce_limit);
     return 0;
 }
 
tools/libxl/xl_cmdtable.c +2 -1
index b4a87ca..5cfdb70 100644
@@ -195,7 +195,8 @@ struct cmd_spec cmd_table[] = {
     { "mem-set",
       &main_memset, 0, 1,
       "Set the current memory usage for a domain",
-      "<Domain> <MemMB['b'[bytes]|'k'[KB]|'m'[MB]|'g'[GB]|'t'[TB]]>",
+      "[-e <0|1>] <Domain> <MemMB['b'[bytes]|'k'[KB]|'m'[MB]|'g'[GB]|'t'[TB]]>",
+      "-e <0|1> (Not)Enforce maximum memory allocation for a domain"
     },
     { "button-press",
       &main_button_press, 0, 1,
Signature · 1 line
1.7.10.4
#24 lore raw

PATCH v3 3/3 xl: Improve xl documentation in regards to guest memory management

In reply to Daniel Kiper

Improve xl documentation in regards to guest memory management.

Some improvements are suggested by Ian Jackson and Konrad Rzeszutek Wilk.

---
 docs/man/xl.conf.pod.5            |   15 +++++++++++++++
 docs/misc/libxl_memory.txt        |   38 +++++++++++++++++++++++--------------
 docs/misc/xenstore-paths.markdown |   15 +++++++--------
 tools/examples/xl.conf            |    4 ++++
 4 files changed, 50 insertions(+), 22 deletions(-)
docs/man/xl.conf.pod.5 +15 -0
index 7b9fcac..34d6a5d 100644
@@ -55,6 +55,21 @@ reduce the amount of memory given to domain 0 by default.
 
 Default: C<1>
 
+=item B<mem_set_enforce_limit=BOOLEAN>
+
+If enabled, then C<xl mem-set> by default will set a memory allocation
+target and enforce the maximum memory allocation for a given domain.
+If disabled, then C<xl mem-set> by default will set a memory allocation
+target only for a given domain.
+
+The memory allocation target is a suggested amount of memory which
+should be allocated by a guest. It is nice when the guest follows this
+value but it may have its own idea about memory requirments. However,
+it could not allocate more memory than a ceiling known as the maximum
+memory allocation.
+
+Default: C<1>
+
 =item B<run_hotplug_scripts=BOOLEAN>
 
 If disabled hotplug scripts will be called from udev, as it used to
docs/misc/libxl_memory.txt +24 -14
index 253476d..f9b9a67 100644
@@ -15,14 +15,19 @@
              |  |          |  |    |        | target  |          |            
              |  | guest    |  |    | build  |         |          |            
              |  |          |  |    | start  |         |          |            
-      static |  |          |  |    |        |         |          |            
-     maximum |  +----------+  |    +        +         +          +            
-             |  |          |  |                                               
-             |  |          |  |                                               
+             |  |          |  |    |        |         |          |            
+             |  +----------+  |    +        +         +          +            
+      static |  |          |  |                                               
+     maximum |  |          |  |                                               
              |  | balloon  |  | build                                         
              |  |          |  | maximum                                       
              |  |          |  |                                               
-             +  +----------+  +                                               
+             |  +----------+  +                                               
+             |  |          |
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+
                 
                 
     extra internal = LIBXL_MAXMEM_CONSTANT
@@ -31,8 +36,8 @@
     static maximum = libxl_domain_build_info.max_memkb
     video = libxl_domain_build_info.video_memkb
     build start = libxl_domain_build_info.target_memkb
-    libxl_domain_setmaxmem -> xen maximum
-    libxl_set_memory_target -> actual target
+    libxl_domain_setmaxmem -> xen maximum & static maximum
+    libxl_set_memory_target -> actual target & optionaly xen maximum
                 
                 
  === Domain memory breakdown: PV guests ==================================
@@ -49,22 +54,27 @@
              |  |          |  |    |        | actual  | maximum  | 
              |  | guest    |  |    | build  | target  |          |            
              |  |          |  |    | start  |         |          |            
-      static |  |          |  |    |        |         |          |            
-     maximum |  +----------+  |    +        +         +          +            
+             |  |          |  |    |        |         |          |            
+             |  +----------+  |    +        +         +          +            
              |  |          |  |                                               
-             |  |          |  |                                               
-             |  | balloon  |  | build                                         
+      static |  |          |  |                                               
+     maximum |  | balloon  |  | build                                         
              |  |          |  | maximum                                       
              |  |          |  |                                               
-             +  +----------+  +                                               
+             |  +----------+  +                                               
+             |  |          |
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+
                 
 
     extra internal = LIBXL_MAXMEM_CONSTANT
     extra external = LIBXL_PV_EXTRA_MEMORY
     static maximum = libxl_domain_build_info.max_memkb
     build start = libxl_domain_build_info.target_memkb
-    libxl_domain_setmaxmem -> xen maximum
-    libxl_set_memory_target -> actual target
+    libxl_domain_setmaxmem -> xen maximum & static maximum
+    libxl_set_memory_target -> actual target & optionaly xen maximum
 
 
  ========================================================================= */
docs/misc/xenstore-paths.markdown +7 -8
index 535830e..911d1ad 100644
@@ -124,18 +124,17 @@ This protocol is not currently well documented.
 
 #### ~/memory/static-max = MEMKB []
 
-Specifies a static maximum amount memory which this domain should
-expect to be given. In the absence of in-guest memory hotplug support
-this set on domain boot and is usually the maximum amount of RAM which
-a guest can make use of. See [docs/misc/libxl_memory.txt][LIBXLMEM]
-for a description of how memory is accounted for in toolstacks using
-the libxl library.
+Specifies a static maximum amount of memory which this domain should
+expect to be given. It is usually the maximum amount of RAM which a guest
+can make use of. This value is set on the domain boot and can be changed by
+xl mem-max. See [docs/misc/libxl_memory.txt][LIBXLMEM] for a description
+of how the memory is accounted for in toolstacks using the libxl library.
 
 #### ~/memory/target = MEMKB []
 
 The current balloon target for the domain. The balloon driver within
-the guest is expected to make every effort to every effort use no more
-than this amount of RAM.
+the guest is expected to make every effort to use no more than this
+amount of RAM.
 
 #### ~/memory/videoram = MEMKB [HVM,INTERNAL]
 
tools/examples/xl.conf +4 -0
index b0caa32..c48caad 100644
@@ -4,6 +4,10 @@
 # memory to create a domain
 #autoballoon=1
 
+# xl mem-set by default will set a memory allocation target and enforce
+# the maximum memory allocation for a given domain
+#mem_set_enforce_limit=1
+
 # full path of the lockfile used by xl during domain creation
 #lockfile="/var/lock/xl"
 
Signature · 1 line
1.7.10.4
#25 lore raw

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

Daniel Kiper writes ("[Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 1 line

Improve xl documentation in regards to guest memory management.

Thanks.

docs/man/xl.conf.pod.5 reduce the amount of memory given to domain 0 by default. · line 63

Quoted 6 lines

+=item B<mem_set_enforce_limit=BOOLEAN>

docs/man/xl.conf.pod.5 +5 -0
+
+If enabled, then C<xl mem-set> by default will set a memory allocation
+target and enforce the maximum memory allocation for a given domain.
+If disabled, then C<xl mem-set> by default will set a memory allocation
+target only for a given domain.

I think you mean "for each domain" rather than "for a given domain", and "will set only a memory allocation target for each domain" ?

docs/man/xl.conf.pod.5 reduce the amount of memory given to domain 0 by default. · line 69

Quoted 5 lines
docs/man/xl.conf.pod.5 +5 -0
+The memory allocation target is a suggested amount of memory which
+should be allocated by a guest. It is nice when the guest follows this
+value but it may have its own idea about memory requirments. However,
+it could not allocate more memory than a ceiling known as the maximum
+memory allocation.

I would write "However the guest is prevented by Xen from allocating more memory than a ceiling known as the "xen maximum" memory allocation" or some such.

Quoted 14 lines
docs/misc/libxl_memory.txt +4 -4
@@ -15,14 +15,19 @@
              |  |          |  |    |        | target  |          | 
              |  | guest    |  |    | build  |         |          | 
              |  |          |  |    | start  |         |          | 
-      static |  |          |  |    |        |         |          | 
-     maximum |  +----------+  |    +        +         +          + 
-             |  |          |  | 
-             |  |          |  | 
+             |  |          |  |    |        |         |          | 
+             |  +----------+  |    +        +         +          + 
+      static |  |          |  | 
+     maximum |  |          |  | 

docs/misc/libxl_memory.txt line 30

Quoted 10 lines
docs/misc/libxl_memory.txt +6 -1
              |  | balloon  |  | build 
              |  |          |  | maximum 
              |  |          |  | 
-             +  +----------+  + 
+             |  +----------+  + 
+             |  |          |
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+

This change to the diagram implies that the target, and and the xen maximum, never include any hotplug memory. I don't think that can be true.

Presumably hotplugged memory can be ballooned too. So perhaps we need another column ?

docs/misc/libxl_memory.txt

Quoted 7 lines
docs/misc/libxl_memory.txt +2 -2
 #### ~/memory/target = MEMKB []
 
 The current balloon target for the domain. The balloon driver within
-the guest is expected to make every effort to every effort use no more
-than this amount of RAM.
+the guest is expected to make every effort to use no more than this
+amount of RAM.

Good fix.

It might be worth adding:

"Guests whose actual allocation exceeds the target may experience
memory allocation failures."

Thanks,
Ian.

#26 lore raw

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

Quoted 5 lines · Ian Jackson

Daniel Kiper writes ("[Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 1 line

Improve xl documentation in regards to guest memory management.

Thanks.

Quoted 9 lines

docs/man/xl.conf.pod.5 reduce the amount of memory given to domain 0 by default. · line 63

Quoted 6 lines

+=item B<mem_set_enforce_limit=BOOLEAN>

docs/man/xl.conf.pod.5 +5 -0
+
+If enabled, then C<xl mem-set> by default will set a memory allocation
+target and enforce the maximum memory allocation for a given domain.
+If disabled, then C<xl mem-set> by default will set a memory allocation
+target only for a given domain.

I think you mean "for each domain" rather than "for a given domain", and "will set only a memory allocation target for each domain" ?

Yes, thanks.

Quoted 30 lines

docs/man/xl.conf.pod.5 reduce the amount of memory given to domain 0 by default. · line 69

Quoted 5 lines
docs/man/xl.conf.pod.5 +5 -0
+The memory allocation target is a suggested amount of memory which
+should be allocated by a guest. It is nice when the guest follows this
+value but it may have its own idea about memory requirments. However,
+it could not allocate more memory than a ceiling known as the maximum
+memory allocation.

I would write "However the guest is prevented by Xen from allocating more memory than a ceiling known as the "xen maximum" memory allocation" or some such.

Quoted 20 lines
docs/misc/libxl_memory.txt +6 -5
@@ -15,14 +15,19 @@
              |  |          |  |    |        | target  |          |
              |  | guest    |  |    | build  |         |          |
              |  |          |  |    | start  |         |          |
-      static |  |          |  |    |        |         |          |
-     maximum |  +----------+  |    +        +         +          +
-             |  |          |  |
-             |  |          |  |
+             |  |          |  |    |        |         |          |
+             |  +----------+  |    +        +         +          +
+      static |  |          |  |
+     maximum |  |          |  |
              |  | balloon  |  | build
              |  |          |  | maximum
              |  |          |  |
-             +  +----------+  +
+             |  +----------+  +
+             |  |          |
Quoted 8 lines

docs/misc/libxl_memory.txt line 30

Quoted 4 lines
docs/misc/libxl_memory.txt +4 -0
+             |  | memory   |
+             |  | hotplug  |
+             |  |          |
+             +  +----------+

This change to the diagram implies that the target, and and the xen maximum, never include any hotplug memory. I don't think that can be true.

Right, "xen maximum" should include hotplug memory. However, I think that "actual target" should be split to "suggested target" and "actual target". "suggested target" could be up to "static maximum". "actual target" always include only video and guest memory (currently alloceted memory by domain).

Quoted 2 lines

Presumably hotplugged memory can be ballooned too. So perhaps we need another column ?

I think that it is better to add comment that after hotplugging memory could be ballooned down.

Quoted 10 lines

docs/misc/libxl_memory.txt

Quoted 7 lines
docs/misc/libxl_memory.txt +2 -2
 #### ~/memory/target = MEMKB []
 
 The current balloon target for the domain. The balloon driver within
-the guest is expected to make every effort to every effort use no more
-than this amount of RAM.
+the guest is expected to make every effort to use no more than this
+amount of RAM.

Good fix.

Quoted 4 lines

It might be worth adding:

"Guests whose actual allocation exceeds the target may experience
memory allocation failures."

... or "Guests which attempts to exceed the target may experience memory allocation failures."

Daniel

#27 lore raw

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 6 lines · Ian Jackson
Quoted 2 lines

Presumably hotplugged memory can be ballooned too. So perhaps we need another column ?

I think that it is better to add comment that after hotplugging memory could be ballooned down.

I'd prefer not to make the diagram wrong. Perhaps it would be better to annotate the "static maximum" to say that it might be increased by memory hotplug.

Quoted 7 lines
Quoted 4 lines

It might be worth adding:

"Guests whose actual allocation exceeds the target may experience
memory allocation failures."

... or "Guests which attempts to exceed the target may experience memory allocation failures."

No, because a guest which is being asked to shrink will experience memory allocation failures if it doesn't shrink monotonically.

Ian.

#28 lore raw

Re: [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management

Quoted 11 lines · Ian Jackson

Daniel Kiper writes ("Re: [Xen-devel] [PATCH v3 3/3] xl: Improve xl documentation in regards to guest memory management"):

Quoted 6 lines · Ian Jackson
Quoted 2 lines

Presumably hotplugged memory can be ballooned too. So perhaps we need another column ?

I think that it is better to add comment that after hotplugging memory could be ballooned down.

I'd prefer not to make the diagram wrong. Perhaps it would be better to annotate the "static maximum" to say that it might be increased by memory hotplug.

No, memory hotplug should not change "static maximum" in any case. "static maximum" should define memory limit for a given guest. If "static maximum" is above "build maximum" then memory hotplug in the guest should be involved (if it is available).

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

It might be worth adding:

"Guests whose actual allocation exceeds the target may experience
memory allocation failures."

... or "Guests which attempts to exceed the target may experience memory allocation failures."

No, because a guest which is being asked to shrink will experience memory allocation failures if it doesn't shrink monotonically.

IIRC if you shrink memory for a given guest and it does not have memory balloon driver installed nothing happens. I did not tested it but it would be crazy when host could steal random page from the guest without its acceptance. It would lead to crash almost immediately. That is why I wrote "attempts" because memory allocation failures may happen when you increase reservation only.

Daniel