Skip to contentSkip to conversation

libxl: memory management patches

PATCH Superseded Daniel Kiper 6 participants 17 messages Mar 15, 2013 – Mar 19, 2013

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover libxl: memory management patches

Hi,

Here are two small but important libxl memory management patches:
- libxl: xl mem-max et consortes must update static-max in xenstore too,
- libxl: xl mem-set should not enforce memory limits.

Daniel

#1 lore raw

PATCH 1/2 libxl: xl mem-max et consortes must update static-max in xenstore too

xl mem-max et consortes must update static-max in xenstore too. Without this patch there is no chance to increase memory reservation for given domain above memory limit defined in config file. It means that memory hotplug is practicaly unusable without this patch.

---
 tools/libxl/libxl.c |    3 +++
 1 file changed, 3 insertions(+)
tools/libxl/libxl.c +3 -0
index a09c0fa..5edeaa6 100644
@@ -3480,6 +3480,9 @@ int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t max_memkb)
         goto out;
     }
 
+    libxl__xs_write(gc, XBT_NULL, libxl__sprintf(gc, "%s/memory/static-max",
+			    dompath), "%"PRIu32, max_memkb);
+
     rc = 0;
 out:
     GC_FREE;
Signature · 1 line
1.7.10.4
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] libxl: xl mem-max et consortes must update static-max in xenstore too

Quoted 4 lines · Daniel Kiper

xl mem-max et consortes must update static-max in xenstore too. Without this patch there is no chance to increase memory reservation for given domain above memory limit defined in config file. It means that memory hotplug is practicaly unusable without this patch.

Quoted 29 lines
---
 tools/libxl/libxl.c |    3 +++
 1 file changed, 3 insertions(+)
tools/libxl/libxl.c +3 -0
index a09c0fa..5edeaa6 100644
@@ -3480,6 +3480,9 @@ int libxl_domain_setmaxmem(libxl_ctx *ctx, uint32_t domid, uint32_t max_memkb)
         goto out;
     }
 
+    libxl__xs_write(gc, XBT_NULL, libxl__sprintf(gc, "%s/memory/static-max",
+			    dompath), "%"PRIu32, max_memkb);
+
     rc = 0;
 out:
     GC_FREE;
Signature · 7 lines
1.7.10.4


_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel
#3 lore raw

PATCH 2/2 libxl: xl mem-set should not enforce memory limits

In reply to Daniel Kiper

xl mem-set should not enforce memory limits. If it does then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

---
 tools/libxl/xl_cmdimpl.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/xl_cmdimpl.c +1 -1
index a98705e..9add281 100644
@@ -2492,7 +2492,7 @@ static void set_memory_target(uint32_t domid, const char *mem)
         exit(3);
     }
 
-    libxl_set_memory_target(ctx, domid, memorykb, 0, /* enforce */ 1);
+    libxl_set_memory_target(ctx, domid, memorykb, 0, 0);
 }
 
 int main_memset(int argc, char **argv)
Signature · 1 line
1.7.10.4
#4 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 1 line · Daniel Kiper

xl mem-set should not enforce memory limits. If it does

NB the title says libxl but this is actually an xl change.

Quoted 3 lines

then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

This is a feature not a bug I think. It is intentional that if the host admin sets a memory target the a guest admin cannot exceed that limit.

Ian.

Quoted 19 lines
---
 tools/libxl/xl_cmdimpl.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/xl_cmdimpl.c +1 -1
index a98705e..9add281 100644
@@ -2492,7 +2492,7 @@ static void set_memory_target(uint32_t domid, const char *mem)
         exit(3);
     }
 
-    libxl_set_memory_target(ctx, domid, memorykb, 0, /* enforce */ 1);
+    libxl_set_memory_target(ctx, domid, memorykb, 0, 0);
 }
 
 int main_memset(int argc, char **argv)
#5 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 11 lines · Ian Campbell
Quoted 1 line · Daniel Kiper

xl mem-set should not enforce memory limits. If it does

NB the title says libxl but this is actually an xl change.

Quoted 3 lines

then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

This is a feature not a bug I think. It is intentional that if the host admin sets a memory target the a guest admin cannot exceed that limit.

Even thought 'xm' did not do it?

Quoted 30 lines

Ian.

Quoted 11 lines
---
 tools/libxl/xl_cmdimpl.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/libxl/xl_cmdimpl.c +0 -0
index a98705e..9add281 100644

tools/libxl/xl_cmdimpl.c static void set_memory_target(uint32_t domid, const char *mem) · line 2498

Quoted 8 lines
tools/libxl/xl_cmdimpl.c +1 -1
@@ -2492,7 +2492,7 @@ static void set_memory_target(uint32_t domid, const char *mem)
         exit(3);
     }
 
-    libxl_set_memory_target(ctx, domid, memorykb, 0, /* enforce */ 1);
+    libxl_set_memory_target(ctx, domid, memorykb, 0, 0);
 }
 
 int main_memset(int argc, char **argv)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#6 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Ian Campbell
Quoted 1 line · Daniel Kiper

xl mem-set should not enforce memory limits. If it does

NB the title says libxl but this is actually an xl change.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

This is a feature not a bug I think. It is intentional that if the host admin sets a memory target the a guest admin cannot exceed that limit.

Even thought 'xm' did not do it?

Yes. Please check the archives, this was discussed at the time (at length IIRC)

Ian.

#7 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 11 lines · Ian Campbell
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Ian Campbell
Quoted 1 line · Daniel Kiper

xl mem-set should not enforce memory limits. If it does

NB the title says libxl but this is actually an xl change.

Quoted 3 lines

then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

This is a feature not a bug I think. It is intentional that if the host admin sets a memory target the a guest admin cannot exceed that limit.

Even thought 'xm' did not do it?

Yes. Please check the archives, this was discussed at the time (at length IIRC)

Could you remind thread subject (more or less)?
It was about ballooning or xm/xl itself?

Daniel

#8 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

In reply to Ian Campbell
Quoted 4 lines · Ian Campbell
Quoted 1 line · Daniel Kiper

xl mem-set should not enforce memory limits. If it does

NB the title says libxl but this is actually an xl change.

OK.

Quoted 6 lines
Quoted 3 lines

then after decreasing reservation there is no chance to increase reservation directly from guest domain by writing to /sys/devices/system/xen_memory/xen_memory0/target*

This is a feature not a bug I think. It is intentional that if the host admin sets a memory target the a guest admin cannot exceed that limit.

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Daniel

#9 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

Ian.

#10 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 7 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

Would it make sense to make the current behavior the default, but have it override-able by a configuration variable in xl.conf?

-George

#11 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

In reply to Ian Jackson
Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 7 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

I think that it is possible to make "everyone else" and Oracle guys happy:
- leave old xl mem-set behavior as is,
- improve comment for this commnd,
- add option which disables memory limit enforcement.

Is it OK for you?

Daniel

#12 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

In reply to Ian Jackson
Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 3 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Daniel is trying to fix an bug that Linux kernel is tripping over b/c of this. Look at the converstation and patch that Daniel posted a week ago for the Linux kernel.

Quoted 4 lines

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

s/Oracle/Xend/. As Xend had this distinction. 'xm mem-set' would only set the target. 'xm mem-max' on the other hand would enforce the limit.

Daniel is just bringing this behavior to 'xl'.

#13 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 7 lines
Quoted 3 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Daniel is trying to fix an bug that Linux kernel is tripping over b/c of this. Look at the converstation and patch that Daniel posted a week ago for the Linux kernel.

It would be useful to mention (or at least) the rationale for a change such as this in the commit message.

However that's rather moot in this case because the rationale is surely wrong. Linux (and indeed balloon drivers generally) are expected to behave correctly whether the toolstack chooses to be enforcing or non-enforcing regarding the balloon target. So you can't "fix" the kernel by simply mandating that all toolstacks are non-enforcing, sorry.

Quoted 8 lines
Quoted 3 lines

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

s/Oracle/Xend/. As Xend had this distinction. 'xm mem-set' would only set the target. 'xm mem-max' on the other hand would enforce the limit.

Daniel is just bringing this behavior to 'xl'.

xl deliberately deviated from xend on this point.

Ian.

#14 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 32 lines · Ian Campbell
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 7 lines
Quoted 3 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Daniel is trying to fix an bug that Linux kernel is tripping over b/c of this. Look at the converstation and patch that Daniel posted a week ago for the Linux kernel.

It would be useful to mention (or at least) the rationale for a change such as this in the commit message.

However that's rather moot in this case because the rationale is surely wrong. Linux (and indeed balloon drivers generally) are expected to behave correctly whether the toolstack chooses to be enforcing or non-enforcing regarding the balloon target. So you can't "fix" the kernel by simply mandating that all toolstacks are non-enforcing, sorry.

Quoted 4 lines
Quoted 3 lines

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

Quoted 6 lines
Quoted 4 lines

s/Oracle/Xend/. As Xend had this distinction. 'xm mem-set' would only set the target. 'xm mem-max' on the other hand would enforce the limit.

Daniel is just bringing this behavior to 'xl'.

xl deliberately deviated from xend on this point.

Ah, I did not dig deep enough in the git annotate to see if there was a story behind it. Perhaps then it would make sense to do two things:

1). Add a comment in the code explicitly mentioning it.

2). If we really want to provide an Xend-type behavior add an global
configuration value that is called "xend-backwarts=1' ? That way we can fit all the other stuff that is inconsistent underneath that (such as timer_mode changing from number to string, etc).

And as part of this global value also explain in the docs what those inconsistencies are?

Quoted 10 lines

Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#15 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

Quoted 1 line · Konrad Rzeszutek Wilk

1). Add a comment in the code explicitly mentioning it.

That seems reasonable. It could/should even be in the manpage I guess?

Quoted 7 lines

2). If we really want to provide an Xend-type behavior add an global
configuration value that is called "xend-backwarts=1' ? That way we can fit all the other stuff that is inconsistent underneath that (such as timer_mode changing from number to string, etc).

And as part of this global value also explain in the docs what those inconsistencies are?

I'd rather see specific options for specific cases where there is a genuine choice to be made rather than a catch-all global option.

I think we take care of the old for of timer_mode already, don't we?

Ian.

Acked-by Jonathan Knowles
#16 lore raw

Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits

In reply to Ian Campbell
Quoted 32 lines · Ian Campbell
Quoted 10 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Ian Jackson

Daniel Kiper writes ("Re: [PATCH 2/2] libxl: xl mem-set should not enforce memory limits"):

Quoted 7 lines

I think that xl mem-max should be used to enforce limits. If admin would like to enforce "hard" limit it should call xl mem-set and xl mem-max in sequence. If we would like to leave old xl mem-set behavior we should change comment for this command because now it does not mention anythig about limit enforcement. Or we should add an option which explicitly disables memory limit enforment (this behavior is in line with xm mem-set behavior).

Quoted 7 lines
Quoted 3 lines

I think this conversation is related to the fact that at Oracle you have a different model of the Xen memory allocation model to everyone else.

Daniel is trying to fix an bug that Linux kernel is tripping over b/c of this. Look at the converstation and patch that Daniel posted a week ago for the Linux kernel.

It would be useful to mention (or at least) the rationale for a change such as this in the commit message.

However that's rather moot in this case because the rationale is surely wrong. Linux (and indeed balloon drivers generally) are expected to behave correctly whether the toolstack chooses to be enforcing or non-enforcing regarding the balloon target. So you can't "fix" the kernel by simply mandating that all toolstacks are non-enforcing, sorry.

Quoted 4 lines
Quoted 3 lines

Outside Oracle, guests are supposed to aim for the balloon target and are not permitted to exceed it (when ballooning up) or to regress (when ballooning down).

Quoted 6 lines
Quoted 4 lines

s/Oracle/Xend/. As Xend had this distinction. 'xm mem-set' would only set the target. 'xm mem-max' on the other hand would enforce the limit.

Daniel is just bringing this behavior to 'xl'.

xl deliberately deviated from xend on this point.

This was :

# HG changeset patch
# User Keir Fraser <[email protected]> # Date 1273819996 -3600
# Node ID b29e42cb4727d718025b4b7039b35a824c8a11d3 # Parent 9b9a277d8a51de049068d3251b5d84b73a24196b
libxl: Adjustments to memset/memmax handling

I think xl memset should change the memory currently used by the guest and xl memmax should change the size of the guest's address space and not the population. For this reason libxl_set_memory_target should provide a way to enforce the memory target, calling xc_domain_setmaxmem. On the other hand xl memmax shouldn't call xc_domain_setmaxmem because that is the upper bound of the memory reservation, it should just change static-max, that at the moment wouldn't do much, but we can imagine that in the future could trigger something useful in the guest.

which just says: "I think". Was there some more background behind this?