Skip to contentSkip to conversation

domain_create: honour global grant/maptrack frame limits...

PATCH In review Paul Durrant 5 participants 8 messages Nov 13, 2019 – Nov 26, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 domain_create: honour global grant/maptrack frame limits...

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit should be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

---
 xen/common/domain.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
xen/common/domain.c +12 -2
index 611116c7fc..aad6d55b82 100644
@@ -335,6 +335,7 @@ struct domain *domain_create(domid_t domid,
     enum { INIT_watchdog = 1u<<1,
            INIT_evtchn = 1u<<3, INIT_gnttab = 1u<<4, INIT_arch = 1u<<5 };
     int err, init_status = 0;
+    unsigned int max_grant_frames, max_maptrack_frames;
 
     if ( config && (err = sanitise_domain_config(config)) )
         return ERR_PTR(err);
@@ -456,8 +457,17 @@ struct domain *domain_create(domid_t domid,
             goto fail;
         init_status |= INIT_evtchn;
 
-        if ( (err = grant_table_init(d, config->max_grant_frames,
-                                     config->max_maptrack_frames)) != 0 )
+        /*
+         * Make sure that the configured values don't reduce any
+         * global command line override.
+         */
+        max_grant_frames = max(config->max_grant_frames,
+                               opt_max_grant_frames);
+        max_maptrack_frames = max(config->max_maptrack_frames,
+                                  opt_max_maptrack_frames);
+
+        if ( (err = grant_table_init(d, max_grant_frames,
+                                     max_maptrack_frames)) != 0 )
             goto fail;
         init_status |= INIT_gnttab;
 
Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 4 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit should be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

Paul

Quoted 40 lines
---
 xen/common/domain.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
xen/common/domain.c +12 -3
index 611116c7fc..aad6d55b82 100644
@@ -335,6 +335,7 @@ struct domain *domain_create(domid_t domid,
     enum { INIT_watchdog = 1u<<1,
            INIT_evtchn = 1u<<3, INIT_gnttab = 1u<<4, INIT_arch = 1u<<5 };
     int err, init_status = 0;
+    unsigned int max_grant_frames, max_maptrack_frames;
 
     if ( config && (err = sanitise_domain_config(config)) )
         return ERR_PTR(err);
@@ -456,8 +457,17 @@ struct domain *domain_create(domid_t domid,
             goto fail;
         init_status |= INIT_evtchn;
 
-        if ( (err = grant_table_init(d, config->max_grant_frames,
-                                     config->max_maptrack_frames)) != 0 )
+        /*
+         * Make sure that the configured values don't reduce any
+         * global command line override.
+         */
+        max_grant_frames = max(config->max_grant_frames,
+                               opt_max_grant_frames);
+        max_maptrack_frames = max(config->max_maptrack_frames,
+                                  opt_max_maptrack_frames);
+
+        if ( (err = grant_table_init(d, max_grant_frames,
+                                     max_maptrack_frames)) != 0 )
             goto fail;
         init_status |= INIT_gnttab;
 
--

2.17.1

Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
Quoted 48 lines · Paul Durrant
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 4 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit should be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

Paul

Quoted 20 lines
---
  xen/common/domain.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
xen/common/domain.c +1 -0
index 611116c7fc..aad6d55b82 100644
@@ -335,6 +335,7 @@ struct domain *domain_create(domid_t domid,
      enum { INIT_watchdog = 1u<<1,
             INIT_evtchn = 1u<<3, INIT_gnttab = 1u<<4, INIT_arch = 1u<<5 };
      int err, init_status = 0;
+    unsigned int max_grant_frames, max_maptrack_frames;
 
      if ( config && (err = sanitise_domain_config(config)) )
          return ERR_PTR(err);
@@ -456,8 +457,17 @@ struct domain *domain_create(domid_t domid,
              goto fail;
          init_status |= INIT_evtchn;
 
Quoted 13 lines

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 470

Quoted 13 lines
xen/common/domain.c +11 -2
-        if ( (err = grant_table_init(d, config->max_grant_frames,
-                                     config->max_maptrack_frames)) != 0 )
+        /*
+         * Make sure that the configured values don't reduce any
+         * global command line override.
+         */
+        max_grant_frames = max(config->max_grant_frames,
+                               opt_max_grant_frames);
+        max_maptrack_frames = max(config->max_maptrack_frames,
+                                  opt_max_maptrack_frames);
+
+        if ( (err = grant_table_init(d, max_grant_frames,
+                                     max_maptrack_frames)) != 0 )

So basically the per-domain settings are ignored.

They are not allowed to be smaller than the global limits (due to using max()).

They are not allowed to be larger than the global limits (due to the test in grant_table_init().

That is _not_ the purpose of being able to control the settings per domain.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 79 lines

-----Original Message-----
From: Jürgen Groß <[email protected]>
Sent: 26 November 2019 11:37
To: Paul Durrant <[email protected]>; Durrant, Paul <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>; xen-devel <[email protected]>
Subject: Re: [Xen-devel] [PATCH] domain_create: honour global grant/maptrack frame limits...

Quoted 15 lines · Paul Durrant
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 2 lines
Quoted 2 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit

should

Quoted 15 lines
Quoted 2 lines

be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

Paul

Quoted 4 lines
---
  xen/common/domain.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
Quoted 7 lines

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 336

Quoted 7 lines
xen/common/domain.c +0 -0
index 611116c7fc..aad6d55b82 100644
@@ -335,6 +335,7 @@ struct domain *domain_create(domid_t domid,
      enum { INIT_watchdog = 1u<<1,
             INIT_evtchn = 1u<<3, INIT_gnttab = 1u<<4, INIT_arch =

1u<<5 };

Quoted 11 lines

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 460

Quoted 11 lines
xen/common/domain.c +1 -2
      int err, init_status = 0;
+    unsigned int max_grant_frames, max_maptrack_frames;
 
      if ( config && (err = sanitise_domain_config(config)) )
          return ERR_PTR(err);
@@ -456,8 +457,17 @@ struct domain *domain_create(domid_t domid,
              goto fail;
          init_status |= INIT_evtchn;
 
-        if ( (err = grant_table_init(d, config->max_grant_frames,
-                                     config->max_maptrack_frames)) !=

0 )

Quoted 11 lines

xen/common/domain.c struct domain *domain_create(domid_t domid, · line 470

Quoted 11 lines
xen/common/domain.c +11 -0
+        /*
+         * Make sure that the configured values don't reduce any
+         * global command line override.
+         */
+        max_grant_frames = max(config->max_grant_frames,
+                               opt_max_grant_frames);
+        max_maptrack_frames = max(config->max_maptrack_frames,
+                                  opt_max_maptrack_frames);
+
+        if ( (err = grant_table_init(d, max_grant_frames,
+                                     max_maptrack_frames)) != 0 )
Quoted 2 lines

So basically the per-domain settings are ignored.

Basically, yes.

Quoted 9 lines

They are not allowed to be smaller than the global limits (due to using max()).

They are not allowed to be larger than the global limits (due to the test in grant_table_init().

That is _not_ the purpose of being able to control the settings per domain.

Ok, if a straight-up return to old behaviour is out then I guess 4.13 will carry the regression.

Paul

Quoted 2 lines

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
In reply to Paul Durrant
Quoted 18 lines · Paul Durrant
Quoted 13 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 4 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit should be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Quoted 5 lines

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

I think I've been very clear on my opinion of this patch, and what I would consider an acceptable way forward.

This patch breaks things in exactly the (opposite) way you are complaining about having happened when the Xen command line options were replaced with xl.conf options for domU.

Yes - it wasn't great to have done things like this.  No - its not acceptable to do the same again and break people now relying on the per domain settings to take effect.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
In reply to Paul Durrant
Quoted 20 lines · Paul Durrant
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 4 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit should be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Quoted 5 lines

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

This would be a bit easier to form a "policy" opinion on (or perhaps alternate solutions to) if more of the situation were outlined here.

Is the problem that the per-domain config is always set, and doesn't take the hypervisor-set config into account? Wouldn't it be better to modify the toolstack to use the hypervisor value if it's not set?

In fact, it looks kind of like things are screwed up anyway -- the "default" value of max_grant_frames, if no value is specified, is set in xl.c. If that were the behavior we wanted, it should be set in libxl.c.

But it doesn't seem like it should be terribly difficult to get a "use the default" sentinel value passed in to Xen, such that:

1. People who don't do anything will get the default currently specified in xl.c

2. People who set the value on the Xen command-line and don't set anything in the guest config file will get the Xen command-line value

3. People who set the value in the config file will get the value they specified (regardless of the global setting).

Is that the behaviour you'd like to see, Paul?

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 58 lines

-----Original Message-----
From: George Dunlap <[email protected]>
Sent: 26 November 2019 12:32
To: Paul Durrant <[email protected]>; Durrant, Paul <[email protected]>
Cc: xen-devel <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH] domain_create: honour global grant/maptrack frame limits...

Quoted 15 lines · Paul Durrant
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 2 lines
Quoted 2 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit

should

Quoted 8 lines
Quoted 2 lines

be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

This would be a bit easier to form a "policy" opinion on (or perhaps alternate solutions to) if more of the situation were outlined here.

Is the problem that the per-domain config is always set, and doesn't take the hypervisor-set config into account? Wouldn't it be better to modify the toolstack to use the hypervisor value if it's not set?

In fact, it looks kind of like things are screwed up anyway -- the "default" value of max_grant_frames, if no value is specified, is set in xl.c. If that were the behavior we wanted, it should be set in libxl.c.

But it doesn't seem like it should be terribly difficult to get a "use the default" sentinel value passed in to Xen, such that:

1. People who don't do anything will get the default currently specified in xl.c

Quoted 7 lines

2. People who set the value on the Xen command-line and don't set anything in the guest config file will get the Xen command-line value

3. People who set the value in the config file will get the value they specified (regardless of the global setting).

Is that the behaviour you'd like to see, Paul?

I think the order should be:

If set in xl.cfg => use that, else
If set in xl.conf => use that, else
Use the command line/default value

I.e. the ultimate value should be set in Xen (and possibly overridden by the command line) and not hardcoded at any other layer.

There is also the issue of limits but I guess the rationale there should be: If a value *is* specified then it should not exceed the value set in Xen.

Does that sound right?

Paul

Quoted 2 lines

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw
Quoted 72 lines · Paul
Quoted 58 lines

-----Original Message-----
From: George Dunlap <[email protected]>
Sent: 26 November 2019 12:32
To: Paul Durrant <[email protected]>; Durrant, Paul <[email protected]>
Cc: xen-devel <[email protected]>; Stefano Stabellini <[email protected]>; Julien Grall <[email protected]>; Wei Liu <[email protected]>; Konrad Rzeszutek Wilk <[email protected]>; George Dunlap <[email protected]>; Andrew Cooper <[email protected]>; Ian Jackson <[email protected]>; Jan Beulich <[email protected]>
Subject: Re: [Xen-devel] [PATCH] domain_create: honour global grant/maptrack frame limits...

Quoted 15 lines · Paul Durrant
Quoted 14 lines · Paul Durrant

...when their values are larger than the per-domain configured limits.

---
Cc: Andrew Cooper <[email protected]>
Cc: George Dunlap <[email protected]>
Cc: Ian Jackson <[email protected]>
Cc: Jan Beulich <[email protected]>
Cc: Julien Grall <[email protected]>
Cc: Konrad Rzeszutek Wilk <[email protected]>
Cc: Stefano Stabellini <[email protected]>
Cc: Wei Liu <[email protected]>

Quoted 2 lines
Quoted 2 lines

After mining through commits it is still unclear to me exactly when Xen stopped honouring the global values, but I really think this commit

should

Quoted 8 lines
Quoted 2 lines

be back-ported to stable trees as it was a behavioural change that can cause domUs to fail in non-obvious ways.

Any other opinions on this? AFAICT questions is still open:

- Do we consider not honouring the command line values to be a regression (since domUs that would have worked before will no longer work after a basic upgrade of Xen)?

This would be a bit easier to form a "policy" opinion on (or perhaps alternate solutions to) if more of the situation were outlined here.

Is the problem that the per-domain config is always set, and doesn't take the hypervisor-set config into account? Wouldn't it be better to modify the toolstack to use the hypervisor value if it's not set?

In fact, it looks kind of like things are screwed up anyway -- the "default" value of max_grant_frames, if no value is specified, is set in xl.c. If that were the behavior we wanted, it should be set in libxl.c.

But it doesn't seem like it should be terribly difficult to get a "use the default" sentinel value passed in to Xen, such that:

1. People who don't do anything will get the default currently specified in xl.c

Quoted 7 lines

2. People who set the value on the Xen command-line and don't set anything in the guest config file will get the Xen command-line value

3. People who set the value in the config file will get the value they specified (regardless of the global setting).

Is that the behaviour you'd like to see, Paul?

I think the order should be:

If set in xl.cfg => use that, else
If set in xl.conf => use that, else
Use the command line/default value

Quoted 5 lines

I.e. the ultimate value should be set in Xen (and possibly overridden by the command line) and not hardcoded at any other layer.

There is also the issue of limits but I guess the rationale there should be: If a value *is* specified then it should not exceed the value set in Xen.

Does that sound right?

So part of the issue here sounds like a terminology issue. Is it the case that there's a default "max", and you want to raise the default "max"; is that right?

But the documentation actually says:

"Specify the maximum number of frames which any domain may use as part of its grant table."

Which makes it sound a lot more like a "maximum max" -- i.e., that any domain which is created with a value higher than this should fail.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel