Skip to contentSkip to conversation

Reset iomem's gfn to LIBXL_INVALID_GFN on reboot

PATCH In review Oleksandr Grytsov 3 participants 6 messages Oct 10, 2019 – Oct 24, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Reset iomem's gfn to LIBXL_INVALID_GFN on reboot

From: Oleksandr Andrushchenko <[email protected]>

During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to LIBXL_INVALID_GFN, thus making Xen to map IOMEM_START to address 0 in the guest's address space.

Workaround it by resseting GFN to LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.

---
 tools/libxl/libxl_domain.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/libxl/libxl_domain.c +9 -0
index 9d0eb5aed1..0ae16a5b12 100644
@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc,
         }
     }
 
+    /* reset IOMEM's GFN to initial value */
+    {
+        int i;
+
+        for (i = 0; i < d_config->b_info.num_iomem; i++)
+            if (d_config->b_info.iomem[i].gfn == 0)
+                d_config->b_info.iomem[i].gfn = LIBXL_INVALID_GFN;
+    }
+
     /* Devices: disk, nic, vtpm, pcidev etc. */
 
     /* The MERGE macro implements following logic:
Signature · 7 lines
2.17.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to LIBXL_INVALID_GFN on reboot"):

Quoted 10 lines

During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to LIBXL_INVALID_GFN, thus making Xen to map IOMEM_START to address 0 in the guest's address space.

Quoted 2 lines

Workaround it by resseting GFN to LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.

Thanks for this patch.

I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.

I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.

BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.

Thanks,
Ian.

Quoted 28 lines
---
 tools/libxl/libxl_domain.c | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/libxl/libxl_domain.c +9 -0
index 9d0eb5aed1..0ae16a5b12 100644
@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc,
         }
     }
 
+    /* reset IOMEM's GFN to initial value */
+    {
+        int i;
+
+        for (i = 0; i < d_config->b_info.num_iomem; i++)
+            if (d_config->b_info.iomem[i].gfn == 0)
+                d_config->b_info.iomem[i].gfn = LIBXL_INVALID_GFN;
+    }
+
     /* Devices: disk, nic, vtpm, pcidev etc. */
 
     /* The MERGE macro implements following logic:
Signature · 1 line
2.17.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw

Hi Ian,

Quoted 16 lines · Ian Jackson

Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to LIBXL_INVALID_GFN on reboot"):

Quoted 10 lines

During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to LIBXL_INVALID_GFN, thus making Xen to map IOMEM_START to address 0 in the guest's address space.

Quoted 2 lines

Workaround it by resseting GFN to LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.

Thanks for this patch.

Quoted 8 lines

I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.

I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.

I found a thread from December 2017 related to the problem described here [1].

Looking at the thread, there were no conclusion on the root causes and some questions were left unanswered by the contributor (see [2]).

Oleksandr, could you look at the thread and see if you can provide more details what's going on? Answering back here would be fine.

Quoted 31 lines

BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.

Thanks,
Ian.

Quoted 24 lines
---
  tools/libxl/libxl_domain.c | 9 +++++++++
  1 file changed, 9 insertions(+)
tools/libxl/libxl_domain.c +9 -0
index 9d0eb5aed1..0ae16a5b12 100644
@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc,
          }
      }
 
+    /* reset IOMEM's GFN to initial value */
+    {
+        int i;
+
+        for (i = 0; i < d_config->b_info.num_iomem; i++)
+            if (d_config->b_info.iomem[i].gfn == 0)
+                d_config->b_info.iomem[i].gfn = LIBXL_INVALID_GFN;
+    }
+
      /* Devices: disk, nic, vtpm, pcidev etc. */
 
Quoted 4 lines
Quoted 4 lines
tools/libxl/libxl_domain.c +0 -0
      /* The MERGE macro implements following logic:
Signature · 1 line
2.17.1

Cheers,

[1] <[email protected]>
[2] <[email protected]>

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw
Quoted 75 lines · Julien Grall

Hi Ian,

Quoted 16 lines · Ian Jackson

Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to LIBXL_INVALID_GFN on reboot"):

Quoted 10 lines

During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to LIBXL_INVALID_GFN, thus making Xen to map IOMEM_START to address 0 in the guest's address space.

Quoted 2 lines

Workaround it by resseting GFN to LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.

Thanks for this patch.

Quoted 8 lines

I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.

I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.

I found a thread from December 2017 related to the problem described here [1].

Looking at the thread, there were no conclusion on the root causes and some questions were left unanswered by the contributor (see [2]).

Oleksandr, could you look at the thread and see if you can provide more details what's going on? Answering back here would be fine.

Quoted 31 lines

BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.

Thanks,
Ian.

Quoted 24 lines
---
  tools/libxl/libxl_domain.c | 9 +++++++++
  1 file changed, 9 insertions(+)
tools/libxl/libxl_domain.c +9 -0
index 9d0eb5aed1..0ae16a5b12 100644
@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc,
          }
      }
 
+    /* reset IOMEM's GFN to initial value */
+    {
+        int i;
+
+        for (i = 0; i < d_config->b_info.num_iomem; i++)
+            if (d_config->b_info.iomem[i].gfn == 0)
+                d_config->b_info.iomem[i].gfn = LIBXL_INVALID_GFN;
+    }
+
      /* Devices: disk, nic, vtpm, pcidev etc. */
 
Quoted 4 lines
Quoted 4 lines
tools/libxl/libxl_domain.c +0 -1
      /* The MERGE macro implements following logic:
--

2.17.1

Cheers,

Quoted 5 lines

[1] <[email protected]>
[2] <[email protected]>

--
Julien Grall

Julien,

Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.

For example, libxl_domain_config_from_json calls libxl_domain_config_init which initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (see libxl__domain_build_info_parse_json iomem parsing):

p->num_iomem = x->u.array->count;
p->iomem = libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));

if (!p->iomem && p->num_iomem != 0) {
    rc = -1;
    goto out;
}
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
    rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
    if (rc)
       goto out;
}

libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:

for (i=0; (t=libxl__json_array_get(x,i)); i++) {
    libxl_iomem_range_init(&p->iomem[i]);
    rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
    if (rc)
       goto out;
}

I've changes gentypes.py as following:

tools/libxl/gentypes.py +1 -0
index 88e5c5f30e..92e28be469 100644
@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = "
 ", parent = None, discrimina
         s += "        goto out;\n"
         s += "    }\n"
         s += "    for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n"
+        if ty.elem_type.init_fn is not None and

ty.elem_type.autogenerate_init_fn:
+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)

         s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]",
                                      indent + "    ", parent)
         s += "    }\n"

I'm not sure is it right and complete fix.

Ian, could you review?

If the fix is ok, I will submit the patch.

Thanks.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Hi Oleksandr,

Apologies for the late answer.

Quoted 31 lines · Oleksandr Grytsov

On Mon, Oct 14, 2019 at 12:28 PM Julien Grall <[email protected]> wrote:
Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.

For example, libxl_domain_config_from_json calls libxl_domain_config_init which initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (see libxl__domain_build_info_parse_json iomem parsing):

p->num_iomem = x->u.array->count;
p->iomem = libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));

if (!p->iomem && p->num_iomem != 0) {
     rc = -1;
     goto out;
}
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
     rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
     if (rc)
        goto out;
}

libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:

Quoted 6 lines
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
     libxl_iomem_range_init(&p->iomem[i]);
     rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
     if (rc)
        goto out;
}

Not initializing the values is fine as long as the JSON describes all the fields of the structure.

The key point here is the GFN is not described in the JSON (see libxl_iomem_range_gen_json) if it is equal to LIBXL_INVALID_GFN. As the field is not described, it will be defaulted to 0.

Quoted 13 lines

I've changes gentypes.py as following:

tools/libxl/gentypes.py +1 -0
index 88e5c5f30e..92e28be469 100644
@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = "
  ", parent = None, discrimina
          s += "        goto out;\n"
          s += "    }\n"
          s += "    for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n"
+        if ty.elem_type.init_fn is not None and
Quoted 1 line

ty.elem_type.autogenerate_init_fn:

My knowledge of libxl is quite limited. But I don't think this is correct, you want to call init_fn whether this has been autogenerated or not.

Quoted 2 lines

+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)

Looking at the other usage (like _libxl_C_type_init), init_fn is called with

            s += "%s(%s);\n" % (ty.init_fn, ty.pass_arg(v, parent is None))

I am also not entirely sure whether we should also cater the ty.init_val != None as well here.

Quoted 9 lines
          s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]",
                                       indent + "    ", parent)
          s += "    }\n"

I'm not sure is it right and complete fix.

Ian, could you review?

If the fix is ok, I will submit the patch.

IHMO, the idea is there. The code may need some modifications (see above). Please post a patch so we can go forward in the process to review it.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw
Quoted 44 lines · Julien Grall

Hi Oleksandr,

Apologies for the late answer.

Quoted 31 lines · Oleksandr Grytsov

On Mon, Oct 14, 2019 at 12:28 PM Julien Grall <[email protected]> wrote:
Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.

For example, libxl_domain_config_from_json calls libxl_domain_config_init which initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (see libxl__domain_build_info_parse_json iomem parsing):

p->num_iomem = x->u.array->count;
p->iomem = libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));

if (!p->iomem && p->num_iomem != 0) {
     rc = -1;
     goto out;
}
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
     rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
     if (rc)
        goto out;
}

libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:

Quoted 6 lines
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
     libxl_iomem_range_init(&p->iomem[i]);
     rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
     if (rc)
        goto out;
}
Quoted 7 lines

Not initializing the values is fine as long as the JSON describes all the fields of the structure.

The key point here is the GFN is not described in the JSON (see libxl_iomem_range_gen_json) if it is equal to LIBXL_INVALID_GFN. As the field is not described, it will be defaulted to 0.

Yes. So, either ..._gen_json should generate entries for default values or ..._parse_json
should set missing entries to its default values. Second solution looks more correct.

Quoted 32 lines
Quoted 13 lines

I've changes gentypes.py as following:

tools/libxl/gentypes.py +1 -0
index 88e5c5f30e..92e28be469 100644
@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = "
  ", parent = None, discrimina
          s += "        goto out;\n"
          s += "    }\n"
          s += "    for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n"
+        if ty.elem_type.init_fn is not None and
Quoted 1 line

ty.elem_type.autogenerate_init_fn:

My knowledge of libxl is quite limited. But I don't think this is correct, you want to call init_fn whether this has been autogenerated or not.

Quoted 2 lines

+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)

Looking at the other usage (like _libxl_C_type_init), init_fn is called with

             s += "%s(%s);\n" % (ty.init_fn, ty.pass_arg(v, parent is None))

I am also not entirely sure whether we should also cater the ty.init_val != None as well here.

Quoted 4 lines
          s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]",
                                       indent + "    ", parent)
          s += "    }\n"
Quoted 8 lines
Quoted 5 lines

I'm not sure is it right and complete fix.

Ian, could you review?

If the fix is ok, I will submit the patch.

IHMO, the idea is there. The code may need some modifications (see above). Please post a patch so we can go forward in the process to review it.

Thanks. I will post the separate path.

Quoted 5 lines

Cheers,

--
Julien Grall

Signature · 7 lines
Best Regards,
Oleksandr Grytsov.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel