Reset iomem's gfn to LIBXL_INVALID_GFN on reboot
PATCH 1/1 Reset iomem's gfn to LIBXL_INVALID_GFN on reboot
From: Oleksandr Andrushchenko <[email protected]>
During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to LIBXL_INVALID_GFN, thus making Xen to map IOMEM_START to address 0 in the guest's address space.
Workaround it by resseting GFN to LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.
--- tools/libxl/libxl_domain.c | 9 +++++++++ 1 file changed, 9 insertions(+)
tools/libxl/libxl_domain.c +9 -0
@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc, } } +/* reset IOMEM's GFN to initial value */ + { +int i; + +for (i =0 ; i < d_config->b_info.num_iomem; i++) +if (d_config->b_info.iomem[i].gfn ==0 ) + d_config->b_info.iomem[i].gfn =LIBXL_INVALID_GFN ; + } +/* Devices: disk, nic, vtpm, pcidev etc. */ /* The MERGE macro implements following logic:
Signature · 7 lines
2.17.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to LIBXL_INVALID_GFN on reboot"):
Quoted 10 lines
During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to
LIBXL_INVALID_GFN, thus making Xen to mapIOMEM_STARTto address 0 in the guest's address space.
Quoted 2 lines
Workaround it by resseting GFN to
LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.
Thanks for this patch.
I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.
I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.
BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.
Thanks,
Ian.
Quoted 28 lines
Signed-off-by Oleksandr Andrushchenko <[email protected]>--- tools/libxl/libxl_domain.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/libxl/libxl_domain.c +9
-0@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc, } } +/* reset IOMEM's GFN to initial value */ + { +int i; + +for (i =0 ; i < d_config->b_info.num_iomem; i++) +if (d_config->b_info.iomem[i].gfn ==0 ) + d_config->b_info.iomem[i].gfn =LIBXL_INVALID_GFN ; + } +/* Devices: disk, nic, vtpm, pcidev etc. */ /* The MERGE macro implements following logic: Signature · 1 line
2.17.1
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Ian,
Quoted 16 lines · Ian Jackson
Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to
LIBXL_INVALID_GFNon reboot"):Quoted 10 lines
During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to
LIBXL_INVALID_GFN, thus making Xen to mapIOMEM_STARTto address 0 in the guest's address space.Quoted 2 lines
Workaround it by resseting GFN to
LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.Thanks for this patch.
Quoted 8 lines
I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.
I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.
I found a thread from December 2017 related to the problem described here [1].
Looking at the thread, there were no conclusion on the root causes and some questions were left unanswered by the contributor (see [2]).
Oleksandr, could you look at the thread and see if you can provide more details what's going on? Answering back here would be fine.
Quoted 31 lines
BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.
Thanks,
Ian.Quoted 24 lines
Signed-off-by Oleksandr Andrushchenko <[email protected]>--- tools/libxl/libxl_domain.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/libxl/libxl_domain.c +9
-0@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc, } } +/* reset IOMEM's GFN to initial value */ + { +int i; + +for (i =0 ; i < d_config->b_info.num_iomem; i++) +if (d_config->b_info.iomem[i].gfn ==0 ) + d_config->b_info.iomem[i].gfn =LIBXL_INVALID_GFN ; + } +/* Devices: disk, nic, vtpm, pcidev etc. */
Quoted 4 lines
Quoted 4 lines
tools/libxl/libxl_domain.c +0
-0/* The MERGE macro implements following logic: Signature · 1 line
2.17.1
Cheers,
[1] <[email protected]>
[2] <[email protected]>
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 75 lines · Julien Grall
Hi Ian,
Quoted 16 lines · Ian Jackson
Oleksandr Grytsov writes ("[PATCH v1] Reset iomem's gfn to
LIBXL_INVALID_GFNon reboot"):Quoted 10 lines
During domain reboot its configuration is partially reused to re-create a new domain, but iomem's GFN field for the iomem is only restored for those memory ranges, which are configured in form of [IOMEM_START,NUM_PAGES[@GFN], but not for those in form of [IOMEM_START,NUM_PAGES], e.g. without GFN. For the latter GFN is reset to 0, but while mapping ranges to a domain during reboot there is a check that GFN treated as valid if it is not equal to
LIBXL_INVALID_GFN, thus making Xen to mapIOMEM_STARTto address 0 in the guest's address space.Quoted 2 lines
Workaround it by resseting GFN to
LIBXL_INVALID_GFN, so xl can set proper values for mapping on reboot.Thanks for this patch.
Quoted 8 lines
I confess that I am not sure what is going on here. Where is this troublesome 0 coming from ? I see that the default value for gfn in the struct is 0 and looking for assignments before this patch, gfn is defaulted from b_info->iomem[i].start, which is presumably non-0.
I suspect that your patch may be fixing this the wrong way. I have addressed this mail to various people who have touched this area of code and hope they will be able to clarify.
I found a thread from December 2017 related to the problem described here [1].
Looking at the thread, there were no conclusion on the root causes and some questions were left unanswered by the contributor (see [2]).
Oleksandr, could you look at the thread and see if you can provide more details what's going on? Answering back here would be fine.
Quoted 31 lines
BTW, please do ping this (and your other patches) by email, if the conversation seems to stall.
Thanks,
Ian.Quoted 24 lines
Signed-off-by Oleksandr Andrushchenko <[email protected]>--- tools/libxl/libxl_domain.c | 9 +++++++++ 1 file changed, 9 insertions(+)tools/libxl/libxl_domain.c +9
-0@@ -2120,6 +2120,15 @@ static void retrieve_domain_configuration_end(libxl__egc *egc, } } +/* reset IOMEM's GFN to initial value */ + { +int i; + +for (i =0 ; i < d_config->b_info.num_iomem; i++) +if (d_config->b_info.iomem[i].gfn ==0 ) + d_config->b_info.iomem[i].gfn =LIBXL_INVALID_GFN ; + } +/* Devices: disk, nic, vtpm, pcidev etc. */ Quoted 4 lines
Quoted 4 lines
tools/libxl/libxl_domain.c +0
-1/* The MERGE macro implements following logic: -- 2.17.1
Cheers,
Quoted 5 lines
[1] <[email protected]>
[2] <[email protected]>--
Julien Grall
Julien,
Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.
For example, libxl_domain_config_from_json calls libxl_domain_config_init which initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (see libxl__domain_build_info_parse_json iomem parsing):
p->num_iomem = x->u.array->count;p->iomem = libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));
if (!p->iomem && p->num_iomem != 0) {
rc = -1;
goto out;
}
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
if (rc)
goto out;
}libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
libxl_iomem_range_init(&p->iomem[i]);
rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
if (rc)
goto out;
}I've changes gentypes.py as following:
tools/libxl/gentypes.py +1 -0
@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = " ", parent = None, discrimina s += " goto out;\n" s += " }\n" s += " for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n" + if ty.elem_type.init_fn is not None and
ty.elem_type.autogenerate_init_fn:
+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)
s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]",
indent + " ", parent)
s += " }\n"I'm not sure is it right and complete fix.
Ian, could you review?
If the fix is ok, I will submit the patch.
Thanks.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Hi Oleksandr,
Apologies for the late answer.
Quoted 31 lines · Oleksandr Grytsov
On Mon, Oct 14, 2019 at 12:28 PM Julien Grall <[email protected]> wrote:
Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.For example,
libxl_domain_config_from_jsoncallslibxl_domain_config_initwhich initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (seelibxl__domain_build_info_parse_jsoniomem parsing):
p->num_iomem=x->u.array->count;p->iomem=libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));if (!p->iomem && p->num_iomem != 0) { rc = -1; goto out; } for (i=0; (t=libxl__json_array_get(x,i)); i++) { rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]); if (rc) goto out; }libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:
Quoted 6 lines
for (i=0; (t=libxl__json_array_get(x,i)); i++) {
libxl_iomem_range_init(&p->iomem[i]);
rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]);
if (rc)
goto out;
}Not initializing the values is fine as long as the JSON describes all the fields of the structure.
The key point here is the GFN is not described in the JSON (see libxl_iomem_range_gen_json) if it is equal to LIBXL_INVALID_GFN. As the field is not described, it will be defaulted to 0.
Quoted 13 lines
I've changes
gentypes.pyas following:tools/libxl/gentypes.py +1
-0@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = " ", parent = None, discrimina s += " goto out;\n" s += " }\n" s += " for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n" + if ty.elem_type.init_fn is not None and
Quoted 1 line
ty.elem_type.autogenerate_init_fn:
My knowledge of libxl is quite limited. But I don't think this is correct, you want to call init_fn whether this has been autogenerated or not.
Quoted 2 lines
+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)
Looking at the other usage (like _libxl_C_type_init), init_fn is called with
s += "%s(%s);\n" % (ty.init_fn, ty.pass_arg(v, parent is None))
I am also not entirely sure whether we should also cater the ty.init_val != None as well here.
Quoted 9 lines
s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]", indent + " ", parent) s += " }\n"I'm not sure is it right and complete fix.
Ian, could you review?
If the fix is ok, I will submit the patch.
IHMO, the idea is there. The code may need some modifications (see above). Please post a patch so we can go forward in the process to review it.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 44 lines · Julien Grall
Hi Oleksandr,
Apologies for the late answer.
Quoted 31 lines · Oleksandr Grytsov
On Mon, Oct 14, 2019 at 12:28 PM Julien Grall <[email protected]> wrote:
Thanks to point me out for this old thread. I completely forgot about it (I haven't worked with xen since long time). I've performed additional investigation
and found the root cause of the issue. It doesn't relate to iomem GFN directly. The problem is in type from json parsing at place where libxl creates array of struct.For example,
libxl_domain_config_from_jsoncallslibxl_domain_config_initwhich initializes all child structures and arrays. But then when libxl parses json and creates the array of structure, it doesn't initialize array elements properly (seelibxl__domain_build_info_parse_jsoniomem parsing):
p->num_iomem=x->u.array->count;p->iomem=libxl__calloc(NOGC, p->num_iomem, sizeof(*p->iomem));if (!p->iomem && p->num_iomem != 0) { rc = -1; goto out; } for (i=0; (t=libxl__json_array_get(x,i)); i++) { rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]); if (rc) goto out; }libxl creates array element with calloc function, so all element fields are initialized
with zero values. Even some of them have default value different from zero. For these purpose dedicated init function should be called for each element. Above example should be:Quoted 6 lines
for (i=0; (t=libxl__json_array_get(x,i)); i++) { libxl_iomem_range_init(&p->iomem[i]); rc = libxl__iomem_range_parse_json(gc, t, &p->iomem[i]); if (rc) goto out; }
Quoted 7 lines
Not initializing the values is fine as long as the JSON describes all the fields of the structure.
The key point here is the GFN is not described in the JSON (see
libxl_iomem_range_gen_json) if it is equal toLIBXL_INVALID_GFN. As the field is not described, it will be defaulted to 0.
Yes. So, either ..._gen_json should generate entries for default values or ..._parse_json
should set missing entries to its default values. Second solution looks more correct.
Quoted 32 lines
Quoted 13 lines
I've changes
gentypes.pyas following:tools/libxl/gentypes.py +1
-0@@ -454,6 +454,8 @@ def libxl_C_type_parse_json(ty, w, v, indent = " ", parent = None, discrimina s += " goto out;\n" s += " }\n" s += " for (i=0; (t=libxl__json_array_get(x,i)); i++) {\n" + if ty.elem_type.init_fn is not None andQuoted 1 line
ty.elem_type.autogenerate_init_fn:
My knowledge of libxl is quite limited. But I don't think this is correct, you want to call
init_fnwhether this has been autogenerated or not.Quoted 2 lines
+ s += indent + " "+"%s_init(&%s[i]);\n" %
(ty.elem_type.typename, v)Looking at the other usage (like
_libxl_C_type_init),init_fnis called withs += "%s(%s);\n" % (ty.init_fn, ty.pass_arg(v, parent is None))I am also not entirely sure whether we should also cater the ty.init_val != None as well here.
Quoted 4 lines
s += libxl_C_type_parse_json(ty.elem_type, "t", v+"[i]", indent + " ", parent) s += " }\n"
Quoted 8 lines
Quoted 5 lines
I'm not sure is it right and complete fix.
Ian, could you review?
If the fix is ok, I will submit the patch.
IHMO, the idea is there. The code may need some modifications (see above). Please post a patch so we can go forward in the process to review it.
Thanks. I will post the separate path.
Quoted 5 lines
Cheers,
--
Julien Grall
Signature · 7 lines
Best Regards, Oleksandr Grytsov. _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel