support >3 level p2m tree
cover support >3 level p2m tree
Pv domains write the mfn of a 3 level p2m tree to arch_shared_info structure. Consumers of this information are the domain save/restore functions of the Xen tools.
Being defined as having 3 levels the maximum supported domain size is 512 GB. The following patch expands the arch_shared_info structure to support more levels.
The Xen tools are not covered in this patch as the patch series "Libxl migration v2 support":
http://lists.xen.org/archives/html/xen-devel/2014-09/msg00427.html
should be applied first.
Juergen Gross (1):
expand x86 arch_shared_info to support >3 level p2m tree
xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.4.5
PATCH 1/1 expand x86 arch_shared_info to support >3 level p2m tree
The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain. To be able to support more RAM an additional level is to be added.
This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).
--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-x86/xen.h +7 -1
@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */ xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains +* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ]; };typedef struct arch_shared_info arch_shared_info_t ;
Signature · 1 line
1.8.4.5
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.
Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 20 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)xen/include/public/arch-x86/xen.h +0
-0@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 233
Quoted 10 lines
xen/include/public/arch-x86/xen.h +7
-1xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains +* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];
This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
~Andrew
Quoted 3 lines
xen/include/public/arch-x86/xen.h +0
-0};typedef struct arch_shared_info arch_shared_info_t ;
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
Quoted 37 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 25 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)xen/include/public/arch-x86/xen.h +2
-1@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */ xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 8 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 233
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5
-0+* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
The padding in this case is a nightmare.
For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[]. This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.
And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.
As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...
What is your preference?
Juergen
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
Quoted 48 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 21 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 9 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Quoted 6 lines
Review
Quoted 6 lines
Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/xen/include/public/arch-x86/xen.hQuoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224
Quoted 5 lines
index f35804b..b7fa2b6 100644
xen/include/public/arch-x86/xen.h +0
-0@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */
Quoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info {
Quoted 5 lines
xen/include/public/arch-x86/xen.h +2
-1xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 6 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5
-0+* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];
Quoted 6 lines
Quoted 2 lines
This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
The padding in this case is a nightmare.
For 32 bits
arch_shared_info_tis 64 bit aligned due touint64_tpad[].
No - uint64_t is 4-byte aligned on 32-bit.
Quoted 11 lines
This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in
shared_info_t). And before pad[] there is another 4-byte hole.And don't forget: on 32 bits
xen_pfn_tis 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.As
arch_shared_info_tis at the end ofshared_info_tI could just ignore the alignment/padding ...What is your preference?
As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.
Jan
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 9 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Quoted 6 lines
Review
Quoted 6 lines
Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/xen/include/public/arch-x86/xen.hQuoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224
Quoted 5 lines
index f35804b..b7fa2b6 100644
xen/include/public/arch-x86/xen.h +0
-0@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */
Quoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info {
Quoted 5 lines
xen/include/public/arch-x86/xen.h +2
-1xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5
-0+* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];Quoted 5 lines
Quoted 2 lines
This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
The padding in this case is a nightmare.
For 32 bits
arch_shared_info_tis 64 bit aligned due touint64_tpad[].No -
uint64_tis 4-byte aligned on 32-bit.Quoted 7 lines
This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in
shared_info_t). And before pad[] there is another 4-byte hole.And don't forget: on 32 bits
xen_pfn_tis 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.
Quoted 9 lines
Quoted 4 lines
As
arch_shared_info_tis at the end ofshared_info_tI could just ignore the alignment/padding ...What is your preference?
As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.
Jan
Which begs the question why the padding exists in the first place?
arch_shared_info is the final field in shared_info, and shared_info is explicitly documented not to necessarily have a consistent size across different versions of Xen.
Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.
~Andrew
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
Quoted 71 lines · Andrew Cooper
Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 9 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Quoted 6 lines
Review
Quoted 6 lines
Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/xen/include/public/arch-x86/xen.hQuoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224
Quoted 5 lines
index f35804b..b7fa2b6 100644
xen/include/public/arch-x86/xen.h +0
-0@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */
Quoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info {
Quoted 5 lines
xen/include/public/arch-x86/xen.h +2
-1xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5
-0+* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];Quoted 5 lines
Quoted 2 lines
This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
The padding in this case is a nightmare.
For 32 bits
arch_shared_info_tis 64 bit aligned due touint64_tpad[].No -
uint64_tis 4-byte aligned on 32-bit.Quoted 7 lines
This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in
shared_info_t). And before pad[] there is another 4-byte hole.And don't forget: on 32 bits
xen_pfn_tis 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.Quoted 9 lines
Quoted 4 lines
As
arch_shared_info_tis at the end ofshared_info_tI could just ignore the alignment/padding ...What is your preference?
As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.
Jan
Quoted 8 lines
Which begs the question why the padding exists in the first place?
arch_shared_infois the final field inshared_info, andshared_infois explicitly documented not to necessarily have a consistent size across different versions of Xen.Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.
Okay. I'll remove the padding.
Juergen
Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree
In reply to Andrew CooperQuoted 69 lines
Quoted 1 line
Quoted 1 line · On 09.09.14 at 11:36, <[email protected]> wrote:
Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross
The x86 struct
arch_shared_infofieldpfn_to_mfn_frame_list_listcurrently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.Specifically only 64bit PV domains have the 512GB limit.
32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.
Quoted 9 lines
To be
able to support more RAM an additional level is to be added.This patch expands struct
arch_shared_infowith a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL intopfn_to_mfn_frame_list_list(this should be done only if more than three levels are needed, of course).Quoted 6 lines
Review
Quoted 6 lines
Signed-off-by Juergen Gross <[email protected]>--- xen/include/public/arch-x86/xen.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)(quoted diff) +0
-0
b/xen/include/public/arch-x86/xen.hQuoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224
Quoted 5 lines
index f35804b..b7fa2b6 100644
xen/include/public/arch-x86/xen.h +0
-0@@ -224,7 +224,13 @@ struct arch_shared_info {/* Frame containing list of mfns containing list of mfns containing p2m. */
Quoted 5 lines
xen/include/public/arch-x86/xen.h struct arch_shared_info {
Quoted 5 lines
xen/include/public/arch-x86/xen.h +2
-1xen_pfn_t pfn_to_mfn_frame_list_list;unsigned long nmi_reason; -uint64_t pad[32 ]; +/* +* Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5
-0+* ~0UL. +*/ +unsigned long p2m_levels;/* number of levels of p2m tree */ +xen_pfn_t p2m_root;/* p2m tree top level mfn */ +uint64_t pad[30 ];Quoted 5 lines
Quoted 2 lines
This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.
The padding in this case is a nightmare.
For 32 bits
arch_shared_info_tis 64 bit aligned due touint64_tpad[].No -
uint64_tis 4-byte aligned on 32-bit.Quoted 7 lines
This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in
shared_info_t). And before pad[] there is another 4-byte hole.And don't forget: on 32 bits
xen_pfn_tis 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.Quoted 6 lines
Quoted 4 lines
As
arch_shared_info_tis at the end ofshared_info_tI could just ignore the alignment/padding ...What is your preference?
As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.
Quoted 8 lines
Which begs the question why the padding exists in the first place?
arch_shared_infois the final field inshared_info, andshared_infois explicitly documented not to necessarily have a consistent size across different versions of Xen.Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.
I agree.
Jan