Skip to contentSkip to conversation

support >3 level p2m tree

PATCH New Juergen Gross 3 participants 8 messages Sep 8, 2014 – Sep 9, 2014

lore
Share & tools
lore
#0 lore raw

cover support >3 level p2m tree

Pv domains write the mfn of a 3 level p2m tree to arch_shared_info structure. Consumers of this information are the domain save/restore functions of the Xen tools.

Being defined as having 3 levels the maximum supported domain size is 512 GB. The following patch expands the arch_shared_info structure to support more levels.

The Xen tools are not covered in this patch as the patch series "Libxl migration v2 support":
http://lists.xen.org/archives/html/xen-devel/2014-09/msg00427.html
should be applied first.

Juergen Gross (1):
expand x86 arch_shared_info to support >3 level p2m tree

 xen/include/public/arch-x86/xen.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Signature · 1 line
1.8.4.5
#1 lore raw

PATCH 1/1 expand x86 arch_shared_info to support >3 level p2m tree

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain. To be able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

---
 xen/include/public/arch-x86/xen.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-x86/xen.h +7 -1
index f35804b..b7fa2b6 100644
@@ -224,7 +224,13 @@ struct arch_shared_info {
     /* Frame containing list of mfns containing list of mfns containing p2m. */
     xen_pfn_t     pfn_to_mfn_frame_list_list;
     unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list contains
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];
 };
 typedef struct arch_shared_info arch_shared_info_t;
 
Signature · 1 line
1.8.4.5
#2 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 20 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

---
 xen/include/public/arch-x86/xen.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-x86/xen.h +0 -0
index f35804b..b7fa2b6 100644
@@ -224,7 +224,13 @@ struct arch_shared_info {
     /* Frame containing list of mfns containing list of mfns containing p2m. */

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 233

Quoted 10 lines
xen/include/public/arch-x86/xen.h +7 -1
     xen_pfn_t     pfn_to_mfn_frame_list_list;
     unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list contains
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

~Andrew

Quoted 3 lines
xen/include/public/arch-x86/xen.h +0 -0
 };
 typedef struct arch_shared_info arch_shared_info_t;
 
#3 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

Quoted 37 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 25 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

---
  xen/include/public/arch-x86/xen.h | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-x86/xen.h +2 -1
index f35804b..b7fa2b6 100644
@@ -224,7 +224,13 @@ struct arch_shared_info {
      /* Frame containing list of mfns containing list of mfns containing p2m. */
      xen_pfn_t     pfn_to_mfn_frame_list_list;
      unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list contains
Quoted 8 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 233

Quoted 5 lines
xen/include/public/arch-x86/xen.h +5 -0
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

The padding in this case is a nightmare.

For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[]. This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.

And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.

As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...

What is your preference?

Juergen

#4 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

Quoted 48 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 21 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 9 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

Quoted 6 lines

Review

Quoted 6 lines
---
  xen/include/public/arch-x86/xen.h | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/xen/include/public/arch-x86/xen.h

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224

Quoted 5 lines

index f35804b..b7fa2b6 100644

xen/include/public/arch-x86/xen.h +0 -0
@@ -224,7 +224,13 @@ struct arch_shared_info {
      /* Frame containing list of mfns containing list of mfns containing 

p2m. */

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info {

Quoted 5 lines
xen/include/public/arch-x86/xen.h +2 -1
      xen_pfn_t     pfn_to_mfn_frame_list_list;
      unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list 

contains

Quoted 6 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5 -0
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];
Quoted 6 lines
Quoted 2 lines

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

The padding in this case is a nightmare.

For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[].

No - uint64_t is 4-byte aligned on 32-bit.

Quoted 11 lines

This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.

And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.

As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...

What is your preference?

As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.

Jan

#5 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 9 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

Quoted 6 lines

Review

Quoted 6 lines
---
  xen/include/public/arch-x86/xen.h | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/xen/include/public/arch-x86/xen.h

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224

Quoted 5 lines

index f35804b..b7fa2b6 100644

xen/include/public/arch-x86/xen.h +0 -0
@@ -224,7 +224,13 @@ struct arch_shared_info {
      /* Frame containing list of mfns containing list of mfns containing 

p2m. */

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info {

Quoted 5 lines
xen/include/public/arch-x86/xen.h +2 -1
      xen_pfn_t     pfn_to_mfn_frame_list_list;
      unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list 

contains

Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5 -0
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];
Quoted 5 lines
Quoted 2 lines

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

The padding in this case is a nightmare.

For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[].

No - uint64_t is 4-byte aligned on 32-bit.

Quoted 7 lines

This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.

And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.

Quoted 9 lines
Quoted 4 lines

As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...

What is your preference?

As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.

Jan

Which begs the question why the padding exists in the first place?

arch_shared_info is the final field in shared_info, and shared_info is explicitly documented not to necessarily have a consistent size across different versions of Xen.

Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.

~Andrew

#6 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

Quoted 71 lines · Andrew Cooper
Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 9 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

Quoted 6 lines

Review

Quoted 6 lines
---
   xen/include/public/arch-x86/xen.h | 8 +++++++-
   1 file changed, 7 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/xen/include/public/arch-x86/xen.h

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224

Quoted 5 lines

index f35804b..b7fa2b6 100644

xen/include/public/arch-x86/xen.h +0 -0
@@ -224,7 +224,13 @@ struct arch_shared_info {
       /* Frame containing list of mfns containing list of mfns containing

p2m. */

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info {

Quoted 5 lines
xen/include/public/arch-x86/xen.h +2 -1
       xen_pfn_t     pfn_to_mfn_frame_list_list;
       unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list

contains

Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5 -0
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];
Quoted 5 lines
Quoted 2 lines

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

The padding in this case is a nightmare.

For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[].

No - uint64_t is 4-byte aligned on 32-bit.

Quoted 7 lines

This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.

And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.

Quoted 9 lines
Quoted 4 lines

As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...

What is your preference?

As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.

Jan

Quoted 8 lines

Which begs the question why the padding exists in the first place?

arch_shared_info is the final field in shared_info, and shared_info is explicitly documented not to necessarily have a consistent size across different versions of Xen.

Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.

Okay. I'll remove the padding.

Juergen

#7 lore raw

Re: [PATCH] expand x86 arch_shared_info to support >3 level p2m tree

In reply to Andrew Cooper
Quoted 69 lines
Quoted 1 line
Quoted 1 line · On 09.09.14 at 11:36, <[email protected]> wrote:
Quoted 60 lines · Jan Beulich
Quoted 46 lines
Quoted 1 line
Quoted 1 line · On 08.09.14 at 18:11, <[email protected]> wrote:
Quoted 20 lines · Andrew Cooper
Quoted 5 lines · Juergen Gross

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list currently contains the mfn of the top level page frame of the 3 level p2m tree, which is used by the Xen tools during saving and restoring (and live migration) of pv domains. With three levels of the p2m tree it is possible to support up to 512 GB of RAM for a pv domain.

Specifically only 64bit PV domains have the 512GB limit.

32bit PV domains have a far larger supported RAM as they can fit twice as many mfns in each p2m page.

Quoted 9 lines

To be
able to support more RAM an additional level is to be added.

This patch expands struct arch_shared_info with a new p2m tree root and the number of levels of the p2m tree. The new information is indicated by the domain to be valid by storing ~0UL into pfn_to_mfn_frame_list_list (this should be done only if more than three levels are needed, of course).

Quoted 6 lines

Review

Quoted 6 lines
---
  xen/include/public/arch-x86/xen.h | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/xen/include/public/arch-x86/xen.h

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info { · line 224

Quoted 5 lines

index f35804b..b7fa2b6 100644

xen/include/public/arch-x86/xen.h +0 -0
@@ -224,7 +224,13 @@ struct arch_shared_info {
      /* Frame containing list of mfns containing list of mfns containing 

p2m. */

Quoted 5 lines

xen/include/public/arch-x86/xen.h struct arch_shared_info {

Quoted 5 lines
xen/include/public/arch-x86/xen.h +2 -1
      xen_pfn_t     pfn_to_mfn_frame_list_list;
      unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following two fields are valid if pfn_to_mfn_frame_list_list 

contains

Quoted 5 lines
Quoted 5 lines
xen/include/public/arch-x86/xen.h +5 -0
+     * ~0UL.
+     */
+    unsigned long p2m_levels;   /* number of levels of p2m tree */
+    xen_pfn_t     p2m_root;     /* p2m tree top level mfn */
+    uint64_t pad[30];
Quoted 5 lines
Quoted 2 lines

This padding is now wrong, as unsigned long is only 4 bytes in 32bit, and it is adjacent to another unsigned long.

The padding in this case is a nightmare.

For 32 bits arch_shared_info_t is 64 bit aligned due to uint64_t pad[].

No - uint64_t is 4-byte aligned on 32-bit.

Quoted 7 lines

This enforces a 4 byte hole before the structure (there are 3 4-byte fields before it in shared_info_t). And before pad[] there is another 4-byte hole.

And don't forget: on 32 bits xen_pfn_t is 4 bytes, too. I could either align each new variable explicitly to 8 bytes or I could use a union.

Quoted 6 lines
Quoted 4 lines

As arch_shared_info_t is at the end of shared_info_t I could just ignore the alignment/padding ...

What is your preference?

As I view it, the exact padding size here doesn't really matter: The shared info lives in a separate page anyway.

Quoted 8 lines

Which begs the question why the padding exists in the first place?

arch_shared_info is the final field in shared_info, and shared_info is explicitly documented not to necessarily have a consistent size across different versions of Xen.

Frankly, I think the padding can just be removed. It serves no purpose, and attempting to maintain it correctly is proving very difficult.

I agree.

Jan