update xenctx to dump pagetables

PATCH In review Olaf Hering 5 participants 18 messages Jun 20, 2011 – Jun 22, 2011

Share & tools

Olaf Hering

author

2011-06-20 15:26 UTC #0 lore raw

PATCH 1/1 update xenctx to dump pagetables

The following series adds an option -P to xenctx to dump the guests pagetable. Another option to work an all vcpus is also added.

The reason for such a change is that on some systems a live migration from a system with a small amount of memory to a system with a huge amount of memory (>100GB) fails. DomU crashes on the target.
To get some debug data I'm trying to dump the pagetable, just to verify that nothing bad to them happend during transit.

The code to walk the pagetables is/was based on xc_translate_foreign_address(), but I still think I have some major bugs in there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped. Any help with getting that output fixed for at least a a 64bit PV guest is much appreciated.

Olaf

part of xenctx output:
...

  0 l4e  ff 00000000a1240067:  x   0     a1240000 0 0 1 A c t u W P
  0 l3e 1ff 00000000bc133067:  x   0     bc133000 0 0 s 1 A c t U W P
  0 l2e  64 00000000a1471067:  x   0     a1471000 0 0 s 1 A c t U W P
  0 l1e  46 80000000bd56f167:  X   0     bd56f000 0 G p s D A c t U W P     7fffcc846000
  0 l1e  47 80000000bd57c167:  X   0     bd57c000 0 G p s D A c t U W P     7fffcc847000
  0 l1e  48 80000000bd5af165:  X   0     bd5af000 0 G p s D A c t U w P     7fffcc848000
  0 l1e  db 00000000a1972125:  x   0     a1972000 0 G p s d A c t U w P     7fffcc8db000
  0 l4e 100 000000013fff8067:  x   0    13fff8000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 101 000000013fff0063:  x   0    13fff0000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 102 00000000bc5b0063:  x   0     bc5b0000 0 0 1 A c t u W P
  0 l3e  fe 00000000a7ed2067:  x   0     a7ed2000 0 0 s 1 A c t U W P
  0 l2e 163 00000000a154a067:  x   0     a154a000 0 0 s 1 A c t U W P
  0 l1e 1eb 00000000bc1a7067:  x   0     bc1a7000 0 g p s D A c t U W P     813fac7eb000
  0 l1e 1ec 00000000a154f067:  x   0     a154f000 0 g p s D A c t U W P     813fac7ec000

...

in dmesg:
...
(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=19 (XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=19
...

Olaf

Signature · 8 lines

[PATCH 1/5] xenctx: recognize also -S option for --stack-trace
[PATCH 2/5] xenctx: move all globals into struct xenctx
[PATCH 3/5] xenctx: move xc_* access out of dump_ctx
[PATCH 4/5] xenctx: add option -C to dump context for all vcpus
[PATCH 5/5] xenctx: dump pagetable

 tools/xentrace/xenctx.c |  405 ++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 329 insertions(+), 76 deletions(-)

Olaf Hering

author

2011-06-20 15:26 UTC #1 lore raw

PATCH 1/1 xenctx: recognize also -S option for --stack-trace

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308312572 -7200
# Node ID 434e772168bc2208e1b9219d690b5e4909b552a7 # Parent eca057e4475ca455ec36f962b9179fd2c9674196
xenctx: recognize also -S option for --stack-trace

Update help text.

Signed-off-by Olaf Hering <[email protected]>

tools/xentrace/xenctx.c +2 -2

diff -r eca057e4475c -r 434e772168bc tools/xentrace/xenctx.c
@@ -924,7 +924,7 @@ static void usage(void)
     printf("                    frame pointers.\n");
     printf("  -s SYMTAB, --symbol-table=SYMTAB\n");
     printf("                    read symbol table from SYMTAB.\n");
-    printf("  --stack-trace     print a complete stack trace.\n");
+    printf("  -S --stack-trace  print a complete stack trace.\n");
     printf("  -k, --kernel-start\n");
     printf("                    set user/kernel split. (default 0xc0000000)\n");
 #ifdef __ia64__

@@ -938,7 +938,7 @@ static void usage(void)
 int main(int argc, char **argv)
 {
     int ch;
-    static const char *sopts = "fs:hak:"
+    static const char *sopts = "fs:hak:S"
 #ifdef __ia64__
         "r:"
 #endif

Olaf Hering

author

2011-06-20 15:26 UTC #2 lore raw

PATCH 1/1 xenctx: move all globals into struct xenctx

In reply to Olaf Hering

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308312618 -7200
# Node ID 132480af62e4cb35d24350cf6ea1c2769f1cd1b5 # Parent 434e772168bc2208e1b9219d690b5e4909b552a7
xenctx: move all globals into struct xenctx

Move all globals used for options and libxc data into a new struct xenctx. This is used in subsequent changes.

Signed-off-by Olaf Hering <[email protected]>

tools/xentrace/xenctx.c +39 ~~-37~~

diff -r 434e772168bc -r 132480af62e4 tools/xentrace/xenctx.c
@@ -29,11 +29,14 @@
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/save.h>
 
-xc_interface *xc_handle = 0;
-int domid = 0;
-int frame_ptrs = 0;
-int stack_trace = 0;
-int disp_all = 0;
+static struct xenctx {
+    xc_interface *xc_handle;
+    int domid;
+    int frame_ptrs;
+    int stack_trace;
+    int disp_all;
+    xc_dominfo_t dominfo;
+} xenctx;
 
 #if defined (__i386__) || defined (__x86_64__)
 typedef unsigned long long guest_word_t;

@@ -300,7 +303,7 @@ static void print_ctx_32(vcpu_guest_cont
     printf(" fs:     %04x\t", regs->fs);
     printf(" gs:     %04x\n", regs->gs);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 4);
         print_special(ctx->debugreg, "dr", 0xcf, 4);
     }

@@ -329,7 +332,7 @@ static void print_ctx_32on64(vcpu_guest_
     printf(" fs:     %04x\t", regs->fs);
     printf(" gs:     %04x\n", regs->gs);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 4);
         print_special(ctx->debugreg, "dr", 0xcf, 4);
     }

@@ -373,7 +376,7 @@ static void print_ctx_64(vcpu_guest_cont
     printf(" gs: %04x @ %016"PRIx64"/%016"PRIx64"\n", regs->gs,
            ctx->gs_base_kernel, ctx->gs_base_user);
 
-    if (disp_all) {
+    if (xenctx.disp_all) {
         print_special(ctx->ctrlreg, "cr", 0x1d, 8);
         print_special(ctx->debugreg, "dr", 0xcf, 8);
     }

@@ -681,7 +684,7 @@ static void *map_page(vcpu_guest_context
     static unsigned long previous_mfn = 0;
     static void *mapped = NULL;
 
-    unsigned long mfn = xc_translate_foreign_address(xc_handle, domid, vcpu, virt);
+    unsigned long mfn = xc_translate_foreign_address(xenctx.xc_handle, xenctx.domid, vcpu, virt);
     unsigned long offset = virt & ~XC_PAGE_MASK;
 
     if (mapped && mfn == previous_mfn)

@@ -692,7 +695,7 @@ static void *map_page(vcpu_guest_context
 
     previous_mfn = mfn;
 
-    mapped = xc_map_foreign_range(xc_handle, domid, XC_PAGE_SIZE, PROT_READ, mfn);
+    mapped = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, XC_PAGE_SIZE, PROT_READ, mfn);
 
     if (mapped == NULL) {
         fprintf(stderr, "failed to map page.\n");

@@ -764,21 +767,21 @@ static void print_stack(vcpu_guest_conte
     }
     printf("\n");
 
-    if(stack_trace)
+    if(xenctx.stack_trace)
         printf("Stack Trace:\n");
     else
         printf("Call Trace:\n");
-    printf("%c [<", stack_trace ? '*' : ' ');
+    printf("%c [<", xenctx.stack_trace ? '*' : ' ');
     print_stack_word(instr_pointer(ctx), width);
     printf(">] ");
 
     print_symbol(instr_pointer(ctx));
     printf(" <--\n");
-    if (frame_ptrs) {
+    if (xenctx.frame_ptrs) {
         stack = stack_pointer(ctx);
         frame = frame_pointer(ctx);
         while(frame && stack < stack_limit) {
-            if (stack_trace) {
+            if (xenctx.stack_trace) {
                 while (stack < frame) {
                     p = map_page(ctx, vcpu, stack);
                     printf("|   ");

@@ -792,7 +795,7 @@ static void print_stack(vcpu_guest_conte
 
             p = map_page(ctx, vcpu, stack);
             frame = read_stack_word(p, width);
-            if (stack_trace) {
+            if (xenctx.stack_trace) {
                 printf("|-- ");
                 print_stack_word(read_stack_word(p, width), width);
                 printf("\n");

@@ -802,7 +805,7 @@ static void print_stack(vcpu_guest_conte
             if (frame) {
                 p = map_page(ctx, vcpu, stack);
                 word = read_stack_word(p, width);
-                printf("%c [<", stack_trace ? '|' : ' ');
+                printf("%c [<", xenctx.stack_trace ? '|' : ' ');
                 print_stack_word(word, width);
                 printf(">] ");
                 print_symbol(word);

@@ -821,7 +824,7 @@ static void print_stack(vcpu_guest_conte
                 printf(">] ");
                 print_symbol(word);
                 printf("\n");
-            } else if (stack_trace) {
+            } else if (xenctx.stack_trace) {
                 printf("    ");
                 print_stack_word(word, width);
                 printf("\n");

@@ -836,37 +839,36 @@ static void dump_ctx(int vcpu)
 {
     int ret;
     vcpu_guest_context_any_t ctx;
-    xc_dominfo_t dominfo;
 
-    xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
+    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
 
-    ret = xc_domain_getinfo(xc_handle, domid, 1, &dominfo);
+    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
     if (ret < 0) {
         perror("xc_domain_getinfo");
         exit(-1);
     }
     
-    ret = xc_domain_pause(xc_handle, domid);
+    ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
     if (ret < 0) {
         perror("xc_domain_pause");
         exit(-1);
     }
 
-    ret = xc_vcpu_getcontext(xc_handle, domid, vcpu, &ctx);
+    ret = xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx);
     if (ret < 0) {
-        if (!dominfo.paused)
-            xc_domain_unpause(xc_handle, domid);
+        if (!xenctx.dominfo.paused)
+            xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
         perror("xc_vcpu_getcontext");
         exit(-1);
     }
 
 #if defined(__i386__) || defined(__x86_64__)
     {
-        if (dominfo.hvm) {
+        if (xenctx.dominfo.hvm) {
             struct hvm_hw_cpu cpuctx;
             xen_capabilities_info_t xen_caps = "";
             if (xc_domain_hvm_getcontext_partial(
-                    xc_handle, domid, HVM_SAVE_CODE(CPU), 
+                    xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
                     vcpu, &cpuctx, sizeof cpuctx) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
                 exit(-1);

@@ -874,7 +876,7 @@ static void dump_ctx(int vcpu)
             guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
             guest_protected_mode = (cpuctx.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
-            if (xc_version(xc_handle, XENVER_capabilities, &xen_caps) != 0) {
+            if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");
                 exit(-1);
             }

@@ -882,9 +884,9 @@ static void dump_ctx(int vcpu)
         } else {
             struct xen_domctl domctl;
             memset(&domctl, 0, sizeof domctl);
-            domctl.domain = domid;
+            domctl.domain = xenctx.domid;
             domctl.cmd = XEN_DOMCTL_get_address_size;
-            if (xc_domctl(xc_handle, &domctl) == 0)
+            if (xc_domctl(xenctx.xc_handle, &domctl) == 0)
                 ctxt_word_size = guest_word_size = domctl.u.address_size.size / 8;
         }
     }

@@ -897,15 +899,15 @@ static void dump_ctx(int vcpu)
         print_stack(&ctx, vcpu, guest_word_size);
 #endif
 
-    if (!dominfo.paused) {
-        ret = xc_domain_unpause(xc_handle, domid);
+    if (!xenctx.dominfo.paused) {
+        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
         if (ret < 0) {
             perror("xc_domain_unpause");
             exit(-1);
         }
     }
 
-    ret = xc_interface_close(xc_handle);
+    ret = xc_interface_close(xenctx.xc_handle);
     if (ret < 0) {
         perror("xc_interface_close");
         exit(-1);

@@ -962,13 +964,13 @@ int main(int argc, char **argv)
     while ((ch = getopt_long(argc, argv, sopts, lopts, NULL)) != -1) {
         switch(ch) {
         case 'f':
-            frame_ptrs = 1;
+            xenctx.frame_ptrs = 1;
             break;
         case 's':
             symbol_table = optarg;
             break;
         case 'S':
-            stack_trace = 1;
+            xenctx.stack_trace = 1;
             break;
 #ifdef __ia64__
         case 'r':

@@ -1004,7 +1006,7 @@ int main(int argc, char **argv)
             break;
 #else
         case 'a':
-            disp_all = 1;
+            xenctx.disp_all = 1;
             break;
 #endif
         case 'k':

@@ -1026,8 +1028,8 @@ int main(int argc, char **argv)
         exit(-1);
     }
 
-    domid = atoi(argv[0]);
-    if (domid==0) {
+    xenctx.domid = atoi(argv[0]);
+    if (xenctx.domid==0) {
             fprintf(stderr, "cannot trace dom0\n");
             exit(-1);
     }

Olaf Hering

author

2011-06-20 15:26 UTC #6 lore raw

PATCH 1/1 xenctx: move xc_* access out of dump_ctx

In reply to Olaf Hering

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308312712 -7200
# Node ID 0e0c9fbedc2e29e2d507e2e1dce9536d06af442d # Parent 132480af62e4cb35d24350cf6ea1c2769f1cd1b5
xenctx: move xc_* access out of dump_ctx

move xc_* access out of dump_ctx.
Update code paths to return an error instead of calling exit(). On error unpause the guest if it was paused by xenctx.

Signed-off-by Olaf Hering <[email protected]>

tools/xentrace/xenctx.c +62 ~~-41~~

diff -r 132480af62e4 -r 0e0c9fbedc2e tools/xentrace/xenctx.c
@@ -35,6 +35,7 @@ static struct xenctx {
     int frame_ptrs;
     int stack_trace;
     int disp_all;
+    int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;

@@ -699,7 +700,7 @@ static void *map_page(vcpu_guest_context
 
     if (mapped == NULL) {
         fprintf(stderr, "failed to map page.\n");
-        exit(-1);
+        return NULL;
     }
 
  out:

@@ -722,7 +723,7 @@ static void print_stack_word(guest_word_
         printf(FMT_64B_WORD, word);
 }
 
-static void print_code(vcpu_guest_context_any_t *ctx, int vcpu)
+static int print_code(vcpu_guest_context_any_t *ctx, int vcpu)
 {
     guest_word_t instr;
     int i;

@@ -732,6 +733,8 @@ static void print_code(vcpu_guest_contex
     instr -= 21;
     for(i=0; i<32; i++) {
         unsigned char *c = map_page(ctx, vcpu, instr+i);
+        if (!c)
+            return -1;
         if (instr+i == instr_pointer(ctx))
             printf("<%02x> ", *c);
         else

@@ -740,9 +743,10 @@ static void print_code(vcpu_guest_contex
     printf("\n");
 
     printf("\n");
+    return 0;
 }
 
-static void print_stack(vcpu_guest_context_any_t *ctx, int vcpu, int width)
+static int print_stack(vcpu_guest_context_any_t *ctx, int vcpu, int width)
 {
     guest_word_t stack = stack_pointer(ctx);
     guest_word_t stack_limit;

@@ -758,6 +762,8 @@ static void print_stack(vcpu_guest_conte
     for (i=1; i<5 && stack < stack_limit; i++) {
         while(stack < stack_limit && stack < stack_pointer(ctx) + i*32) {
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             word = read_stack_word(p, width);
             printf(" ");
             print_stack_word(word, width);

@@ -784,6 +790,8 @@ static void print_stack(vcpu_guest_conte
             if (xenctx.stack_trace) {
                 while (stack < frame) {
                     p = map_page(ctx, vcpu, stack);
+                    if (!p)
+                        return -1;
                     printf("|   ");
                     print_stack_word(read_stack_word(p, width), width);
                     printf("   \n");

@@ -794,6 +802,8 @@ static void print_stack(vcpu_guest_conte
             }
 
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             frame = read_stack_word(p, width);
             if (xenctx.stack_trace) {
                 printf("|-- ");

@@ -804,6 +814,8 @@ static void print_stack(vcpu_guest_conte
 
             if (frame) {
                 p = map_page(ctx, vcpu, stack);
+                if (!p)
+                    return -1;
                 word = read_stack_word(p, width);
                 printf("%c [<", xenctx.stack_trace ? '|' : ' ');
                 print_stack_word(word, width);

@@ -817,6 +829,8 @@ static void print_stack(vcpu_guest_conte
         stack = stack_pointer(ctx);
         while(stack < stack_limit) {
             p = map_page(ctx, vcpu, stack);
+            if (!p)
+                return -1;
             word = read_stack_word(p, width);
             if (is_kernel_text(word)) {
                 printf("  [<");

@@ -832,34 +846,17 @@ static void print_stack(vcpu_guest_conte
             stack += width;
         }
     }
+    return 0;
 }
 #endif
 
 static void dump_ctx(int vcpu)
 {
-    int ret;
     vcpu_guest_context_any_t ctx;
 
-    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
-
-    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
-    if (ret < 0) {
-        perror("xc_domain_getinfo");
-        exit(-1);
-    }
-    
-    ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
-    if (ret < 0) {
-        perror("xc_domain_pause");
-        exit(-1);
-    }
-
-    ret = xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx);
-    if (ret < 0) {
-        if (!xenctx.dominfo.paused)
-            xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
+    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx) < 0) {
         perror("xc_vcpu_getcontext");
-        exit(-1);
+        return;
     }
 
 #if defined(__i386__) || defined(__x86_64__)

@@ -871,14 +868,14 @@ static void dump_ctx(int vcpu)
                     xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
                     vcpu, &cpuctx, sizeof cpuctx) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
-                exit(-1);
+                return;
             }
             guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
             guest_protected_mode = (cpuctx.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
             if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");
-                exit(-1);
+                return;
             }
             ctxt_word_size = (strstr(xen_caps, "xen-3.0-x86_64")) ? 8 : 4;
         } else {

@@ -894,24 +891,12 @@ static void dump_ctx(int vcpu)
 
     print_ctx(&ctx);
 #ifndef NO_TRANSLATION
-    print_code(&ctx, vcpu);
+    if (print_code(&ctx, vcpu))
+        return;
     if (is_kernel_text(instr_pointer(&ctx)))
-        print_stack(&ctx, vcpu, guest_word_size);
+        if (print_stack(&ctx, vcpu, guest_word_size))
+            return;
 #endif
-
-    if (!xenctx.dominfo.paused) {
-        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
-        if (ret < 0) {
-            perror("xc_domain_unpause");
-            exit(-1);
-        }
-    }
-
-    ret = xc_interface_close(xenctx.xc_handle);
-    if (ret < 0) {
-        perror("xc_interface_close");
-        exit(-1);
-    }
 }
 
 static void usage(void)

@@ -940,6 +925,7 @@ static void usage(void)
 int main(int argc, char **argv)
 {
     int ch;
+    int ret;
     static const char *sopts = "fs:hak:S"
 #ifdef __ia64__
         "r:"

@@ -1040,8 +1026,43 @@ int main(int argc, char **argv)
     if (symbol_table)
         read_symbol_table(symbol_table);
 
+    xenctx.xc_handle = xc_interface_open(0,0,0); /* for accessing control interface */
+    if (xenctx.xc_handle < 0) {
+        perror("xc_interface_open");
+        exit(-1);
+    }
+
+    ret = xc_domain_getinfo(xenctx.xc_handle, xenctx.domid, 1, &xenctx.dominfo);
+    if (ret < 0) {
+        perror("xc_domain_getinfo");
+        exit(-1);
+    }
+
+    if (!xenctx.dominfo.paused) {
+        ret = xc_domain_pause(xenctx.xc_handle, xenctx.domid);
+        if (ret < 0) {
+            perror("xc_domain_pause");
+            exit(-1);
+        }
+        xenctx.self_paused = 1;
+    }
+
     dump_ctx(vcpu);
 
+    if (xenctx.self_paused) {
+        ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);
+        if (ret < 0) {
+            perror("xc_domain_unpause");
+            exit(-1);
+        }
+    }
+
+    ret = xc_interface_close(xenctx.xc_handle);
+    if (ret < 0) {
+        perror("xc_interface_close");
+        exit(-1);
+    }
+
     return 0;
 }

Olaf Hering

author

2011-06-20 15:27 UTC #7 lore raw

PATCH 1/1 xenctx: add option -C to dump context for all vcpus

In reply to Olaf Hering

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308313376 -7200
# Node ID ddf16ea954876d8af371f9751fc06eb4c9e78b36 # Parent 0e0c9fbedc2e29e2d507e2e1dce9536d06af442d
xenctx: add option -C to dump context for all vcpus

Signed-off-by Olaf Hering <[email protected]>

tools/xentrace/xenctx.c +24 -2

diff -r 0e0c9fbedc2e -r ddf16ea95487 tools/xentrace/xenctx.c
@@ -35,6 +35,7 @@ static struct xenctx {
     int frame_ptrs;
     int stack_trace;
     int disp_all;
+    int all_vcpus;
     int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;

@@ -899,6 +900,19 @@ static void dump_ctx(int vcpu)
 #endif
 }
 
+static void dump_all_vcpus(void)
+{
+    xc_vcpuinfo_t vinfo;
+    int vcpu;
+    for (vcpu = 0; vcpu <= xenctx.dominfo.max_vcpu_id; vcpu++)
+    {
+        if ( xc_vcpu_getinfo(xenctx.xc_handle, xenctx.domid, vcpu, &vinfo) )
+            continue;
+        if ( vinfo.online )
+            dump_ctx(vcpu);
+    }
+}
+
 static void usage(void)
 {
     printf("usage:\n\n");

@@ -920,13 +934,14 @@ static void usage(void)
 #else
     printf("  -a --all          display more registers\n");
 #endif
+    printf("  -C --all-vcpus    print info for all vcpus\n");
 }
 
 int main(int argc, char **argv)
 {
     int ch;
     int ret;
-    static const char *sopts = "fs:hak:S"
+    static const char *sopts = "fs:hak:SC"
 #ifdef __ia64__
         "r:"
 #endif

@@ -940,6 +955,7 @@ int main(int argc, char **argv)
         {"regs", 1, NULL, 'r'},
 #endif
         {"all", 0, NULL, 'a'},
+        {"all-vcpus", 0, NULL, 'C'},
         {"help", 0, NULL, 'h'},
         {0, 0, 0, 0}
     };

@@ -995,6 +1011,9 @@ int main(int argc, char **argv)
             xenctx.disp_all = 1;
             break;
 #endif
+        case 'C':
+            xenctx.all_vcpus = 1;
+            break;
         case 'k':
             kernel_start = strtoull(optarg, NULL, 0);
             break;

@@ -1047,7 +1066,10 @@ int main(int argc, char **argv)
         xenctx.self_paused = 1;
     }
 
-    dump_ctx(vcpu);
+    if (xenctx.all_vcpus)
+        dump_all_vcpus();
+    else
+        dump_ctx(vcpu);
 
     if (xenctx.self_paused) {
         ret = xc_domain_unpause(xenctx.xc_handle, xenctx.domid);

Olaf Hering

author

2011-06-20 15:27 UTC #8 lore raw

PATCH 1/1 xenctx: dump pagetable

In reply to Olaf Hering

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308582353 -7200
# Node ID 0443de5faea9b904b92219b3b804dec147697366 # Parent ddf16ea954876d8af371f9751fc06eb4c9e78b36
xenctx: dump pagetable

This change is buggy...

tools/xentrace/xenctx.c +219 ~~-11~~

diff -r ddf16ea95487 -r 0443de5faea9 tools/xentrace/xenctx.c
@@ -25,21 +25,34 @@
 #include <getopt.h>
 
 #include "xenctrl.h"
+#include "xg_private.h"
+#include "xc_private.h"
 #include <xen/foreign/x86_32.h>
 #include <xen/foreign/x86_64.h>
 #include <xen/hvm/save.h>
 
+#define CR4_PAE 0x20
+#define EFER_LMA 0x400
+
 static struct xenctx {
     xc_interface *xc_handle;
     int domid;
     int frame_ptrs;
     int stack_trace;
+    int dump_pagetable;
     int disp_all;
     int all_vcpus;
     int self_paused;
     xc_dominfo_t dominfo;
 } xenctx;
 
+struct cpuctx {
+    vcpu_guest_context_any_t any;
+#if defined(__i386__) || defined(__x86_64__)
+    struct hvm_hw_cpu hvm;
+#endif
+} cpuctx;
+
 #if defined (__i386__) || defined (__x86_64__)
 typedef unsigned long long guest_word_t;
 #define FMT_32B_WORD "%08llx"

@@ -436,6 +449,184 @@ static guest_word_t frame_pointer(vcpu_g
         return ctx->x64.user_regs.rbp;
 }
 
+static void walk_l1(int vcpu, uint64_t l1, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l1e, phys_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l1 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l1");
+        return;
+    }
+    for (e = 0; e < L1_PAGETABLE_ENTRIES_X86_64; e++) {
+        l1e = map[e];
+        if (!l1e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l1e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l1e & phys_mask); /* phys base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l1e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_GLOBAL ? 'G' : 'g');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PAT ? 'P' : 'p');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PSE ? 'S' : 's');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_DIRTY ? 'D' : 'd');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l1e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l1e %3llx %016llx: %s %16llx\n", vcpu, e, l1e, buf, (unsigned long long)virt | (e << L1_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l2(int vcpu, uint64_t l2, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l2e, l1_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l2 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l2");
+        return;
+    }
+    for (e = 0; e < L2_PAGETABLE_ENTRIES_X86_64; e++) {
+        l2e = map[e];
+        if (!l2e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l2e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l2e & l1_mask); /* l1 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 8) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", (l2e >> 7) & 0x1 ? 'S' : 's'); /* SP */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l2e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l2e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l2e %3llx %016llx: %s\n", vcpu, e, l2e, buf);
+        if (l2e & _PAGE_PRESENT)
+            walk_l1(vcpu, l2e & l1_mask, virt | (e << L2_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l3(int vcpu, uint64_t l3, uint64_t virt)
+{
+    uint64_t *map;
+    unsigned long long e, l3e, l2_mask = ~(-1ull << 52) & (-1ull << 12);
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l3 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l3");
+        return;
+    }
+    for (e = 0; e < L3_PAGETABLE_ENTRIES_X86_64; e++) {
+        l3e = map[e];
+        if (!l3e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l3e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l3e & l2_mask); /* l2 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 8) & 0x1); /* MBZ */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", (l3e >> 7) & 0x1 ? 'S' : 's'); /* SP */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l3e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_USER ? 'U' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l3e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l3e %3llx %016llx: %s\n", vcpu, e, l3e, buf);
+        if (l3e & _PAGE_PRESENT)
+            walk_l2(vcpu, l3e & l2_mask, virt | (e << L3_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void walk_l4(int vcpu, uint64_t l4)
+{
+    uint64_t *map;
+    unsigned long long e, l4e, l3_mask = ~(-1ull << 52) & (-1ull << 12), virt = 0;
+    char buf[123];
+    fflush(stdout);
+    map = xc_map_foreign_range(xenctx.xc_handle, xenctx.domid, PAGE_SIZE, PROT_READ, l4 >> PAGE_SHIFT);
+    if (!map) {
+        perror("xc_map_foreign_range l4");
+        return;
+    }
+    for (e = 0; e < L4_PAGETABLE_ENTRIES_X86_64; e++) {
+        l4e = map[e];
+        if (!l4e)
+            continue;
+        buf[0] = '\0';
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & ( 1ull << 63) ? 'X' : 'x'); /* NX */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %3llx", (l4e >> 52) & ~( 1ull << 11)); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %12llx", l4e & l3_mask); /* l3 base adress */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 9) & 0x7); /* Available */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 7) & 0x3); /* MBZ */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %llx", (l4e >> 6) & 0x1); /* Ignored */
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_ACCESSED ? 'A' : 'a');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PCD ? 'C' : 'c');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PWT ? 'T' : 't');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_USER ? 'u' : 'u');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_RW ? 'W' : 'w');
+        snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), " %c", l4e & _PAGE_PRESENT ? 'P' : 'p');
+        printf("%3d l4e %3llx %016llx: %s\n", vcpu, e, l4e, buf);
+        if (l4e & _PAGE_PRESENT)
+            walk_l3(vcpu, l4e & l3_mask, virt | (e << L4_PAGETABLE_SHIFT_X86_64));
+    }
+    munmap(map, PAGE_SIZE);
+}
+
+static void dump_pagetable(struct cpuctx *cpuctx, int vcpu)
+{
+    uint64_t paddr;
+    int pt_levels, size;
+
+    printf("\npagetable for vcpu %d\n", vcpu);
+    /* What kind of paging are we dealing with? */
+    if (xenctx.dominfo.hvm) {
+        if (!guest_protected_mode) {
+            printf("protected mode disabled\n");
+            return;
+        }
+        pt_levels = (cpuctx->hvm.msr_efer & EFER_LMA) ? 4 : (cpuctx->hvm.cr4 & CR4_PAE) ? 3 : 2;
+        paddr = cpuctx->hvm.cr3 & ((pt_levels == 3) ? ~0x1full : ~0xfffull);
+    } else {
+        if (guest_word_size == 8) {
+            pt_levels = 4;
+            paddr = cpuctx->any.x64.ctrlreg[3] & ~0xfffull;
+        } else {
+            pt_levels = 3;
+            paddr = ((uint64_t) xen_cr3_to_pfn(cpuctx->any.x32.ctrlreg[3])) << PAGE_SHIFT;
+        }
+    }
+
+    size = (pt_levels == 2 ? 4 : 8);
+    if (pt_levels == 4) {
+        walk_l4(vcpu, paddr);
+    } else if (pt_levels == 3) {
+        walk_l3(vcpu, paddr, 0);
+    } else {
+        walk_l2(vcpu, paddr, 0);
+    }
+    printf("\n");
+    fflush(stdout);
+}
+
 #elif defined(__ia64__)
 
 #define PTE_ED_SHIFT              52

@@ -853,9 +1044,11 @@ static int print_stack(vcpu_guest_contex
 
 static void dump_ctx(int vcpu)
 {
-    vcpu_guest_context_any_t ctx;
+    struct cpuctx cpuctx;
 
-    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &ctx) < 0) {
+    memset(&cpuctx, 0, sizeof(cpuctx));
+
+    if (xc_vcpu_getcontext(xenctx.xc_handle, xenctx.domid, vcpu, &cpuctx.any) < 0) {
         perror("xc_vcpu_getcontext");
         return;
     }

@@ -863,16 +1056,15 @@ static void dump_ctx(int vcpu)
 #if defined(__i386__) || defined(__x86_64__)
     {
         if (xenctx.dominfo.hvm) {
-            struct hvm_hw_cpu cpuctx;
             xen_capabilities_info_t xen_caps = "";
             if (xc_domain_hvm_getcontext_partial(
                     xenctx.xc_handle, xenctx.domid, HVM_SAVE_CODE(CPU),
-                    vcpu, &cpuctx, sizeof cpuctx) != 0) {
+                    vcpu, &cpuctx.hvm, sizeof (cpuctx.hvm)) != 0) {
                 perror("xc_domain_hvm_getcontext_partial");
                 return;
             }
-            guest_word_size = (cpuctx.msr_efer & 0x400) ? 8 : 4;
-            guest_protected_mode = (cpuctx.cr0 & CR0_PE);
+            guest_word_size = (cpuctx.hvm.msr_efer & EFER_LMA) ? 8 : 4;
+            guest_protected_mode = (cpuctx.hvm.cr0 & CR0_PE);
             /* HVM guest context records are always host-sized */
             if (xc_version(xenctx.xc_handle, XENVER_capabilities, &xen_caps) != 0) {
                 perror("xc_version");

@@ -890,14 +1082,18 @@ static void dump_ctx(int vcpu)
     }
 #endif
 
-    print_ctx(&ctx);
+    print_ctx(&cpuctx.any);
 #ifndef NO_TRANSLATION
-    if (print_code(&ctx, vcpu))
+    if (print_code(&cpuctx.any, vcpu))
         return;
-    if (is_kernel_text(instr_pointer(&ctx)))
-        if (print_stack(&ctx, vcpu, guest_word_size))
+    if (is_kernel_text(instr_pointer(&cpuctx.any)))
+        if (print_stack(&cpuctx.any, vcpu, guest_word_size))
             return;
 #endif
+#if defined(__i386__) || defined(__x86_64__)
+    if (xenctx.dump_pagetable)
+        dump_pagetable(&cpuctx, vcpu);
+#endif
 }
 
 static void dump_all_vcpus(void)

@@ -926,6 +1122,10 @@ static void usage(void)
     printf("  -s SYMTAB, --symbol-table=SYMTAB\n");
     printf("                    read symbol table from SYMTAB.\n");
     printf("  -S --stack-trace  print a complete stack trace.\n");
+#if defined(__i386__) || defined(__x86_64__)
+    printf("  -P --dump-pagetable\n");
+    printf("                    print a complete pagetable for the given vcpu.\n");
+#endif
     printf("  -k, --kernel-start\n");
     printf("                    set user/kernel split. (default 0xc0000000)\n");
 #ifdef __ia64__

@@ -941,7 +1141,7 @@ int main(int argc, char **argv)
 {
     int ch;
     int ret;
-    static const char *sopts = "fs:hak:SC"
+    static const char *sopts = "fs:hak:SPC"
 #ifdef __ia64__
         "r:"
 #endif

@@ -951,6 +1151,9 @@ int main(int argc, char **argv)
         {"symbol-table", 1, NULL, 's'},
         {"frame-pointers", 0, NULL, 'f'},
         {"kernel-start", 1, NULL, 'k'},
+#if defined(__i386__) || defined(__x86_64__)
+        {"dump-pagetable", 0, NULL, 'P'},
+#endif
 #ifdef __ia64__
         {"regs", 1, NULL, 'r'},
 #endif

@@ -974,6 +1177,11 @@ int main(int argc, char **argv)
         case 'S':
             xenctx.stack_trace = 1;
             break;
+#if defined(__i386__) || defined(__x86_64__)
+        case 'P':
+            xenctx.dump_pagetable = 1;
+            break;
+#endif
 #ifdef __ia64__
         case 'r':
             {

Tim Deegan

2011-06-20 17:33 UTC #11 lore raw

In reply to Olaf Hering

Hi,

Quoted 5 lines · Olaf Hering

The code to walk the pagetables is/was based on xc_translate_foreign_address(), but I still think I have some major bugs in there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped. Any help with getting that output fixed for at least a a 64bit PV guest is much appreciated.

I didn't spot anything broken in the walker (though it will need a bunch of cleaning up) and the output looks very plausible. I take it this is a PV guest? Is it a well-behaved one or a broken post-migrate one?

One thing that might be causing trouble is if the domain's not paused while you do the walk, then you might see inconsistent tables, though I'd expect them to be garbage rather than looking like this. Your patch 3/5 does seem to make the pausing conditional where before it always happened.

Tim.

Quoted 24 lines

part of xenctx output:
...

  0 l4e  ff 00000000a1240067:  x   0     a1240000 0 0 1 A c t u W P
  0 l3e 1ff 00000000bc133067:  x   0     bc133000 0 0 s 1 A c t U W P
  0 l2e  64 00000000a1471067:  x   0     a1471000 0 0 s 1 A c t U W P
  0 l1e  46 80000000bd56f167:  X   0     bd56f000 0 G p s D A c t U W P     7fffcc846000
  0 l1e  47 80000000bd57c167:  X   0     bd57c000 0 G p s D A c t U W P     7fffcc847000
  0 l1e  48 80000000bd5af165:  X   0     bd5af000 0 G p s D A c t U w P     7fffcc848000
  0 l1e  db 00000000a1972125:  x   0     a1972000 0 G p s d A c t U w P     7fffcc8db000
  0 l4e 100 000000013fff8067:  x   0    13fff8000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 101 000000013fff0063:  x   0    13fff0000 0 0 1 A c t u W P
xc_map_foreign_range l3: Invalid argument
  0 l4e 102 00000000bc5b0063:  x   0     bc5b0000 0 0 1 A c t u W P
  0 l3e  fe 00000000a7ed2067:  x   0     a7ed2000 0 0 s 1 A c t U W P
  0 l2e 163 00000000a154a067:  x   0     a154a000 0 0 s 1 A c t U W P
  0 l1e 1eb 00000000bc1a7067:  x   0     bc1a7000 0 g p s D A c t U W P     813fac7eb000
  0 l1e 1ec 00000000a154f067:  x   0     a154f000 0 g p s D A c t U W P     813fac7ec000

...

Signature · 3 lines

Tim Deegan <[email protected]>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Olaf Hering

author

2011-06-20 17:51 UTC #12 lore raw

Quoted 9 lines · Tim Deegan

Hi,
Quoted 5 lines · Olaf Hering
The code to walk the pagetables is/was based on xc_translate_foreign_address(), but I still think I have some major bugs in there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped. Any help with getting that output fixed for at least a a 64bit PV guest is much appreciated.

Quoted 3 lines

I didn't spot anything broken in the walker (though it will need a bunch of cleaning up) and the output looks very plausible. I take it this is a PV guest? Is it a well-behaved one or a broken post-migrate one?

Yes, its a PV guest. I cant reproduce the migrate crashes, it happens only on very few systems or on a certain configuration. The logs I posted are from my test system.
Any ideas why some mfns are not accessible?

Are there any other paging states maintained outside of the guests memory?

Quoted 5 lines

One thing that might be causing trouble is if the domain's not paused while you do the walk, then you might see inconsistent tables, though I'd expect them to be garbage rather than looking like this. Your patch 3/5 does seem to make the pausing conditional where before it always happened.

It tries it preserve the state, if the guest was paused it probably should not be unpaused. Is the dominfo.paused flag somehow unreliable, I thought it comes right from struct domain?

Konrad Rzeszutek Wilk

2011-06-20 21:56 UTC #13 lore raw

Quoted 11 lines · Olaf Hering

Quoted 9 lines · Tim Deegan
Hi,
Quoted 5 lines · Olaf Hering
The code to walk the pagetables is/was based on xc_translate_foreign_address(), but I still think I have some major bugs in there (the last patch). I cant figure out why some l3/l2/l1 entries can not be mapped. Any help with getting that output fixed for at least a a 64bit PV guest is much appreciated.

Quoted 8 lines

Quoted 3 lines
I didn't spot anything broken in the walker (though it will need a bunch of cleaning up) and the output looks very plausible. I take it this is a PV guest? Is it a well-behaved one or a broken post-migrate one?
Yes, its a PV guest. I cant reproduce the migrate crashes, it happens only on very few systems or on a certain configuration. The logs I posted are from my test system.
Any ideas why some mfns are not accessible?

They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer, PCI IO bars, MP tale.

Quoted 3 lines

Are there any other paging states maintained outside of the guests memory?

They look to be I/O pages.

But not sure why they are mapped to your guest?

Quoted 15 lines

Quoted 6 lines
One thing that might be causing trouble is if the domain's not paused while you do the walk, then you might see inconsistent tables, though I'd expect them to be garbage rather than looking like this. Your patch 3/5 does seem to make the pausing conditional where before it always happened.
It tries it preserve the state, if the guest was paused it probably should not be unpaused. Is the dominfo.paused flag somehow unreliable, I thought it comes right from struct domain?
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

Tim Deegan

2011-06-21 09:59 UTC #14 lore raw

Hi,

Quoted 4 lines · Konrad Rzeszutek Wilk

Quoted 1 line
Any ideas why some mfns are not accessible?
They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer, PCI IO bars, MP tale.

I think they're too high for that, but if you can post the e820 map of the system this happened on then we'll know.

Quoted 6 lines

Quoted 2 lines
Are there any other paging states maintained outside of the guests memory?
They look to be I/O pages.
But not sure why they are mapped to your guest?

But they're not mapped into the guest - from the look of them they're not mapped anywhere. You could add some extra printouts around that warning in mm.c to show whether the MFNs are valid and if so which domain owns them.

Also, interesting that it's the addresses just above 0xffff800000000000 that are different - what lives at that address in the PV kernel you're running?

Tim.

Signature · 3 lines

Tim Deegan <[email protected]>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

Jan Beulich

2011-06-21 10:31 UTC #15 lore raw

Quoted 19 lines

Quoted 1 line
Quoted 1 line · Tim Deegan
Hi,
Quoted 4 lines · Konrad Rzeszutek Wilk
Quoted 1 line
Any ideas why some mfns are not accessible?
They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer, PCI IO bars, MP tale.
I think they're too high for that, but if you can post the e820 map of the system this happened on then we'll know.
Quoted 6 lines
Quoted 2 lines
Are there any other paging states maintained outside of the guests memory?
They look to be I/O pages.
But not sure why they are mapped to your guest?

Quoted 8 lines

But they're not mapped into the guest - from the look of them they're not mapped anywhere. You could add some extra printouts around that warning in mm.c to show whether the MFNs are valid and if so which domain owns them.
Also, interesting that it's the addresses just above 0xffff800000000000 that are different - what lives at that address in the PV kernel you're running?

That's Xen's space, isn't it. Clearly any non-hypervisor based page table walking code has to ignore this range for PV guests' page tables.

Jan

Ian Jackson

2011-06-21 12:33 UTC #9 lore raw

Re: [PATCH 5 of 5] xenctx: dump pagetable

In reply to Olaf Hering

Olaf Hering writes ("[Xen-devel] [PATCH 5 of 5] xenctx: dump pagetable"):

Quoted 8 lines

# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308582353 -7200
# Node ID 0443de5faea9b904b92219b3b804dec147697366 # Parent ddf16ea954876d8af371f9751fc06eb4c9e78b36
xenctx: dump pagetable
This change is buggy...

Uh... ? Did this slip through by mistake somehow ?

Ian.

Ian Jackson

2011-06-21 12:36 UTC #3 lore raw

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

In reply to Olaf Hering

Olaf Hering writes ("[Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):

Quoted 1 line

xenctx: move all globals into struct xenctx

Having read 1-4 I think they look reasonable and I'm happy to apply them when you give the word you think they're ready. It wasn't entirely clear from your messages whether you want me to do so.

Thanks,
Ian.

Olaf Hering

author

2011-06-21 15:25 UTC #10 lore raw

Re: [PATCH 5 of 5] xenctx: dump pagetable

In reply to Ian Jackson

Quoted 11 lines · Ian Jackson

Olaf Hering writes ("[Xen-devel] [PATCH 5 of 5] xenctx: dump pagetable"):
Quoted 8 lines
# HG changeset patch
# User Olaf Hering <[email protected]>
# Date 1308582353 -7200
# Node ID 0443de5faea9b904b92219b3b804dec147697366 # Parent ddf16ea954876d8af371f9751fc06eb4c9e78b36
xenctx: dump pagetable
This change is buggy...
Uh... ? Did this slip through by mistake somehow ?

Its here for review and discussion, I forgot to add a reasonable description. The change itself needs PAE and i386 support at least.
I will work on a better version.

Olaf

Olaf Hering

author

2011-06-21 15:27 UTC #4 lore raw

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

In reply to Ian Jackson

Quoted 6 lines · Ian Jackson

Olaf Hering writes ("[Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):
Quoted 1 line
xenctx: move all globals into struct xenctx
Having read 1-4 I think they look reasonable and I'm happy to apply them when you give the word you think they're ready. It wasn't entirely clear from your messages whether you want me to do so.

1-4 is ready, just last one (the pagetable change) needs more work. If the concept is ok, please apply these changes.

Olaf

Ian Jackson

2011-06-21 15:31 UTC #5 lore raw

Re: [PATCH 2 of 5] xenctx: move all globals into struct xenctx

Olaf Hering writes ("Re: [Xen-devel] [PATCH 2 of 5] xenctx: move all globals into struct xenctx"):

Quoted 2 lines

1-4 is ready, just last one (the pagetable change) needs more work. If the concept is ok, please apply these changes.

Done, thanks.

Ian.

Olaf Hering

author

2011-06-22 13:50 UTC #16 lore raw

In reply to Jan Beulich

Quoted 24 lines · Jan Beulich

Quoted 24 lines
Quoted 1 line
Quoted 1 line · Tim Deegan
Hi,
Quoted 4 lines · Konrad Rzeszutek Wilk
Quoted 1 line
Any ideas why some mfns are not accessible?
They look to be the special I/O PFNs. The ones that cover ACPI, framebuffer, PCI IO bars, MP tale.
I think they're too high for that, but if you can post the e820 map of the system this happened on then we'll know.
Quoted 6 lines
Quoted 2 lines
Are there any other paging states maintained outside of the guests memory?
They look to be I/O pages.
But not sure why they are mapped to your guest?
But they're not mapped into the guest - from the look of them they're not mapped anywhere. You could add some extra printouts around that warning in mm.c to show whether the MFNs are valid and if so which domain owns them.

Quoted 7 lines

Quoted 3 lines
Also, interesting that it's the addresses just above 0xffff800000000000 that are different - what lives at that address in the PV kernel you're running?
That's Xen's space, isn't it. Clearly any non-hypervisor based page table walking code has to ignore this range for PV guests' page tables.

Is there a way to detect that? I cant seem to match any of these ranges to something in guests dmesg or /proc.

After adding some debug to my xenctx from sles11 4.0, I get this on vcpu 0:

xc_map_foreign_range: walk_l3: 0 virt 0000800000000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000808000000000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814000000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814040000000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0000000 mfn 13fff8: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0200000 mfn 13fff0: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0800000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0a00000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l1: 0 virt 00008140a0c00000 mfn bf495: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814100000000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814140000000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l2: 0 virt 0000814180000000 mfn bf495: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000820000000000 mfn 137ff8: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000828000000000 mfn bf49a: Invalid argument
xc_map_foreign_range: walk_l3: 0 virt 0000830000000000 mfn bf495: Invalid argument

And dmesg has:

(XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 13fff8 (pfn 5555555555555555) from L1 entry 800000013fff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 13fff0 (pfn 5555555555555555) from L1 entry 800000013fff0625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn 137ff8 (pfn f180d) from L1 entry 8000000137ff8625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf49a (pfn 5555555555555555) from L1 entry 80000000bf49a625 for l1e_owner=0, pg_owner=1 (XEN) mm.c:880:d0 Error getting mfn bf495 (pfn 5555555555555555) from L1 entry 80000000bf495625 for l1e_owner=0, pg_owner=1

Olaf

Jan Beulich

2011-06-22 14:34 UTC #17 lore raw

Quoted 32 lines

Quoted 1 line
Quoted 1 line · Olaf Hering
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · Tim Deegan
Hi,
Quoted 3 lines · Konrad Rzeszutek Wilk
Quoted 1 line
Any ideas why some mfns are not accessible?
They look to be the special I/O PFNs. The ones that cover ACPI,
framebuffer,
Quoted 21 lines
Quoted 20 lines
Quoted 1 line
PCI IO bars, MP tale.
I think they're too high for that, but if you can post the e820 map of the system this happened on then we'll know.
Quoted 6 lines
Quoted 2 lines
Are there any other paging states maintained outside of the guests memory?
They look to be I/O pages.
But not sure why they are mapped to your guest?
But they're not mapped into the guest - from the look of them they're not mapped anywhere. You could add some extra printouts around that warning in mm.c to show whether the MFNs are valid and if so which domain owns them.
Also, interesting that it's the addresses just above 0xffff800000000000 that are different - what lives at that address in the PV kernel you're running?

Quoted 6 lines

Quoted 3 lines
That's Xen's space, isn't it. Clearly any non-hypervisor based page table walking code has to ignore this range for PV guests' page tables.
Is there a way to detect that? I cant seem to match any of these ranges to something in guests dmesg or /proc.

No need for detection - the hole is part of the ABI (minus its dynamic size in 32-bit pv guests running on 64-bit hypervisor - that may require use of heuristics).

Jan