Skip to contentSkip to conversation

xen,xend,tools: Add NUMA support to Xen

PATCH In review Ryan Harper 3 participants 11 messages Jul 31, 2006 – Aug 7, 2006

lore
Share & tools
lore

Ryan Harper

author
#0 lore raw

cover xen,xend,tools: Add NUMA support to Xen

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

The results for these were done on an x460, single node, with 6G ram. The balloon test was from 512M to ~6G. The NUMA patches include all of the previous optimizations.

Balloon Up:

Without NUMA    With NUMA    NUMA Delta
Try1: 2943ms    2937ms       -6ms
Try2: 2996ms    2992ms       -4ms
Try3: 2971ms    2968ms       -3ms

Increase reservation
Without NUMA:
MemSize 128M 512M 1G 2G 3G 4G
------------------------------------------

Try1:    10ms  38ms 76ms 160ms 237ms 308ms
Try2:     9ms  37ms 76ms 151ms 232ms 308ms
Try3:     9ms  37ms 76ms 151ms 226ms 311ms

With NUMA:
MemSize 128M 512M 1G 2G 3G 4G
------------------------------------------

Try1:    10ms  42ms 84ms 170ms 260ms 346ms
Try2:    10ms  42ms 83ms 167ms 255ms 346ms
Try3:    10ms  42ms 83ms 168ms 251ms 346ms

The only other change has been to the infrastructure patch in which I added the call to Xen's command line parsing to pick up the numa= options. This allows even on NUMA systems to disable NUMA parsing and fake the number of nodes (numa=off fake=1, would fake 1 node).

1. http://lists.xensource.com/archives/html/xen-devel/2006-07/msg00384.html

Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]
#1 lore raw
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

I am trying to use your patch on ia64.

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t node_to_cpumask[];

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, but not both! (ia64 defines cpu_to_node as a macro).

Tristan.

#9 lore raw
In reply to Ryan Harper
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

one more comment:
in dom0_ops.c:
+ case DOM0_AVAILHEAP:
+ {
+ ret = -EINVAL;
+ if ( op->u.availheap.node >= num_online_nodes() )
+ break;
+ if ( op->u.availheap.zone >= NR_ZONES )
+ break;
+
+ /* indicate the number of zones/nodes queried.
+ * NB: -1 is wild card for all zones/nodes */
+ ( op->u.availheap.zone < 0 ) ?
+ (op->u.availheap.nr_zones=NR_ZONES) :
+ (op->u.availheap.nr_zones=1);
+
+ ( op->u.availheap.node < 0 ) ?
+ (op->u.availheap.nr_nodes=num_online_nodes()) :
+ (op->u.availheap.nr_nodes=1);

I am not sure this corresponds to the Xen style.
Either use an if statement or a variable assignment:
op->u.availheap.nr_nodes = ( op->u.availheap.node < 0 ?) num_online_nodes() :
1;

Tristan.

author
#2 lore raw
In reply to Tristan Gingold
Quoted 10 lines · Tristan Gingold
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

I am trying to use your patch on ia64.

Thanks for testing these out on ia64.

Quoted 6 lines

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t node_to_cpumask[];

Quoted 7 lines

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, but not both! (ia64 defines cpu_to_node as a macro).

I'm not sure about this, but the definition of both the variable and macro come from Linux, for example in

linux/include/asm-x86_64/topology.h

extern unsigned char cpu_to_node[];
extern cpumask_t node_to_cpumask[];

#ifdef CONFIG_ACPI_NUMA
extern int __node_distance(int, int);
#define node_distance(a,b) __node_distance(a,b)
/* #else fallback version */
#endif

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (first_cpu(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

AFAIK, this isn't an issue.

Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]
#3 lore raw
Quoted 30 lines · Ryan Harper
Quoted 11 lines · Tristan Gingold
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

I am trying to use your patch on ia64.

Thanks for testing these out on ia64.

Quoted 5 lines

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t node_to_cpumask[];

Quoted 7 lines

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, but not both! (ia64 defines cpu_to_node as a macro).

I'm not sure about this, but the definition of both the variable and macro come from Linux, for example in

[...]

Quoted 1 line

AFAIK, this isn't an issue.

Except you are using both versions mainly the macro but at least the variable once in page_alloc.c:

/* Allocate 2^@order contiguous pages. */
struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
unsigned int order)

{
    unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
    unsigned int request = (1UL << order);

This was a problem for ia64.
Furthermore you define the variable in xen/numa.h:
extern unsigned int cpu_to_node[]; #include <xen/cpumask.h>
extern cpumask_t node_to_cpumask[];

Which one is the API ?

Tristan.

#10 lore raw
In reply to Ryan Harper
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

two more points.

For DOM0_PHYSINFO, sockets_per_node is wrong:

      pi->sockets_per_node = 
            num_online_cpus() / cpus_weight(cpu_core_map[0]);

[On ia64, I return the max of sockets per node].

The node_to_cpu field of struct dom0_physinfo is:

    XEN_GUEST_HANDLE(u64) node_to_cpu;

This means the max cpus is 64.
I know cpumap_t is also 64 bits.
I think you'd better to use cpumap_t.

Here we have NUMA machines with 128 cpus. The question is how to upgrade Xen to handle more than 64 cpus. We can either incease cpumap_t size to 256 (until we reach a new limit) or use an unlimited cpu structure.

Tristan.

author
#4 lore raw
In reply to Tristan Gingold
Quoted 43 lines · Tristan Gingold
Quoted 30 lines · Ryan Harper
Quoted 11 lines · Tristan Gingold
Quoted 6 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Hi,

I am trying to use your patch on ia64.

Thanks for testing these out on ia64.

Quoted 5 lines

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t node_to_cpumask[];

Quoted 7 lines

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, but not both! (ia64 defines cpu_to_node as a macro).

I'm not sure about this, but the definition of both the variable and macro come from Linux, for example in

[...]

Quoted 1 line

AFAIK, this isn't an issue.

Except you are using both versions mainly the macro but at least the variable once in page_alloc.c:

/* Allocate 2^@order contiguous pages. */
struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
unsigned int order)

{
    unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
    unsigned int request = (1UL << order);
Quoted 7 lines

This was a problem for ia64.
Furthermore you define the variable in xen/numa.h:
extern unsigned int cpu_to_node[]; #include <xen/cpumask.h>
extern cpumask_t node_to_cpumask[];

Which one is the API ?

If ia64 is already using macros, then we should use the macros. I should be able to remove those externs from numa.h and include the asm/topology.h (not in xen/numa.h since that is the same thing) in the .c files , like page_alloc.c, and use the macros instead.

Something like this (compiled on x86_64):

xen/arch/x86/dom0_ops.c +2 -1
diff -r 083e69a85080 xen/arch/x86/dom0_ops.c
@@ -26,6 +26,7 @@
 #include <asm/processor.h>
 #include <public/sched_ctl.h>
 #include <asm/numa.h>
+#include <asm/topology.h>
 
 #include <asm/mtrr.h>
 #include "cpu/mtrr/mtrr.h"

@@ -220,7 +221,7 @@ long arch_do_dom0_op(struct dom0_op *op,
         memset(node_to_cpu_64, 0, sizeof(node_to_cpu_64));
         for ( i=0; i<pi->nr_nodes; i++) {
             for ( j=0; j<num_online_cpus(); j++)
-                if ( cpu_isset(j, node_to_cpumask[i]) )
+                if ( cpu_isset(j, node_to_cpumask(i)) )
                     node_to_cpu_64[i] |= (u64)1 << j;
 
             if ( copy_to_guest_offset(op->u.physinfo.node_to_cpu,
xen/common/page_alloc.c +2 -1
diff -r 083e69a85080 xen/common/page_alloc.c
@@ -35,6 +35,7 @@
 #include <xen/domain_page.h>
 #include <xen/keyhandler.h>
 #include <asm/numa.h>
+#include <asm/topology.h>
 #include <asm/page.h>
 
 /*

@@ -317,7 +318,7 @@ struct page_info *alloc_heap_pages(unsig
 struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
                                    unsigned int order)
 {
-    unsigned int i,j, node = cpu_to_node[cpu], num_nodes = num_online_nodes();
+    unsigned int i,j, node = cpu_to_node(cpu), num_nodes = num_online_nodes();
     unsigned int request = (1UL << order);
     struct page_info *pg;
xen/include/xen/numa.h +0 -4
diff -r 083e69a85080 xen/include/xen/numa.h
@@ -23,8 +23,4 @@
 /* needed for drivers/acpi/numa.c */
 #define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
 
-extern unsigned int cpu_to_node[];
-#include <xen/cpumask.h>
-extern cpumask_t node_to_cpumask[];
-
 #endif /* _XEN_NUMA_H */
Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]
#5 lore raw
Quoted 54 lines · Ryan Harper
Quoted 53 lines · Tristan Gingold
Quoted 31 lines · Ryan Harper
Quoted 9 lines · Tristan Gingold
Quoted 7 lines · Ryan Harper

I've respun the NUMA patches against 10874 and I'm re-submitting them with the optimizations mentioned [1]previously on the list. There was a request to see the overhead on non-numa/single-node machines. I've re-run those benchmarks (ballooning up from small mem to multi-gig) as well as timing the initially domain increase_reservation time to gauge the overhead when allocating from the heap.

Quoted 3 lines

Hi,

I am trying to use your patch on ia64.

Thanks for testing these out on ia64.

Quoted 5 lines

In asm-x86/topology.h, you wrote:

extern unsigned int cpu_to_node[];
extern cpumask_t node_to_cpumask[];

Quoted 7 lines

#define cpu_to_node(cpu) (cpu_to_node[cpu])
#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])

I think cpu_to_node and node_to_cpumask must be either a variable or a macro, but not both! (ia64 defines cpu_to_node as a macro).

I'm not sure about this, but the definition of both the variable and macro come from Linux, for example in

[...]

Quoted 1 line

AFAIK, this isn't an issue.

Except you are using both versions mainly the macro but at least the variable once in page_alloc.c:

/* Allocate 2^@order contiguous pages. */
struct page_info *alloc_heap_pages(unsigned int zone, unsigned int cpu,
unsigned int order)
{
unsigned int i,j, node = cpu_to_node[cpu], num_nodes =
num_online_nodes(); unsigned int request = (1UL << order);

This was a problem for ia64.
Furthermore you define the variable in xen/numa.h:
extern unsigned int cpu_to_node[]; #include <xen/cpumask.h>
extern cpumask_t node_to_cpumask[];

Quoted 8 lines
Quoted 1 line

Which one is the API ?

If ia64 is already using macros, then we should use the macros. I should be able to remove those externs from numa.h and include the asm/topology.h (not in xen/numa.h since that is the same thing) in the .c files , like page_alloc.c, and use the macros instead.

Something like this (compiled on x86_64):

[...]
Yes, this would be fine for me.

Thank you for your NUMA work,
Tristan.

#6 lore raw

RE: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000

In reply to Ryan Harper

Ryan,
I tried out these patches on an ES7000.
It's an x86-64 two-node machine with 16 procesors and 32 GB of memory total.

I applied all of the six patches you had posted to the latest changeset as of today(10946:e1a2a8029e9f).

Xen finds the SLIT and SRAT tables. I was able to bring up and down multiple DomUs.

I then tried running a set of tests (bringing up domus and running LTP and Kernbench) and I found that the system was hanging. It wasn't crashed (I could ping it etc). But it was running so slow that ssh wouldn't work, even though I could use telnet to get on port 22.

I found out by accident booting into Dom0 and starting xm top hangs the system in a similar fashion.

Nothing appears on the serial port.

I have attached the serial output.
I am trying to figure out why it is happening.

Is anyone else seeing similar issues? Are there any suggestions on the approach I can take to debugging this?
Thanks
Raj
Xen Development Team
Unisys Corp.

 \ \/ /___ _ __   |___ / / _ \    _   _ _ __  ___| |_ __ _| |__ | | ___ 
  \  // _ \ '_ \    |_ \| | | |__| | | | '_ \/ __| __/ _` | '_ \| |/ _ \
  /  \  __/ | | |  ___) | |_| |__| |_| | | | \__ \ || (_| | |_) | |  __/
 /_/\_\___|_| |_| |____(_)___/    \__,_|_| |_|___/\__\__,_|_.__/|_|\___|

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.0-unstable (root@site) (gcc version 4.1.0 (SUSE Linux))
Thu Aug 3 13:49:43 EDT 2006
Latest ChangeSet: Fri Aug 04 11:36:07 2006 +0100 10946:e1a2a8029e9f

(XEN) Command line: /boot/xen.gz com1=115200,8n1 apic_verbosity=debug (XEN) Physical RAM map:

(XEN)  0000000000000000 - 000000000009dc00 (usable)
(XEN)  000000000009dc00 - 00000000000a0000 (reserved)
(XEN)  00000000000ce000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000002670000 (usable)
(XEN)  0000000002670000 - 00000000026d7000 (ACPI data)
(XEN)  00000000026d7000 - 0000000002700000 (ACPI NVS)
(XEN)  0000000002700000 - 00000000f0000000 (usable)
(XEN)  00000000f8000000 - 00000000fec00000 (reserved)
(XEN)  00000000fffc0000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000001008000000 (usable)

(XEN) System RAM: 65407MB (66976820kB)
(XEN) ACPI: RSDP (v000 PTLTD ) @
0x00000000000f8650
(XEN) ACPI: RSDT (v001 PTLTD RSDT 0x06040000 LTP 0x06040000) @
0x000000000267de53
(XEN) ACPI: FADT (v004 UNISYS ZORRO 0x06040000 PTL 0x00000008) @
0x00000000026d5f7c
(XEN) ACPI: OEM1 (v001 UNISYS OEM1 0x06040000 LTP 0x06040000) @
0x00000000026d6070
(XEN) ACPI: OEM2 (v001 UNISYS OEM2 0x06040000 LTP 0x06040000) @
0x00000000026d60a0
(XEN) ACPI: WSPT (v001 UNISYS WSPT 0x06040000 LTP 0x06040000) @
0x00000000026d6c9c
(XEN) ACPI: SRAT (v001 UNISYS SRAT 0x06040000 LTP 0x06040000) @
0x00000000026d6cc4
(XEN) ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x06040000) @
0x00000000026d6e6c
(XEN) ACPI: SPCR (v001 PTLTD $UCRTBL$ 0x06040000 PTL 0x00000001) @ 0x00000000026d6fb0
(XEN) ACPI: DSDT (v001 Intel 870 SMP 0x06040000 MSFT 0x02000001) @ 0x0000000000000000
(XEN) SRAT: PXM 2 -> APIC 28 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 26 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 30 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 24 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 22 -> Node 0
(XEN) SRAT: PXM 2 -> APIC 16 -> Node 0
(XEN) SRAT: PXM 1 -> APIC 12 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 10 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 4 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 2 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 14 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 8 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 6 -> Node 1
(XEN) SRAT: PXM 1 -> APIC 0 -> Node 1
(XEN) SRAT: Node 1 PXM 1 0-100000
(XEN) SRAT: Node 1 PXM 1 0-808000000
(XEN) SRAT: Node 0 PXM 2 808000000-1008000000
(XEN) NUMA: Using 27 for the hash shift.
(XEN) Reserving non-aligned node boundary @ mfn 8421376 (XEN) Xen heap: 11MB (12280kB)
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) found SMP MP-table at 000f8680
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: OEM1 (v001 UNISYS OEM1 0x06040000 LTP 0x06040000) @
0x00000000026d6070
(XEN) ACPI: DSDT (v001 Intel 870 SMP 0x06040000 MSFT 0x02000001) @ 0x0000000000000000
(XEN) Switched to APIC driver `es7000'.
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) (XEN) Processor #0 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled) (XEN) Processor #6 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x08] enabled) (XEN) Processor #8 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x0e] enabled) (XEN) Processor #14 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled) (XEN) Processor #2 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) (XEN) Processor #4 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled) (XEN) Processor #10 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled) (XEN) Processor #12 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled) (XEN) Processor #16 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x16] enabled) (XEN) Processor #22 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x18] enabled) (XEN) Processor #24 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x1e] enabled) (XEN) Processor #30 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x12] enabled) (XEN) Processor #18 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x14] enabled) (XEN) Processor #20 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1a] enabled) (XEN) Processor #26 15:4 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x1c] enabled) (XEN) Processor #28 15:4 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1]) (XEN) ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1]) (XEN) ACPI: IOAPIC (id[0x10] address[0xfec00000] gsi_base[0]) (XEN) IOAPIC[0]: apic_id 16, version 32, address 0xfec00000, GSI 0-23 (XEN) ACPI: IOAPIC (id[0x1a] address[0xfec04000] gsi_base[72]) (XEN) IOAPIC[1]: apic_id 26, version 32, address 0xfec04000, GSI 72-95 (XEN) ACPI: IOAPIC (id[0x1b] address[0xfec05000] gsi_base[96]) (XEN) IOAPIC[2]: apic_id 27, version 32, address 0xfec05000, GSI 96-119 (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) (XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Phys. Using 3 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information (XEN) mapped APIC to ffff828bfffff000 (fee00000)
(XEN) mapped IOAPIC to ffff828bffffe000 (fec00000)
(XEN) mapped IOAPIC to ffff828bffffd000 (fec04000)
(XEN) mapped IOAPIC to ffff828bffffc000 (fec05000)
(XEN) Initializing CPU#0
(XEN) Detected 3000.207 MHz processor.
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU0: Thermal monitoring enabled
(XEN) CPU0: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Getting VERSION: 50014
(XEN) Getting VERSION: 50014
(XEN) Getting ID: 0
(XEN) Getting LVT0: 10700
(XEN) Getting LVT1: 10400
(XEN) ES7000: Enabling APIC mode.
(XEN) masked ExtINT on CPU#0
(XEN) Mapping cpu 0 to node 1
(XEN) Booting processor 1/6 eip 90000
(XEN) Initializing CPU#1
(XEN) masked ExtINT on CPU#1
(XEN) Mapping cpu 1 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU1: Thermal monitoring enabled
(XEN) CPU1: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 2/8 eip 90000
(XEN) Initializing CPU#2
(XEN) masked ExtINT on CPU#2
(XEN) Mapping cpu 2 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#2.
(XEN) CPU2: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU2: Thermal monitoring enabled
(XEN) CPU2: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 3/14 eip 90000
(XEN) Initializing CPU#3
(XEN) masked ExtINT on CPU#3
(XEN) Mapping cpu 3 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#3.
(XEN) CPU3: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU3: Thermal monitoring enabled
(XEN) CPU3: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 4/2 eip 90000
(XEN) Initializing CPU#4
(XEN) masked ExtINT on CPU#4
(XEN) Mapping cpu 4 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#4.
(XEN) CPU4: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU4: Thermal monitoring enabled
(XEN) CPU4: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 5/4 eip 90000
(XEN) Initializing CPU#5
(XEN) masked ExtINT on CPU#5
(XEN) Mapping cpu 5 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 1
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#5.
(XEN) CPU5: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU5: Thermal monitoring enabled
(XEN) CPU5: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 6/10 eip 90000
(XEN) Initializing CPU#6
(XEN) masked ExtINT on CPU#6
(XEN) Mapping cpu 6 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 2
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#6.
(XEN) CPU6: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU6: Thermal monitoring enabled
(XEN) CPU6: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 7/12 eip 90000
(XEN) Initializing CPU#7
(XEN) masked ExtINT on CPU#7
(XEN) Mapping cpu 7 to node 1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 3
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#7.
(XEN) CPU7: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU7: Thermal monitoring enabled
(XEN) CPU7: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 8/16 eip 90000
(XEN) Initializing CPU#8
(XEN) masked ExtINT on CPU#8
(XEN) Mapping cpu 8 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#8.
(XEN) CPU8: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU8: Thermal monitoring enabled
(XEN) CPU8: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 9/22 eip 90000
(XEN) Initializing CPU#9
(XEN) masked ExtINT on CPU#9
(XEN) Mapping cpu 9 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#9.
(XEN) CPU9: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU9: Thermal monitoring enabled
(XEN) CPU9: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 10/24 eip 90000
(XEN) Initializing CPU#10
(XEN) masked ExtINT on CPU#10
(XEN) Mapping cpu 10 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#10.
(XEN) CPU10: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU10: Thermal monitoring enabled
(XEN) CPU10: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 11/30 eip 90000
(XEN) Initializing CPU#11
(XEN) masked ExtINT on CPU#11
(XEN) Mapping cpu 11 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#11.
(XEN) CPU11: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU11: Thermal monitoring enabled
(XEN) CPU11: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 12/18 eip 90000
(XEN) Initializing CPU#12
(XEN) masked ExtINT on CPU#12
(XEN) Mapping cpu 12 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 4
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#12.
(XEN) CPU12: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU12: Thermal monitoring enabled
(XEN) CPU12: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 13/20 eip 90000
(XEN) Initializing CPU#13
(XEN) masked ExtINT on CPU#13
(XEN) Mapping cpu 13 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 5
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#13.
(XEN) CPU13: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU13: Thermal monitoring enabled
(XEN) CPU13: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 14/26 eip 90000
(XEN) Initializing CPU#14
(XEN) masked ExtINT on CPU#14
(XEN) Mapping cpu 14 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 6
(XEN) CPU: Processor Core ID: 1
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#14.
(XEN) CPU14: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU14: Thermal monitoring enabled
(XEN) CPU14: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Booting processor 15/28 eip 90000
(XEN) Initializing CPU#15
(XEN) masked ExtINT on CPU#15
(XEN) Mapping cpu 15 to node 0
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 7
(XEN) CPU: Processor Core ID: 0
(XEN) VMXON is done
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#15.
(XEN) CPU15: Intel P4/Xeon Extended MCE MSRs (24) available (XEN) CPU15: Thermal monitoring enabled
(XEN) CPU15: Intel Genuine Intel(R) CPU 3.00GHz stepping 08 (XEN) Total of 16 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) init IO_APIC IRQs
(XEN) IO-APIC (apicid-pin) 16-0, 16-16, 16-17, 16-18, 16-19, 16-20, 16-21, 16-22, 16-23, 26-0, 26-1, 26-2, 26-3, 26-4, 26-5, 26-6, 26-7, 26-8, 26-9, 26-10, 26-11, 26-12, 26-13, 26-14, 26-15, 26-16, 26-17, 26-18, 26-19, 26-20, 26-21, 26-22, 26-23, 27-0, 27-1, 27-2, 27-3, 27-4, 27-5, 27-6, 27-7, 27-8, 27-9, 27-10, 27-11, 27-12, 27-13, 27-14, 27-15, 27-16, 27-17, 27-18, 27-19, 27-20, 27-21, 27-22, 27-23 not connected. (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0 (XEN) number of MP IRQ sources: 15.
(XEN) number of IO-APIC #16 registers: 24.
(XEN) number of IO-APIC #26 registers: 24.
(XEN) number of IO-APIC #27 registers: 24.
(XEN) testing the IO APIC.......................
(XEN) IO APIC #16......
(XEN) .... register #00: 00000000

(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020

(XEN) .... register #02: 00000000
(XEN) ....... : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) ....... : Boot DT : 1
(XEN) .... IRQ redirection table:

(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  0    0    0   0   0    0    0    28
(XEN)  02 000 00  0    0    0   0   0    0    0    F0
(XEN)  03 000 00  0    0    0   0   0    0    0    30
(XEN)  04 000 00  0    0    0   0   0    0    0    F1
(XEN)  05 000 00  0    0    0   0   0    0    0    38
(XEN)  06 000 00  0    0    0   0   0    0    0    40
(XEN)  07 000 00  0    0    0   0   0    0    0    48
(XEN)  08 000 00  0    0    0   0   0    0    0    50
(XEN)  09 000 00  1    1    0   0   0    0    0    58
(XEN)  0a 000 00  0    0    0   0   0    0    0    60
(XEN)  0b 000 00  0    0    0   0   0    0    0    68
(XEN)  0c 000 00  0    0    0   0   0    0    0    70
(XEN)  0d 000 00  0    0    0   0   0    0    0    78
(XEN)  0e 000 00  0    0    0   0   0    0    0    88
(XEN)  0f 000 00  0    0    0   0   0    0    0    90
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #26......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020

(XEN) .... register #02: 00000000
(XEN) ....... : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) ....... : Boot DT : 1
(XEN) .... IRQ redirection table:

(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00
(XEN) IO APIC #27......
(XEN) .... register #00: 00000000
(XEN) .......    : physical APIC id: 00
(XEN) .......    : Delivery Type: 0
(XEN) .......    : LTS          : 0
(XEN) .... register #01: 00178020
(XEN) .......     : max redirection entries: 0017
(XEN) .......     : PRQ implemented: 1
(XEN) .......     : IO APIC version: 0020

(XEN) .... register #02: 00000000
(XEN) ....... : arbitration: 00
(XEN) .... register #03: 00000001
(XEN) ....... : Boot DT : 1
(XEN) .... IRQ redirection table:

(XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:   
(XEN)  00 000 00  1    0    0   0   0    0    0    00
(XEN)  01 000 00  1    0    0   0   0    0    0    00
(XEN)  02 000 00  1    0    0   0   0    0    0    00
(XEN)  03 000 00  1    0    0   0   0    0    0    00
(XEN)  04 000 00  1    0    0   0   0    0    0    00
(XEN)  05 000 00  1    0    0   0   0    0    0    00
(XEN)  06 000 00  1    0    0   0   0    0    0    00
(XEN)  07 000 00  1    0    0   0   0    0    0    00
(XEN)  08 000 00  1    0    0   0   0    0    0    00
(XEN)  09 000 00  1    0    0   0   0    0    0    00
(XEN)  0a 000 00  1    0    0   0   0    0    0    00
(XEN)  0b 000 00  1    0    0   0   0    0    0    00
(XEN)  0c 000 00  1    0    0   0   0    0    0    00
(XEN)  0d 000 00  1    0    0   0   0    0    0    00
(XEN)  0e 000 00  1    0    0   0   0    0    0    00
(XEN)  0f 000 00  1    0    0   0   0    0    0    00
(XEN)  10 000 00  1    0    0   0   0    0    0    00
(XEN)  11 000 00  1    0    0   0   0    0    0    00
(XEN)  12 000 00  1    0    0   0   0    0    0    00
(XEN)  13 000 00  1    0    0   0   0    0    0    00
(XEN)  14 000 00  1    0    0   0   0    0    0    00
(XEN)  15 000 00  1    0    0   0   0    0    0    00
(XEN)  16 000 00  1    0    0   0   0    0    0    00
(XEN)  17 000 00  1    0    0   0   0    0    0    00

(XEN) Using vector-based indexing
(XEN) IRQ to pin mappings:
(XEN) IRQ240 -> 0:2
(XEN) IRQ40 -> 0:1
(XEN) IRQ48 -> 0:3
(XEN) IRQ241 -> 0:4
(XEN) IRQ56 -> 0:5
(XEN) IRQ64 -> 0:6
(XEN) IRQ72 -> 0:7
(XEN) IRQ80 -> 0:8
(XEN) IRQ88 -> 0:9
(XEN) IRQ96 -> 0:10
(XEN) IRQ104 -> 0:11
(XEN) IRQ112 -> 0:12
(XEN) IRQ120 -> 0:13
(XEN) IRQ136 -> 0:14
(XEN) IRQ144 -> 0:15
(XEN) .................................... done. (XEN) Using local APIC timer interrupts.
(XEN) calibrating APIC timer ...
(XEN) ..... CPU clock speed is 3000.0389 MHz. (XEN) ..... host bus clock speed is 200.0025 MHz. (XEN) ..... bus_scale = 0x0000CCD7
(XEN) checking TSC synchronization across 16 CPUs:
(XEN) CPU#0 had 683915 usecs TSC skew, fixed it up. (XEN) CPU#1 had 683926 usecs TSC skew, fixed it up. (XEN) CPU#2 had 683932 usecs TSC skew, fixed it up. (XEN) CPU#3 had 683927 usecs TSC skew, fixed it up. (XEN) CPU#4 had 683926 usecs TSC skew, fixed it up. (XEN) CPU#5 had 683934 usecs TSC skew, fixed it up. (XEN) CPU#6 had 683926 usecs TSC skew, fixed it up. (XEN) CPU#7 had 683927 usecs TSC skew, fixed it up. (XEN) CPU#8 had -683927 usecs TSC skew, fixed it up. (XEN) CPU#9 had -683928 usecs TSC skew, fixed it up. (XEN) CPU#10 had -683927 usecs TSC skew, fixed it up. (XEN) CPU#11 had -683927 usecs TSC skew, fixed it up. (XEN) CPU#12 had -683928 usecs TSC skew, fixed it up. (XEN) CPU#13 had -683927 usecs TSC skew, fixed it up. (XEN) CPU#14 had -683928 usecs TSC skew, fixed it up. (XEN) CPU#15 had -683921 usecs TSC skew, fixed it up. (XEN) Platform timer is 1.193MHz PIT
(XEN) CPU 0 APIC 0 -> Node 1
(XEN) CPU 1 APIC 6 -> Node 1
(XEN) CPU 2 APIC 8 -> Node 1
(XEN) CPU 3 APIC 14 -> Node 1
(XEN) CPU 4 APIC 2 -> Node 1
(XEN) CPU 5 APIC 4 -> Node 1
(XEN) CPU 6 APIC 10 -> Node 1
(XEN) CPU 7 APIC 12 -> Node 1
(XEN) CPU 8 APIC 16 -> Node 0
(XEN) CPU 9 APIC 22 -> Node 0
(XEN) CPU 10 APIC 24 -> Node 0
(XEN) CPU 11 APIC 30 -> Node 0
(XEN) CPU 12 APIC 18 -> Node 0
(XEN) CPU 13 APIC 20 -> Node 0
(XEN) CPU 14 APIC 26 -> Node 0
(XEN) CPU 15 APIC 28 -> Node 0
(XEN) Brought up 16 CPUs
(XEN) Machine check exception polling timer started. (XEN) *** LOADING DOMAIN 0 ***
(XEN) Domain 0 kernel supports features = { 0000000f }. (XEN) Domain 0 kernel requires features = { 00000000 }. (XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000030000000->0000000040000000 (16444393 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:

(XEN)  Loaded kernel: ffffffff80100000->ffffffff804b4f08
(XEN)  Init. ramdisk: ffffffff804b5000->ffffffff80b8de00
(XEN)  Phys-Mach map: ffffffff80b8e000->ffffffff88983f48
(XEN)  Start info:    ffffffff88984000->ffffffff88985000
(XEN)  Page tables:   ffffffff88985000->ffffffff889ce000
(XEN)  Boot stack:    ffffffff889ce000->ffffffff889cf000
(XEN)  TOTAL:         ffffffff80000000->ffffffff88c00000
(XEN)  ENTRY ADDRESS: ffffffff80100000

(XEN) Dom0 has maximum 16 VCPUs
(XEN) Initrd len 0x6d8e00, start at 0xffffffff804b5000
(XEN) Scrubbing Free RAM:
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
........................................................................
.........done.
(XEN) Xen trace buffers: disabled
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen).
kernel direct mapping tables up to fbf3e9000 @ 8a0d000-10847000 Bootdata ok (command line is root=/dev/sda1 vga=0x314 selinux=0 resume=/dev/sda2 splash=native showopts)
Linux version 2.6.16.13-xen (root@m1132-xenunstable) (gcc version 4.1.0 (SUSE Linux)) #1 SMP Thu Aug 3 14:13:58 EDT 2006 BIOS-provided physical RAM map:
Xen: 0000000000000000 - 0000000fbf3e9000 (usable)
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x08] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x0e] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0a] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0c] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x16] enabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x18] enabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x1e] enabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x12] enabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x14] enabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x1a] enabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x1c] enabled)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
ACPI: IOAPIC (id[0x10] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 16, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x1a] address[0xfec04000] gsi_base[72]) IOAPIC[1]: apic_id 26, version 32, address 0xfec04000, GSI 72-95
ACPI: IOAPIC (id[0x1b] address[0xfec05000] gsi_base[96]) IOAPIC[2]: apic_id 27, version 32, address 0xfec05000, GSI 96-119
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Setting APIC routing to xen
Using ACPI (MADT) for SMP configuration information Allocating PCI resources starting at f0800000 (gap: f0000000:8000000) Built 1 zonelists
Kernel command line: root=/dev/sda1 vga=0x314 selinux=0 resume=/dev/sda2 splash=native showopts
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 131072 bytes)
Xen reported: 3000.010 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes) Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes) Software IO TLB enabled:
Aperture: 64 megabytes
Kernel range: 0xffff88004fa29000 - 0xffff880053a29000
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Memory: 64698932k/66047908k available (1898k kernel code, 1340164k reserved, 808k data, 168k init)
Calibrating delay using timer specific routine.. 6001.70 BogoMIPS (lpj=30008520)
Security Framework v1.0.0 initialized
Capability LSM initialized
Mount-cache hash table entries: 256
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Initializing CPU#1
Initializing CPU#2
Initializing CPU#3
Initializing CPU#4
Initializing CPU#5
Initializing CPU#6
Initializing CPU#7
Initializing CPU#8
Initializing CPU#9
Initializing CPU#10
Initializing CPU#11
Initializing CPU#12
Initializing CPU#13
Initializing CPU#14
Brought up 16 CPUs
Initializing CPU#15
migration_cost=1555
checking if image is initramfs... it is
Freeing initrd memory: 7011k freed
DMI present.
Grant table initialized
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Subsystem revision 20060127
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: Device [MBDV] status [00000008]: functional but not present; setting present
ACPI: PCI Root Bridge [MBDV] (0000:ff)
ACPI: PCI Root Bridge [S1H0] (0000:00)
PCI quirk: region 0d00-0d7f claimed by ICH4 ACPI/GPIO/TCO PCI quirk: region 0e80-0ebf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 10 *11 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 10 11 14 15) *12
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 10 11 14 15) *12
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 *6 10 11 14 15)
ACPI: PCI Root Bridge [S1H2] (0000:0f)
PCI: Enable I/O Space to 1 KB Granularity
PCI: Enable I/O Space to 1 KB Granularity
ACPI: PCI Interrupt Link [LNKA] (IRQs *10)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 8 devices
xen_mem: Initialising balloon driver.
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
PCI: Bridge: 0000:00:1e.0
IO window: 1000-1fff
MEM window: f6000000-f7efffff
PREFETCH window: f0800000-f08fffff
PCI: Bridge: 0000:0f:1d.0
IO window: 2000-23ff
MEM window: f4f00000-f4ffffff
PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:100000@f4000000 for 0000:16:01.1
PCI: Bridge: 0000:0f:1f.0
IO window: 2400-27ff
MEM window: f4d00000-f4efffff
PREFETCH window: f3e00000-f3ffffff
IA-32 Microcode Update Driver: v1.14-xen <[email protected]> IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
audit: initializing netlink socket (disabled) audit(1154632323.818:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
loop: loaded (max 8 devices)
Xen virtual console successfully installed as ttyS0 Event-channel device installed.
blkif_init: reqs=64, pages=704, mmap_vstart=0xffff880fbe400000 Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) TCP established hash table entries: 262144 (order: 10, 4194304 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536) TCP reno registered
NET: Registered protocol family 1
NET: Registered protocol family 17
scsi_mod: no version for "struct_module" found: kernel tainted. SCSI subsystem initialized
ICH4: IDE controller at PCI slot 0000:00:1f.1
GSI 16 sharing vector 0x98 and IRQ 16
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 2
ICH4: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0x0880-0x0887, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0x0888-0x088f, BIOS settings: hdc:pio, hdd:pio

hda: TOSHIBA CD/DVDW SD-R6472, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Fusion MPT base driver 3.03.07
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SPI Host driver 3.03.07
GSI 17 sharing vector 0xA0 and IRQ 17
ACPI: PCI Interrupt 0000:16:01.0[A] -> GSI 96 (level, low) -> IRQ 17
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator,Target}
scsi0 : ioc0: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=17
Vendor: UNISYS Model: 014228ST118202LC Rev: B603
Type: Direct-Access ANSI SCSI revision: 02
SCSI device sda: 35833870 512-byte hdwr sectors (18347 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back w/ FUA
SCSI device sda: 35833870 512-byte hdwr sectors (18347 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back w/ FUA
sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
GSI 18 sharing vector 0xA8 and IRQ 18
ACPI: PCI Interrupt 0000:16:01.1[B] -> GSI 97 (level, low) -> IRQ 18
mptbase: Initiating ioc1 bringup
ioc1: 53C1030: Capabilities={Initiator,Target}
scsi1 : ioc1: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=18
ReiserFS: sda1: found reiserfs format "3.6" with standard journal
ReiserFS: sda1: using ordered data mode
ReiserFS: sda1: journal params: device sda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: sda1: checking transaction log (sda1)
ReiserFS: sda1: Using r5 hash to sort names
Adding 2176796k swap on /dev/sda2. Priority:-1 extents:1 across:2176796k
Intel(R) PRO/1000 Network Driver - version 6.3.9-k4-NAPI Copyright (c) 1999-2005 Intel Corporation.
GSI 19 sharing vector 0xB0 and IRQ 19
ACPI: PCI Interrupt 0000:10:01.0[A] -> GSI 72 (level, low) -> IRQ 19
e1000: 0000:10:01.0: e1000_probe: (PCI-X:133MHz:64-bit) 08:00:0b:1e:1a:14
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection GSI 20 sharing vector 0xB8 and IRQ 20
ACPI: PCI Interrupt 0000:10:01.1[B] -> GSI 73 (level, low) -> IRQ 20
e1000: 0000:10:01.1: e1000_probe: (PCI-X:133MHz:64-bit) 08:00:0b:1e:1a:15
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
device-mapper: 4.5.0-ioctl (2005-10-04) initialised: [email protected]
ACPI: Power Button (FF) [PWRF]
e1000: eth0: e1000_watchdog_task: NIC Link is Up 100 Mbps Full Duplex (XEN) mtrr: type mismatch for f6000000,400000 old: uncachable new:
write-combining
(XEN) mtrr: type mismatch for f6000000,400000 old: uncachable new:
write-combining
[-- MARK -- Fri Aug 4 19:00:00 2006]

author
#7 lore raw

Re: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000

Quoted 4 lines · Subrahmanian, Raj

Ryan,
I tried out these patches on an ES7000.
It's an x86-64 two-node machine with 16 procesors and 32 GB of memory total.

Thanks for trying the patches out.

Quoted 7 lines

I applied all of the six patches you had posted to the latest changeset as of today(10946:e1a2a8029e9f).

Xen finds the SLIT and SRAT tables. I was able to bring up and down multiple DomUs.

Quoted 7 lines

I then tried running a set of tests (bringing up domus and running LTP and Kernbench) and I found that the system was hanging. It wasn't crashed (I could ping it etc). But it was running so slow that ssh wouldn't work, even though I could use telnet to get on port 22.

I found out by accident booting into Dom0 and starting xm top hangs the system in a similar fashion.

That doesn't look good. And if you roll back the patches, doing the above works without issue? I'll see if I can lock up my box using xm top.

Quoted 5 lines

Nothing appears on the serial port.

I have attached the serial output.
I am trying to figure out why it is happening.

In any case, I'd be interested in seeing the 'H' and 'u' (heapinfo and numa topology) keyhandlers. Also, from dom0, running a xen_numastat -H -d -1 (dump the heap and the memory to node mapping of all running domains).

Signature · 5 lines
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
[email protected]
#8 lore raw

RE: [PATCH 0/6] xen, xend, tools: Add NUMA support to Xen Issues on the ES7000

Ryan,

Quoted 7 lines
Quoted 3 lines

Ryan,
I tried out these patches on an ES7000.
It's an x86-64 two-node machine with 16 procesors and 32 GB

of memory

Quoted 1 line

total.

Thanks for trying the patches out.

We have a vested interest in making sure this works too :-)

Quoted 5 lines
Quoted 2 lines

I found out by accident booting into Dom0 and starting xm top hangs the system in a similar fashion.

That doesn't look good. And if you roll back the patches, doing the above works without issue? I'll see if I can lock up my box using xm top.

Yes. We run LTP and Kernbench on DomUs of different sizes and these tests complete on the main-line tree.

Quoted 10 lines
Quoted 5 lines

Nothing appears on the serial port.

I have attached the serial output.
I am trying to figure out why it is happening.

In any case, I'd be interested in seeing the 'H' and 'u' (heapinfo and numa topology) keyhandlers. Also, from dom0, running a xen_numastat -H -d -1 (dump the heap and the memory to node mapping of all running domains).

I will do this and get back to you.

Thanks
Raj