Skip to contentSkip to conversation

Some small NUMA placement improvements

PATCH New Dario Faggioli 2 participants 7 messages Oct 19, 2012 – Nov 12, 2012

lore
Share & tools
lore

Dario Faggioli

author
#0 lore raw

PATCH v2 1/1 Some small NUMA placement improvements

Hi again,

Take 2 of this series, with the only comment about strstr()-vs-strncmp() in patch 2 addressed, and all the three patches already Acked-by George.

Just as a quick reminder, the series is about:
- use node distances information during automatic placement (patch 1);
- let the user specify a minimum and a maximum number of NUMA nodes they want the automaic placement to use (patch 2);
- enhance the syntax of the "cpus=" config switch so that full nodes can be specified instead of just single CPUs (patch 3).

Thanks and Regards,
Dario

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

author
#1 lore raw

PATCH v2 1/1 libxl: take node distances into account during NUMA placement

In fact, among placement candidates with the same number of nodes, the closer the various nodes are to each others, the better the performances for a domain placed there.

Signed-off-by Dario Faggioli <[email protected]>
Acked-by George Dunlap <[email protected]>
tools/libxl/libxl_dom.c +10 -0
@@ -105,6 +105,9 @@ out:
  *  - the number of vcpus runnable on the candidates is considered, and
  *    candidates with fewer of them are preferred. If two candidate have
  *    the same number of runnable vcpus,
+ *  - the sum of the node distances in the candidates is considered, and
+ *    candidates with smaller total distance are preferred. If total
+ *    distance is the same for the two candidatess,
  *  - the amount of free memory in the candidates is considered, and the
  *    candidate with greater amount of it is preferred.
  *

@@ -114,6 +117,10 @@ out:
  * overloading large (from a memory POV) nodes. That's right the effect
  * that counting the vcpus able to run on the nodes tries to prevent.
  *
+ * The relative distance within the nodes in the candidates is considered
+ * as the closer the nodes, the better for the domain ending up on the
+ * candidate.
+ *
  * Note that this completely ignore the number of nodes each candidate span,
  * as the fact that fewer nodes is better is already accounted for in the
  * algorithm.

@@ -124,6 +131,9 @@ static int numa_cmpf(const libxl__numa_c
     if (c1->nr_vcpus != c2->nr_vcpus)
         return c1->nr_vcpus - c2->nr_vcpus;
 
+    if (c1->dists_sum != c2->dists_sum)
+        return c1->dists_sum - c2->dists_sum;
+
     return c2->free_memkb - c1->free_memkb;
 }
tools/libxl/libxl_internal.h +1 -0
@@ -2735,6 +2735,7 @@ static inline void libxl__ctx_unlock(lib
 typedef struct {
     int nr_cpus, nr_nodes;
     int nr_vcpus;
+    int dists_sum;
     uint32_t free_memkb;
     libxl_bitmap nodemap;
 } libxl__numa_candidate;
tools/libxl/libxl_numa.c +39 -2

63 lines not on this page · raw message

author
#2 lore raw

Re: [PATCH 1 of 3 v2] libxl: take node distances into account during NUMA placement

Quoted 4 lines · Dario Faggioli

In fact, among placement candidates with the same number of nodes, the closer the various nodes are to each others, the better the performances for a domain placed there.

So, it seems we need to discuss a bit more about this patch... Please, ignore it for now.

OTOH, patches 2 and 3 can be considered for further review/ack/commit, even without this one, as they are pretty much unrelated from this one (and within each other). I'd expect them to apply cleanly, but if I need to rebase and resend, let me know.

Thanks and Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

author
#3 lore raw

PATCH v2 1/1 libxl, xl: user can ask for min and max nodes to use during placement

In reply to Dario Faggioli

And the placement algorithm will stick to that, or fail. This happens adding support for "minnodes=" and "maxnodes=" in the domain config file parser.

Signed-off-by Dario Faggioli <[email protected]>
Acked-by George Dunlap <[email protected]>
docs/man/xl.cfg.pod.5 +12 -0
@@ -133,6 +133,18 @@ the same time, achieving efficient utili
 
 =back
 
+=item B<minnodes=N>
+
+Tells libxl to place the new domain on at least `N` nodes. This is only
+effective if automatic NUMA placement occurs (i.e., if no `cpus=` option
+is specified).
+
+=item B<maxnodes=M>
+
+Tells libxl to place the new domain on at most `M` nodes. This is only
+effective if automatic NUMA placement occurs (i.e., if no `cpus=` option
+is specified).
+
 =head3 CPU Scheduling
 
 =over 4
tools/libxl/libxl_create.c +5 -0
@@ -219,6 +219,11 @@ int libxl__domain_build_info_setdefault(
 
     libxl_defbool_setdefault(&b_info->numa_placement, true);
 
+    if (!b_info->min_nodes)
+        b_info->min_nodes = 0;
+    if (!b_info->max_nodes)
+        b_info->max_nodes = 0;
+
     if (b_info->max_memkb == LIBXL_MEMKB_DEFAULT)
         b_info->max_memkb = 32 * 1024;
     if (b_info->target_memkb == LIBXL_MEMKB_DEFAULT)
tools/libxl/libxl_dom.c +2 -1
@@ -174,7 +174,8 @@ static int numa_place_domain(libxl__gc *
     /* Find the best candidate with enough free memory and at least
      * as much pcpus as the domain has vcpus.  */
     rc = libxl__get_numa_candidate(gc, memkb, info->max_vcpus,
-                                   0, 0, &cpupool_info.cpumap,
+                                   info->min_nodes, info->max_nodes,
+                                   &cpupool_info.cpumap,
                                    numa_cmpf, &candidate, &found);
     if (rc)
         goto out;
tools/libxl/libxl_types.idl +2 -0
@@ -262,6 +262,8 @@ libxl_domain_build_info = Struct("domain
     ("avail_vcpus",     libxl_bitmap),
     ("cpumap",          libxl_bitmap),
     ("numa_placement",  libxl_defbool),
+    ("min_nodes",       integer),
+    ("max_nodes",       integer),
     ("tsc_mode",        libxl_tsc_mode),
     ("max_memkb",       MemKB),
     ("target_memkb",    MemKB),
tools/libxl/xl_cmdimpl.c +5 -0
@@ -731,6 +731,11 @@ static void parse_config_data(const char
         libxl_defbool_set(&b_info->numa_placement, false);
     }
 
+    if (!xlu_cfg_get_long (config, "minnodes", &l, 0))
+        b_info->min_nodes = l;
+    if (!xlu_cfg_get_long (config, "maxnodes", &l, 0))
+        b_info->max_nodes = l;
+
     if (!xlu_cfg_get_long (config, "memory", &l, 0)) {
         b_info->max_memkb = l * 1024;
         b_info->target_memkb = b_info->max_memkb;
#4 lore raw

Re: [PATCH 2 of 3 v2] libxl, xl: user can ask for min and max nodes to use during placement

tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault( · line 225

Quoted 8 lines · Dario Faggioli
tools/libxl/libxl_create.c +4 -0
@@ -219,6 +219,11 @@ int libxl__domain_build_info_setdefault(
 
     libxl_defbool_setdefault(&b_info->numa_placement, true);
 
+    if (!b_info->min_nodes)
+        b_info->min_nodes = 0;
+    if (!b_info->max_nodes)
+        b_info->max_nodes = 0;

Aren't both of these if statements tautologous?

Ian.

author
#5 lore raw

Re: [PATCH 2 of 3 v2] libxl, xl: user can ask for min and max nodes to use during placement

Quoted 12 lines · Ian Campbell

tools/libxl/libxl_create.c int libxl__domain_build_info_setdefault( · line 225

Quoted 8 lines · Dario Faggioli
tools/libxl/libxl_create.c +4 -0
@@ -219,6 +219,11 @@ int libxl__domain_build_info_setdefault(
 
     libxl_defbool_setdefault(&b_info->numa_placement, true);
 
+    if (!b_info->min_nodes)
+        b_info->min_nodes = 0;
+    if (!b_info->max_nodes)
+        b_info->max_nodes = 0;

Aren't both of these if statements tautologous?

Looks like they are. I'll double check this before reposting.

I think what I wanted to do was making sure to zero those new fields, if not otherwise initialized by someone else... and, of course, I got that wrong! :-P

Thanks,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

author
#6 lore raw

PATCH v2 1/1 xl: allow for node-wise specification of vcpu pinning

In reply to Dario Faggioli

Making it possible to use something like the following:
* "nodes:0-3": all pCPUs of nodes 0,1,2,3;
* "nodes:0-3,^node:2": all pCPUS of nodes 0,1,3;
* "1,nodes:1-2,^6": pCPU 1 plus all pCPUs of nodes 1,2 but not pCPU 6;
* ...

In both domain config file and `xl vcpu-pin'.

Signed-off-by Dario Faggioli <[email protected]>
Acked-by George Dunlap <[email protected]>

---
Changes since v1:
* strstr() replaced with strncmp().

docs/man/xl.cfg.pod.5 +9 -1
@@ -109,7 +109,7 @@ some cpus on its own (see below). A C<CP
 
 =over 4
 
-=item "all"
+=item "all" (or "nodes:all")
 
 To allow all the vcpus of the guest to run on all the cpus on the host.
 

@@ -117,6 +117,14 @@ To allow all the vcpus of the guest to r
 
 To allow all the vcpus of the guest to run on cpus 0,2,3,5.
 
+=item "nodes:0-3,^node:2"
+
+To allow all the vcpus of the guest to run on the cpus belonging to
+the NUMA nodes 0,1,3 of the host. Notice that it is possible to combine
+this syntax with the one above. That means, something like "1,node:2,^6"
+is possible and means all the vcpus of the guest will run on cpus 1 plus
+on all the cpus of node 2, but never on cpu 6.
+
 =item ["2", "3"] (or [2, 3])
 
 To ask for specific vcpu mapping. That means (in this example), vcpu #0
tools/libxl/xl_cmdimpl.c +71 -23

129 lines not on this page · raw message