Skip to contentSkip to conversation

syscalls: Add set_mempolicy numa tests.

RFC PATCH New Cyril Hrubis 3 participants 6 messages Aug 9, 2018 – Aug 14, 2018 Library code

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

This is initial attempt to replace numa.sh tests that despite having been fixed several times have still many shortcommings that wouldn't easy to fix. It's not finished nor 100% replacement but I'm sending this anyway because I would like to get feedback at this point.

The main selling points of these testcases are:

The memory allocated for the testing is tracked exactly. We make sure that the mapping has separate record in /proc/$PID/numa_maps by mapping a region, then unmapping hole at the start and at the end of the mapping, then we fault the pages in the middle of the original mapping. We carefuly avoid doing anything that would cause the mapping to expand in the child process while the parent takes the measurements, even opening a file with fopen() may cause buffers to be allocated which may expand the mapping which is the reason we fork and take the measurements in the parent after the child process has faulted the pages.

The tests for file based shared interleaved mappings are no longer mapping a single small file but rather than that we accumulate statistic for larger amount of files over longer period of time and we also allow for small offset (currently 10%). We should probably also increase the number of samples we take as currently it's about 5MB in total on x86 although I haven't managed to make this test fail so far. This also fixes the test on Btrfs where the synthetic test that expects the pages to be distributed exactly equally fails.

What is not finished is compilation without libnuma, that will fail currently, but that is only a matter of adding a few ifdefs. And the coverage is still lacking, ideas for interesting testcases are welcomed as well.

CC Michal Hocko <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Jan Stancek <[email protected]>
---
 include/tst_numa.h                                 |  71 +++++++
 lib/tst_numa.c                                     | 231 +++++++++++++++++++++
 runtest/numa                                       |   5 +
 testcases/kernel/syscalls/set_mempolicy/.gitignore |   4 +
 testcases/kernel/syscalls/set_mempolicy/Makefile   |   7 +
 .../kernel/syscalls/set_mempolicy/set_mempolicy.h  |  26 +++
 .../syscalls/set_mempolicy/set_mempolicy01.c       |  98 +++++++++
 .../syscalls/set_mempolicy/set_mempolicy02.c       |  91 ++++++++
 .../syscalls/set_mempolicy/set_mempolicy03.c       |  99 +++++++++
 .../syscalls/set_mempolicy/set_mempolicy04.c       | 112 ++++++++++
 10 files changed, 744 insertions(+)
 create mode 100644 include/tst_numa.h
 create mode 100644 lib/tst_numa.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/.gitignore
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/Makefile
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy.h
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy02.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy04.c
include/tst_numa.h new file +71 -0
new file mode 100644
index 000000000..08833e398
@@ -0,0 +1,71 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef TST_NUMA_H__
+#define TST_NUMA_H__
+
+/**
+ * Numa nodemap.
+ */
+struct tst_nodemap {
+        /** Number of nodes in map */
+	unsigned int cnt;
+	/** Page allocation counters */
+	unsigned int *counters;
+	/** Array of numa ids */
+	unsigned int map[];
+};
+
+/**
+ * Clears numa counters. The counters are lazy-allocated on first call of this function.
+ *
+ * @nodes Numa nodemap.
+ */
+void tst_nodemap_reset_counters(struct tst_nodemap *nodes);
+
+/**
+ * Allocates requested number of pages, using mmap(), faults them and parsers
+ * /proc/$PID/numa_maps and adds amount of pages allocated per node to the
+ * nodemap counters. We also make sure that only the newly allocated pages are
+ * accounted for correctly, which requires forking a separate process to
+ * allocate the memory and checkpoints to synchronize parent and child process.
+ * The nodemap has to have counters initialized which happens on first counter
+ * reset.
+ *
+ * @nodes Nodemap with initialized counters.
+ * @path  If non-NULL the mapping is backed up by a file.
+ * @pages Number of pages to be allocated.
+ */
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages);
+
+/**
+ * Frees nodemap.
+ *
+ * @nodes Numa nodemap to be freed.
+ */
+void tst_nodemap_free(struct tst_nodemap *nodes);
+
+/**
+ * Bitflags for tst_get_nodemap() function.
+ */
+enum tst_numa_types {
+	TST_NUMA_ANY = 0x00,
+	TST_NUMA_MEM = 0x01,
+};
+
+/**
+ * Allocates and returns numa node map, which is an array of numa nodes which
+ * contain desired resources e.g. memory.
+ *
+ * @types Bitflags of enum tst_numa_types specifying desired resources.
+ *
+ * @return On success returns allocated and initialized struct tst_nodemap which contains
+ *         array of numa node ids that contains desired resources.
+ */
+struct tst_nodemap *tst_get_nodemap(int type);
+
+#endif /* TST_NUMA_H__ */
lib/tst_numa.c new file +231 -0

231 lines not on this page · raw message

runtest/numa +5 -0
index 12aedbb4b..7885be90c 100644
@@ -11,3 +11,8 @@ move_pages09 move_pages09
 move_pages10 move_pages10
 move_pages11 move_pages11
 move_pages12 move_pages12
+
+set_mempolicy01 set_mempolicy01
+set_mempolicy02 set_mempolicy02
+set_mempolicy03 set_mempolicy03
+set_mempolicy04 set_mempolicy04
testcases/kernel/syscalls/set_mempolicy/.gitignore new file +4 -0
new file mode 100644
index 000000000..c5e35a405
@@ -0,0 +1,4 @@
+/set_mempolicy01
+/set_mempolicy02
+set_mempolicy03
+set_mempolicy04
testcases/kernel/syscalls/set_mempolicy/Makefile new file +7 -0
new file mode 100644
index 000000000..d273b432b
@@ -0,0 +1,7 @@
+top_srcdir		?= ../../../..
+
+include $(top_srcdir)/include/mk/testcases.mk
+
+LDLIBS  += $(NUMA_LIBS)
+
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
testcases/kernel/syscalls/set_mempolicy/set_mempolicy.h new file +26 -0
new file mode 100644
index 000000000..0539db0b5
@@ -0,0 +1,26 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef SET_MEMPOLICY_H__
+#define SET_MEMPOLICY_H__
+
+static const char *mode_name(int mode)
+{
+	switch (mode) {
+	case MPOL_DEFAULT:
+		return "MPOL_DEFAULT";
+	case MPOL_BIND:
+		return "MPOL_BIND";
+	case MPOL_PREFERRED:
+		return "MPOL_PREFERRED";
+	//case MPOL_LOCAL:
+	//	return "MPOL_LOCAL";
+	default:
+		return "???";
+	}
+}
+
+#endif /* SET_MEMPOLICY_H__ */
testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c new file +98 -0

98 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy02.c new file +91 -0

91 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c new file +99 -0

99 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy04.c new file +112 -0

112 lines not on this page · raw message

Signature · 1 line
2.13.6
#1 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

Hello,

Just some shallow comments on the API.

Quoted 83 lines · Cyril Hrubis

This is initial attempt to replace numa.sh tests that despite having been fixed several times have still many shortcommings that wouldn't easy to fix. It's not finished nor 100% replacement but I'm sending this anyway because I would like to get feedback at this point.

The main selling points of these testcases are:

The memory allocated for the testing is tracked exactly. We make sure that the mapping has separate record in /proc/$PID/numa_maps by mapping a region, then unmapping hole at the start and at the end of the mapping, then we fault the pages in the middle of the original mapping. We carefuly avoid doing anything that would cause the mapping to expand in the child process while the parent takes the measurements, even opening a file with fopen() may cause buffers to be allocated which may expand the mapping which is the reason we fork and take the measurements in the parent after the child process has faulted the pages.

The tests for file based shared interleaved mappings are no longer mapping a single small file but rather than that we accumulate statistic for larger amount of files over longer period of time and we also allow for small offset (currently 10%). We should probably also increase the number of samples we take as currently it's about 5MB in total on x86 although I haven't managed to make this test fail so far. This also fixes the test on Btrfs where the synthetic test that expects the pages to be distributed exactly equally fails.

What is not finished is compilation without libnuma, that will fail currently, but that is only a matter of adding a few ifdefs. And the coverage is still lacking, ideas for interesting testcases are welcomed as well.

CC Michal Hocko <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Jan Stancek <[email protected]>
---
 include/tst_numa.h                                 |  71 +++++++
 lib/tst_numa.c                                     | 231 +++++++++++++++++++++
 runtest/numa                                       |   5 +
 testcases/kernel/syscalls/set_mempolicy/.gitignore |   4 +
 testcases/kernel/syscalls/set_mempolicy/Makefile   |   7 +
 .../kernel/syscalls/set_mempolicy/set_mempolicy.h  |  26 +++
 .../syscalls/set_mempolicy/set_mempolicy01.c       |  98 +++++++++
 .../syscalls/set_mempolicy/set_mempolicy02.c       |  91 ++++++++
 .../syscalls/set_mempolicy/set_mempolicy03.c       |  99 +++++++++
 .../syscalls/set_mempolicy/set_mempolicy04.c       | 112 ++++++++++
 10 files changed, 744 insertions(+)
 create mode 100644 include/tst_numa.h
 create mode 100644 lib/tst_numa.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/.gitignore
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/Makefile
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy.h
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy02.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy04.c
include/tst_numa.h new file +20 -0
new file mode 100644
index 000000000..08833e398
@@ -0,0 +1,71 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef TST_NUMA_H__
+#define TST_NUMA_H__
+
+/**
+ * Numa nodemap.
+ */
+struct tst_nodemap {
+        /** Number of nodes in map */
+	unsigned int cnt;
+	/** Page allocation counters */
+	unsigned int *counters;
+	/** Array of numa ids */
+	unsigned int map[];
+};

include/tst_numa.h line 30

Quoted 10 lines
include/tst_numa.h new file +10 -0
+
+/**
+ * Clears numa counters. The counters are lazy-allocated on first call of this function.
+ *
+ * @nodes Numa nodemap.
+ */
+void tst_nodemap_reset_counters(struct tst_nodemap *nodes);
+
+/**
+ * Allocates requested number of pages, using mmap(), faults them and parsers

^the

include/tst_numa.h line 31

Quoted 1 line
include/tst_numa.h new file +1 -0
+ * /proc/$PID/numa_maps and adds amount of pages allocated per node to the

^the

include/tst_numa.h line 33

Quoted 2 lines
include/tst_numa.h new file +2 -0
+ * nodemap counters. We also make sure that only the newly allocated pages are
+ * accounted for correctly, which requires forking a separate process to

I think you mean that only newly allocated pages are counted? (i.e. previously allocated pages are ignored). The above implies that previously allocated pages are accounted for incorrectly.

Quoted 59 lines
include/tst_numa.h new file +38 -0
+ * allocate the memory and checkpoints to synchronize parent and child process.
+ * The nodemap has to have counters initialized which happens on first counter
+ * reset.
+ *
+ * @nodes Nodemap with initialized counters.
+ * @path  If non-NULL the mapping is backed up by a file.
+ * @pages Number of pages to be allocated.
+ */
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages);
+
+/**
+ * Frees nodemap.
+ *
+ * @nodes Numa nodemap to be freed.
+ */
+void tst_nodemap_free(struct tst_nodemap *nodes);
+
+/**
+ * Bitflags for tst_get_nodemap() function.
+ */
+enum tst_numa_types {
+	TST_NUMA_ANY = 0x00,
+	TST_NUMA_MEM = 0x01,
+};
+
+/**
+ * Allocates and returns numa node map, which is an array of numa nodes which
+ * contain desired resources e.g. memory.
+ *
+ * @types Bitflags of enum tst_numa_types specifying desired resources.
+ *
+ * @return On success returns allocated and initialized struct tst_nodemap which contains
+ *         array of numa node ids that contains desired resources.
+ */
+struct tst_nodemap *tst_get_nodemap(int type);
+
+#endif /* TST_NUMA_H__ */
lib/tst_numa.c new file +15 -0
new file mode 100644
index 000000000..2328c2814
@@ -0,0 +1,231 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#include <stdio.h>
+#include <ctype.h>
+#include "config.h"
+#ifdef HAVE_NUMA_H
+# include <numa.h>
+#endif
+
+#define TST_NO_DEFAULT_MAIN
+#include "tst_test.h"

lib/tst_numa.c line 25

Quoted 10 lines
lib/tst_numa.c new file +10 -0
+#include "tst_numa.h"
+
+static void store_val(unsigned int node, unsigned int val,
+                      struct tst_nodemap *nodes)
+{
+	unsigned int i;
+
+	for (i = 0; i < nodes->cnt; i++) {
+		if (nodes->map[i] == node) {
+			nodes->counters[i] += val;

Maybe this should be called inc_counter or similar? Because you are doing += not =.

Quoted 10 lines
lib/tst_numa.c new file +10 -0
+			break;
+		}
+	}
+
+//	tst_res(TINFO, "Node %u allocated %u pages", node, val);
+}
+
+static void parse_line(char *line, struct tst_nodemap *nodes)
+{
+	char *c;

lib/tst_numa.c line 45

Quoted 10 lines
lib/tst_numa.c new file +10 -0
+	int state = 0;
+	int node;
+	int val;
+
+	for (c = line; *c && *c != '\n'; c++) {
+
+		if (state == 0 && *c == 'N') {
+			state = 1;
+			continue;
+		}

We need to skip the file path because N3=1.txt is a valid file name. Unless it is guaranteed we will never parse a line with that file name in it.

Quoted 86 lines
lib/tst_numa.c new file +86 -0
+
+		if (state == 1) {
+			if (isdigit(*c)) {
+				node = *c - '0';
+				state = 2;
+			} else {
+				state = 0;
+			}
+			continue;
+		}
+
+		if (state == 2) {
+			if (isdigit(*c)) {
+				node *= 10;
+				node += *c - '0';
+			} else {
+				if (*c == '=') {
+					val = 0;
+					state = 3;
+				} else {
+					state = 0;
+				}
+			}
+			continue;
+		}
+
+		if (state == 3) {
+			if (isdigit(*c)) {
+				val *= 10;
+				val += *c - '0';
+			} else {
+				store_val(node, val, nodes);
+				state = 0;
+			}
+		}
+	}
+}
+
+static char *strip_newline(char *str)
+{
+	size_t i;
+
+	for (i = 0; str[i]; i++) {
+		if (str[i] == '\n') {
+			str[i] = 0;
+			break;
+		}
+	}
+
+	return str;
+}
+
+static void tst_parse_numa_maps(void *ptr, int pid, struct tst_nodemap *nodes)
+{
+	FILE *f;
+	char line[2048];
+	char fname[128];
+	void *p;
+
+	snprintf(fname, sizeof(fname), "/proc/%i/numa_maps", pid);
+
+	f = fopen(fname, "r");
+	if (!f)
+		tst_brk(TBROK | TERRNO, "open(/proc/self/numa_maps)");
+
+	while (fgets(line, sizeof(line), f)) {
+		sscanf(line, "%p", &p);
+		if (p == ptr) {
+			//tst_res(TINFO, "Parsing '%s'", strip_newline(line));
+			parse_line(line, nodes);
+			goto out;
+		}
+	}
+
+	tst_res(TWARN, "Mapping %p not found in numa_maps!", ptr);
+out:
+	fclose(f);
+}
+
+void tst_nodemap_reset_counters(struct tst_nodemap *nodes)
+{
+	size_t arr_size = sizeof(unsigned int) * nodes->cnt;
+
+	if (!nodes->counters)
+		nodes->counters = SAFE_MALLOC(arr_size);
+

lib/tst_numa.c line 141

Quoted 10 lines
lib/tst_numa.c new file +10 -0
+	memset(nodes->counters, 0, arr_size);
+}
+
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages)
+{
+	size_t page_size = getpagesize();
+	char *ptr;
+	int fd = -1;
+	int flags = MAP_PRIVATE|MAP_ANONYMOUS;

It might be useful to ensure nodes are allocated here by calling reset_counters if they are not. On the other hand I am not sure this makes sense with the current usage.

--
Thank you,
Richard.

#2 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

Hi!

Quoted 6 lines
Quoted 2 lines

+ * nodemap counters. We also make sure that only the newly allocated pages are
+ * accounted for correctly, which requires forking a separate process to

I think you mean that only newly allocated pages are counted? (i.e. previously allocated pages are ignored). The above implies that previously allocated pages are accounted for incorrectly.

It should say something "We also make sure only and only the pages we allocated for the test are accounted for." or something similar.

Quoted 11 lines

lib/tst_numa.c line 25

Quoted 8 lines
lib/tst_numa.c new file +8 -0
+static void store_val(unsigned int node, unsigned int val,
+                      struct tst_nodemap *nodes)
+{
+	unsigned int i;
+
+	for (i = 0; i < nodes->cnt; i++) {
+		if (nodes->map[i] == node) {
+			nodes->counters[i] += val;

Maybe this should be called inc_counter or similar? Because you are doing += not =.

Given that there are multiple counters, best name would probably be inc_counters().

Quoted 16 lines
Quoted 16 lines
lib/tst_numa.c new file +16 -0
+			break;
+		}
+	}
+
+//	tst_res(TINFO, "Node %u allocated %u pages", node, val);
+}
+
+static void parse_line(char *line, struct tst_nodemap *nodes)
+{
+	char *c;
+	int state = 0;
+	int node;
+	int val;
+
+	for (c = line; *c && *c != '\n'; c++) {
+
Quoted 8 lines

lib/tst_numa.c line 45

Quoted 4 lines
lib/tst_numa.c new file +4 -0
+		if (state == 0 && *c == 'N') {
+			state = 1;
+			continue;
+		}

We need to skip the file path because N3=1.txt is a valid file name. Unless it is guaranteed we will never parse a line with that file name in it.

Theoretically it's possible, but very unlikely since our process would have to open such silly named file somewhere in the libc library without our knowledge. But anyways it looks like Jan has much better way of figuring out on which node are pages allocated on, so I will probably go with that in v2.

Quoted 11 lines

lib/tst_numa.c line 141

Quoted 7 lines
lib/tst_numa.c new file +7 -0
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages)
+{
+	size_t page_size = getpagesize();
+	char *ptr;
+	int fd = -1;
+	int flags = MAP_PRIVATE|MAP_ANONYMOUS;

It might be useful to ensure nodes are allocated here by calling reset_counters if they are not. On the other hand I am not sure this makes sense with the current usage.

Actually it does not, since certain tests accumulate the statistic over several allocatings, so this is supposed to be called in loop while we gather statistic in the counters.

Signature · 2 lines
Cyril Hrubis
[email protected]
#3 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

In reply to Cyril Hrubis

----- Original Message -----

Quoted 4 lines

This is initial attempt to replace numa.sh tests that despite having been fixed several times have still many shortcommings that wouldn't easy to fix. It's not finished nor 100% replacement but I'm sending this anyway because I would like to get feedback at this point.

Hi,

Why not use get_mempolicy(.., MPOL_F_NODE | MPOL_F_ADDR) to get node id, for the page we just allocated?

This kinda overlaps with numa_helper in some aspects. Which reminds me of some issues we had to address there:
- memory-less nodes
- nodes with little to no free memory

Regards,
Jan

Quoted 27 lines

The main selling points of these testcases are:

The memory allocated for the testing is tracked exactly. We make sure that the mapping has separate record in /proc/$PID/numa_maps by mapping a region, then unmapping hole at the start and at the end of the mapping, then we fault the pages in the middle of the original mapping. We carefuly avoid doing anything that would cause the mapping to expand in the child process while the parent takes the measurements, even opening a file with fopen() may cause buffers to be allocated which may expand the mapping which is the reason we fork and take the measurements in the parent after the child process has faulted the pages.

The tests for file based shared interleaved mappings are no longer mapping a single small file but rather than that we accumulate statistic for larger amount of files over longer period of time and we also allow for small offset (currently 10%). We should probably also increase the number of samples we take as currently it's about 5MB in total on x86 although I haven't managed to make this test fail so far. This also fixes the test on Btrfs where the synthetic test that expects the pages to be distributed exactly equally fails.

What is not finished is compilation without libnuma, that will fail currently, but that is only a matter of adding a few ifdefs. And the coverage is still lacking, ideas for interesting testcases are welcomed as well.

#4 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

Hi!

Quoted 2 lines

Why not use get_mempolicy(.., MPOL_F_NODE | MPOL_F_ADDR) to get node id, for the page we just allocated?

Good point, I will try to look into that.

Quoted 4 lines

This kinda overlaps with numa_helper in some aspects. Which reminds me of some issues we had to address there:
- memory-less nodes
- nodes with little to no free memory

I'm aware of the overlap and we should definitely merge these two into a single library later on.

As far as I can tell the library I wrote covers the case of memory-less nodes, these should the the ones that does not have the membind flag set in the default mask that is returned by numa_get_membind().

We should probably add API for specifying minimal amount of free memory per node as well, which would fix the second problem. We may add min_free parameter to the tst_get_nodemap() function that would be used if passed the TST_NUMA_MEM flag.

Signature · 2 lines
Cyril Hrubis
[email protected]
#5 lore raw

RFC PATCH 1/1 syscalls: Add set_mempolicy numa tests.

----- Original Message -----

Quoted 11 lines

Hi!

Quoted 2 lines

Why not use get_mempolicy(.., MPOL_F_NODE | MPOL_F_ADDR) to get node id, for the page we just allocated?

Good point, I will try to look into that.

Quoted 4 lines

This kinda overlaps with numa_helper in some aspects. Which reminds me of some issues we had to address there:
- memory-less nodes
- nodes with little to no free memory

Quoted 6 lines

I'm aware of the overlap and we should definitely merge these two into a single library later on.

As far as I can tell the library I wrote covers the case of memory-less nodes, these should the the ones that does not have the membind flag set in the default mask that is returned by numa_get_membind().

Confirmed.

# ./a.out 
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
node 0 size: 0 MB
node 0 free: 0 MB
node 1 cpus:
node 1 size: 12288 MB
node 1 free: 11018 MB
node distances:
node   0   1 
  0:  10  40 
  1:  40  10 

membind[0] = 0
membind[1] = 1

If this has support in older releases, it looks simpler than what numa_helper is doing (iterating over sysfs).

Regards,
Jan