Skip to contentSkip to conversation

syscalls: Add set_mempolicy numa tests.

PATCH Superseded Cyril Hrubis 2 participants 3 messages Aug 23, 2018 – Aug 24, 2018 Library code

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 syscalls: Add set_mempolicy numa tests.

This is initial attempt to replace numa.sh tests that despite having been fixed several times have still many shortcommings that wouldn't be easy to fix. It's not finished nor 100% replacement but I'm sending this anyway because I would like to get feedback at this point.

The main selling points of these testcases are:

The memory allocated for the testing is tracked exactly. We are using get_mempolicy() with MPOL_F_NODE | MPOL_F_ADDR that returns the node ID on which specified address is allocated on to count pages allocated per node after we set desired memory policy.

The tests for file based shared interleaved mappings are no longer mapping a single small file but rather than that we accumulate statistic for larger amount of files over longer period of time and we also allow for small offset (currently 10%). We should probably also increase the number of samples we take as currently it's about 5MB in total on x86 although I haven't managed to make this test fail so far. This also fixes the test on Btrfs where the synthetic test that expects the pages to be distributed exactly equally fails.

What is not finished is compilation without libnuma, that will fail currently, but that is only a matter of adding a few ifdefs. And the coverage is still lacking, ideas for interesting testcases are welcomed as well.

I tested this testcases back to SLES11, which is basically the oldest distribution we really care for and everything was working fine.

CC Michal Hocko <[email protected]>
CC Vlastimil Babka <[email protected]>
CC Jan Stancek <[email protected]>
---
 include/tst_numa.h                                 |  76 ++++++++++++
 lib/tst_numa.c                                     | 138 +++++++++++++++++++++
 runtest/numa                                       |   5 +
 testcases/kernel/syscalls/set_mempolicy/.gitignore |   4 +
 testcases/kernel/syscalls/set_mempolicy/Makefile   |   7 ++
 .../kernel/syscalls/set_mempolicy/set_mempolicy.h  |  26 ++++
 .../syscalls/set_mempolicy/set_mempolicy01.c       | 107 ++++++++++++++++
 .../syscalls/set_mempolicy/set_mempolicy02.c       | 101 +++++++++++++++
 .../syscalls/set_mempolicy/set_mempolicy03.c       |  99 +++++++++++++++
 .../syscalls/set_mempolicy/set_mempolicy04.c       | 112 +++++++++++++++++
 10 files changed, 675 insertions(+)
 create mode 100644 include/tst_numa.h
 create mode 100644 lib/tst_numa.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/.gitignore
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/Makefile
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy.h
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy02.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c
 create mode 100644 testcases/kernel/syscalls/set_mempolicy/set_mempolicy04.c
include/tst_numa.h new file +76 -0
new file mode 100644
index 0000000..9551681
@@ -0,0 +1,76 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef TST_NUMA_H__
+#define TST_NUMA_H__
+
+/**
+ * Numa nodemap.
+ */
+struct tst_nodemap {
+        /** Number of nodes in map */
+	unsigned int cnt;
+	/** Page allocation counters */
+	unsigned int *counters;
+	/** Array of numa ids */
+	unsigned int map[];
+};
+
+/**
+ * Clears numa counters. The counters are lazy-allocated on first call of this function.
+ *
+ * @nodes Numa nodemap.
+ */
+void tst_nodemap_reset_counters(struct tst_nodemap *nodes);
+
+/**
+ * Prints pages allocated per each node.
+ *
+ * @nodes Numa nodemap.
+ */
+void tst_nodemap_print_counters(struct tst_nodemap *nodes);
+
+/**
+ * Allocates the requested number of pages, using mmap(), faults them and adds
+ * the amount of pages allocated per node to the nodemap counters. We also make
+ * sure that only the newly allocated pages are accounted for correctly. The
+ * nodemap has to have counters initialized which happens on first counter
+ * reset.
+ *
+ * @nodes Nodemap with initialized counters.
+ * @path  If non-NULL the mapping is backed up by a file.
+ * @pages Number of pages to be allocated.
+ */
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages);
+
+/**
+ * Frees nodemap.
+ *
+ * @nodes Numa nodemap to be freed.
+ */
+void tst_nodemap_free(struct tst_nodemap *nodes);
+
+/**
+ * Bitflags for tst_get_nodemap() function.
+ */
+enum tst_numa_types {
+	TST_NUMA_ANY = 0x00,
+	TST_NUMA_MEM = 0x01,
+};
+
+/**
+ * Allocates and returns numa node map, which is an array of numa nodes which
+ * contain desired resources e.g. memory.
+ *
+ * @types Bitflags of enum tst_numa_types specifying desired resources.
+ *
+ * @return On success returns allocated and initialized struct tst_nodemap which contains
+ *         array of numa node ids that contains desired resources.
+ */
+struct tst_nodemap *tst_get_nodemap(int type);
+
+#endif /* TST_NUMA_H__ */
lib/tst_numa.c new file +138 -0

138 lines not on this page · raw message

runtest/numa +5 -0
index 12aedbb..7885be9 100644
@@ -11,3 +11,8 @@ move_pages09 move_pages09
 move_pages10 move_pages10
 move_pages11 move_pages11
 move_pages12 move_pages12
+
+set_mempolicy01 set_mempolicy01
+set_mempolicy02 set_mempolicy02
+set_mempolicy03 set_mempolicy03
+set_mempolicy04 set_mempolicy04
testcases/kernel/syscalls/set_mempolicy/.gitignore new file +4 -0
new file mode 100644
index 0000000..c5e35a4
@@ -0,0 +1,4 @@
+/set_mempolicy01
+/set_mempolicy02
+set_mempolicy03
+set_mempolicy04
testcases/kernel/syscalls/set_mempolicy/Makefile new file +7 -0
new file mode 100644
index 0000000..d273b43
@@ -0,0 +1,7 @@
+top_srcdir		?= ../../../..
+
+include $(top_srcdir)/include/mk/testcases.mk
+
+LDLIBS  += $(NUMA_LIBS)
+
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
testcases/kernel/syscalls/set_mempolicy/set_mempolicy.h new file +26 -0
new file mode 100644
index 0000000..0539db0
@@ -0,0 +1,26 @@
+/*
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ *
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef SET_MEMPOLICY_H__
+#define SET_MEMPOLICY_H__
+
+static const char *mode_name(int mode)
+{
+	switch (mode) {
+	case MPOL_DEFAULT:
+		return "MPOL_DEFAULT";
+	case MPOL_BIND:
+		return "MPOL_BIND";
+	case MPOL_PREFERRED:
+		return "MPOL_PREFERRED";
+	//case MPOL_LOCAL:
+	//	return "MPOL_LOCAL";
+	default:
+		return "???";
+	}
+}
+
+#endif /* SET_MEMPOLICY_H__ */
testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c new file +107 -0

107 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy02.c new file +101 -0

101 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c new file +99 -0

99 lines not on this page · raw message

testcases/kernel/syscalls/set_mempolicy/set_mempolicy04.c new file +112 -0

112 lines not on this page · raw message

Signature · 1 line
2.12.3
#1 lore raw

PATCH v2 1/1 syscalls: Add set_mempolicy numa tests.

----- Original Message -----

Quoted 22 lines

This is initial attempt to replace numa.sh tests that despite having been fixed several times have still many shortcommings that wouldn't be easy to fix. It's not finished nor 100% replacement but I'm sending this anyway because I would like to get feedback at this point.

The main selling points of these testcases are:

The memory allocated for the testing is tracked exactly. We are using get_mempolicy() with MPOL_F_NODE | MPOL_F_ADDR that returns the node ID on which specified address is allocated on to count pages allocated per node after
we set desired memory policy.

The tests for file based shared interleaved mappings are no longer mapping a single small file but rather than that we accumulate statistic for larger amount of files over longer period of time and we also allow for small offset (currently 10%). We should probably also increase the number of samples we take as currently it's about 5MB in total on x86 although I haven't managed to make this test fail so far. This also fixes the test on Btrfs where the synthetic test that expects the pages to be distributed exactly equally fails.

Quoted 8 lines

What is not finished is compilation without libnuma, that will fail currently, but that is only a matter of adding a few ifdefs. And the coverage is still lacking, ideas for interesting testcases are welcomed as well.

I tested this testcases back to SLES11, which is basically the oldest distribution we really care for and everything was working fine.

I don't have objections to this approach. Just want to re-iterate, that we should also check each node has enough free memory. Couple comments below:

<snip>

+static void inc_counter(unsigned int node, struct tst_nodemap *nodes) +{
+ unsigned int i;
+
+ for (i = 0; i < nodes->cnt; i++) {
+ if (nodes->map[i] == node) {
+ nodes->counters[i]++;
+ break;
+ }

I'd add some check here to warn or TBROK, in case we somehow end up with node id that's not in the map.

Quoted 29 lines
lib/tst_numa.c new file +28 -0
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages)
+{
+	size_t page_size = getpagesize();
+	char *ptr;
+	int fd = -1;
+	int flags = MAP_PRIVATE|MAP_ANONYMOUS;
+	int node;
+	unsigned int i;
+
+	if (path) {
+		fd = SAFE_OPEN(path, O_CREAT | O_EXCL | O_RDWR, 0666);
+		SAFE_FTRUNCATE(fd, pages * page_size);
+		flags = MAP_SHARED;
+	}
+
+	ptr = SAFE_MMAP(NULL, pages  * page_size,
+	                PROT_READ|PROT_WRITE, flags, fd, 0);
+
+	if (path) {
+		SAFE_CLOSE(fd);
+		SAFE_UNLINK(path);
+	}
+
+	memset(ptr, 'a', pages * page_size);
+
+	for (i = 0; i < pages; i++) {
+		get_mempolicy(&node, NULL, 0, ptr + i * page_size, MPOL_F_NODE |

MPOL_F_ADDR);

lib/tst_numa.c line 86

Quoted 8 lines
lib/tst_numa.c new file +8 -0
+
+		if (node < 0 || (unsigned int)node >= nodes->cnt) {
+			tst_res(TWARN, "get_mempolicy(...) returned invalid node %i\n", node);
+			continue;
+		}
+
+		inc_counter(node, nodes);
+	}

This seems useful as a separate function, for example: tst_nodemap_count_area(ptr, size). Maybe the user wants to count already allocated area.

Quoted 6 lines

+++ b/testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c
+ * We are testing set_mempolicy() with MPOL_BIND and MPOL_PREFERRED.

+++ b/testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c
+ * We are testing set_mempolicy() with MPOL_BIND and MPOL_PREFERRED backed by a file.

First and third test look very similar, any chance we can combine them?

Regards,
Jan

#2 lore raw

PATCH v2 1/1 syscalls: Add set_mempolicy numa tests.

Hi!

Quoted 2 lines

I don't have objections to this approach. Just want to re-iterate, that we should also check each node has enough free memory. Couple comments below:

Yes, that is still missing here.

Quoted 10 lines
lib/tst_numa.c new file +9 -0
+static void inc_counter(unsigned int node, struct tst_nodemap *nodes)
+{
+        unsigned int i;
+
+        for (i = 0; i < nodes->cnt; i++) {
+                if (nodes->map[i] == node) {
+                        nodes->counters[i]++;
+                        break;
+                }
Quoted 2 lines

I'd add some check here to warn or TBROK, in case we somehow end up with node id that's not in the map.

Well I do that in the tst_numa_alloc_parse() but I agree that this is a more logical place to do the check.

Quoted 32 lines
Quoted 32 lines
lib/tst_numa.c new file +28 -0
+void tst_numa_alloc_parse(struct tst_nodemap *nodes, const char *path,
+                          unsigned int pages)
+{
+	size_t page_size = getpagesize();
+	char *ptr;
+	int fd = -1;
+	int flags = MAP_PRIVATE|MAP_ANONYMOUS;
+	int node;
+	unsigned int i;
+
+	if (path) {
+		fd = SAFE_OPEN(path, O_CREAT | O_EXCL | O_RDWR, 0666);
+		SAFE_FTRUNCATE(fd, pages * page_size);
+		flags = MAP_SHARED;
+	}
+
+	ptr = SAFE_MMAP(NULL, pages  * page_size,
+	                PROT_READ|PROT_WRITE, flags, fd, 0);
+
+	if (path) {
+		SAFE_CLOSE(fd);
+		SAFE_UNLINK(path);
+	}
+
+	memset(ptr, 'a', pages * page_size);
+
+	for (i = 0; i < pages; i++) {
+		get_mempolicy(&node, NULL, 0, ptr + i * page_size, MPOL_F_NODE |

MPOL_F_ADDR);

lib/tst_numa.c new file +3 -0
+
+		if (node < 0 || (unsigned int)node >= nodes->cnt) {
+			tst_res(TWARN, "get_mempolicy(...) returned invalid node %i\n", node);
Quoted 8 lines

lib/tst_numa.c line 86

Quoted 5 lines
lib/tst_numa.c new file +5 -0
+			continue;
+		}
+
+		inc_counter(node, nodes);
+	}

This seems useful as a separate function, for example: tst_nodemap_count_area(ptr, size). Maybe the user wants to count already allocated area.

Sure.

Quoted 8 lines
Quoted 6 lines

+++ b/testcases/kernel/syscalls/set_mempolicy/set_mempolicy01.c
+ * We are testing set_mempolicy() with MPOL_BIND and MPOL_PREFERRED.

+++ b/testcases/kernel/syscalls/set_mempolicy/set_mempolicy03.c
+ * We are testing set_mempolicy() with MPOL_BIND and MPOL_PREFERRED backed by a file.

First and third test look very similar, any chance we can combine them?

I doubt so, the main difference is that the third test allocates the memory by faulting memory mapped file and hence it's executed with all_filesystems flag, while the the first one works only with anonymous memory.

I guess that we may put some pieces of the code to a shared library though.

Signature · 2 lines
Cyril Hrubis
[email protected]