Skip to contentSkip to conversation

Fix selftests/mm build without requiring "make headers"

PATCH Applied John Hubbard 3 participants 7 messages Mar 28, 2024 – Apr 4, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Fix selftests/mm build without requiring "make headers"

Hi,

As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.

This should also make Peter Zijlstra happy, because it directly addresses the root cause of his "NAK NAK NAK" reply [2]. :)

I haven't done much build testing, because selftests are not so easy to build with a cross-compiler. So it's just tested on x86 64-bit so far.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>

John Hubbard (2):
selftests: break the dependency upon local header files selftests/mm: fix additional build errors for selftests

 tools/include/uapi/linux/memfd.h       |  39 +++
 tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++
 tools/testing/selftests/lib.mk         |   9 +
 tools/testing/selftests/mm/Makefile    |   2 +-
 4 files changed, 435 insertions(+), 1 deletion(-)
 create mode 100644 tools/include/uapi/linux/memfd.h
 create mode 100644 tools/include/uapi/linux/userfaultfd.h

base-commit: 98560e9019851bf55b8a4073978a623a3bcf98c0

Signature · 1 line
2.44.0
#1 lore raw

PATCH 1/2 selftests: break the dependency upon local header files

Use tools/include/uapi/ files instead. These are obtained by taking a
snapshot: run "make headers" at the top level, then copy the desired header file into the appropriate subdir in tools/uapi/.

This was discussed and solved in [1].

However, even before copying any additional files there, there are already quite a few in tools/include/uapi already. And these will immediately fix a number of selftests/mm build failures.

So this patch:

a) Adds TOOLS_INCLUDES to selftests/lib.mk, so that all selftests can immediately and easily include the snapshotted header files.

b) Uses $(TOOLS_INCLUDES) in the selftests/mm build. On today's Arch Linux, this already fixes all build errors except for a few userfaultfd.h (those will be addressed in a subsequent patch).

[1] https://lore.kernel.org/all/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/testing/selftests/lib.mk      | 9 +++++++++
 tools/testing/selftests/mm/Makefile | 2 +-
 2 files changed, 10 insertions(+), 1 deletion(-)
tools/testing/selftests/lib.mk +9 -0
index da2cade3bab0..1dae4a02957f 100644
@@ -48,6 +48,15 @@ ifeq ($(KHDR_INCLUDES),)
 KHDR_INCLUDES := -isystem $(top_srcdir)/usr/include
 endif
 
+# In order to use newer items that haven't yet been added to the user's system
+# header files, add $(TOOLS_INCLUDES) to the compiler invocation in each
+# each selftest.
+# You may need to add files to that location, or to refresh an existing file. In
+# order to do that, run "make headers" from $(top_srcdir), then copy the
+# header file that you want from $(top_srcdir)/usr/include/... , to the matching
+# subdir in $(TOOLS_INCLUDE).
+TOOLS_INCLUDES := -isystem $(top_srcdir)/tools/include/uapi
+
 # The following are built by lib.mk common compile rules.
 # TEST_CUSTOM_PROGS should be used by tests that require
 # custom build rule and prevent common build rule use.
tools/testing/selftests/mm/Makefile +1 -1
index eb5f39a2668b..7ca9186a0639 100644
@@ -32,7 +32,7 @@ endif
 # LDLIBS.
 MAKEFLAGS += --no-builtin-rules
 
-CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES)
+CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) $(TOOLS_INCLUDES)
 LDLIBS = -lrt -lpthread -lm
 
 TEST_GEN_FILES = cow
Signature · 1 line
2.44.0
Acked-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] selftests: break the dependency upon local header files

Quoted 21 lines · John Hubbard

Use tools/include/uapi/ files instead. These are obtained by taking a
snapshot: run "make headers" at the top level, then copy the desired header file into the appropriate subdir in tools/uapi/.

This was discussed and solved in [1].

However, even before copying any additional files there, there are already quite a few in tools/include/uapi already. And these will immediately fix a number of selftests/mm build failures.

So this patch:

a) Adds TOOLS_INCLUDES to selftests/lib.mk, so that all selftests can immediately and easily include the snapshotted header files.

b) Uses $(TOOLS_INCLUDES) in the selftests/mm build. On today's Arch Linux, this already fixes all build errors except for a few userfaultfd.h (those will be addressed in a subsequent patch).

[1] https://lore.kernel.org/all/[email protected]/

Quoted 8 lines
Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>

---

Signature · 3 lines
Cheers,

David / dhildenb
#3 lore raw

PATCH 2/2 selftests/mm: fix additional build errors for selftests

In reply to John Hubbard

These build errors only occur if one fails to first run "make headers". However, that is a non-obvious and instrusive requirement, and so there was a discussion on how to get rid of it [1]. This uses that solution.

These two files were created by taking a snapshot of the generated header files that are created via "make headers". These two files were copied from ./usr/include/linux/ to ./tools/include/uapi/linux/ .

That fixes the selftests/mm build on today's Arch Linux (which required the userfaultfd.h) and Ubuntu 23.04 (which additionally required memfd.h).

[1] https://lore.kernel.org/all/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>
---
 tools/include/uapi/linux/memfd.h       |  39 +++
 tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++
 2 files changed, 425 insertions(+)
 create mode 100644 tools/include/uapi/linux/memfd.h
 create mode 100644 tools/include/uapi/linux/userfaultfd.h
tools/include/uapi/linux/memfd.h new file +39 -0
new file mode 100644
index 000000000000..01c0324e7733
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef _LINUX_MEMFD_H
+#define _LINUX_MEMFD_H
+
+#include <asm-generic/hugetlb_encode.h>
+
+/* flags for memfd_create(2) (unsigned int) */
+#define MFD_CLOEXEC		0x0001U
+#define MFD_ALLOW_SEALING	0x0002U
+#define MFD_HUGETLB		0x0004U
+/* not executable and sealed to prevent changing to executable. */
+#define MFD_NOEXEC_SEAL		0x0008U
+/* executable */
+#define MFD_EXEC		0x0010U
+
+/*
+ * Huge page size encoding when MFD_HUGETLB is specified, and a huge page
+ * size other than the default is desired.  See hugetlb_encode.h.
+ * All known huge page size encodings are provided here.  It is the
+ * responsibility of the application to know which sizes are supported on
+ * the running system.  See mmap(2) man page for details.
+ */
+#define MFD_HUGE_SHIFT	HUGETLB_FLAG_ENCODE_SHIFT
+#define MFD_HUGE_MASK	HUGETLB_FLAG_ENCODE_MASK
+
+#define MFD_HUGE_64KB	HUGETLB_FLAG_ENCODE_64KB
+#define MFD_HUGE_512KB	HUGETLB_FLAG_ENCODE_512KB
+#define MFD_HUGE_1MB	HUGETLB_FLAG_ENCODE_1MB
+#define MFD_HUGE_2MB	HUGETLB_FLAG_ENCODE_2MB
+#define MFD_HUGE_8MB	HUGETLB_FLAG_ENCODE_8MB
+#define MFD_HUGE_16MB	HUGETLB_FLAG_ENCODE_16MB
+#define MFD_HUGE_32MB	HUGETLB_FLAG_ENCODE_32MB
+#define MFD_HUGE_256MB	HUGETLB_FLAG_ENCODE_256MB
+#define MFD_HUGE_512MB	HUGETLB_FLAG_ENCODE_512MB
+#define MFD_HUGE_1GB	HUGETLB_FLAG_ENCODE_1GB
+#define MFD_HUGE_2GB	HUGETLB_FLAG_ENCODE_2GB
+#define MFD_HUGE_16GB	HUGETLB_FLAG_ENCODE_16GB
+
+#endif /* _LINUX_MEMFD_H */
tools/include/uapi/linux/userfaultfd.h new file +386 -0

386 lines not on this page · raw message

Signature · 1 line
2.44.0
Acked-by on v1 2/2
#4 lore raw

Re: [PATCH 2/2] selftests/mm: fix additional build errors for selftests

Quoted 14 lines · John Hubbard

These build errors only occur if one fails to first run "make headers". However, that is a non-obvious and instrusive requirement, and so there was a discussion on how to get rid of it [1]. This uses that solution.

These two files were created by taking a snapshot of the generated header files that are created via "make headers". These two files were copied from ./usr/include/linux/ to ./tools/include/uapi/linux/ .

That fixes the selftests/mm build on today's Arch Linux (which required the userfaultfd.h) and Ubuntu 23.04 (which additionally required memfd.h).

[1] https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>

---

Signature · 3 lines
Cheers,

David / dhildenb
#5 lore raw
In reply to John Hubbard
Quoted 5 lines · John Hubbard

Hi,

As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.

There is another way. The headers should be built automatically by make dependency. The topmost make file always builds headers before building kselftest i.e., make kselftest

The tools/testing/selftests/Makefile and others Makefiles in test suites should be updated to build the headers as well. This used to be the behavior until there were un-resolvable bugs in the Makefiles and it was reverted:
https://lore.kernel.org/all/[email protected]/

We should come up with the revert of this series such that all the different scenarios are covered instead of yet again a new solution; resorting to the duplication of header files or any sort of duplication.

Quoted 30 lines

This should also make Peter Zijlstra happy, because it directly addresses the root cause of his "NAK NAK NAK" reply [2]. :)

I haven't done much build testing, because selftests are not so easy to build with a cross-compiler. So it's just tested on x86 64-bit so far.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

Cc David Hildenbrand <[email protected]>
Cc Mark Brown <[email protected]>
Cc Muhammad Usama Anjum <[email protected]>
Cc Suren Baghdasaryan <[email protected]>
Cc Peter Zijlstra <[email protected]>

John Hubbard (2):
selftests: break the dependency upon local header files selftests/mm: fix additional build errors for selftests

 tools/include/uapi/linux/memfd.h       |  39 +++
 tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++
 tools/testing/selftests/lib.mk         |   9 +
 tools/testing/selftests/mm/Makefile    |   2 +-
 4 files changed, 435 insertions(+), 1 deletion(-)
 create mode 100644 tools/include/uapi/linux/memfd.h
 create mode 100644 tools/include/uapi/linux/userfaultfd.h

base-commit: 98560e9019851bf55b8a4073978a623a3bcf98c0

Signature · 2 lines
BR,
Muhammad Usama Anjum
#6 lore raw
Quoted 10 lines · Muhammad Usama Anjum
Quoted 5 lines · John Hubbard

Hi,

As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.

There is another way. The headers should be built automatically by make dependency. The topmost make file always builds headers before building kselftest i.e., make kselftest

I think we talked through this already: Peter Z. pointed out the problems with requiring this kind of prerequisite. And it really it overkill. The approach here is simple, easy to maintain, and avoids breaking the various unusual build setups that people have.

I'll shut up now and let others weigh in, though. :)

thanks,

Signature · 2 lines
John Hubbard
NVIDIA