Fix selftests/mm build without requiring "make headers"
Subsystems (3)
cover Fix selftests/mm build without requiring "make headers"
Hi,
As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.
This should also make Peter Zijlstra happy, because it directly addresses the root cause of his "NAK NAK NAK" reply [2]. :)
I haven't done much build testing, because selftests are not so easy to build with a cross-compiler. So it's just tested on x86 64-bit so far.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
John Hubbard (2):
selftests: break the dependency upon local header files selftests/mm: fix additional build errors for selftests
tools/include/uapi/linux/memfd.h | 39 +++ tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++ tools/testing/selftests/lib.mk | 9 + tools/testing/selftests/mm/Makefile | 2 +- 4 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 tools/include/uapi/linux/memfd.h create mode 100644 tools/include/uapi/linux/userfaultfd.h
base-commit: 98560e9019851bf55b8a4073978a623a3bcf98c0
Signature · 1 line
2.44.0
PATCH 1/2 selftests: break the dependency upon local header files
Use tools/include/uapi/ files instead. These are obtained by taking a
snapshot: run "make headers" at the top level, then copy the desired header file into the appropriate subdir in tools/uapi/.
This was discussed and solved in [1].
However, even before copying any additional files there, there are already quite a few in tools/include/uapi already. And these will immediately fix a number of selftests/mm build failures.
So this patch:
a) Adds TOOLS_INCLUDES to selftests/lib.mk, so that all selftests can immediately and easily include the snapshotted header files.
b) Uses $(TOOLS_INCLUDES) in the selftests/mm build. On today's Arch Linux, this already fixes all build errors except for a few userfaultfd.h (those will be addressed in a subsequent patch).
[1] https://lore.kernel.org/all/[email protected]/
--- tools/testing/selftests/lib.mk | 9 +++++++++ tools/testing/selftests/mm/Makefile | 2 +- 2 files changed, 10 insertions(+), 1 deletion(-)
tools/testing/selftests/lib.mk +9 -0
@@ -48,6 +48,15 @@ ifeq ($(KHDR_INCLUDES),) KHDR_INCLUDES := -isystem $(top_srcdir)/usr/include endif +# In order to use newer items that haven't yet been added to the user's system +# header files, add $(TOOLS_INCLUDES) to the compiler invocation in each +# each selftest. +# You may need to add files to that location, or to refresh an existing file. In +# order to do that, run "make headers" from $(top_srcdir), then copy the +# header file that you want from $(top_srcdir)/usr/include/... , to the matching +# subdir in $(TOOLS_INCLUDE). +TOOLS_INCLUDES := -isystem $(top_srcdir)/tools/include/uapi + # The following are built by lib.mk common compile rules. # TEST_CUSTOM_PROGS should be used by tests that require # custom build rule and prevent common build rule use.
tools/testing/selftests/mm/Makefile +1 -1
@@ -32,7 +32,7 @@ endif # LDLIBS. MAKEFLAGS += --no-builtin-rules -CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) +CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) $(TOOLS_INCLUDES) LDLIBS = -lrt -lpthread -lm TEST_GEN_FILES = cow
Signature · 1 line
2.44.0
Re: [PATCH 1/2] selftests: break the dependency upon local header files
Quoted 21 lines · John Hubbard
Use
tools/include/uapi/files instead. These are obtained by taking a
snapshot: run "make headers" at the top level, then copy the desired header file into the appropriate subdir intools/uapi/.This was discussed and solved in [1].
However, even before copying any additional files there, there are already quite a few in
tools/include/uapialready. And these will immediately fix a number of selftests/mm build failures.So this patch:
a) Adds
TOOLS_INCLUDEStoselftests/lib.mk, so that all selftests can immediately and easily include the snapshotted header files.b) Uses $(TOOLS_INCLUDES) in the selftests/mm build. On today's Arch Linux, this already fixes all build errors except for a few
userfaultfd.h(those will be addressed in a subsequent patch).
Quoted 8 lines
Cc David Hildenbrand <[email protected]>Cc Mark Brown <[email protected]>Cc Muhammad Usama Anjum <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by John Hubbard <[email protected]>---
Signature · 3 lines
Cheers, David / dhildenb
PATCH 2/2 selftests/mm: fix additional build errors for selftests
In reply to John HubbardThese build errors only occur if one fails to first run "make headers". However, that is a non-obvious and instrusive requirement, and so there was a discussion on how to get rid of it [1]. This uses that solution.
These two files were created by taking a snapshot of the generated header files that are created via "make headers". These two files were copied from ./usr/include/linux/ to ./tools/include/uapi/linux/ .
That fixes the selftests/mm build on today's Arch Linux (which required the userfaultfd.h) and Ubuntu 23.04 (which additionally required memfd.h).
[1] https://lore.kernel.org/all/[email protected]/
--- tools/include/uapi/linux/memfd.h | 39 +++ tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++ 2 files changed, 425 insertions(+) create mode 100644 tools/include/uapi/linux/memfd.h create mode 100644 tools/include/uapi/linux/userfaultfd.h
tools/include/uapi/linux/memfd.h new file +39 -0
@@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _LINUX_MEMFD_H +#define _LINUX_MEMFD_H + +#include <asm-generic/hugetlb_encode.h> + +/* flags for memfd_create(2) (unsigned int) */ +#define MFD_CLOEXEC 0x0001U +#define MFD_ALLOW_SEALING 0x0002U +#define MFD_HUGETLB 0x0004U +/* not executable and sealed to prevent changing to executable. */ +#define MFD_NOEXEC_SEAL 0x0008U +/* executable */ +#define MFD_EXEC 0x0010U + +/* +* Huge page size encoding when MFD_HUGETLB is specified, and a huge page +* size other than the default is desired. See hugetlb_encode.h. +* All known huge page size encodings are provided here. It is the +* responsibility of the application to know which sizes are supported on +* the running system. See mmap(2) man page for details. +*/ +#define MFD_HUGE_SHIFT HUGETLB_FLAG_ENCODE_SHIFT +#define MFD_HUGE_MASK HUGETLB_FLAG_ENCODE_MASK + +#define MFD_HUGE_64KB HUGETLB_FLAG_ENCODE_64KB +#define MFD_HUGE_512KB HUGETLB_FLAG_ENCODE_512KB +#define MFD_HUGE_1MB HUGETLB_FLAG_ENCODE_1MB +#define MFD_HUGE_2MB HUGETLB_FLAG_ENCODE_2MB +#define MFD_HUGE_8MB HUGETLB_FLAG_ENCODE_8MB +#define MFD_HUGE_16MB HUGETLB_FLAG_ENCODE_16MB +#define MFD_HUGE_32MB HUGETLB_FLAG_ENCODE_32MB +#define MFD_HUGE_256MB HUGETLB_FLAG_ENCODE_256MB +#define MFD_HUGE_512MB HUGETLB_FLAG_ENCODE_512MB +#define MFD_HUGE_1GB HUGETLB_FLAG_ENCODE_1GB +#define MFD_HUGE_2GB HUGETLB_FLAG_ENCODE_2GB +#define MFD_HUGE_16GB HUGETLB_FLAG_ENCODE_16GB + +#endif /* _LINUX_MEMFD_H */
tools/include/uapi/linux/userfaultfd.h new file +386 -0
386 lines not on this page · raw message
Signature · 1 line
2.44.0
Re: [PATCH 2/2] selftests/mm: fix additional build errors for selftests
Quoted 14 lines · John Hubbard
These build errors only occur if one fails to first run "make headers". However, that is a non-obvious and instrusive requirement, and so there was a discussion on how to get rid of it [1]. This uses that solution.
These two files were created by taking a snapshot of the generated header files that are created via "make headers". These two files were copied from
./usr/include/linux/to./tools/include/uapi/linux/.That fixes the selftests/mm build on today's Arch Linux (which required the
userfaultfd.h) and Ubuntu 23.04 (which additionally requiredmemfd.h).
Quoted 7 lines
Cc David Hildenbrand <[email protected]>Cc Mark Brown <[email protected]>Cc Muhammad Usama Anjum <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Peter Zijlstra <[email protected]>Signed-off-by John Hubbard <[email protected]>---
Signature · 3 lines
Cheers, David / dhildenb
Quoted 5 lines · John Hubbard
Hi,
As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.
There is another way. The headers should be built automatically by make dependency. The topmost make file always builds headers before building kselftest i.e., make kselftest
The tools/testing/selftests/Makefile and others Makefiles in test suites should be updated to build the headers as well. This used to be the behavior until there were un-resolvable bugs in the Makefiles and it was reverted:
https://lore.kernel.org/all/[email protected]/
We should come up with the revert of this series such that all the different scenarios are covered instead of yet again a new solution; resorting to the duplication of header files or any sort of duplication.
Quoted 30 lines
This should also make Peter Zijlstra happy, because it directly addresses the root cause of his "NAK NAK NAK" reply [2]. :)
I haven't done much build testing, because selftests are not so easy to build with a cross-compiler. So it's just tested on x86 64-bit so far.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/Cc David Hildenbrand <[email protected]>Cc Mark Brown <[email protected]>Cc Muhammad Usama Anjum <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Peter Zijlstra <[email protected]>John Hubbard (2):
selftests: break the dependency upon local header files selftests/mm: fix additional build errors for selfteststools/include/uapi/linux/memfd.h | 39 +++ tools/include/uapi/linux/userfaultfd.h | 386 +++++++++++++++++++++++++ tools/testing/selftests/lib.mk | 9 + tools/testing/selftests/mm/Makefile | 2 +- 4 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 tools/include/uapi/linux/memfd.h create mode 100644 tools/include/uapi/linux/userfaultfd.hbase-commit:
98560e9019851bf55b8a4073978a623a3bcf98c0
Signature · 2 lines
BR, Muhammad Usama Anjum
Quoted 10 lines · Muhammad Usama Anjum
Quoted 5 lines · John Hubbard
Hi,
As mentioned in each patch, this implements the solution that we discussed in December 2023, in [1]. This turned out to be very clean and easy. It should also be quite easy to maintain.
There is another way. The headers should be built automatically by make dependency. The topmost make file always builds headers before building kselftest i.e., make kselftest
I think we talked through this already: Peter Z. pointed out the problems with requiring this kind of prerequisite. And it really it overkill. The approach here is simple, easy to maintain, and avoids breaking the various unusual build setups that people have.
I'll shut up now and let others weigh in, though. :)
thanks,
Signature · 2 lines
John Hubbard NVIDIA