mm/highmem: Preemptible variant of kmap_atomic & friends
Subsystems (11)
v4 cover mm/highmem: Preemptible variant of kmap_atomic & friends
This is the outstanding set of patches for the preemptible kmap_*() implementation. The previous version is here:
https://lore.kernel.org/r/[email protected]
The following changes come with it:
- Better debugging for kmap* by enforcing the temporary mappings even for lowmem pages when the architecture allows it.
- Support for enforced mappings on X86 for both 32 and 64 bit. As KMAP_LOCAL is already independent from HIGHMEM this needs only the extra fixmap space on 64bit and no further code changes.
- Removing the RT dependency from migrate_disable/enable(). This patch still needs some help from the scheduler folks vs. the comment in preempt.h.
- Switching from per CPU storage of the kmap index to a per task storage by adding a pteval array to the per task storage which contains the ptevals of the currently active temporary kmaps
- Adding context switch code which checks whether the outgoing or the incoming task has active temporary kmaps. If so, the outgoing task's kmaps are removed and the incoming task's kmaps are restored.
- Adding new interfaces k[un]map_local*() which are not disabling preemption and can be called from any context (except NMI).
Contrary to kmap() which provides preemptible and "persistant" mappings, these interfaces are meant to replace the temporary mappings provided by kmap_atomic*() today.
This allows to get rid of conditional mapping choices and allows to have preemptible short term "mappings" on 64bit which are today enforced to be non-preemptible due to the highmem constraints. It clearly puts overhead on the highmem users, but highmem is slow anyway.
This is not a wholesale conversion which makes kmap_atomic magically preemptible because there might be usage sites which rely on the implicit preempt disable. So this needs to be done on a case by case basis and the call sites converted to kmap_local().
Changes vs. V3:
- Rebased on top of the merged consolidation patches and on top of the latest version of migrate disable support.
- New debug features (enforced mappings)
- Dropped the conversion patches as they need review from the respective subsystems and better go through the relevant trees for 5.12
The series is based on
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/mm
with
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/core
merged in.
It is also available from git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git highmem
Thanks,
tglx
--- Documentation/driver-api/io-mapping.rst | 74 +++++++++++-------- arch/x86/Kconfig | 1 arch/x86/include/asm/fixmap.h | 12 ++- arch/x86/include/asm/pgtable_64_types.h | 6 + arch/x86/kernel/crash_dump_32.c | 48 ++---------- include/asm-generic/kmap_size.h | 2 include/linux/highmem-internal.h | 58 +++++++++++++++ include/linux/highmem.h | 43 +++++++---- include/linux/io-mapping.h | 30 +++++++ include/linux/kernel.h | 21 +++-- include/linux/preempt.h | 38 ---------- include/linux/sched.h | 11 ++ kernel/entry/common.c | 2 kernel/fork.c | 1 kernel/sched/core.c | 63 ++++++++++++++-- kernel/sched/sched.h | 4 - lib/Kconfig.debug | 22 +++++ lib/smp_processor_id.c | 2 mm/highmem.c | 121 ++++++++++++++++++++++++++++---- 19 files changed, 400 insertions(+), 159 deletions(-)
PATCH v4 1/8 mm/highmem: Provide and use CONFIG_DEBUG_KMAP_LOCAL
CONFIG_KMAP_LOCAL can be enabled by x86/32bit even if CONFIG_HIGHMEM is not enabled for temporary MMIO space mappings.
Provide it as a seperate config option which depends on CONFIG_KMAP_LOCAL and let CONFIG_DEBUG_HIGHMEM select it.
This won't increase the debug coverage of this significantly but it paves the way to do so.
---
V4: New patch
--- include/asm-generic/kmap_size.h | 2 +- lib/Kconfig.debug | 8 ++++++++ mm/highmem.c | 4 ++-- 3 files changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/kmap_size.h +1 -1
@@ -3,7 +3,7 @@#define _ASM_GENERIC_KMAP_SIZE_H/* For debug this provides guard pages between the maps */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_MAX_IDX 33#else # define KM_MAX_IDX 16
lib/Kconfig.debug +8 -0
@@ -849,9 +849,17 @@ config DEBUG_PER_CPU_MAPS Say N if unsure. +config DEBUG_KMAP_LOCAL + bool "Debug kmap_local temporary mappings" + depends on DEBUG_KERNEL && KMAP_LOCAL + help + This option enables additional error checking for the kmap_local + infrastructure. Disable for production use. + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory systems. Disable for production systems.
mm/highmem.c +2 -2
@@ -368,10 +368,10 @@ EXPORT_SYMBOL(kunmap_high); static DEFINE_PER_CPU(int, __kmap_local_idx);/* -* With DEBUG_HIGHMEM the stack depth is doubled and every second +* With DEBUG_KMAP_LOCAL the stack depth is doubled and every second * slot is unused which acts as a guard page */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_INCR 2#else # define KM_INCR 1
[tip: core/mm] mm/highmem: Provide and use CONFIG_DEBUG_KMAP_LOCAL
The following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: 6e799cb69a70eedbb41561b750f7180c12cff280 Gitweb: https://git.kernel.org/tip/6e799cb69a70eedbb41561b750f7180c12cff280 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:39 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:08 +01:00
mm/highmem: Provide and use CONFIG_DEBUG_KMAP_LOCAL
CONFIG_KMAP_LOCAL can be enabled by x86/32bit even if CONFIG_HIGHMEM is not enabled for temporary MMIO space mappings.
Provide it as a seperate config option which depends on CONFIG_KMAP_LOCAL and let CONFIG_DEBUG_HIGHMEM select it.
This won't increase the debug coverage of this significantly but it paves the way to do so.
--- include/asm-generic/kmap_size.h | 2 +- lib/Kconfig.debug | 8 ++++++++ mm/highmem.c | 4 ++-- 3 files changed, 11 insertions(+), 3 deletions(-)
include/asm-generic/kmap_size.h +1 -1
@@ -3,7 +3,7 @@#define _ASM_GENERIC_KMAP_SIZE_H/* For debug this provides guard pages between the maps */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_MAX_IDX 33#else # define KM_MAX_IDX 16
lib/Kconfig.debug +8 -0
@@ -849,9 +849,17 @@ config DEBUG_PER_CPU_MAPS Say N if unsure. +config DEBUG_KMAP_LOCAL + bool "Debug kmap_local temporary mappings" + depends on DEBUG_KERNEL && KMAP_LOCAL + help + This option enables additional error checking for the kmap_local + infrastructure. Disable for production use. + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory systems. Disable for production systems.
mm/highmem.c +2 -2
@@ -368,10 +368,10 @@ EXPORT_SYMBOL(kunmap_high); static DEFINE_PER_CPU(int, __kmap_local_idx);/* -* With DEBUG_HIGHMEM the stack depth is doubled and every second +* With DEBUG_KMAP_LOCAL the stack depth is doubled and every second * slot is unused which acts as a guard page */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_INCR 2#else # define KM_INCR 1
PATCH v4 2/8 mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
In reply to Thomas GleixnerCONFIG_DEBUG_KMAP_LOCAL, which is selected by CONFIG_DEBUG_HIGHMEM is only providing guard pages, but does not provide a mechanism to enforce the usage of the kmap_local() infrastructure.
Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP which forces the temporary mapping even for lowmem pages. This needs to be a seperate config switch because this only works on architectures which do not have cache aliasing problems.
---
V4: New patch
--- lib/Kconfig.debug | 14 ++++++++++++++ mm/highmem.c | 12 +++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
lib/Kconfig.debug +14 -0
@@ -856,9 +856,23 @@ config DEBUG_KMAP_LOCAL This option enables additional error checking for the kmap_local infrastructure. Disable for production use. +config ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + bool + +config DEBUG_KMAP_LOCAL_FORCE_MAP + bool "Enforce kmap_local temporary mappings" + depends on DEBUG_KERNEL && ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + select KMAP_LOCAL + select DEBUG_KMAP_LOCAL + help + This option enforces temporary mappings through the kmap_local + mechanism for non-highmem pages and on non-highmem systems. + Disable this for production systems! + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL_FORCE_MAP if ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory
mm/highmem.c +11 -1
@@ -474,7 +474,12 @@ void *__kmap_local_page_prot(struct page {void *kmap; -if (!PageHighMem (page)) +/* +* To broaden the usage of the actual kmap_local() machinery always map +* pages when debugging is enabled and the architecture has no problems +* with alias mappings. +*/ +if (!IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP ) && !PageHighMem (page))return page_address (page);/* Try kmap_high_get() if architecture has it enabled */
@@ -494,6 +499,11 @@ void kunmap_local_indexed(void *vaddr)if (addr <__fix_to_virt (FIX_KMAP_END ) || addr >__fix_to_virt (FIX_KMAP_BEGIN )) { +if (IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP )) { +/* This _should_ never happen! See above. */ +WARN_ON_ONCE (1 ); +return ; + }/* * Handle mappings which were obtained by kmap_high_get() * first as the virtual address of such mappings is below
Re: [patch V4 2/8] mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
Quoted 5 lines · Thomas Gleixner
Provide
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAPwhich forces the temporary mapping even for lowmem pages. This needs to be a seperate config switch because this only works on architectures which do not have cache aliasing problems.
Very good. And you made sure to have a comment to not enable it for production systems.
Hopefully people will even read it ;)
Linus
Mel Gorman
reviewerRe: [patch V4 2/8] mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Thomas Gleixner
Provide
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAPwhich forces the temporary mapping even for lowmem pages. This needs to be a seperate config switch because this only works on architectures which do not have cache aliasing problems.
Quoted 5 lines
Very good. And you made sure to have a comment to not enable it for production systems.
Hopefully people will even read it ;)
And not start thinking it as a security hardening option.
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [patch V4 2/8] mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
Quoted 2 lines · Mel Gorman
And not start thinking it as a security hardening option.
It's probably the reverse of a hardening option, since it will cause mapping of stuff in known and controllable virtual addresses.
Although any kmap'able page is likely to already be something you can read anyway (ie page cache long after security checks have been done). So it probably really doesn't matter either way.
The only thing that is certain is that it's going to slow down important code-paths.
Linus
[tip: core/mm] mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
In reply to Thomas GleixnerThe following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: 0e91a0c6984c837a7c6760e3f28e8e1c532abf87 Gitweb: https://git.kernel.org/tip/0e91a0c6984c837a7c6760e3f28e8e1c532abf87 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:40 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:08 +01:00
mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
CONFIG_DEBUG_KMAP_LOCAL, which is selected by CONFIG_DEBUG_HIGHMEM is only providing guard pages, but does not provide a mechanism to enforce the usage of the kmap_local() infrastructure.
Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP which forces the temporary mapping even for lowmem pages. This needs to be a seperate config switch because this only works on architectures which do not have cache aliasing problems.
--- lib/Kconfig.debug | 14 ++++++++++++++ mm/highmem.c | 12 +++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-)
lib/Kconfig.debug +14 -0
@@ -856,9 +856,23 @@ config DEBUG_KMAP_LOCAL This option enables additional error checking for the kmap_local infrastructure. Disable for production use. +config ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + bool + +config DEBUG_KMAP_LOCAL_FORCE_MAP + bool "Enforce kmap_local temporary mappings" + depends on DEBUG_KERNEL && ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + select KMAP_LOCAL + select DEBUG_KMAP_LOCAL + help + This option enforces temporary mappings through the kmap_local + mechanism for non-highmem pages and on non-highmem systems. + Disable this for production systems! + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL_FORCE_MAP if ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory
mm/highmem.c +11 -1
@@ -474,7 +474,12 @@ void *__kmap_local_page_prot(struct page *page, pgprot_t prot) {void *kmap; -if (!PageHighMem (page)) +/* +* To broaden the usage of the actual kmap_local() machinery always map +* pages when debugging is enabled and the architecture has no problems +* with alias mappings. +*/ +if (!IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP ) && !PageHighMem (page))return page_address (page);/* Try kmap_high_get() if architecture has it enabled */
@@ -494,6 +499,11 @@ void kunmap_local_indexed(void *vaddr)if (addr <__fix_to_virt (FIX_KMAP_END ) || addr >__fix_to_virt (FIX_KMAP_BEGIN )) { +if (IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP )) { +/* This _should_ never happen! See above. */ +WARN_ON_ONCE (1 ); +return ; + }/* * Handle mappings which were obtained by kmap_high_get() * first as the virtual address of such mappings is below
PATCH v4 3/8 x86: Support kmap_local() forced debugging
In reply to Thomas Gleixnerkmap_local() and related interfaces are NOOPs on 64bit and only create temporary fixmaps for highmem pages on 32bit. That means the test coverage for this code is pretty small.
CONFIG_KMAP_LOCAL can be enabled independent from CONFIG_HIGHMEM, which allows to provide support for enforced kmap_local() debugging even on 64bit.
For 32bit the support is unconditional, for 64bit it's only supported when CONFIG_NR_CPUS <= 4096 as supporting it for 8192 CPUs would require to set up yet another fixmap PGT.
If CONFIG_KMAP_LOCAL_FORCE_DEBUG is enabled then kmap_local()/kmap_atomic() will use the temporary fixmap mapping path.
---
V4: New patch
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/fixmap.h | 12 +++++++++--- arch/x86/include/asm/pgtable_64_types.h | 6 +++++- 3 files changed, 15 insertions(+), 4 deletions(-)
arch/x86/Kconfig +1 -0
@@ -93,6 +93,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 + select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS
arch/x86/include/asm/fixmap.h +9 -3
@@ -14,13 +14,20 @@#ifndef _ASM_X86_FIXMAP_H#define _ASM_X86_FIXMAP_H +#include <asm/kmap_size.h> +/* * Exposed to assembly code for setting up initial page tables. Cannot be * calculated in assembly code (fixmap entries are an enum), but is sanity * checked in the actual fixmap C code to make sure that the fixmap is * covered fully. */ -#define FIXMAP_PMD_NUM 2 +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define FIXMAP_PMD_NUM 2 +#else +# define KM_PMDS (KM_MAX_IDX * ((CONFIG_NR_CPUS + 511) / 512)) +# define FIXMAP_PMD_NUM (KM_PMDS + 2) +#endif /* fixmap starts downwards from the 507th entry in level2_fixmap_pgt */ #define FIXMAP_PMD_TOP 507
@@ -31,7 +38,6 @@#include <asm/pgtable_types.h> #ifdef CONFIG_X86_32 #include <linux/threads.h> -#include <asm/kmap_size.h> #else #include <uapi/asm/vsyscall.h> #endif
@@ -92,7 +98,7 @@ enum fixed_addresses { FIX_IO_APIC_BASE_0, FIX_IO_APIC_BASE_END = FIX_IO_APIC_BASE_0 + MAX_IO_APICS - 1,#endif -#ifdef CONFIG_X86_32 +#ifdef CONFIG_KMAP_LOCAL FIX_KMAP_BEGIN ,/* reserved pte's for temporary kernel mappings */ FIX_KMAP_END =FIX_KMAP_BEGIN + (KM_MAX_IDX *NR_CPUS ) -1 , #ifdef CONFIG_PCI_MMCONFIG
arch/x86/include/asm/pgtable_64_types.h +5 -1
@@ -143,7 +143,11 @@ extern unsigned int ptrs_per_p4d;#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)/* The module sections ends with the start of the fixmap */ -#define MODULES_END _AC(0xffffffffff000000, UL) +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define MODULES_END _AC(0xffffffffff000000, UL) +#else +# define MODULES_END _AC(0xfffffffffe000000, UL) +#endif #define MODULES_LEN (MODULES_END - MODULES_VADDR)#define ESPFIX_PGD_ENTRY _AC(-2, UL)
[tip: core/mm] x86: Support kmap_local() forced debugging
The following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: 14df32670291588036a498051a54cd8462d7f611 Gitweb: https://git.kernel.org/tip/14df32670291588036a498051a54cd8462d7f611 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:41 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:09 +01:00
x86: Support kmap_local() forced debugging
kmap_local() and related interfaces are NOOPs on 64bit and only create temporary fixmaps for highmem pages on 32bit. That means the test coverage for this code is pretty small.
CONFIG_KMAP_LOCAL can be enabled independent from CONFIG_HIGHMEM, which allows to provide support for enforced kmap_local() debugging even on 64bit.
For 32bit the support is unconditional, for 64bit it's only supported when CONFIG_NR_CPUS <= 4096 as supporting it for 8192 CPUs would require to set up yet another fixmap PGT.
If CONFIG_KMAP_LOCAL_FORCE_DEBUG is enabled then kmap_local()/kmap_atomic() will use the temporary fixmap mapping path.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/fixmap.h | 12 +++++++++--- arch/x86/include/asm/pgtable_64_types.h | 6 +++++- 3 files changed, 15 insertions(+), 4 deletions(-)
arch/x86/Kconfig +1 -0
@@ -93,6 +93,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 + select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS
arch/x86/include/asm/fixmap.h +9 -3
@@ -14,13 +14,20 @@#ifndef _ASM_X86_FIXMAP_H#define _ASM_X86_FIXMAP_H +#include <asm/kmap_size.h> +/* * Exposed to assembly code for setting up initial page tables. Cannot be * calculated in assembly code (fixmap entries are an enum), but is sanity * checked in the actual fixmap C code to make sure that the fixmap is * covered fully. */ -#define FIXMAP_PMD_NUM 2 +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define FIXMAP_PMD_NUM 2 +#else +# define KM_PMDS (KM_MAX_IDX * ((CONFIG_NR_CPUS + 511) / 512)) +# define FIXMAP_PMD_NUM (KM_PMDS + 2) +#endif /* fixmap starts downwards from the 507th entry in level2_fixmap_pgt */ #define FIXMAP_PMD_TOP 507
@@ -31,7 +38,6 @@#include <asm/pgtable_types.h> #ifdef CONFIG_X86_32 #include <linux/threads.h> -#include <asm/kmap_size.h> #else #include <uapi/asm/vsyscall.h> #endif
@@ -92,7 +98,7 @@ enum fixed_addresses { FIX_IO_APIC_BASE_0, FIX_IO_APIC_BASE_END = FIX_IO_APIC_BASE_0 + MAX_IO_APICS - 1,#endif -#ifdef CONFIG_X86_32 +#ifdef CONFIG_KMAP_LOCAL FIX_KMAP_BEGIN ,/* reserved pte's for temporary kernel mappings */ FIX_KMAP_END =FIX_KMAP_BEGIN + (KM_MAX_IDX *NR_CPUS ) -1 , #ifdef CONFIG_PCI_MMCONFIG
arch/x86/include/asm/pgtable_64_types.h +5 -1
@@ -143,7 +143,11 @@ extern unsigned int ptrs_per_p4d;#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)/* The module sections ends with the start of the fixmap */ -#define MODULES_END _AC(0xffffffffff000000, UL) +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define MODULES_END _AC(0xffffffffff000000, UL) +#else +# define MODULES_END _AC(0xfffffffffe000000, UL) +#endif #define MODULES_LEN (MODULES_END - MODULES_VADDR)#define ESPFIX_PGD_ENTRY _AC(-2, UL)
Steven Rostedt
reviewer[BUG] from x86: Support kmap_local() forced debugging
In reply to Thomas GleixnerQuoted 27 lines · On Wed, 18 Nov 2020 20:48:41 +0100 Thomas Gleixner <[email protected]> wrote:
kmap_local()and related interfaces are NOOPs on 64bit and only create temporary fixmaps for highmem pages on 32bit. That means the test coverage for this code is pretty small.
CONFIG_KMAP_LOCALcan be enabled independent fromCONFIG_HIGHMEM, which allows to provide support for enforcedkmap_local()debugging even on 64bit.For 32bit the support is unconditional, for 64bit it's only supported when
CONFIG_NR_CPUS<= 4096 as supporting it for 8192 CPUs would require to set up yet another fixmap PGT.If
CONFIG_KMAP_LOCAL_FORCE_DEBUGis enabled then kmap_local()/kmap_atomic() will use the temporary fixmap mapping path.Signed-off-by Thomas Gleixner <[email protected]>---
V4: New patch--- arch/x86/Kconfig | 1 + arch/x86/include/asm/fixmap.h | 12 +++++++++--- arch/x86/include/asm/pgtable_64_types.h | 6 +++++- 3 files changed, 15 insertions(+), 4 deletions(-)
arch/x86/Kconfig config X86 · line 95
Quoted 3 lines
arch/x86/Kconfig +0
-0@@ -93,6 +93,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING if X86_64
I triggered the following crash on x86_32 by simply doing a:
(ssh'ing into the box)
# head -100 /tmp/output-file
Where the /tmp/output-file was the output of a trace-cmd report. Even after rebooting and not running the tracing code, simply doing the head command still crashed.
BUG: unable to handle page fault for address: fff58000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page *pdpt = 0000000006de9001 *pde = 0000000001968063 *pte = 0000000000000000 Oops: 0000 [#1] SMP PTI CPU: 3 PID: 3935 Comm: sshd Not tainted 5.11.0-rc2-test+ #2 Hardware name: MSI MS-7823/CSM-H87M-G43 (MS-7823), BIOS V1.6 02/22/2014 EIP: skb_copy_bits+0x10c/0x1b9 Code: 3b 5d e8 0f 47 5d e8 c7 45 e0 00 00 00 00 8b 7d e0 39 7d e8 76 3a 8b 45 d4 e8 a4 e4 ff ff 8b 55 e4 03 55 e0 89 d9 01 c6 89 d7 <f3> a4 e8 c9 e4 ff ff 01 5d e0 8b 5d e8 b8 00 10 00 00 2b 5d e0 83 EAX: fff57000 EBX: 000005a8 ECX: 000000f8 EDX: c77b9900 ESI: fff58000 EDI: c77b9db0 EBP: c6de39ec ESP: c6de39c0 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00210286 CR0: 80050033 CR2: fff58000 CR3: 06de6000 CR4: 001506f0 Call Trace: skb_segment+0x4a3/0x828 ? __tcp_mtu_to_mss+0x2d/0x6b tcp_gso_segment+0xf6/0x336 ? list_add+0x26/0x26 tcp4_gso_segment+0x77/0x7c ? tcp_gso_segment+0x336/0x336 inet_gso_segment+0x1a1/0x2df ? inet_unregister_protosw+0x5e/0x5e skb_mac_gso_segment+0xb9/0x107 __skb_gso_segment+0xdf/0x10f ? netif_skb_features+0x1ca/0x24a ? __qdisc_run+0x1e4/0x418 validate_xmit_skb.constprop.0+0x10f/0x1ad validate_xmit_skb_list+0x25/0x45 sch_direct_xmit+0x5c/0x19d __qdisc_run+0x3e3/0x418 ? qdisc_run_begin+0x53/0x5d qdisc_run+0x26/0x30 __dev_queue_xmit+0x2bd/0x524 ? mark_held_locks+0x40/0x51 dev_queue_xmit+0xf/0x11 ip_finish_output2+0x378/0x3d7 __ip_finish_output+0xd6/0xe2 ip_output+0x8c/0xbb ? ip_mc_output+0x18d/0x18d dst_output+0x27/0x2d ip_local_out+0x2b/0x30 __ip_queue_xmit+0x32e/0x38e ? __copy_skb_header+0x4b/0x98 ? __ip_queue_xmit+0x38e/0x38e ip_queue_xmit+0x16/0x1b __tcp_transmit_skb+0x731/0x794 tcp_transmit_skb+0x16/0x18 tcp_write_xmit+0x7b4/0xa90 __tcp_push_pending_frames+0x2c/0x6b tcp_push+0x8c/0xf1 tcp_sendmsg_locked+0x74a/0x7f2 ? tcp_sendmsg_locked+0x7f2/0x7f2 tcp_sendmsg+0x27/0x38 ? tcp_sendmsg_locked+0x7f2/0x7f2 inet_sendmsg+0x3c/0x5f ? inet_send_prepare+0x3b/0x3b sock_sendmsg_nosec+0x1a/0x2d sock_sendmsg+0x25/0x29 sock_write_iter+0x84/0xa7 vfs_write+0xf5/0x19b ksys_write+0x68/0xaa __ia32_sys_write+0x15/0x17 __do_fast_syscall_32+0x66/0x76 do_fast_syscall_32+0x29/0x5b do_SYSENTER_32+0x15/0x17 entry_SYSENTER_32+0x9f/0xf2 EIP: 0xb7ee3545 Code: c4 01 10 03 03 74 c0 01 10 05 03 74 b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 EAX: ffffffda EBX: 00000003 ECX: 01d12448 EDX: 00002028 ESI: 00002028 EDI: 01d12448 EBP: bff4e388 ESP: bff4e328 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00200246 ? asm_exc_nmi+0xc5/0x2ab Modules linked in: ppdev parport_pc parport CR2: 00000000fff58000 ---[ end trace 3d4582614c9c2a0e ]--- EIP: skb_copy_bits+0x10c/0x1b9 Code: 3b 5d e8 0f 47 5d e8 c7 45 e0 00 00 00 00 8b 7d e0 39 7d e8 76 3a 8b 45 d4 e8 a4 e4 ff ff 8b 55 e4 03 55 e0 89 d9 01 c6 89 d7 <f3> a4 e8 c9 e4 ff ff 01 5d e0 8b 5d e8 b8 00 10 00 00 2b 5d e0 83 EAX: fff57000 EBX: 000005a8 ECX: 000000f8 EDX: c77b9900 ESI: fff58000 EDI: c77b9db0 EBP: c6de39ec ESP: c6de39c0 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00210286 CR0: 80050033 CR2: fff58000 CR3: 06de6000 CR4: 001506f0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
This was against 5.11-rc2.
I bisected it down to the commit that added this patch.
arch/x86/Kconfig config X86 · line 96
Quoted 1 line
arch/x86/Kconfig +1
-0+ select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096
If I remove the above line, it works fine.
Attached is the config file.
-- Steve
Quoted 61 lines
arch/x86/Kconfig +0
-0select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKSarch/x86/include/asm/fixmap.h +9
-3@@ -14,13 +14,20 @@#ifndef _ASM_X86_FIXMAP_H#define _ASM_X86_FIXMAP_H +#include <asm/kmap_size.h> +/* * Exposed to assembly code for setting up initial page tables. Cannot be * calculated in assembly code (fixmap entries are an enum), but is sanity * checked in the actual fixmap C code to make sure that the fixmap is * covered fully. */ -#define FIXMAP_PMD_NUM 2 +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define FIXMAP_PMD_NUM 2 +#else +# define KM_PMDS (KM_MAX_IDX * ((CONFIG_NR_CPUS + 511) / 512)) +# define FIXMAP_PMD_NUM (KM_PMDS + 2) +#endif /* fixmap starts downwards from the 507th entry in level2_fixmap_pgt */ #define FIXMAP_PMD_TOP 507@@ -31,7 +38,6 @@#include <asm/pgtable_types.h> #ifdef CONFIG_X86_32 #include <linux/threads.h> -#include <asm/kmap_size.h> #else #include <uapi/asm/vsyscall.h> #endif @@ -92,7 +98,7 @@ enum fixed_addresses { FIX_IO_APIC_BASE_0, FIX_IO_APIC_BASE_END = FIX_IO_APIC_BASE_0 + MAX_IO_APICS - 1,#endif -#ifdef CONFIG_X86_32 +#ifdef CONFIG_KMAP_LOCAL FIX_KMAP_BEGIN ,/* reserved pte's for temporary kernel mappings */ FIX_KMAP_END =FIX_KMAP_BEGIN + (KM_MAX_IDX *NR_CPUS ) -1 , #ifdef CONFIG_PCI_MMCONFIGarch/x86/include/asm/pgtable_64_types.h +5
-1@@ -143,7 +143,11 @@ extern unsigned int ptrs_per_p4d;#define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE)/* The module sections ends with the start of the fixmap */ -#define MODULES_END _AC(0xffffffffff000000, UL) +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define MODULES_END _AC(0xffffffffff000000, UL) +#else +# define MODULES_END _AC(0xfffffffffe000000, UL) +#endif #define MODULES_LEN (MODULES_END - MODULES_VADDR)#define ESPFIX_PGD_ENTRY _AC(-2, UL)
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 10 lines · Steven Rostedt
I triggered the following crash on
x86_32by simply doing a:(ssh'ing into the box)
# head -100 /tmp/output-fileWhere the
/tmp/output-filewas the output of a trace-cmd report. Even after rebooting and not running the tracing code, simply doing the head command still crashed.
The code decodes to
0: 3b 5d e8 cmp -0x18(%ebp),%ebx 3: 0f 47 5d e8 cmova -0x18(%ebp),%ebx 7: c7 45 e0 00 00 00 00 movl $0x0,-0x20(%ebp) e: 8b 7d e0 mov -0x20(%ebp),%edi 11: 39 7d e8 cmp %edi,-0x18(%ebp) 14: 76 3a jbe 0x50 16: 8b 45 d4 mov -0x2c(%ebp),%eax 19: e8 a4 e4 ff ff call 0xffffe4c2 1e: 8b 55 e4 mov -0x1c(%ebp),%edx 21: 03 55 e0 add -0x20(%ebp),%edx 24: 89 d9 mov %ebx,%ecx 26: 01 c6 add %eax,%esi 28: 89 d7 mov %edx,%edi 2a:* f3 a4 rep movsb %ds:(%esi),%es:(%edi) <-- trapping instruction 2c: e8 c9 e4 ff ff call 0xffffe4fa 31: 01 5d e0 add %ebx,-0x20(%ebp) 34: 8b 5d e8 mov -0x18(%ebp),%ebx 37: b8 00 10 00 00 mov $0x1000,%eax 3c: 2b 5d e0 sub -0x20(%ebp),%ebx
and while it would be good to see the output of scripts/decode_stacktrace.sh, I strongly suspect that the above is
vaddr = kmap_atomic(p);
memcpy(to + copied, vaddr + p_off, p_len);
kunmap_atomic(vaddr);(although I wonder how/why the heck you've enabled CC_OPTIMIZE_FOR_SIZE=y, which is what causes "memcpy()" to be done as that "rep movsb". I thought we disabled it because it's so bad on most cpus).
So that first "call" instruction is the kmap_atomic(), the "rep movs" is the memcpy(), and the "call" instruction immediately after is the kunmap_atomic().
Anyway, you can see vaddr in register state:
EAX: fff57000
so we've kmapped that one page at fff57000, but we're accessing past it into the next page:
Quoted 1 line
BUG: unable to handle page fault for address: fff58000
with the current source address being (ESI: fff58000) and we still have 248 bytes to go (ECX: 000000f8) even though we've already overflowed into the next page.
You can see the original count still (EBX: 000005a8), so it really looks like that skb_frag_foreach_page() logic
skb_frag_foreach_page(f,
skb_frag_off(f) + offset - start,
copy, p, p_off, p_len, copied) {
vaddr = kmap_atomic(p);
memcpy(to + copied, vaddr + p_off, p_len);
kunmap_atomic(vaddr);
}must be wrong, and doesn't handle the "each page" part properly. It must have started in the middle of the page, and p_len (that 0x5a8) was wrong.
IOW, it really looks like p_off + p_len had the value 0x10f8, which is larger than one page. And looking at the code, in skb_frag_foreach_page(), I see:
p_off = (f_off) & (PAGE_SIZE - 1), \
p_len = skb_frag_must_loop(p) ? \
min_t(u32, f_len, PAGE_SIZE - p_off) : f_len, \where that "min_t(u32, f_len, PAGE_SIZE - p_off)" looks correct, but then presumably skb_frag_must_loop() must be wrong.
Oh, and when I look at that, I see
static inline bool skb_frag_must_loop(struct page *p)
{
#if defined(CONFIG_HIGHMEM)
if (PageHighMem(p))
return true;
#endif
return false;
}and that is no longer true. With the kmap debugging, even non-highmem pages need that "do one page at a time" code, because even non-highmem pages get remapped by kmap().
IOW, I think the patch to fix this might be something like the attached.
I wonder whether there is other code that "knows" about kmap() only affecting PageHighmem() pages thing that is no longer true.
Looking at some other code, skb_gro_reset_offset() looks suspiciously like it also thinks highmem pages are special.
Adding the networking people involved in this area to the cc too.
Linus
Steven Rostedt
reviewerRe: [BUG] from x86: Support kmap_local() forced debugging
Quoted 4 lines · On Wed, 6 Jan 2021 17:03:48 -0800 Linus Torvalds…
(although I wonder how/why the heck you've enabled CC_OPTIMIZE_FOR_SIZE=y, which is what causes "
memcpy()" to be done as that "rep movsb". I thought we disabled it because it's so bad on most cpus).
Why?
Because to test x86_32, I have a Fedora Core 13 (yes 13!) partition (baremetal) that I use. And the .config I use for it hasn't changed since that time ;-) (except to add new features that I want to test on x86_32).
Anyway, I'll test out your patch. Thanks for investigating this.
-- Steve
Jakub Kicinski
maintainerRe: [BUG] from x86: Support kmap_local() forced debugging
In reply to Linus TorvaldsQuoted 7 lines · On Wed, 6 Jan 2021 17:03:48 -0800 Linus Torvalds wrote:
I wonder whether there is other code that "knows" about
kmap()only affectingPageHighmem()pages thing that is no longer true.Looking at some other code,
skb_gro_reset_offset()looks suspiciously like it also thinks highmem pages are special.Adding the networking people involved in this area to the cc too.
Thanks for the detailed analysis! skb_gro_reset_offset() checks if kernel can read data in the fragments directly as an optimization, in case the entire header is in a fragment.
IIUC DEBUG_KMAP_LOCAL_FORCE_MAP only affects the mappings from explicit kmap calls, which GRO won't make - it will fall back to pulling the header out of the fragment and end up in skb_copy_bits(), i.e. the loop you fixed. So GRO should be good. I think..
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 10 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 6 Jan 2021 17:03:48 -0800 Linus Torvalds wrote:
I wonder whether there is other code that "knows" about
kmap()only affectingPageHighmem()pages thing that is no longer true.Looking at some other code,
skb_gro_reset_offset()looks suspiciously like it also thinks highmem pages are special.Adding the networking people involved in this area to the cc too.
Quoted 8 lines
Thanks for the detailed analysis!
skb_gro_reset_offset()checks if kernel can read data in the fragments directly as an optimization, in case the entire header is in a fragment.IIUC
DEBUG_KMAP_LOCAL_FORCE_MAPonly affects the mappings from explicit kmap calls, which GRO won't make - it will fall back to pulling the header out of the fragment and end up inskb_copy_bits(), i.e. the loop you fixed. So GRO should be good. I think..
Agreed. That code in skb_gro_reset_offset skips the GRO frag0 optimization in various cases, including if the first fragment is in high mem.
That specific check goes back to the introduction of the frag0 optimization in commit 86911732d399 ("gro: Avoid copying headers of unmerged packets"), at the time in helper skb_gro_header().
Very glad to hear that the fix addresses the crash in skb_frag_foreach_page. Thanks!
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 11 lines · Willem de Bruijn
Quoted 9 lines · Jakub Kicinski
Quoted 7 lines · On Wed, 6 Jan 2021 17:03:48 -0800 Linus Torvalds wrote:
I wonder whether there is other code that "knows" about
kmap()only affectingPageHighmem()pages thing that is no longer true.Looking at some other code,
skb_gro_reset_offset()looks suspiciously like it also thinks highmem pages are special.Adding the networking people involved in this area to the cc too.
But there are three other kmap_atomic callers under net/ that do not loop at all, so assume non-compound pages. In esp_output_head, esp6_output_head and skb_seq_read. The first two directly use skb_page_frag_refill, which can allocate compound (but not __GFP_HIGHMEM) pages, and the third can be inserted with netfilter xt_string in the path of tcp transmit skbs, which can also have compound pages. I think that these could similarly access data beyond the end of the kmap_atomic mapped page. I'll take a closer look.
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 10 lines · Willem de Bruijn
But there are three other
kmap_atomiccallers undernet/that do not loop at all, so assume non-compound pages. Inesp_output_head,esp6_output_headandskb_seq_read. The first two directly useskb_page_frag_refill, which can allocate compound (but not__GFP_HIGHMEM) pages, and the third can be inserted with netfilterxt_stringin the path of tcp transmit skbs, which can also have compound pages. I think that these could similarly access data beyond the end of thekmap_atomicmapped page. I'll take a closer look.
Thanks.
Note that I have flushed my random one-liner patch from my system, and expect to get a proper fix through the normal networking pulls.
And _if_ the networking people feel that my one-liner was the proper fix, you can use it and add my sign-off if you want to, but it really was more of a "this is the quick ugly fix for testing" rather than anything else.
Linus
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 14 lines · On Thu, 7 Jan 2021 11:47:02 -0800 Linus Torvalds…
Quoted 10 lines · Willem de Bruijn
But there are three other
kmap_atomiccallers undernet/that do not loop at all, so assume non-compound pages. Inesp_output_head,esp6_output_headandskb_seq_read. The first two directly useskb_page_frag_refill, which can allocate compound (but not__GFP_HIGHMEM) pages, and the third can be inserted with netfilterxt_stringin the path of tcp transmit skbs, which can also have compound pages. I think that these could similarly access data beyond the end of thekmap_atomicmapped page. I'll take a closer look.Thanks.
Quoted 8 lines
Note that I have flushed my random one-liner patch from my system, and expect to get a proper fix through the normal networking pulls.
And _if_ the networking people feel that my one-liner was the proper fix, you can use it and add my sign-off if you want to, but it really was more of a "this is the quick ugly fix for testing" rather than anything else.
Please add:
And if you take Linus's patch, please add my:
and if you come up with another patch, please send it to me for testing.
Thanks!
-- Steve
Re: [BUG] from x86: Support kmap_local() forced debugging
Quoted 18 lines · Steven Rostedt
Quoted 14 lines · On Thu, 7 Jan 2021 11:47:02 -0800 Linus Torvalds…
Quoted 10 lines · Willem de Bruijn
But there are three other
kmap_atomiccallers undernet/that do not loop at all, so assume non-compound pages. Inesp_output_head,esp6_output_headandskb_seq_read. The first two directly useskb_page_frag_refill, which can allocate compound (but not__GFP_HIGHMEM) pages, and the third can be inserted with netfilterxt_stringin the path of tcp transmit skbs, which can also have compound pages. I think that these could similarly access data beyond the end of thekmap_atomicmapped page. I'll take a closer look.Thanks.
Quoted 7 lines
Quoted 7 lines
Note that I have flushed my random one-liner patch from my system, and expect to get a proper fix through the normal networking pulls.
And _if_ the networking people feel that my one-liner was the proper fix, you can use it and add my sign-off if you want to, but it really was more of a "this is the quick ugly fix for testing" rather than anything else.
I do think it is the proper fix as is. If no one else has comments, I can submit it through the net tree.
It won't address the other issues that became apparent only as a result of this. I'm preparing separate patches for those.
Quoted 5 lines
Please add:
Quoted 7 lines
And if you take Linus's patch, please add my:
and if you come up with another patch, please send it to me for testing.
Thanks!
Will do, thanks.
Re: [BUG] from x86: Support kmap_local() forced debugging
In reply to Linus Torvaldsinclude/linux/skbuff.h static inline void skb_frag_size_sub(skb_frag_t *frag, int delta) · line 372
Quoted 11 lines · On Wed, 6 Jan 2021 17:03:48 -0800 Linus Torvalds…
include/linux/skbuff.h +1
-1@@ -366,7 +366,7 @@ static inline void skb_frag_size_sub(skb_frag_t *frag, int delta)static inline bool skb_frag_must_loop (struct page *p) {#if defined(CONFIG_HIGHMEM ) -if (PageHighMem (p)) +if (IS_ENABLED (CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP ) ||PageHighMem (p))return true;#endif return false;
I applied this and I haven't been able to crash it again.
Thanks,
-- Steve
PATCH v4 4/8 sched: Make migrate_disable/enable() independent of RT
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT.
There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like kmap_atomic() enforce it implicitly.
Making it available independent of RT allows to provide a preemptible variant of kmap_atomic() and makes the code more consistent in general.
FIXME: Rework the comment in preempt.h - Peter?
--- include/linux/kernel.h | 21 ++++++++++++++------- include/linux/preempt.h | 38 +++----------------------------------- include/linux/sched.h | 2 +- kernel/sched/core.c | 45 +++++++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 4 ++-- lib/smp_processor_id.c | 2 +- 6 files changed, 56 insertions(+), 56 deletions(-)
include/linux/kernel.h +14 -7
@@ -204,6 +204,7 @@ extern int _cond_resched(void);extern void ___might_sleep (const char *file,int line,int preempt_offset);extern void __might_sleep (const char *file,int line,int preempt_offset);extern void __cant_sleep (const char *file,int line,int preempt_offset); +extern void __cant_migrate (const char *file,int line);/** * might_sleep - annotation for functions that can sleep
@@ -227,6 +228,18 @@ extern void __cant_sleep(const char *fil# define cant_sleep () \ do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)# define sched_annotate_sleep () (current->task_state_change = 0) + +/** +* cant_migrate - annotation for functions that cannot migrate +* +* Will print a stack trace if executed in code which is migratable +*/ +# define cant_migrate () \ + do { \ + if (IS_ENABLED(CONFIG_SMP)) \ + __cant_migrate(__FILE__, __LINE__); \ + } while (0) +/** * non_block_start - annotate the start of section where sleeping is prohibited *
@@ -251,6 +264,7 @@ extern void __cant_sleep(const char *fil int preempt_offset) { }# define might_sleep () do { might_resched(); } while (0)# define cant_sleep () do { } while (0) +# define cant_migrate () do { } while (0)# define sched_annotate_sleep () do { } while (0)# define non_block_start () do { } while (0)# define non_block_end () do { } while (0)
@@ -258,13 +272,6 @@ extern void __cant_sleep(const char *fil#define might_sleep_if (cond) do { if (cond) might_sleep(); } while (0) -#ifndef CONFIG_PREEMPT_RT -# define cant_migrate () cant_sleep() -#else -/* Placeholder for now */ -# define cant_migrate () do { } while (0) -#endif -/** * abs - return absolute value of an argument * @x: the value. If it is unsigned type, it is converted to signed type first.
include/linux/preempt.h +3 -35
@@ -322,7 +322,7 @@ static inline void preempt_notifier_init#endif -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP /* * Migrate-Disable and why it is undesired.
@@ -382,43 +382,11 @@ static inline void preempt_notifier_initextern void migrate_disable (void );extern void migrate_enable (void ); -#elif defined(CONFIG_PREEMPT_RT ) +#else static inline void migrate_disable (void ) { }static inline void migrate_enable (void ) { } -#else /* !CONFIG_PREEMPT_RT */ - -/** -* migrate_disable - Prevent migration of the current task -* -* Maps to preempt_disable() which also disables preemption. Use -* migrate_disable() to annotate that the intent is to prevent migration, -* but not necessarily preemption. -* -* Can be invoked nested like preempt_disable() and needs the corresponding -* number of migrate_enable() invocations. -*/ -static __always_inline void migrate_disable(void) -{ -preempt_disable (); -} - -/** -* migrate_enable - Allow migration of the current task -* -* Counterpart to migrate_disable(). -* -* As migrate_disable() can be invoked nested, only the outermost invocation -* reenables migration. -* -* Currently mapped to preempt_enable(). -*/ -static __always_inline void migrate_enable(void) -{ -preempt_enable (); -} - -#endif /* CONFIG_SMP && CONFIG_PREEMPT_RT */ +#endif /* CONFIG_SMP */ #endif /* __LINUX_PREEMPT_H */
include/linux/sched.h +1 -1
@@ -715,7 +715,7 @@ struct task_struct {const cpumask_t *cpus_ptr;cpumask_t cpus_mask;void *migration_pending; -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP unsigned short migration_disabled;#endif unsigned short migration_flags;
kernel/sched/core.c +35 -10
81 lines not on this page · raw message
kernel/sched/sched.h +2 -2
@@ -1056,7 +1056,7 @@ struct rq {struct cpuidle_state *idle_state;#endif -#if defined(CONFIG_PREEMPT_RT ) && defined(CONFIG_SMP ) +#ifdef CONFIG_SMP unsigned int nr_pinned;#endif unsigned int push_busy;
@@ -1092,7 +1092,7 @@ static inline int cpu_of(struct rq *rq)static inline bool is_migration_disabled (struct task_struct *p) { -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP return p->migration_disabled;#else return false;
lib/smp_processor_id.c +1 -1
@@ -26,7 +26,7 @@ unsigned int check_preemption_disabled(cif (current->nr_cpus_allowed ==1 ) goto out; -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP if (current->migration_disabled) goto out;#endif
Mel Gorman
reviewerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 5 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT.
Quoted 9 lines
There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like
kmap_atomic()enforce it implicitly.Making it available independent of RT allows to provide a preemptible variant of
kmap_atomic()and makes the code more consistent in general.FIXME: Rework the comment in
preempt.h- Peter?
I didn't keep up to date and there is clearly a dependency on patches in tip for migrate_enable/migrate_disable . It's not 100% clear to me what reworking you're asking for but then again, I'm not Peter!
From tip;
/**
* migrate_disable - Prevent migration of the current task
*
* Maps to preempt_disable() which also disables preemption. Use
* migrate_disable() to annotate that the intent is to prevent migration,
* but not necessarily preemption.
*
* Can be invoked nested like preempt_disable() and needs the corresponding
* number of migrate_enable() invocations.
*/
I assume that the rework is to document the distinction between migrate_disable and preempt_disable() because it may not be clear to some people why one should be used over another and the risk of cut&paste cargo cult programming.
So I assume the rework is for the middle paragraph
* Maps to preempt_disable() which also disables preemption. Use
* migrate_disable() to annotate that the intent is to prevent migration,
* but not necessarily preemption. The distinction is that preemption
* disabling will protect a per-cpu structure from concurrent
* modifications due to preemption. migrate_disable partially protects
* the tasks address space and potentially preserves the TLB entries
* even if preempted such as an needed for a local IO mapping or a
* kmap_atomic() referenced by on-stack pointers to avoid interference
* between user threads or kernel threads sharing the same address space.
I know it can have other examples that are rt-specific and some tricks on percpu page alloc draining that relies on a combination of migrate_disable and interrupt disabling to protect the structures but the above example might be understandable to a non-RT audience.
Signature · 2 lines
Mel Gorman SUSE Labs
Peter Zijlstra
maintainerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 13 lines · Mel Gorman
Quoted 12 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT.
There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like
kmap_atomic()enforce it implicitly.Making it available independent of RT allows to provide a preemptible variant of
kmap_atomic()and makes the code more consistent in general.
Quoted 6 lines
Quoted 2 lines
FIXME: Rework the comment in
preempt.h- Peter?I didn't keep up to date and there is clearly a dependency on patches in tip for migrate_enable/migrate_disable . It's not 100% clear to me what reworking you're asking for but then again, I'm not Peter!
He's talking about the big one: "Migrate-Disable and why it is undesired.".
I still hate all of this, and I really fear that with migrate_disable() available, people will be lazy and usage will increase :/
Case at hand is this series, the only reason we need it here is because per-cpu page-tables are expensive...
I really do think we want to limit the usage and get rid of the implicit migrate_disable() in spinlock_t/rwlock_t for example.
AFAICT the scenario described there is entirely possible; and it has to show up for workloads that rely on multi-cpu bandwidth for correctness.
Switching from preempt_disable() to migrate_disable() hides the immediate / easily visible high priority latency, but you move the interference term into a place where it is much harder to detect, you don't lose the term, it stays in the system.
So no, I don't want to make the comment less scary. Usage is discouraged.
Mel Gorman
reviewerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Mel Gorman
Quoted 14 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT.
There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like
kmap_atomic()enforce it implicitly.Making it available independent of RT allows to provide a preemptible variant of
kmap_atomic()and makes the code more consistent in general.FIXME: Rework the comment in
preempt.h- Peter?
Quoted 7 lines
Quoted 3 lines
I didn't keep up to date and there is clearly a dependency on patches in tip for migrate_enable/migrate_disable . It's not 100% clear to me what reworking you're asking for but then again, I'm not Peter!
He's talking about the big one: "Migrate-Disable and why it is undesired.".
Ah yes, that makes more sense. I was thinking in terms of what is protected but the PREEMPT_RT hazard is severe.
Quoted 6 lines
I still hate all of this, and I really fear that with
migrate_disable()available, people will be lazy and usage will increase :/Case at hand is this series, the only reason we need it here is because per-cpu page-tables are expensive...
I guessed, it was the only thing that made sense.
Quoted 6 lines
I really do think we want to limit the usage and get rid of the implicit
migrate_disable()in spinlock_t/rwlock_t for example.AFAICT the scenario described there is entirely possible; and it has to show up for workloads that rely on multi-cpu bandwidth for correctness.
Quoted 7 lines
Switching from
preempt_disable()tomigrate_disable()hides the immediate / easily visible high priority latency, but you move the interference term into a place where it is much harder to detect, you don't lose the term, it stays in the system.So no, I don't want to make the comment less scary. Usage is discouraged.
More scary then by adding this to the kerneldoc section for migrate_disable?
* Usage of migrate_disable is heavily discouraged as it is extremely
* hazardous on PREEMPT_RT kernels and any usage needs to be heavily
* justified. Before even thinking about using this, read
* "Migrate-Disable and why it is undesired" in
* include/linux/preempt.h and include both a comment and document
* in the changelog why the use case is an exception.
It's not necessary for the current series because the interface hides it and anyone poking at the internals of kmap_atomic probably should be aware of the address space and TLB hazards associated with it. There are few in-tree users and presumably any future preempt-rt related merges already know why migrate_disable is required.
However, with the kerneldoc, there is no excuse for missing it for new users that are not PREEMPT_RT-aware. It makes it easier to NAK/revert a patch without proper justification similar to how undocumented usages of memory barriers tend to get a poor reception.
Signature · 2 lines
Mel Gorman SUSE Labs
Steven Rostedt
reviewerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 2 lines · On Thu, 19 Nov 2020 12:14:13 +0000 Mel Gorman <[email protected]> wrote:
* Usage of
migrate_disableis heavily discouraged as it is extremely
* hazardous onPREEMPT_RTkernels and any usage needs to be heavily
I don't believe it's just PREEMPT_RT. It's RT tasks that are concerned, especially when you are dealing with SCHED_DEADLINE.
PREEMPT_RT just allows better determinism for RT tasks, but the issue with migrate_disable is not limited to just PREEMPT_RT.
-- Steve
Quoted 4 lines
* justified. Before even thinking about using this, read
* "Migrate-Disable and why it is undesired" in
*include/linux/preempt.hand include both a comment and document
* in the changelog why the use case is an exception.
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
In reply to Peter ZijlstraQuoted 6 lines · Peter Zijlstra
I still hate all of this, and I really fear that with
migrate_disable()available, people will be lazy and usage will increase :/Case at hand is this series, the only reason we need it here is because per-cpu page-tables are expensive...
No, I think you as a scheduler person just need to accept it.
Because this is certainly not the only time migration limiting has come up, and no, it has absolutely nothing to do with per-cpu page tables being completely unacceptable.
The scheduler people need to get used to this. Really. Because ASMP is just going to be a fact.
There are few things more futile than railing against reality, Peter.
Honestly, the only argument I've ever heard against limiting migration is the whole "our scheduling theory doesn't cover it".
So either throw the broken theory away, or live with it. Theory that doesn't match reality isn't theory, it's religion.
Linus
Peter Zijlstra
maintainerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Peter Zijlstra
I still hate all of this, and I really fear that with
migrate_disable()available, people will be lazy and usage will increase :/Case at hand is this series, the only reason we need it here is because per-cpu page-tables are expensive...
No, I think you as a scheduler person just need to accept it.
Well, I did do write the patches.
Quoted 3 lines
Because this is certainly not the only time migration limiting has come up, and no, it has absolutely nothing to do with per-cpu page tables being completely unacceptable.
It is for this instance; but sure, it's come up before in other contexts.
Quoted 2 lines
The scheduler people need to get used to this. Really. Because ASMP is just going to be a fact.
ASMP is different in that it is a hardware constraint, you're just not going to be able to run more of X than there's X capable hardware units on (be if FPUs, Vector units, 32bit or whatever)
Quoted 1 line
There are few things more futile than railing against reality, Peter.
But, but, my windmills! :-)
Quoted 5 lines
Honestly, the only argument I've ever heard against limiting migration is the whole "our scheduling theory doesn't cover it".
So either throw the broken theory away, or live with it. Theory that doesn't match reality isn't theory, it's religion.
The first stage of throwing it out is understanding the problem, which is just about where we're at. Next is creating a new formalism (if possible) that covers this new issue. That might take a while.
Thing is though; without a formalism to reason about timeliness guarantees, there is no Real-Time.
So sure, I've written the patches, doesn't mean I have to like the place we're in due to it.
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds
Because this is certainly not the only time migration limiting has come up, and no, it has absolutely nothing to do with per-cpu page tables being completely unacceptable.
It is for this instance; but sure, it's come up before in other contexts.
Indeed. And one of the really bad outcomes of this is that people are forced to use preempt_disable() to prevent migration which entails a slew of consequences:
- Using spinlocks where it wouldn't be needed otherwise
- Spinwaiting instead of sleeping
- The whole crazyness of doing copy_to/from_user_in_atomic() along with the necessary out of line error handling.
- ....
The introduction of per-cpu storage happened almost 20 years ago (2002) and still the only answer we have is preempt_disable().
I know the scheduling theory folks still try to wrap their heads around the introduction of SMP which dates back to 1962 IIRC...
Quoted 6 lines
Quoted 2 lines
The scheduler people need to get used to this. Really. Because ASMP is just going to be a fact.
ASMP is different in that it is a hardware constraint, you're just not going to be able to run more of X than there's X capable hardware units on (be if FPUs, Vector units, 32bit or whatever)
ASMP is as old as SMP. The first SMP systems were in fact ASMP. The reasons for ASMP 60 years ago were not that different from the reasons for ASMP today. Just the scale and the effectivness are different.
Quoted 3 lines
Quoted 1 line
There are few things more futile than railing against reality, Peter.
But, but, my windmills! :-)
At least you have windmills where you live so you can pull off the real Don Quixote while other people have to find substitutes :)
Thanks,
tglx
Peter Zijlstra
maintainerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 13 lines · Thomas Gleixner
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Linus Torvalds
Because this is certainly not the only time migration limiting has come up, and no, it has absolutely nothing to do with per-cpu page tables being completely unacceptable.
It is for this instance; but sure, it's come up before in other contexts.
Indeed. And one of the really bad outcomes of this is that people are forced to use
preempt_disable()to prevent migration which entails a slew of consequences:
Quoted 8 lines
- Using spinlocks where it wouldn't be needed otherwise
- Spinwaiting instead of sleeping
- The whole crazyness of doing copy_to/from_user_in_atomic() along with the necessary out of line error handling.
- ....The introduction of per-cpu storage happened almost 20 years ago (2002) and still the only answer we have is
preempt_disable().
IIRC the first time this migrate_disable() stuff came up was when Chris Lameter did SLUB. Eventually he settled for that cmpxchg_double() approach (which is somewhat similar to userspace rseq) which is vastly superiour and wouldn't have happened had we provided migrate_disable().
As already stated, per-cpu page-tables would allow for a much saner kmap approach, but alas, x86 really can't sanely do that (the archs that have separate kernel and user page-tables could do this, and how we cursed x86 didn't have that when meltdown happened).
[ and using fixmaps in the per-cpu memory space _could_ work, but is a
giant pain because then all accesses need GS prefix and blah... ]
And I'm sure there's creative ways for other problems too, but yes, it's hard.
Anyway, clearly I'm the only one that cares, so I'll just crawl back under my rock...
Andy Lutomirski
maintainerRe: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 7 lines · Peter Zijlstra
As already stated, per-cpu page-tables would allow for a much saner kmap approach, but alas, x86 really can't sanely do that (the archs that have separate kernel and user page-tables could do this, and how we cursed x86 didn't have that when meltdown happened).
Quoted 8 lines
[ and using fixmaps in the per-cpu memory space _could_ work, but is a
giant pain because then all accesses need GS prefix and blah... ]And I'm sure there's creative ways for other problems too, but yes, it's hard.
Anyway, clearly I'm the only one that cares, so I'll just crawl back under my rock...
I'll poke my head out of the rock for a moment, though...
Several years ago, we discussed (in person at some conference IIRC) having percpu pagetables to get sane kmaps, percpu memory, etc. The conclusion was that Linus thought the performance would suck and we shouldn't do it. Since then, though, we added really fancy infrastructure for keeping track of a per-CPU list of recently used mms and efficiently tracking when they need to be invalidated. We called these "ASIDs". It would be fairly straightforward to have an entire pgd for each (cpu, asid) pair. Newly added second-level (p4d/pud/whatever -- have I ever mentioned how much I dislike the Linux pagetable naming conventions and folding tricks?) tables could be lazily faulted in, and copies of the full 2kB mess would only be neeced when a new (cpu,asid) is allocated because either a flush happened while the mm was inactive on the CPU in question or because the mm fell off the percpu cache.
The total overhead would be a bit more cache usage, 4kB * num cpus * num ASIDs per CPU (or 8k for PTI), and a few extra page faults (max num cpus * 256 per mm over the entire lifetime of that mm). The common case of a CPU switching back and forth between a small number of mms would have no significant overhead.
On an unrelated note, what happens if you migrate_disable(), sleep for a looooong time, and someone tries to offline your CPU?
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 8 lines · Andy Lutomirski
Quoted 2 lines · Peter Zijlstra
Anyway, clearly I'm the only one that cares, so I'll just crawl back under my rock...
I'll poke my head out of the rock for a moment, though...
Several years ago, we discussed (in person at some conference IIRC) having percpu pagetables to get sane kmaps, percpu memory, etc.
Yes, I remember. That was our initial reaction in Prague to the looming PTI challenge 3 years ago.
Quoted 2 lines
The conclusion was that Linus thought the performance would suck and we shouldn't do it.
Linus had opinions, but we all agreed that depending on the workload and the CPU features (think !PCID) the copy/pagefault overhead could be significant.
Quoted 11 lines
Since then, though, we added really fancy infrastructure for keeping track of a per-CPU list of recently used mms and efficiently tracking when they need to be invalidated. We called these "ASIDs". It would be fairly straightforward to have an entire pgd for each (cpu, asid) pair. Newly added second-level (p4d/pud/whatever -- have I ever mentioned how much I dislike the Linux pagetable naming conventions and folding tricks?) tables could be lazily faulted in, and copies of the full 2kB mess would only be neeced when a new (cpu,asid) is allocated because either a flush happened while the mm was inactive on the CPU in question or because the mm fell off the percpu cache.
Quoted 6 lines
The total overhead would be a bit more cache usage, 4kB * num cpus * num ASIDs per CPU (or 8k for PTI), and a few extra page faults (max num cpus * 256 per mm over the entire lifetime of that mm).
The common case of a CPU switching back and forth between a small number of mms would have no significant overhead.
For CPUs which do not support PCID this sucks, which is everything pre Westmere and all of 32bit. Yes, 32bit. If we go there then 32bit has to bite the bullet and use the very same mechanism. Not that I care much TBH.
Even for those CPUs which support it we'd need to increase the number of ASIDs significantly. Right now we use only 6 ASIDs, which is not a lot. There are process heavy workloads out there which do quite some context switching so avoiding the copy matters. I'm not worried about fork as the copy will probably be just noise.
That said, I'm not saying it shouldn't be done, but there are quite a few things which need to be looked at.
TBH, I really would love to see that just to make GS kernel usage and the related mess in the ASM code go away completely.
For the task at hand, i.e. replacing kmap_atomic() by kmap_local(), this is not really helpful because we'd need to make all highmem using architectures do the same thing. But if we can pull it off on x86 the required changes for the kmap_local() code are not really significant.
Quoted 2 lines
On an unrelated note, what happens if you
migrate_disable(), sleep for a looooong time, and someone tries to offline your CPU?
The hotplug code will prevent the CPU from going offline in that case, i.e. it waits until the last task left it's migrate disabled section.
But you are not supposed to invoke sleep($ETERNAL) in such a context. Emphasis on 'not supposed' :)
Thanks,
tglx
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 9 lines · Thomas Gleixner
Quoted 3 lines · Andy Lutomirski
On Fri, Nov 20, 2020 at 1:29 AM Peter Zijlstra <[email protected]> wrote:
The common case of a CPU switching back and forth between a small number of mms would have no significant overhead.For CPUs which do not support PCID this sucks, which is everything pre Westmere and all of 32bit. Yes, 32bit. If we go there then 32bit has to bite the bullet and use the very same mechanism. Not that I care much TBH.
Bah, I completely forgot that AMD does not support PCID before Zen3 which is a major showstopper.
Thanks,
tglx
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 10 lines · Thomas Gleixner
Quoted 5 lines · Thomas Gleixner
Quoted 4 lines · Andy Lutomirski
Quoted 1 line · Peter Zijlstra
The common case of a CPU switching back and forth between a small number of mms would have no significant overhead.
Quoted 7 lines
Quoted 4 lines
For CPUs which do not support PCID this sucks, which is everything pre Westmere and all of 32bit. Yes, 32bit. If we go there then 32bit has to bite the bullet and use the very same mechanism. Not that I care much TBH.
Bah, I completely forgot that AMD does not support PCID before Zen3 which is a major showstopper.
Why? Couldn’t we rig up the code so we still track all the ASIDs even if there is no CPU support? We would take the TLB flush hit on every context switch, but we pay that cost anyway. We would avoid the extra copy in the same cases in which we would avoid it if we had PCID.
Quoted 4 lines
Thanks,
tglx
Re: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT
Quoted 12 lines · Andy Lutomirski
Quoted 12 lines · Thomas Gleixner
Quoted 9 lines · Thomas Gleixner
Quoted 4 lines · Andy Lutomirski
The common case of a CPU switching back and forth between a small number of mms would have no significant overhead.
For CPUs which do not support PCID this sucks, which is everything pre Westmere and all of 32bit. Yes, 32bit. If we go there then 32bit has to bite the bullet and use the very same mechanism. Not that I care much TBH.
Quoted 7 lines
Quoted 2 lines
Bah, I completely forgot that AMD does not support PCID before Zen3 which is a major showstopper.
Why? Couldn’t we rig up the code so we still track all the ASIDs even if there is no CPU support? We would take the TLB flush hit on every context switch, but we pay that cost anyway. We would avoid the extra copy in the same cases in which we would avoid it if we had PCID.
Did not think about that indeed. Yes, that should do the trick and should not be worse than what we have now.
Sometimes one just can't see the forest for the trees :)
Thanks,
tglx
[tip: sched/core] sched: Make migrate_disable/enable() independent of RT
In reply to Thomas GleixnerThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 74d862b682f51e45d25b95b1ecf212428a4967b0 Gitweb: https://git.kernel.org/tip/74d862b682f51e45d25b95b1ecf212428a4967b0 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:42 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 11:25:44 +01:00
sched: Make migrate_disable/enable() independent of RT
Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT.
There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like kmap_atomic() enforce it implicitly.
Making it available independent of RT allows to provide a preemptible variant of kmap_atomic() and makes the code more consistent in general.
--- include/linux/kernel.h | 21 ++++++++++++------- include/linux/preempt.h | 38 ++-------------------------------- include/linux/sched.h | 2 +- kernel/sched/core.c | 45 +++++++++++++++++++++++++++++++--------- kernel/sched/sched.h | 4 ++-- lib/smp_processor_id.c | 2 +- 6 files changed, 56 insertions(+), 56 deletions(-)
include/linux/kernel.h +14 -7
@@ -204,6 +204,7 @@ extern int _cond_resched(void);extern void ___might_sleep (const char *file,int line,int preempt_offset);extern void __might_sleep (const char *file,int line,int preempt_offset);extern void __cant_sleep (const char *file,int line,int preempt_offset); +extern void __cant_migrate (const char *file,int line);/** * might_sleep - annotation for functions that can sleep
@@ -227,6 +228,18 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset);# define cant_sleep () \ do { __cant_sleep(__FILE__, __LINE__, 0); } while (0)# define sched_annotate_sleep () (current->task_state_change = 0) + +/** +* cant_migrate - annotation for functions that cannot migrate +* +* Will print a stack trace if executed in code which is migratable +*/ +# define cant_migrate () \ + do { \ + if (IS_ENABLED(CONFIG_SMP)) \ + __cant_migrate(__FILE__, __LINE__); \ + } while (0) +/** * non_block_start - annotate the start of section where sleeping is prohibited *
@@ -251,6 +264,7 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset); int preempt_offset) { }# define might_sleep () do { might_resched(); } while (0)# define cant_sleep () do { } while (0) +# define cant_migrate () do { } while (0)# define sched_annotate_sleep () do { } while (0)# define non_block_start () do { } while (0)# define non_block_end () do { } while (0)
@@ -258,13 +272,6 @@ extern void __cant_sleep(const char *file, int line, int preempt_offset);#define might_sleep_if (cond) do { if (cond) might_sleep(); } while (0) -#ifndef CONFIG_PREEMPT_RT -# define cant_migrate () cant_sleep() -#else -/* Placeholder for now */ -# define cant_migrate () do { } while (0) -#endif -/** * abs - return absolute value of an argument * @x: the value. If it is unsigned type, it is converted to signed type first.
include/linux/preempt.h +3 -35
@@ -322,7 +322,7 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,#endif -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP /* * Migrate-Disable and why it is undesired.
@@ -382,43 +382,11 @@ static inline void preempt_notifier_init(struct preempt_notifier *notifier,extern void migrate_disable (void );extern void migrate_enable (void ); -#elif defined(CONFIG_PREEMPT_RT ) +#else static inline void migrate_disable (void ) { }static inline void migrate_enable (void ) { } -#else /* !CONFIG_PREEMPT_RT */ - -/** -* migrate_disable - Prevent migration of the current task -* -* Maps to preempt_disable() which also disables preemption. Use -* migrate_disable() to annotate that the intent is to prevent migration, -* but not necessarily preemption. -* -* Can be invoked nested like preempt_disable() and needs the corresponding -* number of migrate_enable() invocations. -*/ -static __always_inline void migrate_disable(void) -{ -preempt_disable (); -} - -/** -* migrate_enable - Allow migration of the current task -* -* Counterpart to migrate_disable(). -* -* As migrate_disable() can be invoked nested, only the outermost invocation -* reenables migration. -* -* Currently mapped to preempt_enable(). -*/ -static __always_inline void migrate_enable(void) -{ -preempt_enable (); -} - -#endif /* CONFIG_SMP && CONFIG_PREEMPT_RT */ +#endif /* CONFIG_SMP */ #endif /* __LINUX_PREEMPT_H */
include/linux/sched.h +1 -1
@@ -715,7 +715,7 @@ struct task_struct {const cpumask_t *cpus_ptr;cpumask_t cpus_mask;void *migration_pending; -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP unsigned short migration_disabled;#endif unsigned short migration_flags;
kernel/sched/core.c +35 -10
81 lines not on this page · raw message
kernel/sched/sched.h +2 -2
@@ -1056,7 +1056,7 @@ struct rq {struct cpuidle_state *idle_state;#endif -#if defined(CONFIG_PREEMPT_RT ) && defined(CONFIG_SMP ) +#ifdef CONFIG_SMP unsigned int nr_pinned;#endif unsigned int push_busy;
@@ -1092,7 +1092,7 @@ static inline int cpu_of(struct rq *rq)static inline bool is_migration_disabled (struct task_struct *p) { -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP return p->migration_disabled;#else return false;
lib/smp_processor_id.c +1 -1
@@ -26,7 +26,7 @@ unsigned int check_preemption_disabled(const char *what1, const char *what2)if (current->nr_cpus_allowed ==1 ) goto out; -#if defined(CONFIG_SMP ) && defined(CONFIG_PREEMPT_RT ) +#ifdef CONFIG_SMP if (current->migration_disabled) goto out;#endif
PATCH v4 5/8 sched: highmem: Store local kmaps in task struct
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Instead of storing the map per CPU provide and use per task storage. That prepares for local kmaps which are preemptible.
The context switch code is preparatory and not yet in use because kmap_atomic() runs with preemption disabled. Will be made usable in the next step.
The context switch logic is safe even when an interrupt happens after clearing or before restoring the kmaps. The kmap index in task struct is not modified so any nesting kmap in an interrupt will use unused indices and on return the counter is the same as before.
Also add an assert into the return to user space code. Going back to user space with an active kmap local is a nono.
---
V4: Use the version which actually compiles and works
V3: Handle the debug case correctly
--- include/linux/highmem-internal.h | 10 +++ include/linux/sched.h | 9 +++ kernel/entry/common.c | 2 kernel/fork.c | 1 kernel/sched/core.c | 18 +++++++ mm/highmem.c | 99 +++++++++++++++++++++++++++++++++++---- 6 files changed, 129 insertions(+), 10 deletions(-)
include/linux/highmem-internal.h +10 -0
@@ -9,6 +9,16 @@void *__kmap_local_pfn_prot (unsigned long pfn,pgprot_t prot);void *__kmap_local_page_prot (struct page *page,pgprot_t prot);void kunmap_local_indexed (void *vaddr); +void kmap_local_fork (struct task_struct *tsk); +void __kmap_local_sched_out (void ); +void __kmap_local_sched_in (void ); +static inline void kmap_assert_nomap (void ) +{ +DEBUG_LOCKS_WARN_ON (current->kmap_ctrl.idx); +} +#else +static inline void kmap_local_fork (struct task_struct *tsk) { } +static inline void kmap_assert_nomap (void ) { }#endif #ifdef CONFIG_HIGHMEM
include/linux/sched.h +9 -0
@@ -34,6 +34,7 @@#include <linux/rseq.h> #include <linux/seqlock.h> #include <linux/kcsan.h> +#include <asm/kmap_size.h> /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context ;
@@ -629,6 +630,13 @@ struct wake_q_node {struct wake_q_node *next; }; +struct kmap_ctrl { +#ifdef CONFIG_KMAP_LOCAL +int idx; +pte_t pteval[KM_MAX_IDX ]; +#endif +}; +struct task_struct {#ifdef CONFIG_THREAD_INFO_IN_TASK /*
@@ -1294,6 +1302,7 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +struct kmap_ctrl kmap_ctrl;#ifdef CONFIG_DEBUG_ATOMIC_SLEEP unsigned long task_state_change;#endif
kernel/entry/common.c +2 -0
@@ -2,6 +2,7 @@#include <linux/context_tracking.h> #include <linux/entry-common.h> +#include <linux/highmem.h> #include <linux/livepatch.h> #include <linux/audit.h>
@@ -194,6 +195,7 @@ static void exit_to_user_mode_prepare(st/* Ensure that the address limit is intact and no locks are held */ addr_limit_user_check (); +kmap_assert_nomap ();lockdep_assert_irqs_disabled ();lockdep_sys_exit (); }
kernel/fork.c +1 -0
@@ -930,6 +930,7 @@ static struct task_struct *dup_task_struaccount_kernel_stack (tsk,1 );kcov_task_init (tsk); +kmap_local_fork (tsk);#ifdef CONFIG_FAULT_INJECTION tsk->fail_nth =0 ;
kernel/sched/core.c +18 -0
@@ -4051,6 +4051,22 @@ static inline void finish_lock_switch(st# define finish_arch_post_lock_switch () do { } while (0)#endif +static inline void kmap_local_sched_out (void ) +{ +#ifdef CONFIG_KMAP_LOCAL +if (unlikely (current->kmap_ctrl.idx)) +__kmap_local_sched_out (); +#endif +} + +static inline void kmap_local_sched_in (void ) +{ +#ifdef CONFIG_KMAP_LOCAL +if (unlikely (current->kmap_ctrl.idx)) +__kmap_local_sched_in (); +#endif +} +/** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch
@@ -4073,6 +4089,7 @@ prepare_task_switch(struct rq *rq, strucperf_event_task_sched_out (prev, next);rseq_preempt (prev);fire_sched_out_preempt_notifiers (prev, next); +kmap_local_sched_out ();prepare_task (next);prepare_arch_switch (next); }
@@ -4139,6 +4156,7 @@ static struct rq *finish_task_switch(strfinish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current); +kmap_local_sched_in ();fire_sched_in_preempt_notifiers (current);/*
mm/highmem.c +89 -10
136 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [patch V4 5/8] sched: highmem: Store local kmaps in task struct
kernel/sched/core.c static struct rq *finish_task_switch(str · line 4159
Quoted 13 lines · Thomas Gleixner
kernel/sched/core.c +2
-0@@ -4073,6 +4089,7 @@ prepare_task_switch(struct rq *rq, strucperf_event_task_sched_out (prev, next);rseq_preempt (prev);fire_sched_out_preempt_notifiers (prev, next); +kmap_local_sched_out ();prepare_task (next);prepare_arch_switch (next); }@@ -4139,6 +4156,7 @@ static struct rq *finish_task_switch(strfinish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current); +kmap_local_sched_in ();
This is asymmetric and deserves a comment. You do the sched_out with IRQs disabled and rq->lock held, but do the sched_in with IRQs enabled and rq->lock released.
I suppose doing it here reduces IRQ latency by however long it takes to update and invalidate that handful of pages, is that worth the asymmetry?
It mirrors preempt_notifiers I suppose, and they actually rely on this asymmetry for something IIRC.
Quoted 2 lines
kernel/sched/core.c +0
-0fire_sched_in_preempt_notifiers (current);/*
Peter Zijlstra
maintainerRe: [patch V4 5/8] sched: highmem: Store local kmaps in task struct
In reply to Thomas GleixnerQuoted 28 lines · Thomas Gleixner
kernel/sched/core.c +13
-0@@ -4073,6 +4089,7 @@ prepare_task_switch(struct rq *rq, strucperf_event_task_sched_out (prev, next);rseq_preempt (prev);fire_sched_out_preempt_notifiers (prev, next); +kmap_local_sched_out ();prepare_task (next);prepare_arch_switch (next); }@@ -4139,6 +4156,7 @@ static struct rq *finish_task_switch(strfinish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current); +kmap_local_sched_in ();fire_sched_in_preempt_notifiers (current);/* +void __kmap_local_sched_out(void) +{ +struct task_struct *tsk = current; +pte_t *kmap_pte = kmap_get_pte(); +int i; + +/* Clear kmaps */ +for (i =0 ; i < tsk->kmap_ctrl.idx; i++) { + } +} +
kernel/sched/core.c static struct rq *finish_task_switch(str
Quoted 10 lines
kernel/sched/core.c +10
-0+void __kmap_local_sched_in (void ) +{ +struct task_struct *tsk = current; +pte_t *kmap_pte =kmap_get_pte (); +int i; + +/* Restore kmaps */ +for (i =0 ; i < tsk->kmap_ctrl.idx; i++) { + } +}
So even in the optimal case, this adds an unconditional load of tsk->kmap_ctrl.idx to schedule() (2 misses, one pre and one post).
Munging preempt-notifiers behind a static_branch, which in that same optimal case, avoided touching curr->preempt_notifier, resulted in a measurable performance improvement. See commit:
1cde2930e154 ("sched/preempt: Add static_key() to preempt_notifiers")
Can we fudge some state in a cacheline we're already touching to avoid this?
Peter Zijlstra
maintainerRe: [patch V4 5/8] sched: highmem: Store local kmaps in task struct
Quoted 44 lines · Peter Zijlstra
Quoted 28 lines · Thomas Gleixner
kernel/sched/core.c +13
-0@@ -4073,6 +4089,7 @@ prepare_task_switch(struct rq *rq, strucperf_event_task_sched_out (prev, next);rseq_preempt (prev);fire_sched_out_preempt_notifiers (prev, next); +kmap_local_sched_out ();prepare_task (next);prepare_arch_switch (next); }@@ -4139,6 +4156,7 @@ static struct rq *finish_task_switch(strfinish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current); +kmap_local_sched_in ();fire_sched_in_preempt_notifiers (current);/* +void __kmap_local_sched_out(void) +{ +struct task_struct *tsk = current; +pte_t *kmap_pte = kmap_get_pte(); +int i; + +/* Clear kmaps */ +for (i =0 ; i < tsk->kmap_ctrl.idx; i++) { + } +} +kernel/sched/core.c static struct rq *finish_task_switch(str
Quoted 10 lines
kernel/sched/core.c +10
-0+void __kmap_local_sched_in (void ) +{ +struct task_struct *tsk = current; +pte_t *kmap_pte =kmap_get_pte (); +int i; + +/* Restore kmaps */ +for (i =0 ; i < tsk->kmap_ctrl.idx; i++) { + } +}So even in the optimal case, this adds an unconditional load of
tsk->kmap_ctrl.idxtoschedule()(2 misses, one pre and one post).
Quoted 8 lines
Munging preempt-notifiers behind a
static_branch, which in that same optimal case, avoided touchingcurr->preempt_notifier, resulted in a measurable performance improvement. See commit:
1cde2930e154("sched/preempt: Addstatic_key()topreempt_notifiers")Can we fudge some state in a cacheline we're already touching to avoid this?
The only state we seem to consistently look at after schedule() is need_resched()'s TIF_NEED_RESCHED.
But adding a TIF_flag to all archs and setting/clearing it from kmap_local might be a bit daft.. :/
Re: [patch V4 5/8] sched: highmem: Store local kmaps in task struct
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
mm/highmem.c void kunmap_local_indexed(void *vaddr) · line 599
Quoted 10 lines
mm/highmem.c +10
-0+void __kmap_local_sched_in (void ) +{ +struct task_struct *tsk = current; +pte_t *kmap_pte =kmap_get_pte (); +int i; + +/* Restore kmaps */ +for (i =0 ; i < tsk->kmap_ctrl.idx; i++) { + } +}So even in the optimal case, this adds an unconditional load of
tsk->kmap_ctrl.idxtoschedule()(2 misses, one pre and one post).Munging preempt-notifiers behind a
static_branch, which in that same optimal case, avoided touchingcurr->preempt_notifier, resulted in a measurable performance improvement. See commit:
1cde2930e154("sched/preempt: Addstatic_key()topreempt_notifiers")
Quoted 8 lines
Quoted 2 lines
Can we fudge some state in a cacheline we're already touching to avoid this?
The only state we seem to consistently look at after
schedule()is need_resched()'sTIF_NEED_RESCHED.But adding a
TIF_flagto all archs and setting/clearing it fromkmap_localmight be a bit daft.. :/
Yes, and we all agreed that pushing the burden on CONFIG_HIGHMEM is fine. It's slow, so making it a tad slower is not the end of the world. Remember we want to get rid of it :)
Thanks,
tglx
[tip: core/mm] sched: highmem: Store local kmaps in task struct
In reply to Thomas GleixnerThe following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: 5fbda3ecd14a5343644979c98d6eb65b7e7de9d8 Gitweb: https://git.kernel.org/tip/5fbda3ecd14a5343644979c98d6eb65b7e7de9d8 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:43 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:09 +01:00
sched: highmem: Store local kmaps in task struct
Instead of storing the map per CPU provide and use per task storage. That prepares for local kmaps which are preemptible.
The context switch code is preparatory and not yet in use because kmap_atomic() runs with preemption disabled. Will be made usable in the next step.
The context switch logic is safe even when an interrupt happens after clearing or before restoring the kmaps. The kmap index in task struct is not modified so any nesting kmap in an interrupt will use unused indices and on return the counter is the same as before.
Also add an assert into the return to user space code. Going back to user space with an active kmap local is a nono.
--- include/linux/highmem-internal.h | 10 +++- include/linux/sched.h | 9 +++- kernel/entry/common.c | 2 +- kernel/fork.c | 1 +- kernel/sched/core.c | 25 ++++++++- mm/highmem.c | 99 ++++++++++++++++++++++++++---- 6 files changed, 136 insertions(+), 10 deletions(-)
include/linux/highmem-internal.h +10 -0
@@ -9,6 +9,16 @@void *__kmap_local_pfn_prot (unsigned long pfn,pgprot_t prot);void *__kmap_local_page_prot (struct page *page,pgprot_t prot);void kunmap_local_indexed (void *vaddr); +void kmap_local_fork (struct task_struct *tsk); +void __kmap_local_sched_out (void ); +void __kmap_local_sched_in (void ); +static inline void kmap_assert_nomap (void ) +{ +DEBUG_LOCKS_WARN_ON (current->kmap_ctrl.idx); +} +#else +static inline void kmap_local_fork (struct task_struct *tsk) { } +static inline void kmap_assert_nomap (void ) { }#endif #ifdef CONFIG_HIGHMEM
include/linux/sched.h +9 -0
@@ -34,6 +34,7 @@#include <linux/rseq.h> #include <linux/seqlock.h> #include <linux/kcsan.h> +#include <asm/kmap_size.h> /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context ;
@@ -629,6 +630,13 @@ struct wake_q_node {struct wake_q_node *next; }; +struct kmap_ctrl { +#ifdef CONFIG_KMAP_LOCAL +int idx; +pte_t pteval[KM_MAX_IDX ]; +#endif +}; +struct task_struct {#ifdef CONFIG_THREAD_INFO_IN_TASK /*
@@ -1294,6 +1302,7 @@ struct task_struct {unsigned int sequential_io;unsigned int sequential_io_avg;#endif +struct kmap_ctrl kmap_ctrl;#ifdef CONFIG_DEBUG_ATOMIC_SLEEP unsigned long task_state_change;#endif
kernel/entry/common.c +2 -0
@@ -2,6 +2,7 @@#include <linux/context_tracking.h> #include <linux/entry-common.h> +#include <linux/highmem.h> #include <linux/livepatch.h> #include <linux/audit.h>
@@ -194,6 +195,7 @@ static void exit_to_user_mode_prepare(struct pt_regs *regs)/* Ensure that the address limit is intact and no locks are held */ addr_limit_user_check (); +kmap_assert_nomap ();lockdep_assert_irqs_disabled ();lockdep_sys_exit (); }
kernel/fork.c +1 -0
@@ -930,6 +930,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)account_kernel_stack (tsk,1 );kcov_task_init (tsk); +kmap_local_fork (tsk);#ifdef CONFIG_FAULT_INJECTION tsk->fail_nth =0 ;
kernel/sched/core.c +25 -0
@@ -4094,6 +4094,22 @@ static inline void finish_lock_switch(struct rq *rq)# define finish_arch_post_lock_switch () do { } while (0)#endif +static inline void kmap_local_sched_out (void ) +{ +#ifdef CONFIG_KMAP_LOCAL +if (unlikely (current->kmap_ctrl.idx)) +__kmap_local_sched_out (); +#endif +} + +static inline void kmap_local_sched_in (void ) +{ +#ifdef CONFIG_KMAP_LOCAL +if (unlikely (current->kmap_ctrl.idx)) +__kmap_local_sched_in (); +#endif +} +/** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch
@@ -4116,6 +4132,7 @@ prepare_task_switch(struct rq *rq, struct task_struct *prev,perf_event_task_sched_out (prev, next);rseq_preempt (prev);fire_sched_out_preempt_notifiers (prev, next); +kmap_local_sched_out ();prepare_task (next);prepare_arch_switch (next); }
@@ -4182,6 +4199,14 @@ static struct rq *finish_task_switch(struct task_struct *prev)finish_lock_switch (rq);finish_arch_post_lock_switch ();kcov_finish_switch (current); +/* +* kmap_local_sched_out() is invoked with rq::lock held and +* interrupts disabled. There is no requirement for that, but the +* sched out code does not have an interrupt enabled section. +* Restoring the maps on sched in does not require interrupts being +* disabled either. +*/ +kmap_local_sched_in ();fire_sched_in_preempt_notifiers (current);/*
mm/highmem.c +89 -10
136 lines not on this page · raw message
PATCH v4 6/8 mm/highmem: Provide kmap_local*
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Now that the kmap atomic index is stored in task struct provide a preemptible variant. On context switch the maps of an outgoing task are removed and the map of the incoming task are restored. That's obviously slow, but highmem is slow anyway.
The kmap_local.*() functions can be invoked from both preemptible and atomic context. kmap local sections disable migration to keep the resulting virtual mapping address correct, but disable neither pagefaults nor preemption.
A wholesale conversion of kmap_atomic to be fully preemptible is not possible because some of the usage sites might rely on the preemption disable for serialization or on the implicit pagefault disable. Needs to be done on a case by case basis.
---
V3: Move migrate disable into the actual highmem mapping code so it only
affects real highmem mappings.
V2: Make it more consistent and add commentry
--- include/linux/highmem-internal.h | 48 +++++++++++++++++++++++++++++++++++++++ include/linux/highmem.h | 43 +++++++++++++++++++++------------- mm/highmem.c | 6 ++++ 3 files changed, 81 insertions(+), 16 deletions(-)
include/linux/highmem-internal.h +48 -0
@@ -68,6 +68,26 @@ static inline void kmap_flush_unused(voi__kmap_flush_unused (); } +static inline void *kmap_local_page (struct page *page) +{ +return __kmap_local_page_prot (page, kmap_prot); +} + +static inline void *kmap_local_page_prot (struct page *page,pgprot_t prot) +{ +return __kmap_local_page_prot (page, prot); +} + +static inline void *kmap_local_pfn (unsigned long pfn) +{ +return __kmap_local_pfn_prot (pfn, kmap_prot); +} + +static inline void __kunmap_local (void *vaddr) +{ +kunmap_local_indexed (vaddr); +} +static inline void *kmap_atomic_prot (struct page *page,pgprot_t prot) {preempt_disable ();
@@ -140,6 +160,28 @@ static inline void kunmap(struct page *p#endif } +static inline void *kmap_local_page (struct page *page) +{ +return page_address (page); +} + +static inline void *kmap_local_page_prot (struct page *page,pgprot_t prot) +{ +return kmap_local_page (page); +} + +static inline void *kmap_local_pfn (unsigned long pfn) +{ +return kmap_local_page (pfn_to_page (pfn)); +} + +static inline void __kunmap_local (void *addr) +{ +#ifdef ARCH_HAS_FLUSH_ON_KUNMAP +kunmap_flush_on_unmap (addr); +#endif +} +static inline void *kmap_atomic (struct page *page) {preempt_disable ();
@@ -181,4 +223,10 @@ do { \__kunmap_atomic (__addr); \ } while (0) +#define kunmap_local (__addr) \ +do { \ + BUILD_BUG_ON(__same_type((__addr), struct page *)); \ + __kunmap_local(__addr); \ +} while (0) +#endif
include/linux/highmem.h +27 -16
73 lines not on this page · raw message
mm/highmem.c +6 -0
@@ -453,6 +453,11 @@ void *__kmap_local_pfn_prot(unsigned lonunsigned long vaddr;int idx; +/* +* Disable migration so resulting virtual address is stable +* accross preemption. +*/ +migrate_disable ();preempt_disable (); idx =arch_kmap_local_map_idx (kmap_local_idx_push (), pfn); vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx);
@@ -522,6 +527,7 @@ void kunmap_local_indexed(void *vaddr) current->kmap_ctrl.pteval[kmap_local_idx ()] =__pte (0 );kmap_local_idx_pop ();preempt_enable (); +migrate_enable (); }EXPORT_SYMBOL (kunmap_local_indexed);
[tip: core/mm] mm/highmem: Provide kmap_local*
The following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: f3ba3c710ac5a30cd058615a9eb62d2ad95bb782 Gitweb: https://git.kernel.org/tip/f3ba3c710ac5a30cd058615a9eb62d2ad95bb782 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:44 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:09 +01:00
mm/highmem: Provide kmap_local*
Now that the kmap atomic index is stored in task struct provide a preemptible variant. On context switch the maps of an outgoing task are removed and the map of the incoming task are restored. That's obviously slow, but highmem is slow anyway.
The kmap_local.*() functions can be invoked from both preemptible and atomic context. kmap local sections disable migration to keep the resulting virtual mapping address correct, but disable neither pagefaults nor preemption.
A wholesale conversion of kmap_atomic to be fully preemptible is not possible because some of the usage sites might rely on the preemption disable for serialization or on the implicit pagefault disable. Needs to be done on a case by case basis.
--- include/linux/highmem-internal.h | 48 +++++++++++++++++++++++++++++++- include/linux/highmem.h | 43 +++++++++++++++++----------- mm/highmem.c | 6 ++++- 3 files changed, 81 insertions(+), 16 deletions(-)
include/linux/highmem-internal.h +48 -0
@@ -68,6 +68,26 @@ static inline void kmap_flush_unused(void)__kmap_flush_unused (); } +static inline void *kmap_local_page (struct page *page) +{ +return __kmap_local_page_prot (page, kmap_prot); +} + +static inline void *kmap_local_page_prot (struct page *page,pgprot_t prot) +{ +return __kmap_local_page_prot (page, prot); +} + +static inline void *kmap_local_pfn (unsigned long pfn) +{ +return __kmap_local_pfn_prot (pfn, kmap_prot); +} + +static inline void __kunmap_local (void *vaddr) +{ +kunmap_local_indexed (vaddr); +} +static inline void *kmap_atomic_prot (struct page *page,pgprot_t prot) {preempt_disable ();
@@ -140,6 +160,28 @@ static inline void kunmap(struct page *page)#endif } +static inline void *kmap_local_page (struct page *page) +{ +return page_address (page); +} + +static inline void *kmap_local_page_prot (struct page *page,pgprot_t prot) +{ +return kmap_local_page (page); +} + +static inline void *kmap_local_pfn (unsigned long pfn) +{ +return kmap_local_page (pfn_to_page (pfn)); +} + +static inline void __kunmap_local (void *addr) +{ +#ifdef ARCH_HAS_FLUSH_ON_KUNMAP +kunmap_flush_on_unmap (addr); +#endif +} +static inline void *kmap_atomic (struct page *page) {preempt_disable ();
@@ -181,4 +223,10 @@ do { \__kunmap_atomic (__addr); \ } while (0) +#define kunmap_local (__addr) \ +do { \ + BUILD_BUG_ON(__same_type((__addr), struct page *)); \ + __kunmap_local(__addr); \ +} while (0) +#endif
include/linux/highmem.h +27 -16
73 lines not on this page · raw message
mm/highmem.c +6 -0
@@ -453,6 +453,11 @@ void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot)unsigned long vaddr;int idx; +/* +* Disable migration so resulting virtual address is stable +* accross preemption. +*/ +migrate_disable ();preempt_disable (); idx =arch_kmap_local_map_idx (kmap_local_idx_push (), pfn); vaddr =__fix_to_virt (FIX_KMAP_BEGIN + idx);
@@ -522,6 +527,7 @@ void kunmap_local_indexed(void *vaddr) current->kmap_ctrl.pteval[kmap_local_idx ()] =__pte (0 );kmap_local_idx_pop ();preempt_enable (); +migrate_enable (); }EXPORT_SYMBOL (kunmap_local_indexed);
PATCH v4 7/8 io-mapping: Provide iomap_local variant
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Similar to kmap local provide a iomap local variant which only disables migration, but neither disables pagefaults nor preemption.
---
V3: Restrict migrate disable to the 32bit mapping case and update documentation.
V2: Split out from the large combo patch and add the !IOMAP_ATOMIC variants
--- Documentation/driver-api/io-mapping.rst | 76 +++++++++++++++++++------------- include/linux/io-mapping.h | 30 +++++++++++- 2 files changed, 74 insertions(+), 32 deletions(-)
Documentation/driver-api/io-mapping.rst +46 -30
@@ -20,55 +20,71 @@ as it would consume too much of the kern mappable, while 'size' indicates how large a mapping region to enable. Both are in bytes. -This _wc variant provides a mapping which may only be used -with the io_mapping_map_atomic_wc or io_mapping_map_wc. +This _wc variant provides a mapping which may only be used with +io_mapping_map_atomic_wc(), io_mapping_map_local_wc() or +io_mapping_map_wc(). + +With this mapping object, individual pages can be mapped either temporarily +or long term, depending on the requirements. Of course, temporary maps are +more efficient. They come in two flavours:: -With this mapping object, individual pages can be mapped either atomically -or not, depending on the necessary scheduling environment. Of course, atomic -maps are more efficient:: + void *io_mapping_map_local_wc(struct io_mapping *mapping, + unsigned long offset) void *io_mapping_map_atomic_wc(struct io_mapping *mapping, unsigned long offset) -'offset' is the offset within the defined mapping region. -Accessing addresses beyond the region specified in the -creation function yields undefined results. Using an offset -which is not page aligned yields an undefined result. The -return value points to a single page in CPU address space. - -This _wc variant returns a write-combining map to the -page and may only be used with mappings created by -io_mapping_create_wc +'offset' is the offset within the defined mapping region. Accessing +addresses beyond the region specified in the creation function yields +undefined results. Using an offset which is not page aligned yields an +undefined result. The return value points to a single page in CPU address +space. -Note that the task may not sleep while holding this page -mapped. +This _wc variant returns a write-combining map to the page and may only be +used with mappings created by io_mapping_create_wc() -:: +Temporary mappings are only valid in the context of the caller. The mapping +is not guaranteed to be globaly visible. - void io_mapping_unmap_atomic(void *vaddr) +io_mapping_map_local_wc() has a side effect on X86 32bit as it disables +migration to make the mapping code work. No caller can rely on this side +effect. + +io_mapping_map_atomic_wc() has the side effect of disabling preemption and +pagefaults. Don't use in new code. Use io_mapping_map_local_wc() instead. -'vaddr' must be the value returned by the last -io_mapping_map_atomic_wc call. This unmaps the specified -page and allows the task to sleep once again. +Nested mappings need to be undone in reverse order because the mapping +code uses a stack for keeping track of them:: -If you need to sleep while holding the lock, you can use the non-atomic -variant, although they may be significantly slower. + addr1 = io_mapping_map_local_wc(map1, offset1); + addr2 = io_mapping_map_local_wc(map2, offset2); + ... + io_mapping_unmap_local(addr2); + io_mapping_unmap_local(addr1); -:: +The mappings are released with:: + + void io_mapping_unmap_local(void *vaddr) + void io_mapping_unmap_atomic(void *vaddr) + +'vaddr' must be the value returned by the last io_mapping_map_local_wc() or +io_mapping_map_atomic_wc() call. This unmaps the specified mapping and +undoes the side effects of the mapping functions. + +If you need to sleep while holding a mapping, you can use the regular +variant, although this may be significantly slower:: void *io_mapping_map_wc(struct io_mapping *mapping, unsigned long offset) -This works like io_mapping_map_atomic_wc except it allows -the task to sleep while holding the page mapped. - +This works like io_mapping_map_atomic/local_wc() except it has no side +effects and the pointer is globaly visible. -:: +The mappings are released with:: void io_mapping_unmap(void *vaddr) -This works like io_mapping_unmap_atomic, except it is used -for pages mapped with io_mapping_map_wc. +Use for pages mapped with io_mapping_map_wc(). At driver close time, the io_mapping object must be freed::
include/linux/io-mapping.h +28 -2
@@ -83,6 +83,21 @@ io_mapping_unmap_atomic(void __iomem *va }static inline void __iomem * +io_mapping_map_local_wc (struct io_mapping *mapping,unsigned long offset) +{ +resource_size_t phys_addr; + +BUG_ON (offset >= mapping->size); + phys_addr = mapping->base + offset; +return __iomap_local_pfn_prot (PHYS_PFN (phys_addr), mapping->prot); +} + +static inline void io_mapping_unmap_local (void __iomem *vaddr) +{ +kunmap_local_indexed ((void __force *)vaddr); +} + +static inline void __iomem *io_mapping_map_wc (struct io_mapping *mapping,unsigned long offset,unsigned long size)
@@ -101,7 +116,7 @@ io_mapping_unmap(void __iomem *vaddr) iounmap(vaddr); } -#else +#else /* HAVE_ATOMIC_IOMAP */ #include <linux/uaccess.h>
@@ -166,7 +181,18 @@ io_mapping_unmap_atomic(void __iomem *vapreempt_enable (); } -#endif /* HAVE_ATOMIC_IOMAP */ +static inline void __iomem * +io_mapping_map_local_wc (struct io_mapping *mapping,unsigned long offset) +{ +return io_mapping_map_wc (mapping, offset,PAGE_SIZE ); +} + +static inline void io_mapping_unmap_local (void __iomem *vaddr) +{ +io_mapping_unmap (vaddr); +} + +#endif /* !HAVE_ATOMIC_IOMAP */ static inline struct io_mapping * io_mapping_create_wc(resource_size_t base,
[tip: core/mm] io-mapping: Provide iomap_local variant
The following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: e66f6e095486f0210fcf3c5eb3ecf13fa348be4c Gitweb: https://git.kernel.org/tip/e66f6e095486f0210fcf3c5eb3ecf13fa348be4c Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:45 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:09 +01:00
io-mapping: Provide iomap_local variant
Similar to kmap local provide a iomap local variant which only disables migration, but neither disables pagefaults nor preemption.
--- Documentation/driver-api/io-mapping.rst | 74 ++++++++++++++---------- include/linux/io-mapping.h | 30 +++++++++- 2 files changed, 73 insertions(+), 31 deletions(-)
Documentation/driver-api/io-mapping.rst +45 -29
@@ -20,55 +20,71 @@ A mapping object is created during driver initialization using:: mappable, while 'size' indicates how large a mapping region to enable. Both are in bytes. -This _wc variant provides a mapping which may only be used -with the io_mapping_map_atomic_wc or io_mapping_map_wc. +This _wc variant provides a mapping which may only be used with +io_mapping_map_atomic_wc(), io_mapping_map_local_wc() or +io_mapping_map_wc(). -With this mapping object, individual pages can be mapped either atomically -or not, depending on the necessary scheduling environment. Of course, atomic -maps are more efficient:: +With this mapping object, individual pages can be mapped either temporarily +or long term, depending on the requirements. Of course, temporary maps are +more efficient. They come in two flavours:: + + void *io_mapping_map_local_wc(struct io_mapping *mapping, + unsigned long offset) void *io_mapping_map_atomic_wc(struct io_mapping *mapping, unsigned long offset) -'offset' is the offset within the defined mapping region. -Accessing addresses beyond the region specified in the -creation function yields undefined results. Using an offset -which is not page aligned yields an undefined result. The -return value points to a single page in CPU address space. +'offset' is the offset within the defined mapping region. Accessing +addresses beyond the region specified in the creation function yields +undefined results. Using an offset which is not page aligned yields an +undefined result. The return value points to a single page in CPU address +space. -This _wc variant returns a write-combining map to the -page and may only be used with mappings created by -io_mapping_create_wc +This _wc variant returns a write-combining map to the page and may only be +used with mappings created by io_mapping_create_wc() -Note that the task may not sleep while holding this page -mapped. +Temporary mappings are only valid in the context of the caller. The mapping +is not guaranteed to be globaly visible. -:: +io_mapping_map_local_wc() has a side effect on X86 32bit as it disables +migration to make the mapping code work. No caller can rely on this side +effect. - void io_mapping_unmap_atomic(void *vaddr) +io_mapping_map_atomic_wc() has the side effect of disabling preemption and +pagefaults. Don't use in new code. Use io_mapping_map_local_wc() instead. -'vaddr' must be the value returned by the last -io_mapping_map_atomic_wc call. This unmaps the specified -page and allows the task to sleep once again. +Nested mappings need to be undone in reverse order because the mapping +code uses a stack for keeping track of them:: -If you need to sleep while holding the lock, you can use the non-atomic -variant, although they may be significantly slower. + addr1 = io_mapping_map_local_wc(map1, offset1); + addr2 = io_mapping_map_local_wc(map2, offset2); + ... + io_mapping_unmap_local(addr2); + io_mapping_unmap_local(addr1); -:: +The mappings are released with:: + + void io_mapping_unmap_local(void *vaddr) + void io_mapping_unmap_atomic(void *vaddr) + +'vaddr' must be the value returned by the last io_mapping_map_local_wc() or +io_mapping_map_atomic_wc() call. This unmaps the specified mapping and +undoes the side effects of the mapping functions. + +If you need to sleep while holding a mapping, you can use the regular +variant, although this may be significantly slower:: void *io_mapping_map_wc(struct io_mapping *mapping, unsigned long offset) -This works like io_mapping_map_atomic_wc except it allows -the task to sleep while holding the page mapped. - +This works like io_mapping_map_atomic/local_wc() except it has no side +effects and the pointer is globaly visible. -:: +The mappings are released with:: void io_mapping_unmap(void *vaddr) -This works like io_mapping_unmap_atomic, except it is used -for pages mapped with io_mapping_map_wc. +Use for pages mapped with io_mapping_map_wc(). At driver close time, the io_mapping object must be freed::
include/linux/io-mapping.h +28 -2
@@ -83,6 +83,21 @@ io_mapping_unmap_atomic(void __iomem *vaddr) }static inline void __iomem * +io_mapping_map_local_wc (struct io_mapping *mapping,unsigned long offset) +{ +resource_size_t phys_addr; + +BUG_ON (offset >= mapping->size); + phys_addr = mapping->base + offset; +return __iomap_local_pfn_prot (PHYS_PFN (phys_addr), mapping->prot); +} + +static inline void io_mapping_unmap_local (void __iomem *vaddr) +{ +kunmap_local_indexed ((void __force *)vaddr); +} + +static inline void __iomem *io_mapping_map_wc (struct io_mapping *mapping,unsigned long offset,unsigned long size)
@@ -101,7 +116,7 @@ io_mapping_unmap(void __iomem *vaddr) iounmap(vaddr); } -#else +#else /* HAVE_ATOMIC_IOMAP */ #include <linux/uaccess.h>
@@ -166,7 +181,18 @@ io_mapping_unmap_atomic(void __iomem *vaddr)preempt_enable (); } -#endif /* HAVE_ATOMIC_IOMAP */ +static inline void __iomem * +io_mapping_map_local_wc (struct io_mapping *mapping,unsigned long offset) +{ +return io_mapping_map_wc (mapping, offset,PAGE_SIZE ); +} + +static inline void io_mapping_unmap_local (void __iomem *vaddr) +{ +io_mapping_unmap (vaddr); +} + +#endif /* !HAVE_ATOMIC_IOMAP */ static inline struct io_mapping * io_mapping_create_wc(resource_size_t base,
PATCH v4 8/8 x86/crashdump/32: Simplify copy_oldmem_page()
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Replace kmap_atomic_pfn() with kmap_local_pfn() which is preemptible and can take page faults.
Remove the indirection of the dump page and the related cruft which is not longer required.
---
V3: New patch
--- arch/x86/kernel/crash_dump_32.c | 48 ++++++++-------------------------------- 1 file changed, 10 insertions(+), 38 deletions(-)
arch/x86/kernel/crash_dump_32.c +10 -38
@@ -13,8 +13,6 @@#include <linux/uaccess.h> -static void *kdump_buf_page; -static inline bool is_crashed_pfn_valid (unsigned long pfn) {#ifndef CONFIG_X86_PAE
@@ -41,15 +39,11 @@ static inline bool is_crashed_pfn_valid(* @userbuf: if set, @buf is in user address space, use copy_to_user(), * otherwise @buf is in kernel address space, use memcpy(). * -* Copy a page from "oldmem". For this page, there is no pte mapped -* in the current kernel. We stitch up a pte, similar to kmap_atomic. -* -* Calling copy_to_user() in atomic context is not desirable. Hence first -* copying the data to a pre-allocated kernel page and then copying to user -* space in non-atomic context. +* Copy a page from "oldmem". For this page, there might be no pte mapped +* in the current kernel. */ -ssize_t copy_oldmem_page (unsigned long pfn,char *buf, -size_t csize,unsigned long offset,int userbuf) +ssize_t copy_oldmem_page (unsigned long pfn,char *buf,size_t csize, +unsigned long offset,int userbuf) {void *vaddr;
@@ -59,38 +53,16 @@ ssize_t copy_oldmem_page(unsigned long pif (!is_crashed_pfn_valid (pfn))return -EFAULT ; - vaddr =kmap_atomic_pfn (pfn); + vaddr =kmap_local_pfn (pfn);if (!userbuf) { -memcpy (buf, (vaddr + offset), csize); -kunmap_atomic (vaddr); +memcpy (buf, vaddr + offset, csize); }else { -if (!kdump_buf_page) { -printk (KERN_WARNING "Kdump: Kdump buffer page not" -" allocated\n" ); -kunmap_atomic (vaddr); -return -EFAULT ; - } -copy_page (kdump_buf_page, vaddr); -kunmap_atomic (vaddr); -if (copy_to_user (buf, (kdump_buf_page + offset), csize)) -return -EFAULT ; +if (copy_to_user (buf, vaddr + offset, csize)) + csize = -EFAULT ; } -return csize; -} +kunmap_local (vaddr); -static int __init kdump_buf_page_init (void ) -{ -int ret =0 ; - - kdump_buf_page =kmalloc (PAGE_SIZE ,GFP_KERNEL ); -if (!kdump_buf_page) { -printk (KERN_WARNING "Kdump: Failed to allocate kdump buffer" -" page\n" ); - ret = -ENOMEM ; - } - -return ret; +return csize; } -arch_initcall (kdump_buf_page_init);
[tip: core/mm] x86/crashdump/32: Simplify copy_oldmem_page()
The following commit has been merged into the core/mm branch of tip:
The following commit has been merged into the core/mm branch of tip:
Commit-ID: 7e015a279853e747f5d4f957855ec5310848c501 Gitweb: https://git.kernel.org/tip/7e015a279853e747f5d4f957855ec5310848c501 Author: Thomas Gleixner <[email protected]> AuthorDate: Wed, 18 Nov 2020 20:48:46 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Tue, 24 Nov 2020 14:42:09 +01:00
x86/crashdump/32: Simplify copy_oldmem_page()
Replace kmap_atomic_pfn() with kmap_local_pfn() which is preemptible and can take page faults.
Remove the indirection of the dump page and the related cruft which is not longer required.
--- arch/x86/kernel/crash_dump_32.c | 48 ++++++-------------------------- 1 file changed, 10 insertions(+), 38 deletions(-)
arch/x86/kernel/crash_dump_32.c +10 -38
@@ -13,8 +13,6 @@#include <linux/uaccess.h> -static void *kdump_buf_page; -static inline bool is_crashed_pfn_valid (unsigned long pfn) {#ifndef CONFIG_X86_PAE
@@ -41,15 +39,11 @@ static inline bool is_crashed_pfn_valid(unsigned long pfn)* @userbuf: if set, @buf is in user address space, use copy_to_user(), * otherwise @buf is in kernel address space, use memcpy(). * -* Copy a page from "oldmem". For this page, there is no pte mapped -* in the current kernel. We stitch up a pte, similar to kmap_atomic. -* -* Calling copy_to_user() in atomic context is not desirable. Hence first -* copying the data to a pre-allocated kernel page and then copying to user -* space in non-atomic context. +* Copy a page from "oldmem". For this page, there might be no pte mapped +* in the current kernel. */ -ssize_t copy_oldmem_page (unsigned long pfn,char *buf, -size_t csize,unsigned long offset,int userbuf) +ssize_t copy_oldmem_page (unsigned long pfn,char *buf,size_t csize, +unsigned long offset,int userbuf) {void *vaddr;
@@ -59,38 +53,16 @@ ssize_t copy_oldmem_page(unsigned long pfn, char *buf,if (!is_crashed_pfn_valid (pfn))return -EFAULT ; - vaddr =kmap_atomic_pfn (pfn); + vaddr =kmap_local_pfn (pfn);if (!userbuf) { -memcpy (buf, (vaddr + offset), csize); -kunmap_atomic (vaddr); +memcpy (buf, vaddr + offset, csize); }else { -if (!kdump_buf_page) { -printk (KERN_WARNING "Kdump: Kdump buffer page not" -" allocated\n" ); -kunmap_atomic (vaddr); -return -EFAULT ; - } -copy_page (kdump_buf_page, vaddr); -kunmap_atomic (vaddr); -if (copy_to_user (buf, (kdump_buf_page + offset), csize)) -return -EFAULT ; +if (copy_to_user (buf, vaddr + offset, csize)) + csize = -EFAULT ; } -return csize; -} +kunmap_local (vaddr); -static int __init kdump_buf_page_init (void ) -{ -int ret =0 ; - - kdump_buf_page =kmalloc (PAGE_SIZE ,GFP_KERNEL ); -if (!kdump_buf_page) { -printk (KERN_WARNING "Kdump: Failed to allocate kdump buffer" -" page\n" ); - ret = -ENOMEM ; - } - -return ret; +return csize; } -arch_initcall (kdump_buf_page_init);
Peter Zijlstra
maintainerQuoted 12 lines · Thomas Gleixner
Documentation/driver-api/io-mapping.rst | 74 +++++++++++-------- arch/x86/Kconfig | 1 arch/x86/include/asm/fixmap.h | 12 ++- arch/x86/include/asm/pgtable_64_types.h | 6 + arch/x86/kernel/crash_dump_32.c | 48 ++---------- include/asm-generic/kmap_size.h | 2 include/linux/highmem-internal.h | 58 +++++++++++++++ include/linux/highmem.h | 43 +++++++---- include/linux/io-mapping.h | 30 +++++++ include/linux/kernel.h | 21 +++-- include/linux/preempt.h | 38 ---------- include/linux/sched.h | 11 ++
Quoted 8 lines
kernel/entry/common.c | 2 kernel/fork.c | 1 kernel/sched/core.c | 63 ++++++++++++++-- kernel/sched/sched.h | 4 - lib/Kconfig.debug | 22 +++++ lib/smp_processor_id.c | 2 mm/highmem.c | 121 ++++++++++++++++++++++++++++---- 19 files changed, 400 insertions(+), 159 deletions(-)