incoming

---

 tools/accounting/getdelays.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

tools/accounting/getdelays.c renamed from tools/accounting/getdelays.c~getdelays-fix-netlink-attribute-length +1 -1

@@ -136,7 +136,7 @@ static int send_cmd(int sd, __u16 nlmsg_
 	msg.g.version = 0x1;
 	na = (struct nlattr *) GENLMSG_DATA(&msg);
 	na->nla_type = nla_type;
-	na->nla_len = nla_len + 1 + NLA_HDRLEN;
+	na->nla_len = nla_len + NLA_HDRLEN;
 	memcpy(NLA_DATA(na), nla_data, nla_len);
 	msg.n.nlmsg_len += NLMSG_ALIGN(na->nla_len);

Andrew Morton

author maintainer

2020-04-02 04:02 UTC #2 lore raw

PATCH 2/155 kthread: mark timer used by delayed kthread works as IRQ safe

From: Petr Mladek <[email protected]>
Subject: kthread: mark timer used by delayed kthread works as IRQ safe

The timer used by delayed kthread works are IRQ safe because the used kthread_delayed_work_timer_fn() is IRQ safe.

It is properly marked when initialized by KTHREAD_DELAYED_WORK_INIT(). But TIMER_IRQSAFE flag is missing when initialized by kthread_init_delayed_work().

The missing flag might trigger invalid warning from del_timer_sync() when kthread_mod_delayed_work() is called with interrupts disabled.

This patch is result of a discussion about using the API, see
https://lkml.kernel.org/r/[email protected]

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Petr Mladek <[email protected]>

Reported-by Grygorii Strashko <[email protected]>

Tested-by Grygorii Strashko <[email protected]>

Cc Thomas Gleixner <[email protected]>

---

 include/linux/kthread.h |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

include/linux/kthread.h renamed from include/linux/kthread.h~kthread-mark-timer-used-by-delayed-kthread-works-as-irq-safe +2 -1

@@ -165,7 +165,8 @@ extern void __kthread_init_worker(struct
 	do {								\
 		kthread_init_work(&(dwork)->work, (fn));		\
 		timer_setup(&(dwork)->timer,				\
-			     kthread_delayed_work_timer_fn, 0);		\
+			     kthread_delayed_work_timer_fn,		\
+			     TIMER_IRQSAFE);				\
 	} while (0)
 
 int kthread_worker_fn(void *worker_ptr);

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #3 lore raw

PATCH 3/155 asm-generic: make more kernel-space headers mandatory

From: Masahiro Yamada <[email protected]>
Subject: asm-generic: make more kernel-space headers mandatory

Change a header to mandatory-y if both of the following are met:

[1] At least one architecture (except um) specifies it as generic-y in arch/*/include/asm/Kbuild

[2] Every architecture (except um) either has its own implementation (arch/*/include/asm/*.h) or specifies it as generic-y in arch/*/include/asm/Kbuild

This commit was generated by the following shell script.

----------------------------------->8-----------------------------------

arches=$(cd arch; ls -1 | sed -e '/Kconfig/d' -e '/um/d')

tmpfile=$(mktemp)

grep "^mandatory-y +=" include/asm-generic/Kbuild > $tmpfile

find arch -path 'arch/*/include/asm/Kbuild' |

	xargs sed -n 's/^generic-y += \(.*\)/\1/p' | sort -u |

while read header
do
mandatory=yes

for arch in $arches
do
if ! grep -q "generic-y += $header" arch/$arch/include/asm/Kbuild &&
! [ -f arch/$arch/include/asm/$header ]; then mandatory=no
break
fi
done

if [ "$mandatory" = yes ]; then
echo "mandatory-y += $header" >> $tmpfile

for arch in $arches
do
sed -i "/generic-y += $header/d" arch/$arch/include/asm/Kbuild
done
fi

done

sed -i '/^mandatory-y +=/d' include/asm-generic/Kbuild

LANG=C sort $tmpfile >> include/asm-generic/Kbuild

----------------------------------->8-----------------------------------

One obvious benefit is the diff stat:

 25 files changed, 52 insertions(+), 557 deletions(-)

It is tedious to list generic-y for each arch that needs it.

So, mandatory-y works like a fallback default (by just wrapping asm-generic one) when arch does not have a specific header implementation.

See the following commits:

def3f7cefe4e81c296090e1722a76551142c227c
a1b39bae16a62ce4aae02d958224f19316d98b24

It is tedious to convert headers one by one, so I processed by a shell script.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Masahiro Yamada <[email protected]>

Cc Michal Simek <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Arnd Bergmann <[email protected]>

---

 arch/alpha/include/asm/Kbuild      |   11 -----
 arch/arc/include/asm/Kbuild        |   21 ----------
 arch/arm/include/asm/Kbuild        |   12 ------
 arch/arm64/include/asm/Kbuild      |   18 ---------
 arch/c6x/include/asm/Kbuild        |   37 -------------------
 arch/csky/include/asm/Kbuild       |   36 ------------------
 arch/h8300/include/asm/Kbuild      |   46 -----------------------
 arch/hexagon/include/asm/Kbuild    |   33 -----------------
 arch/ia64/include/asm/Kbuild       |    7 ---
 arch/m68k/include/asm/Kbuild       |   24 ------------
 arch/microblaze/include/asm/Kbuild |   29 ---------------
 arch/mips/include/asm/Kbuild       |   13 ------
 arch/nds32/include/asm/Kbuild      |   37 -------------------
 arch/nios2/include/asm/Kbuild      |   38 -------------------
 arch/openrisc/include/asm/Kbuild   |   36 ------------------
 arch/parisc/include/asm/Kbuild     |   18 ---------
 arch/powerpc/include/asm/Kbuild    |    4 --
 arch/riscv/include/asm/Kbuild      |   28 --------------
 arch/s390/include/asm/Kbuild       |   15 -------
 arch/sh/include/asm/Kbuild         |   16 --------
 arch/sparc/include/asm/Kbuild      |   14 -------
 arch/unicore32/include/asm/Kbuild  |   34 -----------------
 arch/x86/include/asm/Kbuild        |    2 -
 arch/xtensa/include/asm/Kbuild     |   26 -------------
 include/asm-generic/Kbuild         |   52 +++++++++++++++++++++++++++
 25 files changed, 52 insertions(+), 555 deletions(-)

arch/alpha/include/asm/Kbuild renamed from arch/alpha/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-11~~

@@ -1,17 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 
 generated-y += syscall_table.h
-generic-y += compat.h
-generic-y += exec.h
 generic-y += export.h
-generic-y += fb.h
-generic-y += irq_work.h
 generic-y += kvm_para.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += trace_clock.h
-generic-y += current.h
-generic-y += kprobes.h

arch/arc/include/asm/Kbuild renamed from arch/arc/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-21~~

@@ -1,28 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
 generic-y += extable.h
-generic-y += ftrace.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += parport.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += topology.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/arm64/include/asm/Kbuild renamed from arch/arm64/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-18~~

@@ -1,26 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += bugs.h
-generic-y += delay.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
 generic-y += early_ioremap.h
-generic-y += emergency-restart.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += qrwlock.h
 generic-y += qspinlock.h
-generic-y += serial.h
 generic-y += set_memory.h
-generic-y += switch_to.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h

arch/arm/include/asm/Kbuild renamed from arch/arm/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-12~~

@@ -1,22 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += compat.h
-generic-y += current.h
 generic-y += early_ioremap.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
 generic-y += flat.h
-generic-y += irq_regs.h
-generic-y += kdebug.h
-generic-y += local.h
 generic-y += local64.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += parport.h
-generic-y += preempt.h
 generic-y += seccomp.h
-generic-y += serial.h
-generic-y += trace_clock.h
 
 generated-y += mach-types.h
 generated-y += unistd-nr.h

arch/c6x/include/asm/Kbuild renamed from arch/c6x/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-37~~

@@ -1,42 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += atomic.h
-generic-y += barrier.h
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += futex.h
-generic-y += hw_irq.h
-generic-y += io.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += mmu.h
-generic-y += mmu_context.h
-generic-y += pci.h
-generic-y += percpu.h
-generic-y += pgalloc.h
-generic-y += preempt.h
-generic-y += serial.h
-generic-y += shmparam.h
-generic-y += tlbflush.h
-generic-y += topology.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/csky/include/asm/Kbuild renamed from arch/csky/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-36~~

@@ -1,44 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 generic-y += asm-offsets.h
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += delay.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
-generic-y += fb.h
-generic-y += futex.h
 generic-y += gpio.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += linkage.h
-generic-y += local.h
 generic-y += local64.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += module.h
-generic-y += percpu.h
-generic-y += preempt.h
 generic-y += qrwlock.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += timex.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h
 generic-y += vmlinux.lds.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/h8300/include/asm/Kbuild renamed from arch/h8300/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-46~~

54 lines not on this page · raw message

arch/hexagon/include/asm/Kbuild renamed from arch/hexagon/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-33~~

@@ -1,39 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += barrier.h
-generic-y += bug.h
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += ftrace.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
 generic-y += iomap.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += pci.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += shmparam.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/ia64/include/asm/Kbuild renamed from arch/ia64/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 -7

@@ -1,12 +1,5 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += syscall_table.h
-generic-y += compat.h
-generic-y += exec.h
-generic-y += irq_work.h
 generic-y += kvm_para.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += preempt.h
-generic-y += trace_clock.h
 generic-y += vtime.h
-generic-y += word-at-a-time.h

arch/m68k/include/asm/Kbuild renamed from arch/m68k/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-24~~

@@ -1,32 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += syscall_table.h
-generic-y += barrier.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += futex.h
 generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += shmparam.h
 generic-y += spinlock.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/microblaze/include/asm/Kbuild renamed from arch/microblaze/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-29~~

@@ -1,40 +1,11 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += syscall_table.h
-generic-y += bitops.h
-generic-y += bug.h
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += hardirq.h
 generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += linkage.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += parport.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += serial.h
-generic-y += shmparam.h
 generic-y += syscalls.h
 generic-y += tlb.h
-generic-y += topology.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/mips/include/asm/Kbuild renamed from arch/mips/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-13~~

@@ -4,23 +4,10 @@ generated-y += syscall_table_32_o32.h
 generated-y += syscall_table_64_n32.h
 generated-y += syscall_table_64_n64.h
 generated-y += syscall_table_64_o32.h
-generic-y += current.h
-generic-y += device.h
-generic-y += emergency-restart.h
 generic-y += export.h
-generic-y += irq_work.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
 generic-y += parport.h
-generic-y += percpu.h
-generic-y += preempt.h
 generic-y += qrwlock.h
 generic-y += qspinlock.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/nds32/include/asm/Kbuild renamed from arch/nds32/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-37~~

@@ -1,46 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
 generic-y += asm-offsets.h
-generic-y += atomic.h
-generic-y += bitops.h
-generic-y += bug.h
-generic-y += bugs.h
-generic-y += checksum.h
 generic-y += cmpxchg.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += export.h
-generic-y += fb.h
 generic-y += gpio.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += parport.h
-generic-y += pci.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += switch_to.h
-generic-y += timex.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += xor.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h

arch/nios2/include/asm/Kbuild renamed from arch/nios2/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-38~~

@@ -1,45 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += atomic.h
-generic-y += barrier.h
-generic-y += bitops.h
-generic-y += bug.h
-generic-y += bugs.h
 generic-y += cmpxchg.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += ftrace.h
-generic-y += futex.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += module.h
-generic-y += pci.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += serial.h
 generic-y += spinlock.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/openrisc/include/asm/Kbuild renamed from arch/openrisc/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-36~~

@@ -1,45 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += barrier.h
-generic-y += bug.h
-generic-y += bugs.h
-generic-y += checksum.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += ftrace.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += module.h
-generic-y += pci.h
-generic-y += percpu.h
-generic-y += preempt.h
 generic-y += qspinlock_types.h
 generic-y += qspinlock.h
 generic-y += qrwlock_types.h
 generic-y += qrwlock.h
-generic-y += sections.h
-generic-y += shmparam.h
-generic-y += switch_to.h
-generic-y += topology.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/parisc/include/asm/Kbuild renamed from arch/parisc/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-18~~

@@ -2,26 +2,8 @@
 generated-y += syscall_table_32.h
 generated-y += syscall_table_64.h
 generated-y += syscall_table_c32.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += emergency-restart.h
-generic-y += exec.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += percpu.h
-generic-y += preempt.h
 generic-y += seccomp.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/powerpc/include/asm/Kbuild renamed from arch/powerpc/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 -4

@@ -3,12 +3,8 @@ generated-y += syscall_table_32.h
 generated-y += syscall_table_64.h
 generated-y += syscall_table_c32.h
 generated-y += syscall_table_spu.h
-generic-y += div64.h
-generic-y += dma-mapping.h
 generic-y += export.h
-generic-y += irq_regs.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += preempt.h
 generic-y += vtime.h
 generic-y += early_ioremap.h

arch/riscv/include/asm/Kbuild renamed from arch/riscv/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-28~~

@@ -1,35 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += bugs.h
-generic-y += checksum.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += div64.h
 generic-y += extable.h
 generic-y += flat.h
-generic-y += dma.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
-generic-y += fb.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
-generic-y += mm-arch-hooks.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += shmparam.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h
 generic-y += vmlinux.lds.h
-generic-y += xor.h

arch/s390/include/asm/Kbuild renamed from arch/s390/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-15~~

@@ -5,21 +5,6 @@ generated-y += syscall_table.h
 generated-y += unistd_nr.h
 
 generic-y += asm-offsets.h
-generic-y += cacheflush.h
-generic-y += device.h
-generic-y += dma-mapping.h
-generic-y += div64.h
-generic-y += emergency-restart.h
 generic-y += export.h
-generic-y += fb.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kmap_types.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
-generic-y += word-at-a-time.h

arch/sh/include/asm/Kbuild renamed from arch/sh/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-16~~

@@ -1,22 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += syscall_table.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += delay.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
 generic-y += parport.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += serial.h
-generic-y += trace_clock.h
-generic-y += xor.h

arch/sparc/include/asm/Kbuild renamed from arch/sparc/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-14~~

@@ -4,21 +4,7 @@
 generated-y += syscall_table_32.h
 generated-y += syscall_table_64.h
 generated-y += syscall_table_c32.h
-generic-y += div64.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += export.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
 generic-y += kvm_para.h
-generic-y += linkage.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += module.h
-generic-y += preempt.h
-generic-y += serial.h
-generic-y += trace_clock.h
-generic-y += word-at-a-time.h

arch/unicore32/include/asm/Kbuild renamed from arch/unicore32/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-34~~

@@ -1,41 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
-generic-y += atomic.h
-generic-y += bugs.h
-generic-y += compat.h
-generic-y += current.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += ftrace.h
-generic-y += futex.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
-generic-y += module.h
 generic-y += parport.h
-generic-y += percpu.h
-generic-y += preempt.h
-generic-y += sections.h
-generic-y += serial.h
-generic-y += shmparam.h
 generic-y += syscalls.h
-generic-y += topology.h
-generic-y += trace_clock.h
-generic-y += unaligned.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

arch/x86/include/asm/Kbuild renamed from arch/x86/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 -2

@@ -10,5 +10,3 @@ generated-y += xen-hypercalls.h
 generic-y += early_ioremap.h
 generic-y += export.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h

arch/xtensa/include/asm/Kbuild renamed from arch/xtensa/include/asm/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +0 ~~-26~~

@@ -1,36 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 generated-y += syscall_table.h
-generic-y += bug.h
-generic-y += compat.h
-generic-y += device.h
-generic-y += div64.h
-generic-y += dma-mapping.h
-generic-y += emergency-restart.h
-generic-y += exec.h
 generic-y += extable.h
-generic-y += fb.h
-generic-y += hardirq.h
-generic-y += hw_irq.h
-generic-y += irq_regs.h
-generic-y += irq_work.h
-generic-y += kdebug.h
-generic-y += kmap_types.h
-generic-y += kprobes.h
 generic-y += kvm_para.h
-generic-y += local.h
 generic-y += local64.h
 generic-y += mcs_spinlock.h
-generic-y += mm-arch-hooks.h
-generic-y += mmiowb.h
 generic-y += param.h
-generic-y += percpu.h
-generic-y += preempt.h
 generic-y += qrwlock.h
 generic-y += qspinlock.h
-generic-y += sections.h
-generic-y += topology.h
-generic-y += trace_clock.h
 generic-y += user.h
-generic-y += vga.h
-generic-y += word-at-a-time.h
-generic-y += xor.h

include/asm-generic/Kbuild renamed from include/asm-generic/Kbuild~asm-generic-make-more-kernel-space-headers-mandatory +52 -0

58 lines not on this page · raw message

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #4 lore raw

PATCH 4/155 scripts/spelling.txt: add syfs/sysfs pattern

From: Jonathan Neuschäfer <[email protected]>
Subject: scripts/spelling.txt: add syfs/sysfs pattern

There are a few cases in the tree where "sysfs" is misspelled as "syfs".

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Jonathan Neuschäfer <[email protected]>

Cc Colin Ian King <[email protected]>

Cc Xiong <[email protected]>

Cc Stephen Boyd <[email protected]>

Cc Paul Walmsley <[email protected]>

Cc Chris Paterson <[email protected]>

Cc Luca Ceresoli <[email protected]>

---

 scripts/spelling.txt |    1 +
 1 file changed, 1 insertion(+)

scripts/spelling.txt renamed from scripts/spelling.txt~scripts-spellingtxt-add-syfs-sysfs-pattern +1 -0

@@ -1328,6 +1328,7 @@ swithcing||switching
 swithed||switched
 swithing||switching
 swtich||switch
+syfs||sysfs
 symetric||symmetric
 synax||syntax
 synchonized||synchronized

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #5 lore raw

PATCH 5/155 scripts/spelling.txt: add more spellings to spelling.txt

From: Colin Ian King <[email protected]>
Subject: scripts/spelling.txt: add more spellings to spelling.txt

Here are some of the more common spelling mistakes and typos that I've found while fixing up spelling mistakes in the kernel since November 2019

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Colin Ian King <[email protected]>

Cc Joe Perches <[email protected]>

---

 scripts/spelling.txt |   20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

scripts/spelling.txt renamed from scripts/spelling.txt~scripts-spellingtxt-add-more-spellings-to-spellingtxt +19 -1

@@ -177,7 +177,9 @@ atomatically||automatically
 atomicly||atomically
 atempt||attempt
 attachement||attachment
+attatch||attach
 attched||attached
+attemp||attempt
 attemps||attempts
 attemping||attempting
 attepmpt||attempt

@@ -235,11 +237,13 @@ brievely||briefly
 brigde||bridge
 broadcase||broadcast
 broadcat||broadcast
+bufer||buffer
 bufufer||buffer
 cacluated||calculated
 caculate||calculate
 caculation||calculation
 cadidate||candidate
+cahces||caches
 calender||calendar
 calescing||coalescing
 calle||called

@@ -338,7 +342,6 @@ conditon||condition
 condtion||condition
 conected||connected
 conector||connector
-connecetd||connected
 configration||configuration
 configuartion||configuration
 configuation||configuration

@@ -349,7 +352,9 @@ configuretion||configuration
 configutation||configuration
 conider||consider
 conjuction||conjunction
+connecetd||connected
 connectinos||connections
+connetor||connector
 connnection||connection
 connnections||connections
 consistancy||consistency

@@ -469,6 +474,7 @@ difinition||definition
 digial||digital
 dimention||dimension
 dimesions||dimensions
+disgest||digest
 dispalying||displaying
 diplay||display
 directon||direction

@@ -553,6 +559,7 @@ etsablishment||establishment
 etsbalishment||establishment
 excecutable||executable
 exceded||exceeded
+exceeed||exceed
 excellant||excellent
 execeeded||exceeded
 execeeds||exceeds

@@ -742,6 +749,7 @@ initialzing||initializing
 initilization||initialization
 initilize||initialize
 initliaze||initialize
+initilized||initialized
 inofficial||unofficial
 inrerface||interface
 insititute||institute

@@ -802,6 +810,7 @@ irrelevent||irrelevant
 isnt||isn't
 isssue||issue
 issus||issues
+iteraions||iterations
 iternations||iterations
 itertation||iteration
 itslef||itself

@@ -828,6 +837,7 @@ libary||library
 librairies||libraries
 libraris||libraries
 licenceing||licencing
+limted||limited
 logaritmic||logarithmic
 loggging||logging
 loggin||login

@@ -924,6 +934,7 @@ nerver||never
 nescessary||necessary
 nessessary||necessary
 noticable||noticeable
+notication||notification
 notications||notifications
 notifcations||notifications
 notifed||notified

@@ -1007,6 +1018,7 @@ pendantic||pedantic
 peprocessor||preprocessor
 perfoming||performing
 perfomring||performing
+periperal||peripheral
 peripherial||peripheral
 permissons||permissions
 peroid||period

@@ -1043,6 +1055,7 @@ prefferably||preferably
 premption||preemption
 prepaired||prepared
 preperation||preparation
+preprare||prepare
 pressre||pressure
 primative||primitive
 princliple||principle

@@ -1064,6 +1077,7 @@ processsed||processed
 processsing||processing
 procteted||protected
 prodecure||procedure
+progamming||programming
 progams||programs
 progess||progress
 programers||programmers

@@ -1151,12 +1165,14 @@ replys||replies
 reponse||response
 representaion||representation
 reqeust||request
+reqister||register
 requestied||requested
 requiere||require
 requirment||requirement
 requred||required
 requried||required
 requst||request
+requsted||requested
 reregisteration||reregistration
 reseting||resetting
 reseved||reserved

@@ -1227,10 +1243,12 @@ seqeuncer||sequencer
 seqeuencer||sequencer
 sequece||sequence
 sequencial||sequential
+serivce||service
 serveral||several
 servive||service
 setts||sets
 settting||setting
+shapshot||snapshot
 shotdown||shutdown
 shoud||should
 shouldnt||shouldn't

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #6 lore raw

PATCH 6/155 ocfs2: remove FS_OCFS2_NM

From: Alex Shi <[email protected]>
Subject: ocfs2: remove FS_OCFS2_NM

This macro is unused since commit ab09203e302b ("sysctl fs: Remove dead binary sysctl support").

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Alex Shi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/stackglue.c |    2 --
 1 file changed, 2 deletions(-)

fs/ocfs2/stackglue.c renamed from fs/ocfs2/stackglue.c~ocfs2-remove-fs_ocfs2_nm +0 -1

@@ -656,8 +656,6 @@ error:
  * and easier to preserve the name.
  */
 
-#define FS_OCFS2_NM		1

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #7 lore raw

PATCH 7/155 ocfs2: remove unused macros

From: Alex Shi <[email protected]>
Subject: ocfs2: remove unused macros

O2HB_DEFAULT_BLOCK_BITS/DLM_THREAD_MAX_ASTS/DLM_MIGRATION_RETRY_MS and OCFS2_MAX_RESV_WINDOW_BITS/OCFS2_MIN_RESV_WINDOW_BITS have been unused since commit 66effd3c6812 ("ocfs2/dlm: Do not migrate resource to a node that is leaving the domain").

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Alex Shi <[email protected]>

Cc Greg Kroah-Hartman <[email protected]>

Cc ChenGang <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Richard Fontana <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Joseph Qi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/cluster/heartbeat.c |    2 --
 fs/ocfs2/dlm/dlmmaster.c     |    2 --
 fs/ocfs2/dlm/dlmthread.c     |    1 -
 fs/ocfs2/reservations.c      |    3 ---
 4 files changed, 8 deletions(-)

fs/ocfs2/cluster/heartbeat.c renamed from fs/ocfs2/cluster/heartbeat.c~ocfs2-remove-unused-macros +0 -2

@@ -101,8 +101,6 @@ static struct o2hb_callback {
 
 static struct o2hb_callback *hbcall_from_type(enum o2hb_callback_type type);
 
-#define O2HB_DEFAULT_BLOCK_BITS       9
-
 enum o2hb_heartbeat_modes {
 	O2HB_HEARTBEAT_LOCAL		= 0,
 	O2HB_HEARTBEAT_GLOBAL,

fs/ocfs2/dlm/dlmmaster.c renamed from fs/ocfs2/dlm/dlmmaster.c~ocfs2-remove-unused-macros +0 -2

@@ -2749,8 +2749,6 @@ leave:
 	return ret;
 }
 
-#define DLM_MIGRATION_RETRY_MS  100
-
 /*
  * Should be called only after beginning the domain leave process.
  * There should not be any remaining locks on nonlocal lock resources,

fs/ocfs2/dlm/dlmthread.c renamed from fs/ocfs2/dlm/dlmthread.c~ocfs2-remove-unused-macros +0 -1

@@ -680,7 +680,6 @@ static void dlm_flush_asts(struct dlm_ct
 
 #define DLM_THREAD_TIMEOUT_MS (4 * 1000)
 #define DLM_THREAD_MAX_DIRTY  100
-#define DLM_THREAD_MAX_ASTS   10
 
 static int dlm_thread(void *data)
 {

fs/ocfs2/reservations.c renamed from fs/ocfs2/reservations.c~ocfs2-remove-unused-macros +0 -2

@@ -33,9 +33,6 @@
 
 static DEFINE_SPINLOCK(resv_lock);
 
-#define	OCFS2_MIN_RESV_WINDOW_BITS	8
-#define	OCFS2_MAX_RESV_WINDOW_BITS	1024

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #8 lore raw

PATCH 8/155 ocfs2: use OCFS2_SEC_BITS in macro

From: Alex Shi <[email protected]>
Subject: ocfs2: use OCFS2_SEC_BITS in macro

This macro should be used.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Alex Shi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/dlmglue.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

fs/ocfs2/dlmglue.c renamed from fs/ocfs2/dlmglue.c~ocfs2-use-ocfs2_sec_bits-in-macro +1 -1

@@ -2133,7 +2133,7 @@ static void ocfs2_downconvert_on_unlock(
 }
 
 #define OCFS2_SEC_BITS   34
-#define OCFS2_SEC_SHIFT  (64 - 34)
+#define OCFS2_SEC_SHIFT  (64 - OCFS2_SEC_BITS)
 #define OCFS2_NSEC_MASK  ((1ULL << OCFS2_SEC_SHIFT) - 1)
 
 /* LVB only has room for 64 bits of time here so we pack it for

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #9 lore raw

PATCH 9/155 ocfs2: remove dlm_lock_is_remote

From: Alex Shi <[email protected]>
Subject: ocfs2: remove dlm_lock_is_remote

This macro has been unused since it was introduced.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Alex Shi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/dlm/dlmthread.c |    2 --
 1 file changed, 2 deletions(-)

fs/ocfs2/dlm/dlmthread.c renamed from fs/ocfs2/dlm/dlmthread.c~ocfs2-remove-dlm_lock_is_remote +0 -1

@@ -39,8 +39,6 @@
 static int dlm_thread(void *data);
 static void dlm_flush_asts(struct dlm_ctxt *dlm);
 
-#define dlm_lock_is_remote(dlm, lock)     ((lock)->ml.node != (dlm)->node_num)

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #10 lore raw

PATCH 10/155 ocfs2: there is no need to log twice in several functions

From: wangyan <[email protected]>
Subject: ocfs2: there is no need to log twice in several functions

There is no need to log twice in several functions.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Yan Wang <[email protected]>

Reviewed-by Jun Piao <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

---

 fs/ocfs2/alloc.c    |    1 -
 fs/ocfs2/suballoc.c |    5 -----
 2 files changed, 6 deletions(-)

fs/ocfs2/alloc.c renamed from fs/ocfs2/alloc.c~ocfs2-there-is-no-need-to-log-twice-in-several-functions +0 -1

@@ -1060,7 +1060,6 @@ bail:
 			brelse(bhs[i]);
 			bhs[i] = NULL;
 		}
-		mlog_errno(status);
 	}
 	return status;
 }

fs/ocfs2/suballoc.c renamed from fs/ocfs2/suballoc.c~ocfs2-there-is-no-need-to-log-twice-in-several-functions +0 -0

@@ -2509,9 +2509,6 @@ static int _ocfs2_free_suballoc_bits(han
 
 bail:
 	brelse(group_bh);

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #11 lore raw

PATCH 11/155 ocfs2: correct annotation from "l_next_rec" to "l_next_free_rec"

From: wangyan <[email protected]>
Subject: ocfs2: correct annotation from "l_next_rec" to "l_next_free_rec"

Correct annotation from "l_next_rec" to "l_next_free_rec"

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Yan Wang <[email protected]>

Reviewed-by Jun Piao <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

---

 fs/ocfs2/alloc.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

fs/ocfs2/alloc.c renamed from fs/ocfs2/alloc.c~ocfs2-correct-annotation-from-l_next_rec-to-l_next_free_rec +1 -1

@@ -3941,7 +3941,7 @@ rotate:
 	 * above.
 	 *
 	 * This leaf needs to have space, either by the empty 1st
-	 * extent record, or by virtue of an l_next_rec < l_count.
+	 * extent record, or by virtue of an l_next_free_rec < l_count.
 	 */
 	ocfs2_rotate_leaf(el, insert_rec);
 }

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #12 lore raw

PATCH 12/155 ocfs2: remove useless err

From: Alex Shi <[email protected]>
Subject: ocfs2: remove useless err

We don't need 'err' in these 2 places, better to remove them.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Alex Shi <[email protected]>

Reviewed-by Joseph Qi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Greg Kroah-Hartman <[email protected]>

Cc Kate Stewart <[email protected]>

Cc ChenGang <[email protected]>

Cc Richard Fontana <[email protected]>

Cc Thomas Gleixner <[email protected]>

---

 fs/ocfs2/cluster/tcp.c |    3 +--
 fs/ocfs2/dir.c         |    4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

fs/ocfs2/cluster/tcp.c renamed from fs/ocfs2/cluster/tcp.c~ocfs2-remove-useless-err +1 -2

@@ -1948,7 +1948,6 @@ static void o2net_accept_many(struct wor
 {
 	struct socket *sock = o2net_listen_sock;
 	int	more;
-	int	err;
 
 	/*
 	 * It is critical to note that due to interrupt moderation

@@ -1963,7 +1962,7 @@ static void o2net_accept_many(struct wor
 	 */
 
 	for (;;) {
-		err = o2net_accept_one(sock, &more);
+		o2net_accept_one(sock, &more);
 		if (!more)
 			break;
 		cond_resched();

fs/ocfs2/dir.c renamed from fs/ocfs2/dir.c~ocfs2-remove-useless-err +2 -2

@@ -676,7 +676,7 @@ static struct buffer_head *ocfs2_find_en
 	int ra_ptr = 0;		/* Current index into readahead
 				   buffer */
 	int num = 0;
-	int nblocks, i, err;
+	int nblocks, i;
 
 	sb = dir->i_sb;

@@ -708,7 +708,7 @@ restart:
 				num++;
 
 				bh = NULL;
-				err = ocfs2_read_dir_block(dir, b++, &bh,
+				ocfs2_read_dir_block(dir, b++, &bh,
 							   OCFS2_BH_READAHEAD);
 				bh_use[ra_max] = bh;
 			}

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #13 lore raw

PATCH 13/155 ocfs2: add missing annotations for ocfs2_refcount_cache_lock() and ocfs2_refcount_cache_unlock()

From: Jules Irenge <[email protected]>
Subject: ocfs2: add missing annotations for ocfs2_refcount_cache_lock() and ocfs2_refcount_cache_unlock()

Sparse reports warnings at ocfs2_refcount_cache_lock()
and ocfs2_refcount_cache_unlock()

warning: context imbalance in ocfs2_refcount_cache_lock()
- wrong count at exit
warning: context imbalance in ocfs2_refcount_cache_unlock()
- unexpected unlock

The root cause is the missing annotation at ocfs2_refcount_cache_lock() and at ocfs2_refcount_cache_unlock()

Add the missing __acquires(&rf->rf_lock) annotation to ocfs2_refcount_cache_lock()

Add the missing __releases(&rf->rf_lock) annotation to ocfs2_refcount_cache_unlock()

Signed-off-by Jules Irenge <[email protected]>

Link http://lkml.kernel.org/r/[email protected] open here

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/refcounttree.c |    2 ++
 1 file changed, 2 insertions(+)

fs/ocfs2/refcounttree.c renamed from fs/ocfs2/refcounttree.c~ocfs2-add-missing-annotations-for-ocfs2_refcount_cache_lock-and-ocfs2_refcount_cache_unlock +2 -0

@@ -154,6 +154,7 @@ ocfs2_refcount_cache_get_super(struct oc
 }
 
 static void ocfs2_refcount_cache_lock(struct ocfs2_caching_info *ci)
+__acquires(&rf->rf_lock)
 {
 	struct ocfs2_refcount_tree *rf = cache_info_to_refcount(ci);

@@ -161,6 +162,7 @@ static void ocfs2_refcount_cache_lock(st
 }
 
 static void ocfs2_refcount_cache_unlock(struct ocfs2_caching_info *ci)
+__releases(&rf->rf_lock)
 {
 	struct ocfs2_refcount_tree *rf = cache_info_to_refcount(ci);

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #14 lore raw

PATCH 14/155 ocfs2: replace zero-length array with flexible-array member

Link http://lkml.kernel.org/r/20200213160244.GA6088@embeddedor open here

From: "Gustavo A. R. Silva" <[email protected]>
Subject: ocfs2: replace zero-length array with flexible-array member

The current codebase makes use of the zero-length array language extension to the C90 standard, but the preferred mechanism to declare variable-length types such as these ones is a flexible array member[1][2], introduced in C99:

struct foo {
        int stuff;
        struct boo array[];
};

By making use of the mechanism above, we will get a compiler warning in case the flexible array does not occur last in the structure, which will help us prevent some kind of undefined behavior bugs from being inadvertently introduced[3] to the codebase from now on.

Also, notice that, dynamic memory allocations won't be affected by this change:

"Flexible array members have incomplete type, and so the sizeof operator may not be applied. As a quirk of the original implementation of zero-length arrays, sizeof evaluates to zero."[1]

This issue was found with the help of Coccinelle.

[1] https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html
[2] https://github.com/KSPP/linux/issues/21
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by Gustavo A. R. Silva <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/journal.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

fs/ocfs2/journal.c renamed from fs/ocfs2/journal.c~ocfs2-replace-zero-length-array-with-flexible-array-member +1 -1

@@ -91,7 +91,7 @@ enum ocfs2_replay_state {
 struct ocfs2_replay_map {
 	unsigned int rm_slots;
 	enum ocfs2_replay_state rm_state;
-	unsigned char rm_replay_slots[0];
+	unsigned char rm_replay_slots[];
 };
 
 static void ocfs2_replay_map_set_state(struct ocfs2_super *osb, int state)

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #15 lore raw

PATCH 15/155 ocfs2: cluster: replace zero-length array with flexible-array member

Link http://lkml.kernel.org/r/20200309201907.GA8005@embeddedor open here

From: "Gustavo A. R. Silva" <[email protected]>
Subject: ocfs2: cluster: replace zero-length array with flexible-array member

struct foo {
        int stuff;
        struct boo array[];
};

Also, notice that, dynamic memory allocations won't be affected by this change:

"Flexible array members have incomplete type, and so the sizeof operator may not be applied. As a quirk of the original implementation of zero-length arrays, sizeof evaluates to zero."[1]

This issue was found with the help of Coccinelle.

[1] https://urldefense.com/v3/__https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html__;!!GqivPVa7Brio!NzMr-YRl2zy-K3lwLVVatz7x0uD2z7-ykQag4GrGigxmfWU8TWzDy6xrkTiW3hYl00czlw$
[2] https://urldefense.com/v3/__https://github.com/KSPP/linux/issues/21__;!!GqivPVa7Brio!NzMr-YRl2zy-K3lwLVVatz7x0uD2z7-ykQag4GrGigxmfWU8TWzDy6xrkTiW3hYHG1nAnw$
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by Gustavo A. R. Silva <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/cluster/tcp.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

fs/ocfs2/cluster/tcp.h renamed from fs/ocfs2/cluster/tcp.h~ocfs2-cluster-replace-zero-length-array-with-flexible-array-member +1 -1

@@ -32,7 +32,7 @@ struct o2net_msg
 	__be32 status;
 	__be32 key;
 	__be32 msg_num;
-	__u8  buf[0];
+	__u8  buf[];
 };
 
 typedef int (o2net_msg_handler_func)(struct o2net_msg *msg, u32 len, void *data,

Andrew Morton

author maintainer

2020-04-02 04:03 UTC #16 lore raw

PATCH 16/155 ocfs2: dlm: replace zero-length array with flexible-array member

Link http://lkml.kernel.org/r/20200309202016.GA8210@embeddedor open here

From: "Gustavo A. R. Silva" <[email protected]>
Subject: ocfs2: dlm: replace zero-length array with flexible-array member

struct foo {
        int stuff;
        struct boo array[];
};

Also, notice that, dynamic memory allocations won't be affected by this change:

"Flexible array members have incomplete type, and so the sizeof operator may not be applied. As a quirk of the original implementation of zero-length arrays, sizeof evaluates to zero."[1]

This issue was found with the help of Coccinelle.

[1] https://urldefense.com/v3/__https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html__;!!GqivPVa7Brio!OVOYL_CouISa5L1Lw-20EEFQntw6cKMx-j8UdY4z78uYgzKBUFcfpn50GaurvbV5v7YiUA$
[2] https://urldefense.com/v3/__https://github.com/KSPP/linux/issues/21__;!!GqivPVa7Brio!OVOYL_CouISa5L1Lw-20EEFQntw6cKMx-j8UdY4z78uYgzKBUFcfpn50GaurvbXs8Eh8eg$
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by Gustavo A. R. Silva <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/dlm/dlmcommon.h |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

fs/ocfs2/dlm/dlmcommon.h renamed from fs/ocfs2/dlm/dlmcommon.h~ocfs2-dlm-replace-zero-length-array-with-flexible-array-member +4 -4

@@ -564,7 +564,7 @@ struct dlm_migratable_lockres
 	// 48 bytes
 	u8 lvb[DLM_LVB_LEN];
 	// 112 bytes
-	struct dlm_migratable_lock ml[0];  // 16 bytes each, begins at byte 112
+	struct dlm_migratable_lock ml[];  // 16 bytes each, begins at byte 112
 };
 #define DLM_MIG_LOCKRES_MAX_LEN  \
 	(sizeof(struct dlm_migratable_lockres) + \

@@ -601,7 +601,7 @@ struct dlm_convert_lock
 
 	u8 name[O2NM_MAX_NAME_LEN];
 
-	s8 lvb[0];
+	s8 lvb[];
 };
 #define DLM_CONVERT_LOCK_MAX_LEN  (sizeof(struct dlm_convert_lock)+DLM_LVB_LEN)

@@ -616,7 +616,7 @@ struct dlm_unlock_lock
 
 	u8 name[O2NM_MAX_NAME_LEN];
 
-	s8 lvb[0];
+	s8 lvb[];
 };
 #define DLM_UNLOCK_LOCK_MAX_LEN  (sizeof(struct dlm_unlock_lock)+DLM_LVB_LEN)

@@ -632,7 +632,7 @@ struct dlm_proxy_ast
 
 	u8 name[O2NM_MAX_NAME_LEN];
 
-	s8 lvb[0];
+	s8 lvb[];
 };
 #define DLM_PROXY_AST_MAX_LEN  (sizeof(struct dlm_proxy_ast)+DLM_LVB_LEN)

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #17 lore raw

PATCH 17/155 ocfs2: ocfs2_fs.h: replace zero-length array with flexible-array member

Link http://lkml.kernel.org/r/20200309202155.GA8432@embeddedor open here

From: "Gustavo A. R. Silva" <[email protected]>
Subject: ocfs2: ocfs2_fs.h: replace zero-length array with flexible-array member

struct foo {
        int stuff;
        struct boo array[];
};

Also, notice that, dynamic memory allocations won't be affected by this change:

"Flexible array members have incomplete type, and so the sizeof operator may not be applied. As a quirk of the original implementation of zero-length arrays, sizeof evaluates to zero."[1]

This issue was found with the help of Coccinelle.

[1] https://urldefense.com/v3/__https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html__;!!GqivPVa7Brio!OKPotRhYhHbCG2kibo8Q6_6CuKaa28d_74h1svxyR6rbshrK2L_BdrQpNbvJWBWb40QCkg$
[2] https://urldefense.com/v3/__https://github.com/KSPP/linux/issues/21__;!!GqivPVa7Brio!OKPotRhYhHbCG2kibo8Q6_6CuKaa28d_74h1svxyR6rbshrK2L_BdrQpNbvJWBUhNn9M6g$
[3] commit 76497732932f ("cxgb3/l2t: Fix undefined behaviour")

Signed-off-by Gustavo A. R. Silva <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/ocfs2_fs.h |   18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

fs/ocfs2/ocfs2_fs.h renamed from fs/ocfs2/ocfs2_fs.h~ocfs2-ocfs2_fsh-replace-zero-length-array-with-flexible-array-member +9 -9

@@ -470,7 +470,7 @@ struct ocfs2_extent_list {
 	__le16 l_reserved1;
 	__le64 l_reserved2;		/* Pad to
 					   sizeof(ocfs2_extent_rec) */
-/*10*/	struct ocfs2_extent_rec l_recs[0];	/* Extent records */
+/*10*/	struct ocfs2_extent_rec l_recs[];	/* Extent records */
 };
 
 /*

@@ -484,7 +484,7 @@ struct ocfs2_chain_list {
 	__le16 cl_count;		/* Total chains in this list */
 	__le16 cl_next_free_rec;	/* Next unused chain slot */
 	__le64 cl_reserved1;
-/*10*/	struct ocfs2_chain_rec cl_recs[0];	/* Chain records */
+/*10*/	struct ocfs2_chain_rec cl_recs[];	/* Chain records */
 };
 
 /*

@@ -496,7 +496,7 @@ struct ocfs2_truncate_log {
 /*00*/	__le16 tl_count;		/* Total records in this log */
 	__le16 tl_used;			/* Number of records in use */
 	__le32 tl_reserved1;
-/*08*/	struct ocfs2_truncate_rec tl_recs[0];	/* Truncate records */
+/*08*/	struct ocfs2_truncate_rec tl_recs[];	/* Truncate records */
 };
 
 /*

@@ -640,7 +640,7 @@ struct ocfs2_local_alloc
 	__le16 la_size;		/* Size of included bitmap, in bytes */
 	__le16 la_reserved1;
 	__le64 la_reserved2;
-/*10*/	__u8   la_bitmap[0];
+/*10*/	__u8   la_bitmap[];
 };
 
 /*

@@ -653,7 +653,7 @@ struct ocfs2_inline_data
 				 * for data, starting at id_data */
 	__le16	id_reserved0;
 	__le32	id_reserved1;
-	__u8	id_data[0];	/* Start of user data */
+	__u8	id_data[];	/* Start of user data */
 };
 
 /*

@@ -798,7 +798,7 @@ struct ocfs2_dx_entry_list {
 					 * possible in de_entries */
 	__le16		de_num_used;	/* Current number of
 					 * de_entries entries */
-	struct	ocfs2_dx_entry		de_entries[0];	/* Indexed dir entries
+	struct	ocfs2_dx_entry		de_entries[];	/* Indexed dir entries
 							 * in a packed array of
 							 * length de_num_used */
 };

@@ -935,7 +935,7 @@ struct ocfs2_refcount_list {
 	__le16 rl_used;		/* Current number of used records */
 	__le32 rl_reserved2;
 	__le64 rl_reserved1;	/* Pad to sizeof(ocfs2_refcount_record) */
-/*10*/	struct ocfs2_refcount_rec rl_recs[0];	/* Refcount records */
+/*10*/	struct ocfs2_refcount_rec rl_recs[];	/* Refcount records */
 };

@@ -1021,7 +1021,7 @@ struct ocfs2_xattr_header {
 						    buckets.  A block uses
 						    xb_check and sets
 						    this field to zero.) */
-	struct ocfs2_xattr_entry xh_entries[0]; /* xattr entry list. */
+	struct ocfs2_xattr_entry xh_entries[]; /* xattr entry list. */
 };
 
 /*

@@ -1207,7 +1207,7 @@ struct ocfs2_local_disk_dqinfo {
 /* Header of one chunk of a quota file */
 struct ocfs2_local_disk_chunk {
 	__le32 dqc_free;	/* Number of free entries in the bitmap */
-	__u8 dqc_bitmap[0];	/* Bitmap of entries in the corresponding
+	__u8 dqc_bitmap[];	/* Bitmap of entries in the corresponding
 				 * chunk of quota file */
 };

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #18 lore raw

PATCH 18/155 ocfs2: roll back the reference count modification of the parent directory if an error occurs

From: wangjian <[email protected]>
Subject: ocfs2: roll back the reference count modification of the parent directory if an error occurs

Under some conditions, the directory cannot be deleted. The specific scenarios are as follows: (for example, /mnt/ocfs2 is the mount point)

1. Create the /mnt/ocfs2/p_dir directory. At this time, the i_nlink corresponding to the inode of the /mnt/ocfs2/p_dir directory is equal to 2.

2. During the process of creating the /mnt/ocfs2/p_dir/s_dir directory, if the call to the inc_nlink function in ocfs2_mknod succeeds, the functions such as ocfs2_init_acl, ocfs2_init_security_set, and ocfs2_dentry_attach_lock fail. At this time, the i_nlink corresponding to the inode of the /mnt/ocfs2/p_dir directory is equal to 3, but /mnt/ocfs2/p_dir/s_dir is not added to the /mnt/ocfs2/p_dir directory entry.

3. Delete the /mnt/ocfs2/p_dir directory (rm -rf /mnt/ocfs2/p_dir). At this time, it is found that the i_nlink corresponding to the inode corresponding to the /mnt/ocfs2/p_dir directory is equal to 3. Therefore, the /mnt/ocfs2/p_dir directory cannot be deleted.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Jian wang <[email protected]>

Reviewed-by Jun Piao <[email protected]>

Reviewed-by Joseph Qi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/namei.c |   15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

fs/ocfs2/namei.c renamed from fs/ocfs2/namei.c~ocfs2-roll-back-the-reference-count-modification-of-the-parent-directory-if-an-error-occurs +11 -4

@@ -406,7 +406,7 @@ static int ocfs2_mknod(struct inode *dir
 
 	if (status < 0) {
 		mlog_errno(status);
-		goto leave;
+		goto roll_back;
 	}
 
 	if (si.enable) {

@@ -414,7 +414,7 @@ static int ocfs2_mknod(struct inode *dir
 						 meta_ac, data_ac);
 		if (status < 0) {
 			mlog_errno(status);
-			goto leave;
+			goto roll_back;
 		}
 	}

@@ -427,7 +427,7 @@ static int ocfs2_mknod(struct inode *dir
 					  OCFS2_I(dir)->ip_blkno);
 	if (status) {
 		mlog_errno(status);
-		goto leave;
+		goto roll_back;
 	}
 
 	dl = dentry->d_fsdata;

@@ -437,12 +437,19 @@ static int ocfs2_mknod(struct inode *dir
 				 &lookup);
 	if (status < 0) {
 		mlog_errno(status);
-		goto leave;
+		goto roll_back;
 	}
 
 	insert_inode_hash(inode);
 	d_instantiate(dentry, inode);
 	status = 0;
+
+roll_back:
+	if (status < 0 && S_ISDIR(mode)) {
+		ocfs2_add_links_count(dirfe, -1);
+		drop_nlink(dir);
+	}
+
 leave:
 	if (status < 0 && did_quota_inode)
 		dquot_free_inode(inode);

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #19 lore raw

PATCH 19/155 ocfs2: use scnprintf() for avoiding potential buffer overflow

From: Takashi Iwai <[email protected]>
Subject: ocfs2: use scnprintf() for avoiding potential buffer overflow

Since snprintf() returns the would-be-output size instead of the actual output size, the succeeding calls may go beyond the given buffer limit. Fix it by replacing with scnprintf().

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Takashi Iwai <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Joseph Qi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/cluster/heartbeat.c |   10 +--
 fs/ocfs2/cluster/netdebug.c  |    4 -
 fs/ocfs2/dlm/dlmdebug.c      |  100 ++++++++++++++++-----------------
 fs/ocfs2/super.c             |   46 +++++++--------
 4 files changed, 80 insertions(+), 80 deletions(-)

fs/ocfs2/cluster/heartbeat.c renamed from fs/ocfs2/cluster/heartbeat.c~ocfs2-use-scnprintf-for-avoiding-potential-buffer-overflow +5 -5

@@ -1307,7 +1307,7 @@ static int o2hb_debug_open(struct inode
 
 	case O2HB_DB_TYPE_REGION_NUMBER:
 		reg = (struct o2hb_region *)db->db_data;
-		out += snprintf(buf + out, PAGE_SIZE - out, "%d\n",
+		out += scnprintf(buf + out, PAGE_SIZE - out, "%d\n",
 				reg->hr_region_num);
 		goto done;

@@ -1317,12 +1317,12 @@ static int o2hb_debug_open(struct inode
 		/* If 0, it has never been set before */
 		if (lts)
 			lts = jiffies_to_msecs(jiffies - lts);
-		out += snprintf(buf + out, PAGE_SIZE - out, "%lu\n", lts);
+		out += scnprintf(buf + out, PAGE_SIZE - out, "%lu\n", lts);
 		goto done;
 
 	case O2HB_DB_TYPE_REGION_PINNED:
 		reg = (struct o2hb_region *)db->db_data;
-		out += snprintf(buf + out, PAGE_SIZE - out, "%u\n",
+		out += scnprintf(buf + out, PAGE_SIZE - out, "%u\n",
 				!!reg->hr_item_pinned);
 		goto done;

@@ -1331,8 +1331,8 @@ static int o2hb_debug_open(struct inode
 	}
 
 	while ((i = find_next_bit(map, db->db_len, i + 1)) < db->db_len)
-		out += snprintf(buf + out, PAGE_SIZE - out, "%d ", i);
-	out += snprintf(buf + out, PAGE_SIZE - out, "\n");
+		out += scnprintf(buf + out, PAGE_SIZE - out, "%d ", i);
+	out += scnprintf(buf + out, PAGE_SIZE - out, "\n");
 
 done:
 	i_size_write(inode, out);

fs/ocfs2/cluster/netdebug.c renamed from fs/ocfs2/cluster/netdebug.c~ocfs2-use-scnprintf-for-avoiding-potential-buffer-overflow +2 -2

@@ -443,8 +443,8 @@ static int o2net_fill_bitmap(char *buf,
 	o2net_fill_node_map(map, sizeof(map));
 
 	while ((i = find_next_bit(map, O2NM_MAX_NODES, i + 1)) < O2NM_MAX_NODES)
-		out += snprintf(buf + out, PAGE_SIZE - out, "%d ", i);
-	out += snprintf(buf + out, PAGE_SIZE - out, "\n");
+		out += scnprintf(buf + out, PAGE_SIZE - out, "%d ", i);
+	out += scnprintf(buf + out, PAGE_SIZE - out, "\n");
 
 	return out;
 }

fs/ocfs2/dlm/dlmdebug.c renamed from fs/ocfs2/dlm/dlmdebug.c~ocfs2-use-scnprintf-for-avoiding-potential-buffer-overflow +50 ~~-50~~

288 lines not on this page · raw message

fs/ocfs2/super.c renamed from fs/ocfs2/super.c~ocfs2-use-scnprintf-for-avoiding-potential-buffer-overflow +23 ~~-23~~

127 lines not on this page · raw message

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #20 lore raw

PATCH 20/155 ocfs2: use memalloc_nofs_save instead of memalloc_noio_save

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: ocfs2: use memalloc_nofs_save instead of memalloc_noio_save

OCFS2 doesn't mind if memory reclaim makes I/Os happen; it just cares that it won't be reentered, so it can use memalloc_nofs_save() instead of memalloc_noio_save().

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Reviewed-by Joseph Qi <[email protected]>

Cc Mark Fasheh <[email protected]>

Cc Joel Becker <[email protected]>

Cc Junxiao Bi <[email protected]>

Cc Changwei Ge <[email protected]>

Cc Gang He <[email protected]>

Cc Jun Piao <[email protected]>

---

 fs/ocfs2/cluster/tcp.c |   24 ++++++++++--------------
 1 file changed, 10 insertions(+), 14 deletions(-)

fs/ocfs2/cluster/tcp.c renamed from fs/ocfs2/cluster/tcp.c~ocfs2-use-memalloc_nofs_save-instead-of-memalloc_noio_save +10 ~~-14~~

@@ -1570,15 +1570,13 @@ static void o2net_start_connect(struct w
 	struct sockaddr_in myaddr = {0, }, remoteaddr = {0, };
 	int ret = 0, stop;
 	unsigned int timeout;
-	unsigned int noio_flag;
+	unsigned int nofs_flag;
 
 	/*
-	 * sock_create allocates the sock with GFP_KERNEL. We must set
-	 * per-process flag PF_MEMALLOC_NOIO so that all allocations done
-	 * by this process are done as if GFP_NOIO was specified. So we
-	 * are not reentering filesystem while doing memory reclaim.
+	 * sock_create allocates the sock with GFP_KERNEL. We must
+	 * prevent the filesystem from being reentered by memory reclaim.
 	 */
-	noio_flag = memalloc_noio_save();
+	nofs_flag = memalloc_nofs_save();
 	/* if we're greater we initiate tx, otherwise we accept */
 	if (o2nm_this_node() <= o2net_num_from_nn(nn))
 		goto out;

@@ -1683,7 +1681,7 @@ out:
 	if (mynode)
 		o2nm_node_put(mynode);
 
-	memalloc_noio_restore(noio_flag);
+	memalloc_nofs_restore(nofs_flag);
 	return;
 }

@@ -1810,15 +1808,13 @@ static int o2net_accept_one(struct socke
 	struct o2nm_node *local_node = NULL;
 	struct o2net_sock_container *sc = NULL;
 	struct o2net_node *nn;
-	unsigned int noio_flag;
+	unsigned int nofs_flag;
 
 	/*
-	 * sock_create_lite allocates the sock with GFP_KERNEL. We must set
-	 * per-process flag PF_MEMALLOC_NOIO so that all allocations done
-	 * by this process are done as if GFP_NOIO was specified. So we
-	 * are not reentering filesystem while doing memory reclaim.
+	 * sock_create_lite allocates the sock with GFP_KERNEL. We must
+	 * prevent the filesystem from being reentered by memory reclaim.
 	 */
-	noio_flag = memalloc_noio_save();
+	nofs_flag = memalloc_nofs_save();
 
 	BUG_ON(sock == NULL);
 	*more = 0;

@@ -1934,7 +1930,7 @@ out:
 	if (sc)
 		sc_put(sc);
 
-	memalloc_noio_restore(noio_flag);
+	memalloc_nofs_restore(nofs_flag);
 	return ret;
 }

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #21 lore raw

PATCH 21/155 fs_parse: remove pr_notice() about each validation

Link http://lkml.kernel.org/r/202003061617.A8835CAAF@keescook open here

From: Kees Cook <[email protected]>
Subject: fs_parse: remove pr_notice() about each validation

This notice fills my boot logs with scary-looking asterisks but doesn't really tell me anything. Let's just remove it; validation errors are already reported separately, so this is just a redundant list of filesystems.

$ dmesg | grep VALIDATE
[    0.306256] *** VALIDATE tmpfs ***
[    0.307422] *** VALIDATE proc ***
[    0.308355] *** VALIDATE cgroup ***
[    0.308741] *** VALIDATE cgroup2 ***
[    0.813256] *** VALIDATE bpf ***
[    0.815272] *** VALIDATE ramfs ***
[    0.815665] *** VALIDATE hugetlbfs ***
[    0.876970] *** VALIDATE nfs ***
[    0.877383] *** VALIDATE nfs4 ***

Signed-off-by Kees Cook <[email protected]>

Cc Alexander Viro <[email protected]>

Reviewed-by Seth Arnold <[email protected]>

---

 fs/fs_parser.c |    2 --
 1 file changed, 2 deletions(-)

fs/fs_parser.c renamed from fs/fs_parser.c~fs_parse-remove-pr_notice-about-each-validation +0 -1

@@ -368,8 +368,6 @@ bool fs_validate_description(const char
 	const struct fs_parameter_spec *param, *p2;
 	bool good = true;
 
-	pr_notice("*** VALIDATE %s ***\n", name);

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #22 lore raw

PATCH 22/155 mm/slub.c: replace cpu_slab->partial with wrapped APIs

From: chenqiwu <[email protected]>
Subject: mm/slub.c: replace cpu_slab->partial with wrapped APIs

There are slub_percpu_partial() and slub_set_percpu_partial() APIs to wrap kmem_cache->cpu_partial. This patch will use the two to replace cpu_slab->partial in slub code.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by chenqiwu <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Pekka Enberg <[email protected]>

Cc David Rientjes <[email protected]>

Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

mm/slub.c renamed from mm/slub.c~mm-slubc-replace-cpu_slab-partial-with-wrapped-apis +2 -2

@@ -2205,11 +2205,11 @@ static void unfreeze_partials(struct kme
 	struct kmem_cache_node *n = NULL, *n2 = NULL;
 	struct page *page, *discard_page = NULL;
 
-	while ((page = c->partial)) {
+	while ((page = slub_percpu_partial(c))) {
 		struct page new;
 		struct page old;
 
-		c->partial = page->next;
+		slub_set_percpu_partial(c, page);
 
 		n2 = get_node(s, page_to_nid(page));
 		if (n != n2) {

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #23 lore raw

PATCH 23/155 mm/slub.c: replace kmem_cache->cpu_partial with wrapped APIs

From: chenqiwu <[email protected]>
Subject: mm/slub.c: replace kmem_cache->cpu_partial with wrapped APIs

There are slub_cpu_partial() and slub_set_cpu_partial() APIs to wrap kmem_cache->cpu_partial. This patch will use the two APIs to replace kmem_cache->cpu_partial in slub code.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by chenqiwu <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Pekka Enberg <[email protected]>

Cc David Rientjes <[email protected]>

Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

mm/slub.c renamed from mm/slub.c~mm-slubc-replace-kmem_cache-cpu_partial-with-wrapped-apis +7 -7

@@ -2282,7 +2282,7 @@ static void put_cpu_partial(struct kmem_
 		if (oldpage) {
 			pobjects = oldpage->pobjects;
 			pages = oldpage->pages;
-			if (drain && pobjects > s->cpu_partial) {
+			if (drain && pobjects > slub_cpu_partial(s)) {
 				unsigned long flags;
 				/*
 				 * partial array is full. Move the existing

@@ -2307,7 +2307,7 @@ static void put_cpu_partial(struct kmem_
 
 	} while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page)
 								!= oldpage);
-	if (unlikely(!s->cpu_partial)) {
+	if (unlikely(!slub_cpu_partial(s))) {
 		unsigned long flags;
 
 		local_irq_save(flags);

@@ -3512,15 +3512,15 @@ static void set_cpu_partial(struct kmem_
 	 *    50% to keep some capacity around for frees.
 	 */
 	if (!kmem_cache_has_cpu_partial(s))
-		s->cpu_partial = 0;
+		slub_set_cpu_partial(s, 0);
 	else if (s->size >= PAGE_SIZE)
-		s->cpu_partial = 2;
+		slub_set_cpu_partial(s, 2);
 	else if (s->size >= 1024)
-		s->cpu_partial = 6;
+		slub_set_cpu_partial(s, 6);
 	else if (s->size >= 256)
-		s->cpu_partial = 13;
+		slub_set_cpu_partial(s, 13);
 	else
-		s->cpu_partial = 30;
+		slub_set_cpu_partial(s, 30);
 #endif
 }

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #24 lore raw

PATCH 24/155 slub: improve bit diffusion for freelist ptr obfuscation

[1] https://blog.infosectcbr.com.au/2020/03/weaknesses-in-linux-kernel-heap.html

From: Kees Cook <[email protected]>
Subject: slub: improve bit diffusion for freelist ptr obfuscation

Under CONFIG_SLAB_FREELIST_HARDENED=y, the obfuscation was relatively weak in that the ptr and ptr address were usually so close that the first XOR would result in an almost entirely 0-byte value[1], leaving most of the "secret" number ultimately being stored after the third XOR. A single blind memory content exposure of the freelist was generally sufficient to learn the secret.

Add a swab() call to mix bits a little more. This is a cheap way (1 cycle) to make attacks need more than a single exposure to learn the secret (or to know _where_ the exposure is in memory).

kmalloc-32 freelist walk, before:

ptr ptr_addr stored value secret
ffff90c22e019020@ffff90c22e019000 is 86528eb656b3b5bd (86528eb656b3b59d) ffff90c22e019040@ffff90c22e019020 is 86528eb656b3b5fd (86528eb656b3b59d) ffff90c22e019060@ffff90c22e019040 is 86528eb656b3b5bd (86528eb656b3b59d) ffff90c22e019080@ffff90c22e019060 is 86528eb656b3b57d (86528eb656b3b59d) ffff90c22e0190a0@ffff90c22e019080 is 86528eb656b3b5bd (86528eb656b3b59d)
...

after:

ptr ptr_addr stored value secret
ffff9eed6e019020@ffff9eed6e019000 is 793d1135d52cda42 (86528eb656b3b59d) ffff9eed6e019040@ffff9eed6e019020 is 593d1135d52cda22 (86528eb656b3b59d) ffff9eed6e019060@ffff9eed6e019040 is 393d1135d52cda02 (86528eb656b3b59d) ffff9eed6e019080@ffff9eed6e019060 is 193d1135d52cdae2 (86528eb656b3b59d) ffff9eed6e0190a0@ffff9eed6e019080 is f93d1135d52cdac2 (86528eb656b3b59d)

Link http://lkml.kernel.org/r/202003051623.AF4F8CB@keescook open here

Fixes 2482ddec670f ("mm: add SLUB free list pointer obfuscation")

Reported-by Silvio Cesare <[email protected]>

Signed-off-by Kees Cook <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Pekka Enberg <[email protected]>

Cc David Rientjes <[email protected]>

Cc Joonsoo Kim <[email protected]>

Cc <[email protected]>

---

 mm/slub.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/slub.c renamed from mm/slub.c~slub-improve-bit-diffusion-for-freelist-ptr-obfuscation +1 -1

@@ -259,7 +259,7 @@ static inline void *freelist_ptr(const s
 	 * freepointer to be restored incorrectly.
 	 */
 	return (void *)((unsigned long)ptr ^ s->random ^
-			(unsigned long)kasan_reset_tag((void *)ptr_addr));
+			swab((unsigned long)kasan_reset_tag((void *)ptr_addr)));
 #else
 	return ptr;
 #endif

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #25 lore raw

PATCH 25/155 slub: relocate freelist pointer to middle of object

[1] https://twitter.com/vnik5287/status/1235113523098685440

From: Kees Cook <[email protected]>
Subject: slub: relocate freelist pointer to middle of object

In a recent discussion[1] with Vitaly Nikolenko and Silvio Cesare, it became clear that moving the freelist pointer away from the edge of allocations would likely improve the overall defensive posture of the inline freelist pointer. My benchmarks show no meaningful change to performance (they seem to show it being faster), so this looks like a reasonable change to make.

Instead of having the freelist pointer at the very beginning of an allocation (offset 0) or at the very end of an allocation (effectively offset -sizeof(void *) from the next allocation), move it away from the edges of the allocation and into the middle. This provides some protection against small-sized neighboring overflows (or underflows), for which the freelist pointer is commonly the target. (Large or well controlled overwrites are much more likely to attack live object contents, instead of attempting freelist corruption.)

The vaunted kernel build benchmark, across 5 runs. Before:

	Mean: 250.05
	Std Dev: 1.85

and after, which appears mysteriously faster:

	Mean: 247.13
	Std Dev: 0.76

Attempts at running "sysbench --test=memory" show the change to be well in the noise (sysbench seems to be pretty unstable here -- it's not really measuring allocation).

Hackbench is more allocation-heavy, and while the std dev is above the difference, it looks like may manifest as an improvement as well:

20 runs of "hackbench -g 20 -l 1000", before:

	Mean: 36.322
	Std Dev: 0.577

and after:

	Mean: 36.056
	Std Dev: 0.598

Link http://lkml.kernel.org/r/202003051624.AAAC9AECC@keescook open here

Signed-off-by Kees Cook <[email protected]>

Acked-by Christoph Lameter <[email protected]>

Cc Vitaly Nikolenko <[email protected]>

Cc Silvio Cesare <[email protected]>

Cc Christoph Lameter <[email protected]>Cc: Pekka Enberg <[email protected]>

Cc David Rientjes <[email protected]>

Cc Joonsoo Kim <[email protected]>

---

 mm/slub.c |    7 +++++++
 1 file changed, 7 insertions(+)

mm/slub.c renamed from mm/slub.c~slub-relocate-freelist-pointer-to-middle-of-object +7 -0

@@ -3581,6 +3581,13 @@ static int calculate_sizes(struct kmem_c
 		 */
 		s->offset = size;
 		size += sizeof(void *);
+	} else if (size > sizeof(void *)) {
+		/*
+		 * Store freelist pointer near middle of object to keep
+		 * it away from the edges of the object to avoid small
+		 * sized over/underflows from neighboring allocations.
+		 */
+		s->offset = ALIGN(size / 2, sizeof(void *));
 	}
 
 #ifdef CONFIG_SLUB_DEBUG

Marco Elver

2020-04-15 16:47 UTC #26 lore raw

Re: [patch 025/155] slub: relocate freelist pointer to middle of object

Hello,

Quoted 57 lines · Andrew Morton

From: Kees Cook <[email protected]>
Subject: slub: relocate freelist pointer to middle of object
In a recent discussion[1] with Vitaly Nikolenko and Silvio Cesare, it became clear that moving the freelist pointer away from the edge of allocations would likely improve the overall defensive posture of the inline freelist pointer. My benchmarks show no meaningful change to performance (they seem to show it being faster), so this looks like a reasonable change to make.
Instead of having the freelist pointer at the very beginning of an allocation (offset 0) or at the very end of an allocation (effectively offset -sizeof(void *) from the next allocation), move it away from the edges of the allocation and into the middle. This provides some protection against small-sized neighboring overflows (or underflows), for which the freelist pointer is commonly the target. (Large or well controlled overwrites are much more likely to attack live object contents, instead of attempting freelist corruption.)
The vaunted kernel build benchmark, across 5 runs. Before:
	Mean: 250.05
	Std Dev: 1.85
and after, which appears mysteriously faster:
	Mean: 247.13
	Std Dev: 0.76
Attempts at running "sysbench --test=memory" show the change to be well in the noise (sysbench seems to be pretty unstable here -- it's not really measuring allocation).
Hackbench is more allocation-heavy, and while the std dev is above the difference, it looks like may manifest as an improvement as well:
20 runs of "hackbench -g 20 -l 1000", before:
	Mean: 36.322
	Std Dev: 0.577
and after:
	Mean: 36.056
	Std Dev: 0.598
[1] https://twitter.com/vnik5287/status/1235113523098685440
Link http://lkml.kernel.org/r/202003051624.AAAC9AECC@keescook open here
Signed-off-by Kees Cook <[email protected]>
Acked-by Christoph Lameter <[email protected]>
Cc Vitaly Nikolenko <[email protected]>
Cc Silvio Cesare <[email protected]>
Cc Christoph Lameter <[email protected]>Cc: Pekka Enberg <[email protected]>
Cc David Rientjes <[email protected]>
Cc Joonsoo Kim <[email protected]>
Signed-off-by Andrew Morton <[email protected]>

Quoted 5 lines

---

 mm/slub.c |    7 +++++++
 1 file changed, 7 insertions(+)

With kernel v5.7-rc1 I am unable to boot when using the SLUB allocator and red zoning (slub_debug=Z), but otherwise a default config. Bisect points to this patch, and when reverting it, the kernel boots again.

Splat:

	[...]
	[    0.328713] rcu: Hierarchical RCU implementation.
	[    0.329169] rcu:     RCU event tracing is enabled.
	[    0.329611] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8.
	[    0.330251] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
	[    0.330984] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
	[    0.332130] NR_IRQS: 4352, nr_irqs: 488, preallocated irqs: 16
	[    0.332713] general protection fault, probably for non-canonical address 0xccccccccccccccd4: 0000 [#1] SMP PTI
	[    0.333680] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.7.0-rc1+ #3
	[    0.334280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014
	[    0.335079] RIP: 0010:deactivate_slab.isra.0+0x5b/0x460
	[    0.335582] Code: 48 8b b4 c7 e0 00 00 00 49 8b 44 24 20 31 ff 48 85 c0 40 0f 95 c7 83 c7 0f 89 7c 24 18 48 85 d2 0f 84 a0 00 00 00 41 8b 4e 20 <48> 8b 3c 0b 48 85 ff 0f 84 8c 00 00 00 49 8b 54 24 28 48 89 04 0b
	[    0.337385] RSP: 0000:ffffffffb7e03c80 EFLAGS: 00010086
	[    0.337907] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 0000000000000008
	[    0.338688] RDX: cccccccccccccccc RSI: ffff91241c800f40 RDI: 000000000000000f
	[    0.339473] RBP: ffffffffb7e03d20 R08: ffff91241fc2d230 R09: 0000000000000000
	[    0.340256] R10: ffff91241c89c010 R11: 0000000000000000 R12: ffffcf2f20722700
	[    0.341041] R13: cccccccccccccccc R14: ffff91241c802180 R15: ffffcf2f20722700
	[    0.341833] FS:  0000000000000000(0000) GS:ffff91241fc00000(0000) knlGS:0000000000000000
	[    0.342727] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[    0.343359] CR2: ffff911e74e01000 CR3: 000000027460a001 CR4: 00000000000606b0
	[    0.344146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
	[    0.344929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
	[    0.345727] Call Trace:
	[    0.345999]  ? setup_object_debug.isra.0+0x1d/0x40
	[    0.346525]  ? new_slab+0x195/0x340
	[    0.346909]  ? init_object+0x2f/0x80
	[    0.347305]  ___slab_alloc+0x526/0x570
	[    0.347717]  ? kasprintf+0x4e/0x70
	[    0.348092]  ? init_object+0x2f/0x80
	[    0.348488]  ? string+0x42/0x50
	[    0.348834]  ? kasprintf+0x4e/0x70
	[    0.349217]  __kmalloc_track_caller+0x1d2/0x200
	[    0.349720]  kvasprintf+0x64/0xc0
	[    0.350085]  kasprintf+0x4e/0x70
	[    0.350442]  ? kmem_cache_alloc_trace+0x188/0x1b0
	[    0.350962]  __irq_domain_alloc_fwnode+0x8f/0xd0
	[    0.351474]  arch_early_irq_init+0x16/0x90
	[    0.351923]  start_kernel+0x2aa/0x4c2
	[    0.352325]  secondary_startup_64+0xb6/0xc0
	[    0.352784] Modules linked in:
	[    0.353124] random: get_random_bytes called from print_oops_end_marker+0x21/0x40 with crng_init=0
	[    0.353126] ---[ end trace 186486c23e10986d ]---
	[    0.354613] RIP: 0010:deactivate_slab.isra.0+0x5b/0x460
	[    0.355186] Code: 48 8b b4 c7 e0 00 00 00 49 8b 44 24 20 31 ff 48 85 c0 40 0f 95 c7 83 c7 0f 89 7c 24 18 48 85 d2 0f 84 a0 00 00 00 41 8b 4e 20 <48> 8b 3c 0b 48 85 ff 0f 84 8c 00 00 00 49 8b 54 24 28 48 89 04 0b
	[    0.357255] RSP: 0000:ffffffffb7e03c80 EFLAGS: 00010086
	[    0.357829] RAX: 0000000000000000 RBX: cccccccccccccccc RCX: 0000000000000008
	[    0.358613] RDX: cccccccccccccccc RSI: ffff91241c800f40 RDI: 000000000000000f
	[    0.359398] RBP: ffffffffb7e03d20 R08: ffff91241fc2d230 R09: 0000000000000000
	[    0.360181] R10: ffff91241c89c010 R11: 0000000000000000 R12: ffffcf2f20722700
	[    0.360965] R13: cccccccccccccccc R14: ffff91241c802180 R15: ffffcf2f20722700
	[    0.361755] FS:  0000000000000000(0000) GS:ffff91241fc00000(0000) knlGS:0000000000000000
	[    0.362645] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[    0.363275] CR2: ffff911e74e01000 CR3: 000000027460a001 CR4: 00000000000606b0
	[    0.364060] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
	[    0.364844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
	[    0.365636] Kernel panic - not syncing: Attempted to kill the idle task!
	[    0.366393] ---[ end Kernel panic - not syncing: Attempted to kill the idle task! ]---

Can you reproduce this? Let me know if you need more information.

Thanks,
-- Marco

Quoted 17 lines

mm/slub.c renamed from mm/slub.c~slub-relocate-freelist-pointer-to-middle-of-object +7 -0

@@ -3581,6 +3581,13 @@ static int calculate_sizes(struct kmem_c
 		 */
 		s->offset = size;
 		size += sizeof(void *);
+	} else if (size > sizeof(void *)) {
+		/*
+		 * Store freelist pointer near middle of object to keep
+		 * it away from the edges of the object to avoid small
+		 * sized over/underflows from neighboring allocations.
+		 */
+		s->offset = ALIGN(size / 2, sizeof(void *));
 	}
 
 #ifdef CONFIG_SLUB_DEBUG

Kees Cook

2020-04-15 17:08 UTC #27 lore raw

Re: [patch 025/155] slub: relocate freelist pointer to middle of object

Quoted 10 lines · Marco Elver

Quoted 3 lines · Andrew Morton
From: Kees Cook <[email protected]>
Subject: slub: relocate freelist pointer to middle of object [...]
With kernel v5.7-rc1 I am unable to boot when using the SLUB allocator and red zoning (slub_debug=Z), but otherwise a default config. Bisect points to this patch, and when reverting it, the kernel boots again.
Splat:

Quoted 11 lines

	[...]
	[    0.328713] rcu: Hierarchical RCU implementation.
	[    0.329169] rcu:     RCU event tracing is enabled.
	[    0.329611] rcu:     RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=8.
	[    0.330251] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
	[    0.330984] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
	[    0.332130] NR_IRQS: 4352, nr_irqs: 488, preallocated irqs: 16
	[    0.332713] general protection fault, probably for non-canonical address 0xccccccccccccccd4: 0000 [#1] SMP PTI
	[    0.333680] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.7.0-rc1+ #3
	[    0.334280] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014
	[    0.335079] RIP: 0010:deactivate_slab.isra.0+0x5b/0x460

Thanks for the report! It seems something isn't using get_freepointer() (and is missing the s->offset calculation). I will try to track it down...

Quoted 1 line

Can you reproduce this? Let me know if you need more information.

Yup! I see a crash in the same place with slub_debug=Z. Since I'm building with CONFIG_SLAB_FREELIST_HARDENED=y, I see a random number instead of 0xccccccccccccccd4. I'll keep digging...

Signature · 1 line

Kees Cook

Kees Cook

2020-04-15 18:00 UTC #28 lore raw

Re: [patch 025/155] slub: relocate freelist pointer to middle of object

In reply to Marco Elver

Quoted 3 lines · Marco Elver

With kernel v5.7-rc1 I am unable to boot when using the SLUB allocator and red zoning (slub_debug=Z), but otherwise a default config. Bisect points to this patch, and when reverting it, the kernel boots again.

Okay, I believe this patch should fix the problem:
https://lore.kernel.org/lkml/202004151054.BD695840@keescook

Signature · 1 line

Kees Cook

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #29 lore raw

PATCH 26/155 revert "topology: add support for node_to_mem_node() to determine the fallback node"

From: Vlastimil Babka <[email protected]>
Subject: revert "topology: add support for node_to_mem_node() to determine the fallback node"

This reverts commit ad2c8144418c6a81cefe65379fd47bbe8344cef2.

The function node_to_mem_node() was introduced by that commit for use in SLUB on systems with memoryless nodes, but it turned out to be unreliable on some architectures/configurations and a simpler solution exists than fixing it up.

Thus commit 0715e6c516f1 ("mm, slub: prevent kmalloc_node crashes and memory leaks") removed the only user of node_to_mem_node() and we can revert the commit that introduced the function.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Vlastimil Babka <[email protected]>

Reviewed-by Srikar Dronamraju <[email protected]>

Cc Joonsoo Kim <[email protected]>

Cc Bharata B Rao <[email protected]>

Cc Christopher Lameter <[email protected]>

Cc David Rientjes <[email protected]>

Cc Kirill Tkhai <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Michael Ellerman <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Nathan Lynch <[email protected]>

Cc Pekka Enberg <[email protected]>

Cc PUVICHAKRAVARTHY RAMACHANDRAN <[email protected]>

Cc Sachin Sant <[email protected]>

---

 include/linux/topology.h |   17 -----------------
 mm/page_alloc.c          |    1 -
 2 files changed, 18 deletions(-)

include/linux/topology.h renamed from include/linux/topology.h~revert-topology-add-support-for-node_to_mem_node-to-determine-the-fallback-node +0 ~~-17~~

@@ -130,20 +130,11 @@ static inline int numa_node_id(void)
  * Use the accessor functions set_numa_mem(), numa_mem_id() and cpu_to_mem().
  */
 DECLARE_PER_CPU(int, _numa_mem_);
-extern int _node_numa_mem_[MAX_NUMNODES];
 
 #ifndef set_numa_mem
 static inline void set_numa_mem(int node)
 {
 	this_cpu_write(_numa_mem_, node);
-	_node_numa_mem_[numa_node_id()] = node;
-}
-#endif
-
-#ifndef node_to_mem_node
-static inline int node_to_mem_node(int node)
-{
-	return _node_numa_mem_[node];
 }
 #endif

@@ -166,7 +157,6 @@ static inline int cpu_to_mem(int cpu)
 static inline void set_cpu_numa_mem(int cpu, int node)
 {
 	per_cpu(_numa_mem_, cpu) = node;
-	_node_numa_mem_[cpu_to_node(cpu)] = node;
 }
 #endif

@@ -180,13 +170,6 @@ static inline int numa_mem_id(void)
 }
 #endif
 
-#ifndef node_to_mem_node
-static inline int node_to_mem_node(int node)
-{
-	return node;
-}
-#endif
-
 #ifndef cpu_to_mem
 static inline int cpu_to_mem(int cpu)
 {

mm/page_alloc.c renamed from mm/page_alloc.c~revert-topology-add-support-for-node_to_mem_node-to-determine-the-fallback-node +0 -1

@@ -95,7 +95,6 @@ DEFINE_STATIC_KEY_TRUE(vm_numa_stat_key)
  */
 DEFINE_PER_CPU(int, _numa_mem_);		/* Kernel "local memory" node */
 EXPORT_PER_CPU_SYMBOL(_numa_mem_);
-int _node_numa_mem_[MAX_NUMNODES];
 #endif
 
 /* work_structs for global per-cpu drains */

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #30 lore raw

PATCH 27/155 mm/kmemleak.c: use address-of operator on section symbols

Link https://github.com/ClangBuiltLinux/linux/issues/895

From: Nathan Chancellor <[email protected]>
Subject: mm/kmemleak.c: use address-of operator on section symbols

Clang warns:

../mm/kmemleak.c:1955:28: warning: array comparison always evaluates to a constant [-Wtautological-compare]

        if (__start_ro_after_init < _sdata || __end_ro_after_init > _edata)
                                  ^
../mm/kmemleak.c:1955:60: warning: array comparison always evaluates to a constant [-Wtautological-compare]
        if (__start_ro_after_init < _sdata || __end_ro_after_init > _edata)

These are not true arrays, they are linker defined symbols, which are just addresses. Using the address of operator silences the warning and does not change the resulting assembly with either clang/ld.lld or gcc/ld (tested with diff + objdump -Dr).

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Nathan Chancellor <[email protected]>

Acked-by Catalin Marinas <[email protected]>

---

 mm/kmemleak.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/kmemleak.c renamed from mm/kmemleak.c~mm-kmemleak-use-address-of-operator-on-section-symbols +1 -1

@@ -1947,7 +1947,7 @@ void __init kmemleak_init(void)
 	create_object((unsigned long)__bss_start, __bss_stop - __bss_start,
 		      KMEMLEAK_GREY, GFP_ATOMIC);
 	/* only register .data..ro_after_init if not within .data */
-	if (__start_ro_after_init < _sdata || __end_ro_after_init > _edata)
+	if (&__start_ro_after_init < &_sdata || &__end_ro_after_init > &_edata)
 		create_object((unsigned long)__start_ro_after_init,
 			      __end_ro_after_init - __start_ro_after_init,
 			      KMEMLEAK_GREY, GFP_ATOMIC);

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #31 lore raw

PATCH 28/155 mm/Makefile: disable KCSAN for kmemleak

From: Qian Cai <[email protected]>
Subject: mm/Makefile: disable KCSAN for kmemleak

Kmemleak could scan task stacks while plain writes happens to those stack variables which could results in data races. For example, in sys_rt_sigaction and do_sigaction(), it could have plain writes in a 32-byte size. Since the kmemleak does not care about the actual values of a non-pointer and all do_sigaction() call sites only copy to stack variables, just disable KCSAN for kmemleak to avoid annotating anything outside Kmemleak just because Kmemleak scans everything.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Qian Cai <[email protected]>

Acked-by Marco Elver <[email protected]>

Acked-by Catalin Marinas <[email protected]>

---

 mm/Makefile |    1 +
 1 file changed, 1 insertion(+)

mm/Makefile renamed from mm/Makefile~mm-disable-kcsan-for-kmemleak +1 -0

@@ -6,6 +6,7 @@
 KASAN_SANITIZE_slab_common.o := n
 KASAN_SANITIZE_slab.o := n
 KASAN_SANITIZE_slub.o := n
+KCSAN_SANITIZE_kmemleak.o := n
 
 # These files are disabled because they produce non-interesting and/or
 # flaky coverage that is not a function of syscall inputs. E.g. slab is out of

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #32 lore raw

PATCH 29/155 mm/filemap.c: don't bother dropping mmap_sem for zero size readahead

From: Jan Kara <[email protected]>
Subject: mm/filemap.c: don't bother dropping mmap_sem for zero size readahead

When handling a page fault, we drop mmap_sem to start async readahead so that we don't block on IO submission with mmap_sem held. However there's no point to drop mmap_sem in case readahead is disabled. Handle that case to avoid pointless dropping of mmap_sem and retrying the fault. This was actually reported to block mlockall(MCL_CURRENT) indefinitely.

Link http://lkml.kernel.org/r/[email protected] open here

Fixes 6b4c9f446981 ("filemap: drop the mmap_sem for all blocking operations")

Signed-off-by Jan Kara <[email protected]>

Reported-by Minchan Kim <[email protected]>

Reported-by Robert Stupp <[email protected]>

Reviewed-by Josef Bacik <[email protected]>

Reviewed-by Minchan Kim <[email protected]>

---

 mm/filemap.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/filemap.c renamed from mm/filemap.c~mm-dont-bother-dropping-mmap_sem-for-zero-size-readahead +1 -1

@@ -2416,7 +2416,7 @@ static struct file *do_async_mmap_readah
 	pgoff_t offset = vmf->pgoff;
 
 	/* If we don't want any read-ahead, don't bother */
-	if (vmf->vma->vm_flags & VM_RAND_READ)
+	if (vmf->vma->vm_flags & VM_RAND_READ || !ra->ra_pages)
 		return fpin;
 	if (ra->mmap_miss > 0)
 		ra->mmap_miss--;

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #33 lore raw

PATCH 30/155 mm/page-writeback.c: write_cache_pages(): deduplicate identical checks

From: Mauricio Faria de Oliveira <[email protected]>
Subject: mm/page-writeback.c: write_cache_pages(): deduplicate identical checks

There used to be a 'retry' label in between the two (identical) checks when first introduced in commit f446daaea9d4 ("mm: implement writeback livelock avoidance using page tagging"), and later modified/updated in commit 6e6938b6d313 ("writeback: introduce .tagged_writepages for the WB_SYNC_NONE sync stage").

The label has been removed in commit 64081362e8ff ("mm/page-writeback.c:
fix range_cyclic writeback vs writepages deadlock"), and the (identical) checks are now present / performed immediately one after another.

So, remove/deduplicate the latter check, moving tag_pages_for_writeback() into the former check before the 'tag' variable assignment, so it's clear that it's not used in this (similarly-named) function call but only later in pagevec_lookup_range_tag().

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Mauricio Faria de Oliveira <[email protected]>

Cc Jan Kara <[email protected]>

---

 mm/page-writeback.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

mm/page-writeback.c renamed from mm/page-writeback.c~mm-page-writebackc-write_cache_pages-deduplicate-identical-checks +4 -4

@@ -2182,12 +2182,12 @@ int write_cache_pages(struct address_spa
 		if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 			range_whole = 1;
 	}
-	if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages)
+	if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages) {
+		tag_pages_for_writeback(mapping, index, end);
 		tag = PAGECACHE_TAG_TOWRITE;
-	else
+	} else {
 		tag = PAGECACHE_TAG_DIRTY;
-	if (wbc->sync_mode == WB_SYNC_ALL || wbc->tagged_writepages)
-		tag_pages_for_writeback(mapping, index, end);
+	}
 	done_index = index;
 	while (!done && (index <= end)) {
 		int i;

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #34 lore raw

PATCH 31/155 mm/filemap.c: clear page error before actual read

From: Xianting Tian <[email protected]>
Subject: mm/filemap.c: clear page error before actual read

Mount failure issue happens under the scenario: Application forked dozens of threads to mount the same number of cramfs images separately in docker, but several mounts failed with high probability. Mount failed due to the checking result of the page(read from the superblock of loop dev) is not uptodate after wait_on_page_locked(page) returned in function cramfs_read:

   wait_on_page_locked(page);
   if (!PageUptodate(page)) {
      ...
   }

The reason of the checking result of the page not uptodate: systemd-udevd read the loopX dev before mount, because the status of loopX is Lo_unbound at this time, so loop_make_request directly trigger the calling of io_end handler end_buffer_async_read, which called SetPageError(page). So It caused the page can't be set to uptodate in function end_buffer_async_read:

   if(page_uptodate && !PageError(page)) {
      SetPageUptodate(page);
   }

Then mount operation is performed, it used the same page which is just accessed by systemd-udevd above, Because this page is not uptodate, it will launch a actual read via submit_bh, then wait on this page by calling wait_on_page_locked(page). When the I/O of the page done, io_end handler end_buffer_async_read is called, because no one cleared the page error(during the whole read path of mount), which is caused by systemd-udevd reading, so this page is still in "PageError" status, which can't be set to uptodate in function end_buffer_async_read, then caused mount failure.

But sometimes mount succeed even through systemd-udeved read loopX dev just before, The reason is systemd-udevd launched other loopX read just between step 3.1 and 3.2, the steps as below:

1, loopX dev default status is Lo_unbound;
2, systemd-udved read loopX dev (page is set to PageError); 3, mount operation
1) set loopX status to Lo_bound; ==>systemd-udevd read loopX dev<==
2) read loopX dev(page has no error)
3) mount succeed

As the loopX dev status is set to Lo_bound after step 3.1, so the other loopX dev read by systemd-udevd will go through the whole I/O stack, part of the call trace as below:

   SYS_read
      vfs_read
          do_sync_read
              blkdev_aio_read
                 generic_file_aio_read
                     do_generic_file_read:
                        ClearPageError(page);
                        mapping->a_ops->readpage(filp, page);

here, mapping->a_ops->readpage() is blkdev_readpage. In latest kernel, some function name changed, the call trace as below:

blkdev_read_iter

      generic_file_read_iter
         generic_file_buffered_read:
            /*
             * A previous I/O error may have been due to temporary
             * failures, eg. mutipath errors.
             * Pg_error will be set again if readpage fails.
             */
            ClearPageError(page);
            /* Start the actual read. The read will unlock the page*/
            error=mapping->a_ops->readpage(flip, page);

We can see ClearPageError(page) is called before the actual read, then the read in step 3.2 succeed.

This patch is to add the calling of ClearPageError just before the actual read of read path of cramfs mount. Without the patch, the call trace as below when performing cramfs mount:

   do_mount
      cramfs_read
         cramfs_blkdev_read
            read_cache_page
               do_read_cache_page:
                  filler(data, page);
                  or
                  mapping->a_ops->readpage(data, page);

With the patch, the call trace as below when performing mount:

   do_mount
      cramfs_read
         cramfs_blkdev_read
            read_cache_page:
               do_read_cache_page:
                  ClearPageError(page); <== new add
                  filler(data, page);
                  or
                  mapping->a_ops->readpage(data, page);

With the patch, mount operation trigger the calling of ClearPageError(page) before the actual read, the page has no error if no additional page error happen when I/O done.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Xianting Tian <[email protected]>

Reviewed-by Matthew Wilcox (Oracle) <[email protected]>

Cc Jan Kara <[email protected]>

Cc <[email protected]>

---

 mm/filemap.c |    8 ++++++++
 1 file changed, 8 insertions(+)

mm/filemap.c renamed from mm/filemap.c~mm-filemapc-clear-page-error-before-actual-read +8 -0

@@ -2823,6 +2823,14 @@ filler:
 		unlock_page(page);
 		goto out;
 	}
+
+	/*
+	 * A previous I/O error may have been due to temporary
+	 * failures.
+	 * Clear page error before actual read, PG_error will be
+	 * set again if read page fails.
+	 */
+	ClearPageError(page);
 	goto filler;
 
 out:

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #35 lore raw

PATCH 32/155 mm/filemap.c: remove unused argument from shrink_readahead_size_eio()

From: Souptick Joarder <[email protected]>
Subject: mm/filemap.c: remove unused argument from shrink_readahead_size_eio()

The first argument of shrink_readahead_size_eio() is not used. Hence remove it from the function definition and from all the callers.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Souptick Joarder <[email protected]>

---

 mm/filemap.c |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

mm/filemap.c renamed from mm/filemap.c~mm-filemapc-remove-unused-argument-from-shrink_readahead_size_eio +3 -4

@@ -1962,8 +1962,7 @@ EXPORT_SYMBOL(find_get_pages_range_tag);
  *
  * It is going insane. Fix it by quickly scaling down the readahead size.
  */
-static void shrink_readahead_size_eio(struct file *filp,
-					struct file_ra_state *ra)
+static void shrink_readahead_size_eio(struct file_ra_state *ra)
 {
 	ra->ra_pages /= 4;
 }

@@ -2188,7 +2187,7 @@ readpage:
 					goto find_page;
 				}
 				unlock_page(page);
-				shrink_readahead_size_eio(filp, ra);
+				shrink_readahead_size_eio(ra);
 				error = -EIO;
 				goto readpage_error;
 			}

@@ -2560,7 +2559,7 @@ page_not_uptodate:
 		goto retry_find;
 
 	/* Things didn't work out. Return zero to tell the mm layer so. */
-	shrink_readahead_size_eio(file, ra);
+	shrink_readahead_size_eio(ra);
 	return VM_FAULT_SIGBUS;
 
 out_retry:

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #36 lore raw

PATCH 33/155 mm/filemap.c: use vm_fault error code directly

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/filemap.c: use vm_fault error code directly

Use VM_FAULT_OOM instead of indirecting through vmf_error(-ENOMEM).

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Cc Aneesh Kumar K.V <[email protected]>

Cc Pankaj Gupta <[email protected]>

---

 mm/filemap.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/filemap.c renamed from mm/filemap.c~mm-use-vm_fault-error-code-directly +1 -1

@@ -2490,7 +2490,7 @@ retry_find:
 		if (!page) {
 			if (fpin)
 				goto out_retry;
-			return vmf_error(-ENOMEM);
+			return VM_FAULT_OOM;
 		}
 	}

Andrew Morton

author maintainer

2020-04-02 04:04 UTC #37 lore raw

PATCH 34/155 include/linux/pagemap.h: rename arguments to find_subpage

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: include/linux/pagemap.h: rename arguments to find_subpage

This isn't just a random struct page, it's known to be a head page, and calling it head makes the function better self-documenting. The pgoff_t is less confusing if it's named index instead of offset. Also add a couple of comments to explain why we're doing various things.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Acked-by Pankaj Gupta <[email protected]>

Cc Aneesh Kumar K.V <[email protected]>

---

 include/linux/pagemap.h |   15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

include/linux/pagemap.h renamed from include/linux/pagemap.h~mm-rename-arguments-to-find_subpage +10 -5

@@ -333,14 +333,19 @@ static inline struct page *grab_cache_pa
 			mapping_gfp_mask(mapping));
 }
 
-static inline struct page *find_subpage(struct page *page, pgoff_t offset)
+/*
+ * Given the page we found in the page cache, return the page corresponding
+ * to this index in the file
+ */
+static inline struct page *find_subpage(struct page *head, pgoff_t index)
 {
-	if (PageHuge(page))
-		return page;
+	/* HugeTLBfs wants the head page regardless */
+	if (PageHuge(head))
+		return head;
 
-	VM_BUG_ON_PAGE(PageTail(page), page);
+	VM_BUG_ON_PAGE(PageTail(head), head);
 
-	return page + (offset & (compound_nr(page) - 1));
+	return head + (index & (compound_nr(head) - 1));
 }
 
 struct page *find_get_entry(struct address_space *mapping, pgoff_t offset);

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #38 lore raw

PATCH 35/155 mm/page-writeback.c: use VM_BUG_ON_PAGE in clear_page_dirty_for_io

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/page-writeback.c: use VM_BUG_ON_PAGE in clear_page_dirty_for_io

Dumping the page information in this circumstance helps for debugging.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Cc Aneesh Kumar K.V <[email protected]>

Cc Pankaj Gupta <[email protected]>

---

 mm/page-writeback.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/page-writeback.c renamed from mm/page-writeback.c~mm-use-vm_bug_on_page-in-clear_page_dirty_for_io +1 -1

@@ -2655,7 +2655,7 @@ int clear_page_dirty_for_io(struct page
 	struct address_space *mapping = page_mapping(page);
 	int ret = 0;
 
-	BUG_ON(!PageLocked(page));
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
 
 	if (mapping && mapping_cap_account_dirty(mapping)) {
 		struct inode *inode = mapping->host;

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #39 lore raw

PATCH 36/155 mm/filemap.c: unexport find_get_entry

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/filemap.c: unexport find_get_entry

No in-tree users (proc, madvise, memcg, mincore) can be built as a module.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Cc Aneesh Kumar K.V <[email protected]>

Cc Pankaj Gupta <[email protected]>

---

 mm/filemap.c |    1 -
 1 file changed, 1 deletion(-)

mm/filemap.c renamed from mm/filemap.c~mm-unexport-find_get_entry +0 -1

@@ -1536,7 +1536,6 @@ out:
 
 	return page;
 }
-EXPORT_SYMBOL(find_get_entry);
 
 /**
  * find_lock_entry - locate, pin and lock a page cache entry

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #40 lore raw

PATCH 37/155 mm/filemap.c: rewrite pagecache_get_page documentation

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm/filemap.c: rewrite pagecache_get_page documentation

- These were never called PCG flags; they've been called FGP flags since their introduction in 2014.
- The FGP_FOR_MMAP flag was misleadingly documented as if it was an alternative to FGP_CREAT instead of an option to it.
- Rename the 'offset' parameter to 'index'.
- Capitalisation, formatting, rewording.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Cc Aneesh Kumar K.V <[email protected]>

Cc Pankaj Gupta <[email protected]>

---

 mm/filemap.c |   55 +++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 29 deletions(-)

mm/filemap.c renamed from mm/filemap.c~mm-rewrite-pagecache_get_page-documentation +26 ~~-29~~

@@ -1574,42 +1574,39 @@ repeat:
 EXPORT_SYMBOL(find_lock_entry);
 
 /**
- * pagecache_get_page - find and get a page reference
- * @mapping: the address_space to search
- * @offset: the page index
- * @fgp_flags: PCG flags
- * @gfp_mask: gfp mask to use for the page cache data page allocation
- *
- * Looks up the page cache slot at @mapping & @offset.
- *
- * PCG flags modify how the page is returned.
- *
- * @fgp_flags can be:
- *
- * - FGP_ACCESSED: the page will be marked accessed
- * - FGP_LOCK: Page is return locked
- * - FGP_CREAT: If page is not present then a new page is allocated using
- *   @gfp_mask and added to the page cache and the VM's LRU
- *   list. The page is returned locked and with an increased
- *   refcount.
- * - FGP_FOR_MMAP: Similar to FGP_CREAT, only we want to allow the caller to do
- *   its own locking dance if the page is already in cache, or unlock the page
- *   before returning if we had to add the page to pagecache.
+ * pagecache_get_page - Find and get a reference to a page.
+ * @mapping: The address_space to search.
+ * @index: The page index.
+ * @fgp_flags: %FGP flags modify how the page is returned.
+ * @gfp_mask: Memory allocation flags to use if %FGP_CREAT is specified.
+ *
+ * Looks up the page cache entry at @mapping & @index.
+ *
+ * @fgp_flags can be zero or more of these flags:
+ *
+ * * %FGP_ACCESSED - The page will be marked accessed.
+ * * %FGP_LOCK - The page is returned locked.
+ * * %FGP_CREAT - If no page is present then a new page is allocated using
+ *   @gfp_mask and added to the page cache and the VM's LRU list.
+ *   The page is returned locked and with an increased refcount.
+ * * %FGP_FOR_MMAP - The caller wants to do its own locking dance if the
+ *   page is already in cache.  If the page was allocated, unlock it before
+ *   returning so the caller can do the same dance.
  *
- * If FGP_LOCK or FGP_CREAT are specified then the function may sleep even
- * if the GFP flags specified for FGP_CREAT are atomic.
+ * If %FGP_LOCK or %FGP_CREAT are specified then the function may sleep even
+ * if the %GFP flags specified for %FGP_CREAT are atomic.
  *
  * If there is a page cache page, it is returned with an increased refcount.
  *
- * Return: the found page or %NULL otherwise.
+ * Return: The found page or %NULL otherwise.
  */
-struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
-	int fgp_flags, gfp_t gfp_mask)
+struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index,
+		int fgp_flags, gfp_t gfp_mask)
 {
 	struct page *page;
 
 repeat:
-	page = find_get_entry(mapping, offset);
+	page = find_get_entry(mapping, index);
 	if (xa_is_value(page))
 		page = NULL;
 	if (!page)

@@ -1631,7 +1628,7 @@ repeat:
 			put_page(page);
 			goto repeat;
 		}
-		VM_BUG_ON_PAGE(page->index != offset, page);
+		VM_BUG_ON_PAGE(page->index != index, page);
 	}
 
 	if (fgp_flags & FGP_ACCESSED)

@@ -1656,7 +1653,7 @@ no_page:
 		if (fgp_flags & FGP_ACCESSED)
 			__SetPageReferenced(page);
 
-		err = add_to_page_cache_lru(page, mapping, offset, gfp_mask);
+		err = add_to_page_cache_lru(page, mapping, index, gfp_mask);
 		if (unlikely(err)) {
 			put_page(page);
 			page = NULL;

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #41 lore raw

PATCH 38/155 mm/gup: split get_user_pages_remote() into two routines

From: John Hubbard <[email protected]>
Subject: mm/gup: split get_user_pages_remote() into two routines

Patch series "mm/gup: track FOLL_PIN pages", v6.

This activates tracking of FOLL_PIN pages. This is in support of fixing the get_user_pages()+DMA problem described in [1]-[4].

FOLL_PIN support is now in the main linux tree. However, the patch to use FOLL_PIN to track pages was *not* submitted, because Leon saw an RDMA test suite failure that involved (I think) page refcount overflows when huge pages were used.

This patch definitively solves that kind of overflow problem, by adding an exact pincount, for compound pages (of order > 1), in the 3rd struct page of a compound page. If available, that form of pincounting is used, instead of the GUP_PIN_COUNTING_BIAS approach. Thanks again to Jan Kara for that idea.

Other interesting changes:

* dump_page(): added one, or two new things to report for compound
pages: head refcount (for all compound pages), and map_pincount (for compound pages of order > 1).

* Documentation/core-api/pin_user_pages.rst: removed the "TODO" for the huge page refcount upper limit problems, and added notes about how it works now. Also added a note about the dump_page() enhancements.

* Added some comments in gup.c and mm.h, to explain that there are two ways to count pinned pages: exact (for compound pages of order > 1) and fuzzy (GUP_PIN_COUNTING_BIAS: for all other pages).

============================================================
General notes about the tracking patch:

This is a prerequisite to solving the problem of proper interactions between file-backed pages, and [R]DMA activities, as discussed in [1], [2], [3], [4] and in a remarkable number of email threads since about 2017. :)

In contrast to earlier approaches, the page tracking can be incrementally applied to the kernel call sites that, until now, have been simply calling get_user_pages() ("gup"). In other words, opt-in by changing from this:

get_user_pages() (sets FOLL_GET)
put_page()

to this:
pin_user_pages() (sets FOLL_PIN)
unpin_user_page()

============================================================
Future steps:

* Convert more subsystems from get_user_pages() to pin_user_pages(). The first probably needs to be bio/biovecs, because any filesystem testing is too difficult without those in place.

* Change VFS and filesystems to respond appropriately when encountering dma-pinned pages.

* Work with Ira and others to connect this all up with file system leases.

[1] Some slow progress on get_user_pages() (Apr 2, 2019):

    https://lwn.net/Articles/784574/

[2] DMA and get_user_pages() (LPC: Dec 12, 2018):

    https://lwn.net/Articles/774411/

[3] The trouble with get_user_pages() (Apr 30, 2018):

    https://lwn.net/Articles/753027/

[4] LWN kernel index: get_user_pages()
https://lwn.net/Kernel/Index/#Memory_management-get_user_pages

This patch (of 12):

An upcoming patch requires reusing the implementation of get_user_pages_remote(). Split up get_user_pages_remote() into an outer routine that checks flags, and an implementation routine that will be reused. This makes subsequent changes much easier to understand.

There should be no change in behavior due to this patch.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/gup.c |   56 +++++++++++++++++++++++++++++++----------------------
 1 file changed, 33 insertions(+), 23 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-split-get_user_pages_remote-into-two-routines +31 ~~-19~~

@@ -1557,6 +1557,37 @@ static __always_inline long __gup_longte
 }
 #endif /* CONFIG_FS_DAX || CONFIG_CMA */
 
+#ifdef CONFIG_MMU
+static long __get_user_pages_remote(struct task_struct *tsk,
+				    struct mm_struct *mm,
+				    unsigned long start, unsigned long nr_pages,
+				    unsigned int gup_flags, struct page **pages,
+				    struct vm_area_struct **vmas, int *locked)
+{
+	/*
+	 * Parts of FOLL_LONGTERM behavior are incompatible with
+	 * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on
+	 * vmas. However, this only comes up if locked is set, and there are
+	 * callers that do request FOLL_LONGTERM, but do not set locked. So,
+	 * allow what we can.
+	 */
+	if (gup_flags & FOLL_LONGTERM) {
+		if (WARN_ON_ONCE(locked))
+			return -EINVAL;
+		/*
+		 * This will check the vmas (even if our vmas arg is NULL)
+		 * and return -ENOTSUPP if DAX isn't allowed in this case:
+		 */
+		return __gup_longterm_locked(tsk, mm, start, nr_pages, pages,
+					     vmas, gup_flags | FOLL_TOUCH |
+					     FOLL_REMOTE);
+	}
+
+	return __get_user_pages_locked(tsk, mm, start, nr_pages, pages, vmas,
+				       locked,
+				       gup_flags | FOLL_TOUCH | FOLL_REMOTE);
+}
+
 /*
  * get_user_pages_remote() - pin user pages in memory
  * @tsk:	the task_struct to use for page fault accounting, or

@@ -1619,7 +1650,6 @@ static __always_inline long __gup_longte
  * should use get_user_pages because it cannot pass
  * FAULT_FLAG_ALLOW_RETRY to handle_mm_fault.
  */
-#ifdef CONFIG_MMU
 long get_user_pages_remote(struct task_struct *tsk, struct mm_struct *mm,
 		unsigned long start, unsigned long nr_pages,
 		unsigned int gup_flags, struct page **pages,

@@ -1632,28 +1662,8 @@ long get_user_pages_remote(struct task_s
 	if (WARN_ON_ONCE(gup_flags & FOLL_PIN))
 		return -EINVAL;
 
-	/*
-	 * Parts of FOLL_LONGTERM behavior are incompatible with
-	 * FAULT_FLAG_ALLOW_RETRY because of the FS DAX check requirement on
-	 * vmas. However, this only comes up if locked is set, and there are
-	 * callers that do request FOLL_LONGTERM, but do not set locked. So,
-	 * allow what we can.
-	 */
-	if (gup_flags & FOLL_LONGTERM) {
-		if (WARN_ON_ONCE(locked))
-			return -EINVAL;
-		/*
-		 * This will check the vmas (even if our vmas arg is NULL)
-		 * and return -ENOTSUPP if DAX isn't allowed in this case:
-		 */
-		return __gup_longterm_locked(tsk, mm, start, nr_pages, pages,
-					     vmas, gup_flags | FOLL_TOUCH |
-					     FOLL_REMOTE);
-	}

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #42 lore raw

PATCH 39/155 mm/gup: pass a flags arg to __gup_device_* functions

From: John Hubbard <[email protected]>
Subject: mm/gup: pass a flags arg to __gup_device_* functions

A subsequent patch requires access to gup flags, so pass the flags argument through to the __gup_device_* functions.

Also placate checkpatch.pl by shortening a nearby line.

Link http://lkml.kernel.org/r/[email protected] open here

Reviewed-by Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/gup.c |   28 ++++++++++++++++++----------
 1 file changed, 18 insertions(+), 10 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-pass-a-flags-arg-to-__gup_device_-functions +18 ~~-10~~

@@ -1963,7 +1963,8 @@ static int gup_pte_range(pmd_t pmd, unsi
 
 #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE)
 static int __gup_device_huge(unsigned long pfn, unsigned long addr,
-		unsigned long end, struct page **pages, int *nr)
+			     unsigned long end, unsigned int flags,
+			     struct page **pages, int *nr)
 {
 	int nr_start = *nr;
 	struct dev_pagemap *pgmap = NULL;

@@ -1989,13 +1990,14 @@ static int __gup_device_huge(unsigned lo
 }
 
 static int __gup_device_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,
-		unsigned long end, struct page **pages, int *nr)
+				 unsigned long end, unsigned int flags,
+				 struct page **pages, int *nr)
 {
 	unsigned long fault_pfn;
 	int nr_start = *nr;
 
 	fault_pfn = pmd_pfn(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
-	if (!__gup_device_huge(fault_pfn, addr, end, pages, nr))
+	if (!__gup_device_huge(fault_pfn, addr, end, flags, pages, nr))
 		return 0;
 
 	if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {

@@ -2006,13 +2008,14 @@ static int __gup_device_huge_pmd(pmd_t o
 }
 
 static int __gup_device_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,
-		unsigned long end, struct page **pages, int *nr)
+				 unsigned long end, unsigned int flags,
+				 struct page **pages, int *nr)
 {
 	unsigned long fault_pfn;
 	int nr_start = *nr;
 
 	fault_pfn = pud_pfn(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
-	if (!__gup_device_huge(fault_pfn, addr, end, pages, nr))
+	if (!__gup_device_huge(fault_pfn, addr, end, flags, pages, nr))
 		return 0;
 
 	if (unlikely(pud_val(orig) != pud_val(*pudp))) {

@@ -2023,14 +2026,16 @@ static int __gup_device_huge_pud(pud_t o
 }
 #else
 static int __gup_device_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr,
-		unsigned long end, struct page **pages, int *nr)
+				 unsigned long end, unsigned int flags,
+				 struct page **pages, int *nr)
 {
 	BUILD_BUG();
 	return 0;
 }
 
 static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr,
-		unsigned long end, struct page **pages, int *nr)
+				 unsigned long end, unsigned int flags,
+				 struct page **pages, int *nr)
 {
 	BUILD_BUG();
 	return 0;

@@ -2146,7 +2151,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_
 	if (pmd_devmap(orig)) {
 		if (unlikely(flags & FOLL_LONGTERM))
 			return 0;
-		return __gup_device_huge_pmd(orig, pmdp, addr, end, pages, nr);
+		return __gup_device_huge_pmd(orig, pmdp, addr, end, flags,
+					     pages, nr);
 	}
 
 	page = pmd_page(orig) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);

@@ -2167,7 +2173,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_
 }
 
 static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr,
-		unsigned long end, unsigned int flags, struct page **pages, int *nr)
+			unsigned long end, unsigned int flags,
+			struct page **pages, int *nr)
 {
 	struct page *head, *page;
 	int refs;

@@ -2178,7 +2185,8 @@ static int gup_huge_pud(pud_t orig, pud_
 	if (pud_devmap(orig)) {
 		if (unlikely(flags & FOLL_LONGTERM))
 			return 0;
-		return __gup_device_huge_pud(orig, pudp, addr, end, pages, nr);
+		return __gup_device_huge_pud(orig, pudp, addr, end, flags,
+					     pages, nr);
 	}
 
 	page = pud_page(orig) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #43 lore raw

PATCH 40/155 mm: introduce page_ref_sub_return()

From: John Hubbard <[email protected]>
Subject: mm: introduce page_ref_sub_return()

An upcoming patch requires subtracting a large chunk of refcounts from a page, and checking what the resulting refcount is. This is a little different than the usual "check for zero refcount" that many of the page ref functions already do. However, it is similar to a few other routines that (like this one) are generally useful for things such as 1-based refcounting.

Add page_ref_sub_return(), that subtracts a chunk of refcounts atomically, and returns an atomic snapshot of the result.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 include/linux/page_ref.h |    9 +++++++++
 1 file changed, 9 insertions(+)

include/linux/page_ref.h renamed from include/linux/page_ref.h~mm-introduce-page_ref_sub_return +9 -0

@@ -102,6 +102,15 @@ static inline void page_ref_sub(struct p
 		__page_ref_mod(page, -nr);
 }
 
+static inline int page_ref_sub_return(struct page *page, int nr)
+{
+	int ret = atomic_sub_return(nr, &page->_refcount);
+
+	if (page_ref_tracepoint_active(__tracepoint_page_ref_mod_and_return))
+		__page_ref_mod_and_return(page, -nr, ret);
+	return ret;
+}
+
 static inline void page_ref_inc(struct page *page)
 {
 	atomic_inc(&page->_refcount);

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #44 lore raw

PATCH 41/155 mm/gup: pass gup flags to two more routines

From: John Hubbard <[email protected]>
Subject: mm/gup: pass gup flags to two more routines

In preparation for an upcoming patch, send gup flags args to two more
routines: put_compound_head(), and undo_dev_pagemap().

Link http://lkml.kernel.org/r/[email protected] open here

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/gup.c |   19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-pass-gup-flags-to-two-more-routines +10 -9

@@ -1870,6 +1870,7 @@ static inline pte_t gup_get_pte(pte_t *p
 #endif /* CONFIG_GUP_GET_PTE_LOW_HIGH */
 
 static void __maybe_unused undo_dev_pagemap(int *nr, int nr_start,
+					    unsigned int flags,
 					    struct page **pages)
 {
 	while ((*nr) - nr_start) {

@@ -1909,7 +1910,7 @@ static int gup_pte_range(pmd_t pmd, unsi
 
 			pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
 			if (unlikely(!pgmap)) {
-				undo_dev_pagemap(nr, nr_start, pages);
+				undo_dev_pagemap(nr, nr_start, flags, pages);
 				goto pte_unmap;
 			}
 		} else if (pte_special(pte))

@@ -1974,7 +1975,7 @@ static int __gup_device_huge(unsigned lo
 
 		pgmap = get_dev_pagemap(pfn, pgmap);
 		if (unlikely(!pgmap)) {
-			undo_dev_pagemap(nr, nr_start, pages);
+			undo_dev_pagemap(nr, nr_start, flags, pages);
 			return 0;
 		}
 		SetPageReferenced(page);

@@ -2001,7 +2002,7 @@ static int __gup_device_huge_pmd(pmd_t o
 		return 0;
 
 	if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
-		undo_dev_pagemap(nr, nr_start, pages);
+		undo_dev_pagemap(nr, nr_start, flags, pages);
 		return 0;
 	}
 	return 1;

@@ -2019,7 +2020,7 @@ static int __gup_device_huge_pud(pud_t o
 		return 0;
 
 	if (unlikely(pud_val(orig) != pud_val(*pudp))) {
-		undo_dev_pagemap(nr, nr_start, pages);
+		undo_dev_pagemap(nr, nr_start, flags, pages);
 		return 0;
 	}
 	return 1;

@@ -2053,7 +2054,7 @@ static int record_subpages(struct page *
 	return nr;
 }
 
-static void put_compound_head(struct page *page, int refs)
+static void put_compound_head(struct page *page, int refs, unsigned int flags)
 {
 	VM_BUG_ON_PAGE(page_ref_count(page) < refs, page);
 	/*

@@ -2103,7 +2104,7 @@ static int gup_hugepte(pte_t *ptep, unsi
 		return 0;
 
 	if (unlikely(pte_val(pte) != pte_val(*ptep))) {
-		put_compound_head(head, refs);
+		put_compound_head(head, refs, flags);
 		return 0;
 	}

@@ -2163,7 +2164,7 @@ static int gup_huge_pmd(pmd_t orig, pmd_
 		return 0;
 
 	if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
-		put_compound_head(head, refs);
+		put_compound_head(head, refs, flags);
 		return 0;
 	}

@@ -2197,7 +2198,7 @@ static int gup_huge_pud(pud_t orig, pud_
 		return 0;
 
 	if (unlikely(pud_val(orig) != pud_val(*pudp))) {
-		put_compound_head(head, refs);
+		put_compound_head(head, refs, flags);
 		return 0;
 	}

@@ -2226,7 +2227,7 @@ static int gup_huge_pgd(pgd_t orig, pgd_
 		return 0;
 
 	if (unlikely(pgd_val(orig) != pgd_val(*pgdp))) {
-		put_compound_head(head, refs);
+		put_compound_head(head, refs, flags);
 		return 0;
 	}

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #45 lore raw

PATCH 42/155 mm/gup: require FOLL_GET for get_user_pages_fast()

From: John Hubbard <[email protected]>
Subject: mm/gup: require FOLL_GET for get_user_pages_fast()

Internal to mm/gup.c, require that get_user_pages_fast() and __get_user_pages_fast() identify themselves, by setting FOLL_GET. This is required in order to be able to make decisions based on "FOLL_PIN, or FOLL_GET, or both or neither are set", in upcoming patches.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/gup.c |   19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-require-foll_get-for-get_user_pages_fast +17 -2

@@ -2390,6 +2390,14 @@ int __get_user_pages_fast(unsigned long
 	unsigned long len, end;
 	unsigned long flags;
 	int nr = 0;
+	/*
+	 * Internally (within mm/gup.c), gup fast variants must set FOLL_GET,
+	 * because gup fast is always a "pin with a +1 page refcount" request.
+	 */
+	unsigned int gup_flags = FOLL_GET;
+
+	if (write)
+		gup_flags |= FOLL_WRITE;
 
 	start = untagged_addr(start) & PAGE_MASK;
 	len = (unsigned long) nr_pages << PAGE_SHIFT;

@@ -2415,7 +2423,7 @@ int __get_user_pages_fast(unsigned long
 	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP) &&
 	    gup_fast_permitted(start, end)) {
 		local_irq_save(flags);
-		gup_pgd_range(start, end, write ? FOLL_WRITE : 0, pages, &nr);
+		gup_pgd_range(start, end, gup_flags, pages, &nr);
 		local_irq_restore(flags);
 	}

@@ -2454,7 +2462,7 @@ static int internal_get_user_pages_fast(
 	int nr = 0, ret = 0;
 
 	if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM |
-				       FOLL_FORCE | FOLL_PIN)))
+				       FOLL_FORCE | FOLL_PIN | FOLL_GET)))
 		return -EINVAL;
 
 	start = untagged_addr(start) & PAGE_MASK;

@@ -2521,6 +2529,13 @@ int get_user_pages_fast(unsigned long st
 	if (WARN_ON_ONCE(gup_flags & FOLL_PIN))
 		return -EINVAL;
 
+	/*
+	 * The caller may or may not have explicitly set FOLL_GET; either way is
+	 * OK. However, internally (within mm/gup.c), gup fast variants must set
+	 * FOLL_GET, because gup fast is always a "pin with a +1 page refcount"
+	 * request.
+	 */
+	gup_flags |= FOLL_GET;
 	return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages);
 }
 EXPORT_SYMBOL_GPL(get_user_pages_fast);

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #46 lore raw

PATCH 43/155 mm/gup: track FOLL_PIN pages

From: John Hubbard <[email protected]>
Subject: mm/gup: track FOLL_PIN pages

Add tracking of pages that were pinned via FOLL_PIN. This tracking is implemented via overloading of page->_refcount: pins are added by adding GUP_PIN_COUNTING_BIAS (1024) to the refcount. This provides a fuzzy indication of pinning, and it can have false positives (and that's OK). Please see the pre-existing Documentation/core-api/pin_user_pages.rst for details.

As mentioned in pin_user_pages.rst, callers who effectively set FOLL_PIN (typically via pin_user_pages*()) are required to ultimately free such pages via unpin_user_page().

Please also note the limitation, discussed in pin_user_pages.rst under the "TODO: for 1GB and larger huge pages" section. (That limitation will be removed in a following patch.)

The effect of a FOLL_PIN flag is similar to that of FOLL_GET, and may be thought of as "FOLL_GET for DIO and/or RDMA use".

Pages that have been pinned via FOLL_PIN are identifiable via a new function call:

   bool page_maybe_dma_pinned(struct page *page);

What to do in response to encountering such a page, is left to later patchsets. There is discussion about this in [1], [2], [3], and [4].

This also changes a BUG_ON(), to a WARN_ON(), in follow_page_mask().

[1] Some slow progress on get_user_pages() (Apr 2, 2019):

    https://lwn.net/Articles/784574/
[2] DMA and get_user_pages() (LPC: Dec 12, 2018):
    https://lwn.net/Articles/774411/
[3] The trouble with get_user_pages() (Apr 30, 2018):
    https://lwn.net/Articles/753027/

[4] LWN kernel index: get_user_pages():
https://lwn.net/Kernel/Index/#Memory_management-get_user_pages

[[email protected]: add kerneldoc]

Link http://lkml.kernel.org/r/[email protected] open here

[[email protected]: if pin fails, we need to unpin, a simple put_page will not be enough]

Link http://lkml.kernel.org/r/[email protected] open here

[[email protected]: fix put_compound_head defined but not used]

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Claudio Imbrenda <[email protected]>

Suggested-by Jan Kara <[email protected]>

Cc Ira Weiny <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 Documentation/core-api/pin_user_pages.rst |    6 
 include/linux/mm.h                        |   84 ++++-
 mm/gup.c                                  |  312 ++++++++++++++++----
 mm/huge_memory.c                          |   29 +
 mm/hugetlb.c                              |   54 ++-
 5 files changed, 380 insertions(+), 105 deletions(-)

Documentation/core-api/pin_user_pages.rst renamed from Documentation/core-api/pin_user_pages.rst~mm-gup-track-foll_pin-pages +3 -3

@@ -173,8 +173,8 @@ CASE 4: Pinning for struct page manipula
 -------------------------------------------------
 Here, normal GUP calls are sufficient, so neither flag needs to be set.
 
-page_dma_pinned(): the whole point of pinning
-=============================================
+page_maybe_dma_pinned(): the whole point of pinning
+===================================================
 
 The whole point of marking pages as "DMA-pinned" or "gup-pinned" is to be able
 to query, "is this page DMA-pinned?" That allows code such as page_mkclean()

@@ -186,7 +186,7 @@ and debates (see the References at the e
 here: fill in the details once that's worked out. Meanwhile, it's safe to say
 that having this available: ::
 
-        static inline bool page_dma_pinned(struct page *page)
+        static inline bool page_maybe_dma_pinned(struct page *page)
 
 ...is a prerequisite to solving the long-running gup+DMA problem.

include/linux/mm.h renamed from include/linux/mm.h~mm-gup-track-foll_pin-pages +68 ~~-16~~

103 lines not on this page · raw message

mm/gup.c renamed from mm/gup.c~mm-gup-track-foll_pin-pages +250 ~~-62~~

443 lines not on this page · raw message

mm/huge_memory.c renamed from mm/huge_memory.c~mm-gup-track-foll_pin-pages +23 -6

@@ -958,6 +958,11 @@ struct page *follow_devmap_pmd(struct vm
 	 */
 	WARN_ONCE(flags & FOLL_COW, "mm: In follow_devmap_pmd with FOLL_COW set");
 
+	/* FOLL_GET and FOLL_PIN are mutually exclusive. */
+	if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
+			 (FOLL_PIN | FOLL_GET)))
+		return NULL;
+
 	if (flags & FOLL_WRITE && !pmd_write(*pmd))
 		return NULL;

@@ -973,7 +978,7 @@ struct page *follow_devmap_pmd(struct vm
 	 * device mapped pages can only be returned if the
 	 * caller will manage the page reference count.
 	 */
-	if (!(flags & FOLL_GET))
+	if (!(flags & (FOLL_GET | FOLL_PIN)))
 		return ERR_PTR(-EEXIST);
 
 	pfn += (addr & ~PMD_MASK) >> PAGE_SHIFT;

@@ -981,7 +986,8 @@ struct page *follow_devmap_pmd(struct vm
 	if (!*pgmap)
 		return ERR_PTR(-EFAULT);
 	page = pfn_to_page(pfn);
-	get_page(page);
+	if (!try_grab_page(page, flags))
+		page = ERR_PTR(-ENOMEM);
 
 	return page;
 }

@@ -1101,6 +1107,11 @@ struct page *follow_devmap_pud(struct vm
 	if (flags & FOLL_WRITE && !pud_write(*pud))
 		return NULL;
 
+	/* FOLL_GET and FOLL_PIN are mutually exclusive. */
+	if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==
+			 (FOLL_PIN | FOLL_GET)))
+		return NULL;
+
 	if (pud_present(*pud) && pud_devmap(*pud))
 		/* pass */;
 	else

@@ -1112,8 +1123,10 @@ struct page *follow_devmap_pud(struct vm
 	/*
 	 * device mapped pages can only be returned if the
 	 * caller will manage the page reference count.
+	 *
+	 * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here:
 	 */
-	if (!(flags & FOLL_GET))
+	if (!(flags & (FOLL_GET | FOLL_PIN)))
 		return ERR_PTR(-EEXIST);
 
 	pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT;

@@ -1121,7 +1134,8 @@ struct page *follow_devmap_pud(struct vm
 	if (!*pgmap)
 		return ERR_PTR(-EFAULT);
 	page = pfn_to_page(pfn);
-	get_page(page);
+	if (!try_grab_page(page, flags))
+		page = ERR_PTR(-ENOMEM);
 
 	return page;
 }

@@ -1497,8 +1511,13 @@ struct page *follow_trans_huge_pmd(struc
 
 	page = pmd_page(*pmd);
 	VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page);
+
+	if (!try_grab_page(page, flags))
+		return ERR_PTR(-ENOMEM);
+
 	if (flags & FOLL_TOUCH)
 		touch_pmd(vma, addr, pmd, flags);
+
 	if ((flags & FOLL_MLOCK) && (vma->vm_flags & VM_LOCKED)) {
 		/*
 		 * We don't mlock() pte-mapped THPs. This way we can avoid

@@ -1535,8 +1554,6 @@ struct page *follow_trans_huge_pmd(struc
 skip_mlock:
 	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
 	VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page);
-	if (flags & FOLL_GET)
-		get_page(page);
 
 out:
 	return page;

mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-track-foll_pin-pages +36 ~~-18~~

90 lines not on this page · raw message

Tetsuo Handa

2020-04-09 06:09 UTC #47 lore raw

Re: [patch 043/155] mm/gup: track FOLL_PIN pages

Hello.

I'm hitting WARN_ON() at try_get_page() from try_grab_page() due to page_ref_count(page) == -1021 (which is "3 - GUP_PIN_COUNTING_BIAS") if I use loadable kernel module version of TOMOYO security module. Since I don't see recent changes in security/tomoyo regarding get_user_pages_remote(), I'm wondering what is happening.

[   10.427414][    T1] ------------[ cut here ]------------
[   10.427425][    T1] WARNING: CPU: 3 PID: 1 at ./include/linux/mm.h:1009 try_grab_page+0x77/0x80
[   10.427426][    T1] Modules linked in: caitsith(O) akari(O) xfs libcrc32c crc32c_generic sd_mod ata_generic pata_acpi mptspi scsi_transport_spi vmwgfx mptscsih drm_kms_helper mptbase cfbfillrect syscopyarea cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea fb ahci fbdev libahci ttm ata_piix nvme drm drm_panel_orientation_quirks libata nvme_core t10_pi agpgart e1000 i2c_core scsi_mod serio_raw atkbd libps2 i8042 serio unix
[   10.427451][    T1] CPU: 3 PID: 1 Comm: akari-init Tainted: G           O      5.6.0-05654-g3faa52c03f44 #984
[   10.427452][    T1] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   10.427454][    T1] RIP: 0010:try_grab_page+0x77/0x80
[   10.427456][    T1] Code: 34 85 c0 7e 25 f0 ff 47 34 b8 01 00 00 00 5d c3 0f 0b eb b1 48 8d 78 ff eb c6 0f 0b 31 c0 5d 0f 1f 40 00 c3 48 8d 78 ff eb d4 <0f> 0b 31 c0 5d c3 0f 1f 00 55 48 89 e5 41 57 49 89 cf 41 56 49 89
[   10.427457][    T1] RSP: 0018:ffffa859c0013ac0 EFLAGS: 00010286
[   10.427459][    T1] RAX: 00000000fffffc03 RBX: ffffcb6208c71dc0 RCX: 8000000231c77067
[   10.427460][    T1] RDX: 8000000231c77067 RSI: 0000000000002016 RDI: ffffcb6208c71dc0
[   10.427461][    T1] RBP: ffffa859c0013ac0 R08: 0000000000000001 R09: 0000000000000000
[   10.427462][    T1] R10: 0000000000000001 R11: ffff95d91c615340 R12: 0000000000002016
[   10.427463][    T1] R13: ffff95d91c615328 R14: 0000000000000ff0 R15: ffff95d91e0c0ff0
[   10.427465][    T1] FS:  00007fbfc3772740(0000) GS:ffff95d927a00000(0000) knlGS:0000000000000000
[   10.427479][    T1] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.427483][    T1] CR2: 0000000000ac5190 CR3: 000000022090a002 CR4: 00000000003606e0
[   10.427491][    T1] Call Trace:
[   10.427494][    T1]  follow_page_pte+0x329/0x443
[   10.427503][    T1]  follow_p4d_mask+0x756/0x7c1
[   10.427511][    T1]  follow_page_mask+0x6a/0x70
[   10.427516][    T1]  __get_user_pages+0x110/0x880
[   10.427518][    T1]  ? ___slab_alloc.constprop.95+0x929/0x980
[   10.427532][    T1]  __get_user_pages_remote+0xce/0x230
[   10.427540][    T1]  get_user_pages_remote+0x27/0x40
[   10.427545][    T1]  ccs_dump_page+0x6a/0x140 [akari]
[   10.427552][    T1]  ccs_start_execve+0x28c/0x490 [akari]
[   10.427555][    T1]  ? ccs_start_execve+0x90/0x490 [akari]
[   10.427561][    T1]  ? ccs_load_policy+0xee/0x150 [akari]
[   10.427568][    T1]  ccs_bprm_check_security+0x4e/0x70 [akari]
[   10.427572][    T1]  security_bprm_check+0x26/0x40
[   10.427576][    T1]  search_binary_handler+0x22/0x1c0
[   10.427580][    T1]  __do_execve_file.isra.41+0x723/0xac0
[   10.427581][    T1]  ? __do_execve_file.isra.41+0x665/0xac0
[   10.427590][    T1]  __x64_sys_execve+0x44/0x50
[   10.427614][    T1]  do_syscall_64+0x4a/0x1e0
[   10.427618][    T1]  entry_SYSCALL_64_after_hwframe+0x49/0xb3
[   10.427649][    T1] RIP: 0033:0x7fbfc2e36c37
[   10.427651][    T1] Code: ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de 64 41 89 32 eb db 66 2e 0f 1f 84 00 00 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 08 12 30 00 f7 d8 64 89 02
[   10.427652][    T1] RSP: 002b:00007fff25f9c0f8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[   10.427654][    T1] RAX: ffffffffffffffda RBX: 0000000000ac64f0 RCX: 00007fbfc2e36c37
[   10.427672][    T1] RDX: 0000000000ac6be0 RSI: 0000000000ac6ac0 RDI: 0000000000ac64f0
[   10.427673][    T1] RBP: 0000000000000000 R08: 00007fff25f9c0e0 R09: 0000000000000000
[   10.427674][    T1] R10: 00007fff25f9bb60 R11: 0000000000000202 R12: 0000000000ac6be0
[   10.427675][    T1] R13: 0000000000ac6ac0 R14: 0000000000ac6be0 R15: 0000000000ac6820
[   10.427701][    T1] irq event stamp: 10135190
[   10.427704][    T1] hardirqs last  enabled at (10135189): [<ffffffffb223092a>] __slab_alloc.constprop.94+0x48/0x5e
[   10.427706][    T1] hardirqs last disabled at (10135190): [<ffffffffb2001eb7>] trace_hardirqs_off_thunk+0x1a/0x1c
[   10.427707][    T1] softirqs last  enabled at (10135114): [<ffffffffb2a0032b>] __do_softirq+0x32b/0x455
[   10.427710][    T1] softirqs last disabled at (10135107): [<ffffffffb2072985>] irq_exit+0xa5/0xb0
[   10.427711][    T1] ---[ end trace 984e9bd0ce5a1e09 ]---

+bool __must_check try_grab_page(struct page *page, unsigned int flags) +{
+ WARN_ON_ONCE((flags & (FOLL_GET | FOLL_PIN)) == (FOLL_GET | FOLL_PIN));
+
+ if (flags & FOLL_GET)
+ return try_get_page(page);
+ else if (flags & FOLL_PIN) {
+ page = compound_head(page);
+
+ if (WARN_ON_ONCE(page_ref_count(page) <= 0))
+ return false;
+
+ page_ref_add(page, GUP_PIN_COUNTING_BIAS);
+ }
+
+ return true;
+}

Bisection says 3faa52c03f440d1b ("mm/gup: track FOLL_PIN pages") is the bad commit.

$ git bisect log
# bad: [3faa52c03f440d1b9ddef18c4f189f4790d52d7e] mm/gup: track FOLL_PIN pages
# good: [7111951b8d4973bda27ff663f2cf18b663d15b48] Linux 5.6
# good: [d5226fa6dbae0569ee43ecfc08bdcd6770fc4755] Linux 5.5
# good: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
# good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
# good: [0ecfebd2b52404ae0c54a878c872bb93363ada36] Linux 5.2
# good: [e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd] Linux 5.1
# good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
# good: [86dfbed49f88fd87ce8a12d2314b1f93266da7a7] mm/gup: pass a flags arg to __gup_device_* functions
# good: [83daf837884cc44c3cc0e4f8a096c5d1461cbcc2] mm/filemap.c: unexport find_get_entry
# good: [cc7b8f6245f0042a232c7f6807dc130d87233164] mm/page-writeback.c: write_cache_pages(): deduplicate identical checks
# good: [29d9f30d4ce6c7a38745a54a8cddface10013490] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
# good: [4afdb733b1606c6cb86e7833f9335f4870cf7ddd] io-uring: drop completion when removing file
git bisect start '3faa52c03f440d1b9ddef18c4f189f4790d52d7e' 'v5.6' 'v5.5' 'v5.4' 'v5.3' 'v5.2' 'v5.1' 'v5.0' '86dfbed49f88fd87ce8a12d2314b1f93266da7a7' '83daf837884cc44c3cc0e4f8a096c5d1461cbcc2' 'cc7b8f6245f0042a232c7f6807dc130d87233164' '29d9f30d4ce6c7a38745a54a8cddface10013490' '4afdb733b1606c6cb86e7833f9335f4870cf7ddd'
# good: [3b78d8347d31a050bb4f378a5f42cf495d873796] mm/gup: pass gup flags to two more routines
git bisect good 3b78d8347d31a050bb4f378a5f42cf495d873796
# good: [94202f126f698691f8865906ad6a68203e5dde8c] mm/gup: require FOLL_GET for get_user_pages_fast()
git bisect good 94202f126f698691f8865906ad6a68203e5dde8c
# first bad commit: [3faa52c03f440d1b9ddef18c4f189f4790d52d7e] mm/gup: track FOLL_PIN pages

John Hubbard

reviewer

2020-04-09 06:38 UTC #48 lore raw

Re: [patch 043/155] mm/gup: track FOLL_PIN pages

Quoted 6 lines · Tetsuo Handa

Hello.
I'm hitting WARN_ON() at try_get_page() from try_grab_page() due to page_ref_count(page) == -1021 (which is "3 - GUP_PIN_COUNTING_BIAS") if I use loadable kernel module version of TOMOYO security module. Since I don't see recent changes in security/tomoyo regarding get_user_pages_remote(), I'm wondering what is happening.

Hi Tetsuo,

Yes, commit 3faa52c03f440d1b ("mm/gup: track FOLL_PIN pages") is the one that turns everything on, so if any problems with the whole FOLL_PIN scheme are to be found, it's natural that git bisect would point to that commit.

Thanks for all the details here. One of the first questions I normally have is: is anything in the system even using FOLL_PIN? And the way to answer that is to monitor the two new *foll_pin* entries in /proc/vmstat, approximately like this:

$ cat /proc/vmstat |grep foll_pin
nr_foll_pin_acquired 0
nr_foll_pin_released 0

If you could do that before, during and after (ideally...or whatever you can get) the problem, I'd love to see that data.

Also, if you happen to know if anything is calling pin_user_page*() and/or unpin_user_page*(), that is extra credit. :)

I don't see anything here that jumps out at me, yet. The call stack below is as expected, and the WARN often picks up problems that happened in some other calling path (something else leaked a FOLL_PIN page for example, etc).

quick question below:

Quoted 22 lines

[   10.427414][    T1] ------------[ cut here ]------------
[   10.427425][    T1] WARNING: CPU: 3 PID: 1 at ./include/linux/mm.h:1009 try_grab_page+0x77/0x80
[   10.427426][    T1] Modules linked in: caitsith(O) akari(O) xfs libcrc32c crc32c_generic sd_mod ata_generic pata_acpi mptspi scsi_transport_spi vmwgfx mptscsih drm_kms_helper mptbase cfbfillrect syscopyarea cfbimgblt sysfillrect sysimgblt fb_sys_fops cfbcopyarea fb ahci fbdev libahci ttm ata_piix nvme drm drm_panel_orientation_quirks libata nvme_core t10_pi agpgart e1000 i2c_core scsi_mod serio_raw atkbd libps2 i8042 serio unix
[   10.427451][    T1] CPU: 3 PID: 1 Comm: akari-init Tainted: G           O      5.6.0-05654-g3faa52c03f44 #984
[   10.427452][    T1] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
[   10.427454][    T1] RIP: 0010:try_grab_page+0x77/0x80
[   10.427456][    T1] Code: 34 85 c0 7e 25 f0 ff 47 34 b8 01 00 00 00 5d c3 0f 0b eb b1 48 8d 78 ff eb c6 0f 0b 31 c0 5d 0f 1f 40 00 c3 48 8d 78 ff eb d4 <0f> 0b 31 c0 5d c3 0f 1f 00 55 48 89 e5 41 57 49 89 cf 41 56 49 89
[   10.427457][    T1] RSP: 0018:ffffa859c0013ac0 EFLAGS: 00010286
[   10.427459][    T1] RAX: 00000000fffffc03 RBX: ffffcb6208c71dc0 RCX: 8000000231c77067
[   10.427460][    T1] RDX: 8000000231c77067 RSI: 0000000000002016 RDI: ffffcb6208c71dc0
[   10.427461][    T1] RBP: ffffa859c0013ac0 R08: 0000000000000001 R09: 0000000000000000
[   10.427462][    T1] R10: 0000000000000001 R11: ffff95d91c615340 R12: 0000000000002016
[   10.427463][    T1] R13: ffff95d91c615328 R14: 0000000000000ff0 R15: ffff95d91e0c0ff0
[   10.427465][    T1] FS:  00007fbfc3772740(0000) GS:ffff95d927a00000(0000) knlGS:0000000000000000
[   10.427479][    T1] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.427483][    T1] CR2: 0000000000ac5190 CR3: 000000022090a002 CR4: 00000000003606e0
[   10.427491][    T1] Call Trace:
[   10.427494][    T1]  follow_page_pte+0x329/0x443
[   10.427503][    T1]  follow_p4d_mask+0x756/0x7c1
[   10.427511][    T1]  follow_page_mask+0x6a/0x70
[   10.427516][    T1]  __get_user_pages+0x110/0x880

Quoted 8 lines

[   10.427518][    T1]  ? ___slab_alloc.constprop.95+0x929/0x980
[   10.427532][    T1]  __get_user_pages_remote+0xce/0x230
[   10.427540][    T1]  get_user_pages_remote+0x27/0x40
[   10.427545][    T1]  ccs_dump_page+0x6a/0x140 [akari]
[   10.427552][    T1]  ccs_start_execve+0x28c/0x490 [akari]
[   10.427555][    T1]  ? ccs_start_execve+0x90/0x490 [akari]
[   10.427561][    T1]  ? ccs_load_policy+0xee/0x150 [akari]
[   10.427568][    T1]  ccs_bprm_check_security+0x4e/0x70 [akari]

...say, I can't find the ccs_*() routines in my tree, can you point me to them? Probably not important but I'm curious now.

thanks,

Signature · 2 lines

John Hubbard
NVIDIA

Quoted 64 lines

[   10.427572][    T1]  security_bprm_check+0x26/0x40
[   10.427576][    T1]  search_binary_handler+0x22/0x1c0
[   10.427580][    T1]  __do_execve_file.isra.41+0x723/0xac0
[   10.427581][    T1]  ? __do_execve_file.isra.41+0x665/0xac0
[   10.427590][    T1]  __x64_sys_execve+0x44/0x50
[   10.427614][    T1]  do_syscall_64+0x4a/0x1e0
[   10.427618][    T1]  entry_SYSCALL_64_after_hwframe+0x49/0xb3
[   10.427649][    T1] RIP: 0033:0x7fbfc2e36c37
[   10.427651][    T1] Code: ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de 64 41 89 32 eb db 66 2e 0f 1f 84 00 00 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 08 12 30 00 f7 d8 64 89 02
[   10.427652][    T1] RSP: 002b:00007fff25f9c0f8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[   10.427654][    T1] RAX: ffffffffffffffda RBX: 0000000000ac64f0 RCX: 00007fbfc2e36c37
[   10.427672][    T1] RDX: 0000000000ac6be0 RSI: 0000000000ac6ac0 RDI: 0000000000ac64f0
[   10.427673][    T1] RBP: 0000000000000000 R08: 00007fff25f9c0e0 R09: 0000000000000000
[   10.427674][    T1] R10: 00007fff25f9bb60 R11: 0000000000000202 R12: 0000000000ac6be0
[   10.427675][    T1] R13: 0000000000ac6ac0 R14: 0000000000ac6be0 R15: 0000000000ac6820
[   10.427701][    T1] irq event stamp: 10135190
[   10.427704][    T1] hardirqs last  enabled at (10135189): [<ffffffffb223092a>] __slab_alloc.constprop.94+0x48/0x5e
[   10.427706][    T1] hardirqs last disabled at (10135190): [<ffffffffb2001eb7>] trace_hardirqs_off_thunk+0x1a/0x1c
[   10.427707][    T1] softirqs last  enabled at (10135114): [<ffffffffb2a0032b>] __do_softirq+0x32b/0x455
[   10.427710][    T1] softirqs last disabled at (10135107): [<ffffffffb2072985>] irq_exit+0xa5/0xb0
[   10.427711][    T1] ---[ end trace 984e9bd0ce5a1e09 ]---

mm/gup.c renamed from mm/gup.c~mm-gup-track-foll_pin-pages +17 -0

+bool __must_check try_grab_page(struct page *page, unsigned int flags)
+{
+       WARN_ON_ONCE((flags & (FOLL_GET | FOLL_PIN)) == (FOLL_GET | FOLL_PIN));
+
+       if (flags & FOLL_GET)
+               return try_get_page(page);
+       else if (flags & FOLL_PIN) {
+               page = compound_head(page);
+
+               if (WARN_ON_ONCE(page_ref_count(page) <= 0))
+                       return false;
+
+               page_ref_add(page, GUP_PIN_COUNTING_BIAS);
+       }
+
+       return true;
+}

Bisection says 3faa52c03f440d1b ("mm/gup: track FOLL_PIN pages") is the bad commit.

$ git bisect log
# bad: [3faa52c03f440d1b9ddef18c4f189f4790d52d7e] mm/gup: track FOLL_PIN pages
# good: [7111951b8d4973bda27ff663f2cf18b663d15b48] Linux 5.6
# good: [d5226fa6dbae0569ee43ecfc08bdcd6770fc4755] Linux 5.5
# good: [219d54332a09e8d8741c1e1982f5eae56099de85] Linux 5.4
# good: [4d856f72c10ecb060868ed10ff1b1453943fc6c8] Linux 5.3
# good: [0ecfebd2b52404ae0c54a878c872bb93363ada36] Linux 5.2
# good: [e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd] Linux 5.1
# good: [1c163f4c7b3f621efff9b28a47abb36f7378d783] Linux 5.0
# good: [86dfbed49f88fd87ce8a12d2314b1f93266da7a7] mm/gup: pass a flags arg to __gup_device_* functions
# good: [83daf837884cc44c3cc0e4f8a096c5d1461cbcc2] mm/filemap.c: unexport find_get_entry
# good: [cc7b8f6245f0042a232c7f6807dc130d87233164] mm/page-writeback.c: write_cache_pages(): deduplicate identical checks
# good: [29d9f30d4ce6c7a38745a54a8cddface10013490] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next
# good: [4afdb733b1606c6cb86e7833f9335f4870cf7ddd] io-uring: drop completion when removing file
git bisect start '3faa52c03f440d1b9ddef18c4f189f4790d52d7e' 'v5.6' 'v5.5' 'v5.4' 'v5.3' 'v5.2' 'v5.1' 'v5.0' '86dfbed49f88fd87ce8a12d2314b1f93266da7a7' '83daf837884cc44c3cc0e4f8a096c5d1461cbcc2' 'cc7b8f6245f0042a232c7f6807dc130d87233164' '29d9f30d4ce6c7a38745a54a8cddface10013490' '4afdb733b1606c6cb86e7833f9335f4870cf7ddd'
# good: [3b78d8347d31a050bb4f378a5f42cf495d873796] mm/gup: pass gup flags to two more routines
git bisect good 3b78d8347d31a050bb4f378a5f42cf495d873796
# good: [94202f126f698691f8865906ad6a68203e5dde8c] mm/gup: require FOLL_GET for get_user_pages_fast()
git bisect good 94202f126f698691f8865906ad6a68203e5dde8c
# first bad commit: [3faa52c03f440d1b9ddef18c4f189f4790d52d7e] mm/gup: track FOLL_PIN pages

Tetsuo Handa

2020-04-09 07:21 UTC #49 lore raw

Re: [patch 043/155] mm/gup: track FOLL_PIN pages

Hello.

Quoted 2 lines · John Hubbard

Also, if you happen to know if anything is calling pin_user_page*() and/or unpin_user_page*(), that is extra credit. :)

Ah, I got it. I can see that only nr_foll_pin_released is increasing after I hit WARN_ON(), and indeed AKARI is calling unpin_user_page() when put_user() should be used.

https://osdn.net/projects/akari/scm/svn/blobs/head/trunk/akari/permission.c line 1480

Sorry for the noise. Thank you.

John Hubbard

reviewer

2020-04-09 07:46 UTC #50 lore raw

Re: [patch 043/155] mm/gup: track FOLL_PIN pages

Quoted 6 lines · Tetsuo Handa

Hello.
Quoted 2 lines · John Hubbard
Also, if you happen to know if anything is calling pin_user_page*() and/or unpin_user_page*(), that is extra credit. :)

Quoted 7 lines

Ah, I got it. I can see that only nr_foll_pin_released is increasing after I hit WARN_ON(), and indeed AKARI is calling unpin_user_page() when put_user() should be used.
https://osdn.net/projects/akari/scm/svn/blobs/head/trunk/akari/permission.c line 1480
Sorry for the noise. Thank you.

Thanks for the quick investigation, it's a relief to hear that. :)

thanks,

Signature · 2 lines

John Hubbard
NVIDIA

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #51 lore raw

PATCH 44/155 mm/gup: page->hpage_pinned_refcount: exact pin counts for huge pages

From: John Hubbard <[email protected]>
Subject: mm/gup: page->hpage_pinned_refcount: exact pin counts for huge pages

For huge pages (and in fact, any compound page), the GUP_PIN_COUNTING_BIAS scheme tends to overflow too easily, each tail page increments the head page->_refcount by GUP_PIN_COUNTING_BIAS (1024). That limits the number of huge pages that can be pinned.

This patch removes that limitation, by using an exact form of pin counting for compound pages of order > 1. The "order > 1" is required because this approach uses the 3rd struct page in the compound page, and order 1 compound pages only have two pages, so that won't work there.

A new struct page field, hpage_pinned_refcount, has been added, replacing a padding field in the union (so no new space is used).

This enhancement also has a useful side effect: huge pages and compound pages (of order > 1) do not suffer from the "potential false positives" problem that is discussed in the page_dma_pinned() comment block. That is because these compound pages have extra space for tracking things, so they get exact pin counts instead of overloading page->_refcount.

Documentation/core-api/pin_user_pages.rst is updated accordingly.

Link http://lkml.kernel.org/r/[email protected] open here

Suggested-by Jan Kara <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 Documentation/core-api/pin_user_pages.rst |   40 ++++------
 include/linux/mm.h                        |   26 ++++++
 include/linux/mm_types.h                  |    7 +
 mm/gup.c                                  |   78 +++++++++++++++++---
 mm/hugetlb.c                              |    6 +
 mm/page_alloc.c                           |    2 
 mm/rmap.c                                 |    6 +
 7 files changed, 133 insertions(+), 32 deletions(-)

Documentation/core-api/pin_user_pages.rst renamed from Documentation/core-api/pin_user_pages.rst~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +17 ~~-23~~

@@ -52,8 +52,22 @@ Which flags are set by each wrapper
 
 For these pin_user_pages*() functions, FOLL_PIN is OR'd in with whatever gup
 flags the caller provides. The caller is required to pass in a non-null struct
-pages* array, and the function then pin pages by incrementing each by a special
-value. For now, that value is +1, just like get_user_pages*().::
+pages* array, and the function then pins pages by incrementing each by a special
+value: GUP_PIN_COUNTING_BIAS.
+
+For huge pages (and in fact, any compound page of more than 2 pages), the
+GUP_PIN_COUNTING_BIAS scheme is not used. Instead, an exact form of pin counting
+is achieved, by using the 3rd struct page in the compound page. A new struct
+page field, hpage_pinned_refcount, has been added in order to support this.
+
+This approach for compound pages avoids the counting upper limit problems that
+are discussed below. Those limitations would have been aggravated severely by
+huge pages, because each tail page adds a refcount to the head page. And in
+fact, testing revealed that, without a separate hpage_pinned_refcount field,
+page overflows were seen in some huge page stress tests.
+
+This also means that huge pages and compound pages (of order > 1) do not suffer
+from the false positives problem that is mentioned below.::
 
  Function
  --------

@@ -99,27 +113,6 @@ pages:
 This also leads to limitations: there are only 31-10==21 bits available for a
 counter that increments 10 bits at a time.
 
-TODO: for 1GB and larger huge pages, this is cutting it close. That's because
-when pin_user_pages() follows such pages, it increments the head page by "1"
-(where "1" used to mean "+1" for get_user_pages(), but now means "+1024" for
-pin_user_pages()) for each tail page. So if you have a 1GB huge page:
-
-* There are 256K (18 bits) worth of 4 KB tail pages.
-* There are 21 bits available to count up via GUP_PIN_COUNTING_BIAS (that is,
-  10 bits at a time)
-* There are 21 - 18 == 3 bits available to count. Except that there aren't,
-  because you need to allow for a few normal get_page() calls on the head page,
-  as well. Fortunately, the approach of using addition, rather than "hard"
-  bitfields, within page->_refcount, allows for sharing these bits gracefully.
-  But we're still looking at about 8 references.
-
-This, however, is a missing feature more than anything else, because it's easily
-solved by addressing an obvious inefficiency in the original get_user_pages()
-approach of retrieving pages: stop treating all the pages as if they were
-PAGE_SIZE. Retrieve huge pages as huge pages. The callers need to be aware of
-this, so some work is required. Once that's in place, this limitation mostly
-disappears from view, because there will be ample refcounting range available.
-
 * Callers must specifically request "dma-pinned tracking of pages". In other
   words, just calling get_user_pages() will not suffice; a new set of functions,
   pin_user_page() and related, must be used.

@@ -228,5 +221,6 @@ References
 * `Some slow progress on get_user_pages() (Apr 2, 2019) <https://lwn.net/Articles/784574/>`_
 * `DMA and get_user_pages() (LPC: Dec 12, 2018) <https://lwn.net/Articles/774411/>`_
 * `The trouble with get_user_pages() (Apr 30, 2018) <https://lwn.net/Articles/753027/>`_
+* `LWN kernel index: get_user_pages() <https://lwn.net/Kernel/Index/#Memory_management-get_user_pages>`_
 
 John Hubbard, October, 2019

include/linux/mm.h renamed from include/linux/mm.h~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +26 -0

@@ -770,6 +770,24 @@ static inline unsigned int compound_orde
 	return page[1].compound_order;
 }
 
+static inline bool hpage_pincount_available(struct page *page)
+{
+	/*
+	 * Can the page->hpage_pinned_refcount field be used? That field is in
+	 * the 3rd page of the compound page, so the smallest (2-page) compound
+	 * pages cannot support it.
+	 */
+	page = compound_head(page);
+	return PageCompound(page) && compound_order(page) > 1;
+}
+
+static inline int compound_pincount(struct page *page)
+{
+	VM_BUG_ON_PAGE(!hpage_pincount_available(page), page);
+	page = compound_head(page);
+	return atomic_read(compound_pincount_ptr(page));
+}
+
 static inline void set_compound_order(struct page *page, unsigned int order)
 {
 	page[1].compound_order = order;

@@ -1084,6 +1102,11 @@ void unpin_user_pages(struct page **page
  * refcounts, and b) all the callers of this routine are expected to be able to
  * deal gracefully with a false positive.
  *
+ * For huge pages, the result will be exactly correct. That's because we have
+ * more tracking data available: the 3rd struct page in the compound page is
+ * used to track the pincount (instead using of the GUP_PIN_COUNTING_BIAS
+ * scheme).
+ *
  * For more information, please see Documentation/vm/pin_user_pages.rst.
  *
  * @page:	pointer to page to be queried.

@@ -1092,6 +1115,9 @@ void unpin_user_pages(struct page **page
  */
 static inline bool page_maybe_dma_pinned(struct page *page)
 {
+	if (hpage_pincount_available(page))
+		return compound_pincount(page) > 0;
+
 	/*
 	 * page_ref_count() is signed. If that refcount overflows, then
 	 * page_ref_count() returns a negative value, and callers will avoid

include/linux/mm_types.h renamed from include/linux/mm_types.h~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +6 -1

@@ -137,7 +137,7 @@ struct page {
 		};
 		struct {	/* Second tail page of compound page */
 			unsigned long _compound_pad_1;	/* compound_head */
-			unsigned long _compound_pad_2;
+			atomic_t hpage_pinned_refcount;
 			/* For both global and memcg */
 			struct list_head deferred_list;
 		};

@@ -226,6 +226,11 @@ static inline atomic_t *compound_mapcoun
 	return &page[1].compound_mapcount;
 }
 
+static inline atomic_t *compound_pincount_ptr(struct page *page)
+{
+	return &page[2].hpage_pinned_refcount;
+}
+
 /*
  * Used for sizing the vmemmap region on some architectures
  */

mm/gup.c renamed from mm/gup.c~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +70 -8

129 lines not on this page · raw message

mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +6 -0

@@ -1009,6 +1009,9 @@ static void destroy_compound_gigantic_pa
 	struct page *p = page + 1;
 
 	atomic_set(compound_mapcount_ptr(page), 0);
+	if (hpage_pincount_available(page))
+		atomic_set(compound_pincount_ptr(page), 0);
+
 	for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) {
 		clear_compound_head(p);
 		set_page_refcounted(p);

@@ -1287,6 +1290,9 @@ static void prep_compound_gigantic_page(
 		set_compound_head(p, page);
 	}
 	atomic_set(compound_mapcount_ptr(page), -1);
+
+	if (hpage_pincount_available(page))
+		atomic_set(compound_pincount_ptr(page), 0);
 }
 
 /*

mm/page_alloc.c renamed from mm/page_alloc.c~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +2 -0

@@ -688,6 +688,8 @@ void prep_compound_page(struct page *pag
 		set_compound_head(p, page);
 	}
 	atomic_set(compound_mapcount_ptr(page), -1);
+	if (hpage_pincount_available(page))
+		atomic_set(compound_pincount_ptr(page), 0);
 }
 
 #ifdef CONFIG_DEBUG_PAGEALLOC

mm/rmap.c renamed from mm/rmap.c~mm-gup-page-hpage_pinned_refcount-exact-pin-counts-for-huge-pages +6 -0

@@ -1178,6 +1178,9 @@ void page_add_new_anon_rmap(struct page
 		VM_BUG_ON_PAGE(!PageTransHuge(page), page);
 		/* increment count (starts at -1) */
 		atomic_set(compound_mapcount_ptr(page), 0);
+		if (hpage_pincount_available(page))
+			atomic_set(compound_pincount_ptr(page), 0);
+
 		__inc_node_page_state(page, NR_ANON_THPS);
 	} else {
 		/* Anon THP always mapped first with PMD */

@@ -1974,6 +1977,9 @@ void hugepage_add_new_anon_rmap(struct p
 {
 	BUG_ON(address < vma->vm_start || address >= vma->vm_end);
 	atomic_set(compound_mapcount_ptr(page), 0);
+	if (hpage_pincount_available(page))
+		atomic_set(compound_pincount_ptr(page), 0);
+
 	__page_set_anon_rmap(page, vma, address, 1);
 }
 #endif /* CONFIG_HUGETLB_PAGE */

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #52 lore raw

PATCH 45/155 mm/gup: /proc/vmstat: pin_user_pages (FOLL_PIN) reporting

From: John Hubbard <[email protected]>
Subject: mm/gup: /proc/vmstat: pin_user_pages (FOLL_PIN) reporting

Now that pages are "DMA-pinned" via pin_user_page*(), and unpinned via unpin_user_pages*(), we need some visibility into whether all of this is working correctly.

Add two new fields to /proc/vmstat:

nr_foll_pin_acquired
nr_foll_pin_released

These are documented in Documentation/core-api/pin_user_pages.rst. They represent the number of pages (since boot time) that have been pinned ("nr_foll_pin_acquired") and unpinned ("nr_foll_pin_released"), via pin_user_pages*() and unpin_user_pages*().

In the absence of long-running DMA or RDMA operations that hold pages pinned, the above two fields will normally be equal to each other.

Also: update Documentation/core-api/pin_user_pages.rst, to remove an earlier (now confirmed untrue) claim about a performance problem with /proc/vmstat.

Also: update Documentation/core-api/pin_user_pages.rst to rename the new /proc/vmstat entries, to the names listed here.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 Documentation/core-api/pin_user_pages.rst |   33 ++++++++++++++++----
 include/linux/mmzone.h                    |    2 +
 mm/gup.c                                  |   13 +++++++
 mm/vmstat.c                               |    2 +
 4 files changed, 45 insertions(+), 5 deletions(-)

Documentation/core-api/pin_user_pages.rst renamed from Documentation/core-api/pin_user_pages.rst~mm-gup-proc-vmstat-pin_user_pages-foll_pin-reporting +28 -5

@@ -208,12 +208,35 @@ has the following new calls to exercise
 You can monitor how many total dma-pinned pages have been acquired and released
 since the system was booted, via two new /proc/vmstat entries: ::
 
-    /proc/vmstat/nr_foll_pin_requested
-    /proc/vmstat/nr_foll_pin_requested
+    /proc/vmstat/nr_foll_pin_acquired
+    /proc/vmstat/nr_foll_pin_released
 
-Those are both going to show zero, unless CONFIG_DEBUG_VM is set. This is
-because there is a noticeable performance drop in unpin_user_page(), when they
-are activated.
+Under normal conditions, these two values will be equal unless there are any
+long-term [R]DMA pins in place, or during pin/unpin transitions.
+
+* nr_foll_pin_acquired: This is the number of logical pins that have been
+  acquired since the system was powered on. For huge pages, the head page is
+  pinned once for each page (head page and each tail page) within the huge page.
+  This follows the same sort of behavior that get_user_pages() uses for huge
+  pages: the head page is refcounted once for each tail or head page in the huge
+  page, when get_user_pages() is applied to a huge page.
+
+* nr_foll_pin_released: The number of logical pins that have been released since
+  the system was powered on. Note that pages are released (unpinned) on a
+  PAGE_SIZE granularity, even if the original pin was applied to a huge page.
+  Becaused of the pin count behavior described above in "nr_foll_pin_acquired",
+  the accounting balances out, so that after doing this::
+
+    pin_user_pages(huge_page);
+    for (each page in huge_page)
+        unpin_user_page(page);
+
+...the following is expected::
+
+    nr_foll_pin_released == nr_foll_pin_acquired
+
+(...unless it was already out of balance due to a long-term RDMA pin being in
+place.)
 
 References
 ==========

include/linux/mmzone.h renamed from include/linux/mmzone.h~mm-gup-proc-vmstat-pin_user_pages-foll_pin-reporting +2 -0

@@ -243,6 +243,8 @@ enum node_stat_item {
 	NR_DIRTIED,		/* page dirtyings since bootup */
 	NR_WRITTEN,		/* page writings since bootup */
 	NR_KERNEL_MISC_RECLAIMABLE,	/* reclaimable non-slab kernel pages */
+	NR_FOLL_PIN_ACQUIRED,	/* via: pin_user_page(), gup flag: FOLL_PIN */
+	NR_FOLL_PIN_RELEASED,	/* pages returned via unpin_user_page() */
 	NR_VM_NODE_STAT_ITEMS
 };

mm/gup.c renamed from mm/gup.c~mm-gup-proc-vmstat-pin_user_pages-foll_pin-reporting +13 -0

@@ -86,6 +86,8 @@ static __maybe_unused struct page *try_g
 	if (flags & FOLL_GET)
 		return try_get_compound_head(page, refs);
 	else if (flags & FOLL_PIN) {
+		int orig_refs = refs;
+
 		/*
 		 * When pinning a compound page of order > 1 (which is what
 		 * hpage_pincount_available() checks for), use an exact count to

@@ -104,6 +106,9 @@ static __maybe_unused struct page *try_g
 		if (hpage_pincount_available(page))
 			hpage_pincount_add(page, refs);
 
+		mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_ACQUIRED,
+				    orig_refs);
+
 		return page;
 	}

@@ -158,6 +163,8 @@ bool __must_check try_grab_page(struct p
 		 * once, so that the page really is pinned.
 		 */
 		page_ref_add(page, refs);
+
+		mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_ACQUIRED, 1);
 	}
 
 	return true;

@@ -178,6 +185,7 @@ static bool __unpin_devmap_managed_user_
 
 	count = page_ref_sub_return(page, refs);
 
+	mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, 1);
 	/*
 	 * devmap page refcounts are 1-based, rather than 0-based: if
 	 * refcount is 1, then the page is free and the refcount is

@@ -228,6 +236,8 @@ void unpin_user_page(struct page *page)
 
 	if (page_ref_sub_and_test(page, refs))
 		__put_page(page);
+
+	mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, 1);
 }
 EXPORT_SYMBOL(unpin_user_page);

@@ -2014,6 +2024,9 @@ EXPORT_SYMBOL(get_user_pages_unlocked);
 static void put_compound_head(struct page *page, int refs, unsigned int flags)
 {
 	if (flags & FOLL_PIN) {
+		mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED,
+				    refs);
+
 		if (hpage_pincount_available(page))
 			hpage_pincount_sub(page, refs);
 		else

mm/vmstat.c renamed from mm/vmstat.c~mm-gup-proc-vmstat-pin_user_pages-foll_pin-reporting +2 -0

@@ -1168,6 +1168,8 @@ const char * const vmstat_text[] = {
 	"nr_dirtied",
 	"nr_written",
 	"nr_kernel_misc_reclaimable",
+	"nr_foll_pin_acquired",
+	"nr_foll_pin_released",
 
 	/* enum writeback_stat_item counters */
 	"nr_dirty_threshold",

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #53 lore raw

PATCH 46/155 mm/gup_benchmark: support pin_user_pages() and related calls

From: John Hubbard <[email protected]>
Subject: mm/gup_benchmark: support pin_user_pages() and related calls

Up until now, gup_benchmark supported testing of the following kernel functions:

* get_user_pages(): via the '-U' command line option
* get_user_pages_longterm(): via the '-L' command line option
* get_user_pages_fast(): as the default (no options required)

Add test coverage for the new corresponding pin_*() functions:

* pin_user_pages_fast(): via the '-a' command line option
* pin_user_pages(): via the '-b' command line option

Also, add an option for clarity: '-u' for what is now (still) the default
choice: get_user_pages_fast().

Also, for the commands that set FOLL_PIN, verify that the pages really are dma-pinned, via the new is_dma_pinned() routine. Those commands are:

PIN_FAST_BENCHMARK : calls pin_user_pages_fast()
PIN_BENCHMARK : calls pin_user_pages()

In between the calls to pin_*() and unpin_user_pages(), check each page:
if page_maybe_dma_pinned() returns false, then WARN and return.

Do this outside of the benchmark timestamps, so that it doesn't affect reported times.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Jan Kara <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/gup_benchmark.c                         |   71 +++++++++++++++++--
 tools/testing/selftests/vm/gup_benchmark.c |   15 +++-
 2 files changed, 80 insertions(+), 6 deletions(-)

mm/gup_benchmark.c renamed from mm/gup_benchmark.c~mm-gup_benchmark-support-pin_user_pages-and-related-calls +66 -5

@@ -8,6 +8,8 @@
 #define GUP_FAST_BENCHMARK	_IOWR('g', 1, struct gup_benchmark)
 #define GUP_LONGTERM_BENCHMARK	_IOWR('g', 2, struct gup_benchmark)
 #define GUP_BENCHMARK		_IOWR('g', 3, struct gup_benchmark)
+#define PIN_FAST_BENCHMARK	_IOWR('g', 4, struct gup_benchmark)
+#define PIN_BENCHMARK		_IOWR('g', 5, struct gup_benchmark)
 
 struct gup_benchmark {
 	__u64 get_delta_usec;

@@ -19,6 +21,48 @@ struct gup_benchmark {
 	__u64 expansion[10];	/* For future use */
 };
 
+static void put_back_pages(unsigned int cmd, struct page **pages,
+			   unsigned long nr_pages)
+{
+	unsigned long i;
+
+	switch (cmd) {
+	case GUP_FAST_BENCHMARK:
+	case GUP_LONGTERM_BENCHMARK:
+	case GUP_BENCHMARK:
+		for (i = 0; i < nr_pages; i++)
+			put_page(pages[i]);
+		break;
+
+	case PIN_FAST_BENCHMARK:
+	case PIN_BENCHMARK:
+		unpin_user_pages(pages, nr_pages);
+		break;
+	}
+}
+
+static void verify_dma_pinned(unsigned int cmd, struct page **pages,
+			      unsigned long nr_pages)
+{
+	unsigned long i;
+	struct page *page;
+
+	switch (cmd) {
+	case PIN_FAST_BENCHMARK:
+	case PIN_BENCHMARK:
+		for (i = 0; i < nr_pages; i++) {
+			page = pages[i];
+			if (WARN(!page_maybe_dma_pinned(page),
+				 "pages[%lu] is NOT dma-pinned\n", i)) {
+
+				dump_page(page, "gup_benchmark failure");
+				break;
+			}
+		}
+		break;
+	}
+}
+
 static int __gup_benchmark_ioctl(unsigned int cmd,
 		struct gup_benchmark *gup)
 {

@@ -66,6 +110,14 @@ static int __gup_benchmark_ioctl(unsigne
 			nr = get_user_pages(addr, nr, gup->flags, pages + i,
 					    NULL);
 			break;
+		case PIN_FAST_BENCHMARK:
+			nr = pin_user_pages_fast(addr, nr, gup->flags,
+						 pages + i);
+			break;
+		case PIN_BENCHMARK:
+			nr = pin_user_pages(addr, nr, gup->flags, pages + i,
+					    NULL);
+			break;
 		default:
 			kvfree(pages);
 			ret = -EINVAL;

@@ -78,15 +130,22 @@ static int __gup_benchmark_ioctl(unsigne
 	}
 	end_time = ktime_get();
 
+	/* Shifting the meaning of nr_pages: now it is actual number pinned: */
+	nr_pages = i;
+
 	gup->get_delta_usec = ktime_us_delta(end_time, start_time);
 	gup->size = addr - gup->addr;
 
+	/*
+	 * Take an un-benchmark-timed moment to verify DMA pinned
+	 * state: print a warning if any non-dma-pinned pages are found:
+	 */
+	verify_dma_pinned(cmd, pages, nr_pages);
+
 	start_time = ktime_get();
-	for (i = 0; i < nr_pages; i++) {
-		if (!pages[i])
-			break;
-		put_page(pages[i]);
-	}
+
+	put_back_pages(cmd, pages, nr_pages);
+
 	end_time = ktime_get();
 	gup->put_delta_usec = ktime_us_delta(end_time, start_time);

@@ -105,6 +164,8 @@ static long gup_benchmark_ioctl(struct f
 	case GUP_FAST_BENCHMARK:
 	case GUP_LONGTERM_BENCHMARK:
 	case GUP_BENCHMARK:
+	case PIN_FAST_BENCHMARK:
+	case PIN_BENCHMARK:
 		break;
 	default:
 		return -EINVAL;

tools/testing/selftests/vm/gup_benchmark.c renamed from tools/testing/selftests/vm/gup_benchmark.c~mm-gup_benchmark-support-pin_user_pages-and-related-calls +14 -1

@@ -18,6 +18,10 @@
 #define GUP_LONGTERM_BENCHMARK	_IOWR('g', 2, struct gup_benchmark)
 #define GUP_BENCHMARK		_IOWR('g', 3, struct gup_benchmark)
 
+/* Similar to above, but use FOLL_PIN instead of FOLL_GET. */
+#define PIN_FAST_BENCHMARK	_IOWR('g', 4, struct gup_benchmark)
+#define PIN_BENCHMARK		_IOWR('g', 5, struct gup_benchmark)
+
 /* Just the flags we need, copied from mm.h: */
 #define FOLL_WRITE	0x01	/* check pte is writable */

@@ -40,8 +44,14 @@ int main(int argc, char **argv)
 	char *file = "/dev/zero";
 	char *p;
 
-	while ((opt = getopt(argc, argv, "m:r:n:f:tTLUwSH")) != -1) {
+	while ((opt = getopt(argc, argv, "m:r:n:f:abtTLUuwSH")) != -1) {
 		switch (opt) {
+		case 'a':
+			cmd = PIN_FAST_BENCHMARK;
+			break;
+		case 'b':
+			cmd = PIN_BENCHMARK;
+			break;
 		case 'm':
 			size = atoi(optarg) * MB;
 			break;

@@ -63,6 +73,9 @@ int main(int argc, char **argv)
 		case 'U':
 			cmd = GUP_BENCHMARK;
 			break;
+		case 'u':
+			cmd = GUP_FAST_BENCHMARK;
+			break;
 		case 'w':
 			write = 1;
 			break;

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #54 lore raw

PATCH 47/155 selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN coverage

From: John Hubbard <[email protected]>
Subject: selftests/vm: run_vmtests: invoke gup_benchmark with basic FOLL_PIN coverage

It's good to have basic unit test coverage of the new FOLL_PIN behavior. Fortunately, the gup_benchmark unit test is extremely fast (a few milliseconds), so adding it the the run_vmtests suite is going to cause no noticeable change in running time.

So, add two new invocations to run_vmtests:

1) Run gup_benchmark with normal get_user_pages().

2) Run gup_benchmark with pin_user_pages(). This is much like the first call, except that it sets FOLL_PIN.

Running these two in quick succession also provide a visual comparison of the running times, which is convenient.

The new invocations are fairly early in the run_vmtests script, because with test suites, it's usually preferable to put the shorter, faster tests first, all other things being equal.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Jan Kara <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc Kirill A. Shutemov <[email protected]>

Cc "Matthew Wilcox (Oracle)" <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 tools/testing/selftests/vm/run_vmtests |   22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

tools/testing/selftests/vm/run_vmtests renamed from tools/testing/selftests/vm/run_vmtests~selftests-vm-run_vmtests-invoke-gup_benchmark-with-basic-foll_pin-coverage +22 -0

@@ -123,6 +123,28 @@ else
 	echo "[PASS]"
 fi
 
+echo "--------------------------------------------"
+echo "running 'gup_benchmark -U' (normal/slow gup)"
+echo "--------------------------------------------"
+./gup_benchmark -U
+if [ $? -ne 0 ]; then
+	echo "[FAIL]"
+	exitcode=1
+else
+	echo "[PASS]"
+fi
+
+echo "------------------------------------------"
+echo "running gup_benchmark -b (pin_user_pages)"
+echo "------------------------------------------"
+./gup_benchmark -b
+if [ $? -ne 0 ]; then
+	echo "[FAIL]"
+	exitcode=1
+else
+	echo "[PASS]"
+fi
+
 echo "-------------------"
 echo "running userfaultfd"
 echo "-------------------"

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #55 lore raw

PATCH 48/155 mm: improve dump_page() for compound pages

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: improve dump_page() for compound pages

There was no protection against a corrupted struct page having an implausible compound_head(). Sanity check that a compound page has a head within reach of the maximum allocatable page (this will need to be adjusted if one of the plans to allocate 1GB pages comes to fruition). In addition,

- Print the mapping pointer using %p insted of %px. The actual value of the pointer can be read out of the raw page dump and using %p gives a chance to correlate it with an earlier printk of the mapping pointer
- Print the mapping pointer from the head page, not the tail page (the tail ->mapping pointer may be in use for other purposes, eg part of a list_head)
- Print the order of the page for compound pages
- Dump the raw head page as well as the raw page
- Print the refcount from the head page, not the tail page

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Co-developed-by John Hubbard <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Jan Kara <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 mm/debug.c |   33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)

mm/debug.c renamed from mm/debug.c~mm-improve-dump_page-for-compound-pages +23 ~~-10~~

@@ -44,8 +44,10 @@ const struct trace_print_flags vmaflag_n
 
 void __dump_page(struct page *page, const char *reason)
 {
+	struct page *head = compound_head(page);
 	struct address_space *mapping;
 	bool page_poisoned = PagePoisoned(page);
+	bool compound = PageCompound(page);
 	/*
 	 * Accessing the pageblock without the zone lock. It could change to
 	 * "isolate" again in the meantime, but since we are just dumping the

@@ -66,25 +68,32 @@ void __dump_page(struct page *page, cons
 		goto hex_only;
 	}
 
-	mapping = page_mapping(page);
+	if (page < head || (page >= head + MAX_ORDER_NR_PAGES)) {
+		/* Corrupt page, cannot call page_mapping */
+		mapping = page->mapping;
+		head = page;
+		compound = false;
+	} else {
+		mapping = page_mapping(page);
+	}
 
 	/*
 	 * Avoid VM_BUG_ON() in page_mapcount().
 	 * page->_mapcount space in struct page is used by sl[aou]b pages to
 	 * encode own info.
 	 */
-	mapcount = PageSlab(page) ? 0 : page_mapcount(page);
+	mapcount = PageSlab(head) ? 0 : page_mapcount(page);
 
-	if (PageCompound(page))
-		pr_warn("page:%px refcount:%d mapcount:%d mapping:%px "
-			"index:%#lx compound_mapcount: %d\n",
-			page, page_ref_count(page), mapcount,
-			page->mapping, page_to_pgoff(page),
-			compound_mapcount(page));
+	if (compound)
+		pr_warn("page:%px refcount:%d mapcount:%d mapping:%p "
+			"index:%#lx head:%px order:%u compound_mapcount:%d\n",
+			page, page_ref_count(head), mapcount,
+			mapping, page_to_pgoff(page), head,
+			compound_order(head), compound_mapcount(page));
 	else
-		pr_warn("page:%px refcount:%d mapcount:%d mapping:%px index:%#lx\n",
+		pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n",
 			page, page_ref_count(page), mapcount,
-			page->mapping, page_to_pgoff(page));
+			mapping, page_to_pgoff(page));
 	if (PageKsm(page))
 		type = "ksm ";
 	else if (PageAnon(page))

@@ -106,6 +115,10 @@ hex_only:
 	print_hex_dump(KERN_WARNING, "raw: ", DUMP_PREFIX_NONE, 32,
 			sizeof(unsigned long), page,
 			sizeof(struct page), false);
+	if (head != page)
+		print_hex_dump(KERN_WARNING, "head: ", DUMP_PREFIX_NONE, 32,
+			sizeof(unsigned long), head,
+			sizeof(struct page), false);
 
 	if (reason)
 		pr_warn("page dumped because: %s\n", reason);

Andrew Morton

author maintainer

2020-04-02 04:05 UTC #56 lore raw

PATCH 49/155 mm: dump_page(): additional diagnostics for huge pinned pages

From: John Hubbard <[email protected]>
Subject: mm: dump_page(): additional diagnostics for huge pinned pages

As part of pin_user_pages() and related API calls, pages are "dma-pinned". For the case of compound pages of order > 1, the per-page accounting of dma pins is accomplished via the 3rd struct page in the compound page. In order to support debugging of any pin_user_pages()- related problems, enhance dump_page() so as to report the pin count in that case.

Documentation/core-api/pin_user_pages.rst is also updated accordingly.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Jan Kara <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 Documentation/core-api/pin_user_pages.rst |    7 ++++++
 mm/debug.c                                |   21 +++++++++++++++-----
 2 files changed, 23 insertions(+), 5 deletions(-)

Documentation/core-api/pin_user_pages.rst renamed from Documentation/core-api/pin_user_pages.rst~mm-dump_page-additional-diagnostics-for-huge-pinned-pages +7 -0

@@ -238,6 +238,13 @@ long-term [R]DMA pins in place, or durin
 (...unless it was already out of balance due to a long-term RDMA pin being in
 place.)
 
+Other diagnostics
+=================
+
+dump_page() has been enhanced slightly, to handle these new counting fields, and
+to better report on compound pages in general. Specifically, for compound pages
+with order > 1, the exact (hpage_pinned_refcount) pincount is reported.
+
 References
 ==========

mm/debug.c renamed from mm/debug.c~mm-dump_page-additional-diagnostics-for-huge-pinned-pages +16 -5

@@ -85,11 +85,22 @@ void __dump_page(struct page *page, cons
 	mapcount = PageSlab(head) ? 0 : page_mapcount(page);
 
 	if (compound)
-		pr_warn("page:%px refcount:%d mapcount:%d mapping:%p "
-			"index:%#lx head:%px order:%u compound_mapcount:%d\n",
-			page, page_ref_count(head), mapcount,
-			mapping, page_to_pgoff(page), head,
-			compound_order(head), compound_mapcount(page));
+		if (hpage_pincount_available(page)) {
+			pr_warn("page:%px refcount:%d mapcount:%d mapping:%p "
+				"index:%#lx head:%px order:%u "
+				"compound_mapcount:%d compound_pincount:%d\n",
+				page, page_ref_count(head), mapcount,
+				mapping, page_to_pgoff(page), head,
+				compound_order(head), compound_mapcount(page),
+				compound_pincount(page));
+		} else {
+			pr_warn("page:%px refcount:%d mapcount:%d mapping:%p "
+				"index:%#lx head:%px order:%u "
+				"compound_mapcount:%d\n",
+				page, page_ref_count(head), mapcount,
+				mapping, page_to_pgoff(page), head,
+				compound_order(head), compound_mapcount(page));
+		}
 	else
 		pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n",
 			page, page_ref_count(page), mapcount,

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #57 lore raw

PATCH 50/155 mm/gup/writeback: add callbacks for inaccessible pages

From: Claudio Imbrenda <[email protected]>
Subject: mm/gup/writeback: add callbacks for inaccessible pages

With the introduction of protected KVM guests on s390 there is now a concept of inaccessible pages. These pages need to be made accessible before the host can access them.

While cpu accesses will trigger a fault that can be resolved, I/O accesses will just fail. We need to add a callback into architecture code for places that will do I/O, namely when writeback is started or when a page reference is taken.

This is not only to enable paging, file backing etc, it is also necessary to protect the host against a malicious user space. For example a bad QEMU could simply start direct I/O on such protected memory. We do not want userspace to be able to trigger I/O errors and thus the logic is "whenever somebody accesses that page (gup) or does I/O, make sure that this page can be accessed". When the guest tries to access that page we will wait in the page fault handler for writeback to have finished and for the page_ref to be the expected value.

On s390x the function is not supposed to fail, so it is ok to use a WARN_ON on failure. If we ever need some more finegrained handling we can tackle this when we know the details.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Claudio Imbrenda <[email protected]>

Acked-by Will Deacon <[email protected]>

Reviewed-by Christian Borntraeger <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Cc Jan Kara <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Jérôme Glisse <[email protected]>

Cc Al Viro <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Dan Williams <[email protected]>

Cc Dave Chinner <[email protected]>

Cc Jason Gunthorpe <[email protected]>

Cc Jonathan Corbet <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Vlastimil Babka <[email protected]>

---

 include/linux/gfp.h |    6 ++++++
 mm/gup.c            |   30 +++++++++++++++++++++++++++---
 mm/page-writeback.c |    9 ++++++++-
 3 files changed, 41 insertions(+), 4 deletions(-)

include/linux/gfp.h renamed from include/linux/gfp.h~mm-gup-writeback-add-callbacks-for-inaccessible-pages +6 -0

@@ -485,6 +485,12 @@ static inline void arch_free_page(struct
 #ifndef HAVE_ARCH_ALLOC_PAGE
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
+#ifndef HAVE_ARCH_MAKE_PAGE_ACCESSIBLE
+static inline int arch_make_page_accessible(struct page *page)
+{
+	return 0;
+}
+#endif
 
 struct page *
 __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,

mm/gup.c renamed from mm/gup.c~mm-gup-writeback-add-callbacks-for-inaccessible-pages +27 -3

@@ -390,6 +390,7 @@ static struct page *follow_page_pte(stru
 	struct page *page;
 	spinlock_t *ptl;
 	pte_t *ptep, pte;
+	int ret;
 
 	/* FOLL_GET and FOLL_PIN are mutually exclusive. */
 	if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) ==

@@ -448,8 +449,6 @@ retry:
 		if (is_zero_pfn(pte_pfn(pte))) {
 			page = pte_page(pte);
 		} else {
-			int ret;
-
 			ret = follow_pfn_pte(vma, address, ptep, flags);
 			page = ERR_PTR(ret);
 			goto out;

@@ -457,7 +456,6 @@ retry:
 	}
 
 	if (flags & FOLL_SPLIT && PageTransCompound(page)) {
-		int ret;
 		get_page(page);
 		pte_unmap_unlock(ptep, ptl);
 		lock_page(page);

@@ -474,6 +472,19 @@ retry:
 		page = ERR_PTR(-ENOMEM);
 		goto out;
 	}
+	/*
+	 * We need to make the page accessible if and only if we are going
+	 * to access its content (the FOLL_PIN case).  Please see
+	 * Documentation/core-api/pin_user_pages.rst for details.
+	 */
+	if (flags & FOLL_PIN) {
+		ret = arch_make_page_accessible(page);
+		if (ret) {
+			unpin_user_page(page);
+			page = ERR_PTR(ret);
+			goto out;
+		}
+	}
 	if (flags & FOLL_TOUCH) {
 		if ((flags & FOLL_WRITE) &&
 		    !pte_dirty(pte) && !PageDirty(page))

@@ -2163,6 +2174,19 @@ static int gup_pte_range(pmd_t pmd, unsi
 
 		VM_BUG_ON_PAGE(compound_head(page) != head, page);
 
+		/*
+		 * We need to make the page accessible if and only if we are
+		 * going to access its content (the FOLL_PIN case).  Please
+		 * see Documentation/core-api/pin_user_pages.rst for
+		 * details.
+		 */
+		if (flags & FOLL_PIN) {
+			ret = arch_make_page_accessible(page);
+			if (ret) {
+				unpin_user_page(page);
+				goto pte_unmap;
+			}
+		}
 		SetPageReferenced(page);
 		pages[*nr] = page;
 		(*nr)++;

mm/page-writeback.c renamed from mm/page-writeback.c~mm-gup-writeback-add-callbacks-for-inaccessible-pages +8 -1

@@ -2764,7 +2764,7 @@ int test_clear_page_writeback(struct pag
 int __test_set_page_writeback(struct page *page, bool keep_write)
 {
 	struct address_space *mapping = page_mapping(page);
-	int ret;
+	int ret, access_ret;
 
 	lock_page_memcg(page);
 	if (mapping && mapping_use_writeback_tags(mapping)) {

@@ -2807,6 +2807,13 @@ int __test_set_page_writeback(struct pag
 		inc_zone_page_state(page, NR_ZONE_WRITE_PENDING);
 	}
 	unlock_page_memcg(page);
+	access_ret = arch_make_page_accessible(page);
+	/*
+	 * If writeback has been triggered on a page that cannot be made
+	 * accessible, it is too late to recover here.
+	 */
+	VM_BUG_ON_PAGE(access_ret != 0, page);
+
 	return ret;
 
 }

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #58 lore raw

PATCH 51/155 mm/gup: rename nr as nr_pinned in get_user_pages_fast()

From: Pingfan Liu <[email protected]>
Subject: mm/gup: rename nr as nr_pinned in get_user_pages_fast()

To better reflect the held state of pages and make code self-explaining, rename nr as nr_pinned.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Pingfan Liu <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Dan Williams <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Jason Gunthorpe <[email protected]>

---

 mm/gup.c |   26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-rename-nr-as-nr_pinned-in-get_user_pages_fast +13 ~~-13~~

@@ -2637,7 +2637,7 @@ int __get_user_pages_fast(unsigned long
 {
 	unsigned long len, end;
 	unsigned long flags;
-	int nr = 0;
+	int nr_pinned = 0;
 	/*
 	 * Internally (within mm/gup.c), gup fast variants must set FOLL_GET,
 	 * because gup fast is always a "pin with a +1 page refcount" request.

@@ -2671,11 +2671,11 @@ int __get_user_pages_fast(unsigned long
 	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP) &&
 	    gup_fast_permitted(start, end)) {
 		local_irq_save(flags);
-		gup_pgd_range(start, end, gup_flags, pages, &nr);
+		gup_pgd_range(start, end, gup_flags, pages, &nr_pinned);
 		local_irq_restore(flags);
 	}
 
-	return nr;
+	return nr_pinned;
 }
 EXPORT_SYMBOL_GPL(__get_user_pages_fast);

@@ -2707,7 +2707,7 @@ static int internal_get_user_pages_fast(
 					struct page **pages)
 {
 	unsigned long addr, len, end;
-	int nr = 0, ret = 0;
+	int nr_pinned = 0, ret = 0;
 
 	if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM |
 				       FOLL_FORCE | FOLL_PIN | FOLL_GET)))

@@ -2726,25 +2726,25 @@ static int internal_get_user_pages_fast(
 	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP) &&
 	    gup_fast_permitted(start, end)) {
 		local_irq_disable();
-		gup_pgd_range(addr, end, gup_flags, pages, &nr);
+		gup_pgd_range(addr, end, gup_flags, pages, &nr_pinned);
 		local_irq_enable();
-		ret = nr;
+		ret = nr_pinned;
 	}
 
-	if (nr < nr_pages) {
+	if (nr_pinned < nr_pages) {
 		/* Try to get the remaining pages with get_user_pages */
-		start += nr << PAGE_SHIFT;
-		pages += nr;
+		start += nr_pinned << PAGE_SHIFT;
+		pages += nr_pinned;
 
-		ret = __gup_longterm_unlocked(start, nr_pages - nr,
+		ret = __gup_longterm_unlocked(start, nr_pages - nr_pinned,
 					      gup_flags, pages);
 
 		/* Have to be a bit careful with return values */
-		if (nr > 0) {
+		if (nr_pinned > 0) {
 			if (ret < 0)
-				ret = nr;
+				ret = nr_pinned;
 			else
-				ret += nr;
+				ret += nr_pinned;
 		}
 	}

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #59 lore raw

PATCH 52/155 mm/gup: fix omission of check on FOLL_LONGTERM in gup fast path

From: Pingfan Liu <[email protected]>
Subject: mm/gup: fix omission of check on FOLL_LONGTERM in gup fast path

FOLL_LONGTERM is a special case of FOLL_PIN. It suggests a pin which is going to be given to hardware and can't move. It would truncate CMA permanently and should be excluded.

In gup slow path, where __gup_longterm_locked->check_and_migrate_cma_pages() handles FOLL_LONGTERM, but in fast path, there lacks such a check, which means a possible leak of CMA page to longterm pinned.

Place a check in try_grab_compound_head() in the fast path to fix the leak, and if FOLL_LONGTERM happens on CMA, it will fall back to slow path to migrate the page.

Some note about the check: Huge page's subpages have the same migrate type due to either allocation from a free_list[] or alloc_contig_range() with param MIGRATE_MOVABLE. So it is enough to check on a single subpage by is_migrate_cma_page(subpage)

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Pingfan Liu <[email protected]>

Reviewed-by Jason Gunthorpe <[email protected]>

Cc Ira Weiny <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Dan Williams <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc John Hubbard <[email protected]>

Cc "Aneesh Kumar K.V" <[email protected]>

Cc Christoph Hellwig <[email protected]>

Cc Shuah Khan <[email protected]>

Cc Jason Gunthorpe <[email protected]>

---

 mm/gup.c |    8 ++++++++
 1 file changed, 8 insertions(+)

mm/gup.c renamed from mm/gup.c~mm-gup-fix-omission-of-check-on-foll_longterm-in-gup-fast-path +8 -0

@@ -89,6 +89,14 @@ static __maybe_unused struct page *try_g
 		int orig_refs = refs;
 
 		/*
+		 * Can't do FOLL_LONGTERM + FOLL_PIN with CMA in the gup fast
+		 * path, so fail and let the caller fall back to the slow path.
+		 */
+		if (unlikely(flags & FOLL_LONGTERM) &&
+				is_migrate_cma_page(page))
+			return NULL;
+
+		/*
 		 * When pinning a compound page of order > 1 (which is what
 		 * hpage_pincount_available() checks for), use an exact count to
 		 * track it, via hpage_pincount_add/_sub().

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #60 lore raw

PATCH 53/155 mm/swapfile.c: fix comments for swapcache_prepare

From: Chen Wandun <[email protected]>
Subject: mm/swapfile.c: fix comments for swapcache_prepare

The -EEXIST returned by __swap_duplicate means there is a swap cache instead -EBUSY

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Chen Wandun <[email protected]>

---

 mm/swapfile.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/swapfile.c renamed from mm/swapfile.c~mm-swapfilec-fix-comments-for-swapcache_prepare +1 -1

@@ -3475,7 +3475,7 @@ int swap_duplicate(swp_entry_t entry)
  *
  * Called when allocating swap cache for existing swap entry,
  * This can return error codes. Returns 0 at success.
- * -EBUSY means there is a swap cache.
+ * -EEXIST means there is a swap cache.
  * Note: return code is different from swap_duplicate().
  */
 int swapcache_prepare(swp_entry_t entry)

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #61 lore raw

PATCH 54/155 mm/swap.c: not necessary to export __pagevec_lru_add()

From: Wei Yang <[email protected]>
Subject: mm/swap.c: not necessary to export __pagevec_lru_add()

__pagevec_lru_add() is only used in mm directory now.

Remove the export symbol.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Wei Yang <[email protected]>

---

 mm/swap.c |    1 -
 1 file changed, 1 deletion(-)

mm/swap.c renamed from mm/swap.c~mm-swapc-not-necessary-to-export-__pagevec_lru_add +0 -1

@@ -986,7 +986,6 @@ void __pagevec_lru_add(struct pagevec *p
 {
 	pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn, NULL);
 }
-EXPORT_SYMBOL(__pagevec_lru_add);
 
 /**
  * pagevec_lookup_entries - gang pagecache lookup

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #62 lore raw

PATCH 55/155 mm/swapfile: fix data races in try_to_unuse()

From: Qian Cai <[email protected]>
Subject: mm/swapfile: fix data races in try_to_unuse()

si->inuse_pages could be accessed concurrently as noticed by KCSAN,

write to 0xffff98b00ebd04dc of 4 bytes by task 82262 on cpu 92:

  swap_range_free+0xbe/0x230
  swap_range_free at mm/swapfile.c:719
  swapcache_free_entries+0x1be/0x250
  free_swap_slot+0x1c8/0x220
  __swap_entry_free.constprop.19+0xa3/0xb0
  free_swap_and_cache+0x53/0xa0
  unmap_page_range+0x7e0/0x1ce0
  unmap_single_vma+0xcd/0x170
  unmap_vmas+0x18b/0x220
  exit_mmap+0xee/0x220
  mmput+0xe7/0x240
  do_exit+0x598/0xfd0
  do_group_exit+0x8b/0x180
  get_signal+0x293/0x13d0
  do_signal+0x37/0x5d0
  prepare_exit_to_usermode+0x1b7/0x2c0
  ret_from_intr+0x32/0x42

read to 0xffff98b00ebd04dc of 4 bytes by task 82499 on cpu 46:

  try_to_unuse+0x86b/0xc80
  try_to_unuse at mm/swapfile.c:2185
  __x64_sys_swapoff+0x372/0xd40
  do_syscall_64+0x91/0xb05
  entry_SYSCALL_64_after_hwframe+0x49/0xbe

The plain reads in try_to_unuse() are outside si->lock critical section which result in data races that could be dangerous to be used in a loop. Fix them by adding READ_ONCE().

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Qian Cai <[email protected]>

Cc Marco Elver <[email protected]>

Cc Hugh Dickins <[email protected]>

---

 mm/swapfile.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

mm/swapfile.c renamed from mm/swapfile.c~mm-swapfile-fix-data-races-in-try_to_unuse +4 -4

@@ -2132,7 +2132,7 @@ int try_to_unuse(unsigned int type, bool
 	swp_entry_t entry;
 	unsigned int i;
 
-	if (!si->inuse_pages)
+	if (!READ_ONCE(si->inuse_pages))
 		return 0;
 
 	if (!frontswap)

@@ -2148,7 +2148,7 @@ retry:
 
 	spin_lock(&mmlist_lock);
 	p = &init_mm.mmlist;
-	while (si->inuse_pages &&
+	while (READ_ONCE(si->inuse_pages) &&
 	       !signal_pending(current) &&
 	       (p = p->next) != &init_mm.mmlist) {

@@ -2177,7 +2177,7 @@ retry:
 	mmput(prev_mm);
 
 	i = 0;
-	while (si->inuse_pages &&
+	while (READ_ONCE(si->inuse_pages) &&
 	       !signal_pending(current) &&
 	       (i = find_next_to_unuse(si, i, frontswap)) != 0) {

@@ -2219,7 +2219,7 @@ retry:
 	 * been preempted after get_swap_page(), temporarily hiding that swap.
 	 * It's easy and robust (though cpu-intensive) just to keep retrying.
 	 */
-	if (si->inuse_pages) {
+	if (READ_ONCE(si->inuse_pages)) {
 		if (!signal_pending(current))
 			goto retry;
 		retval = -EINTR;

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #63 lore raw

PATCH 56/155 mm/swap_slots.c: assign|reset cache slot by value directly

From: Wei Yang <[email protected]>
Subject: mm/swap_slots.c: assign|reset cache slot by value directly

Currently we use a tmp pointer, pentry, to transfer and reset swap cache slot, which is a little redundant. Swap cache slot stores the entry value directly, assign and reset it by value would be straight forward.

Also this patch merges the else and if, since this is the only case we refill and repeat swap cache.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Wei Yang <[email protected]>

Acked-by Tim Chen <[email protected]>

---

 mm/swap_slots.c |   12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

mm/swap_slots.c renamed from mm/swap_slots.c~mm-swap_slotsc-assignreset-cache-slot-by-value-directly +5 -7

@@ -309,7 +309,7 @@ direct_free:
 
 swp_entry_t get_swap_page(struct page *page)
 {
-	swp_entry_t entry, *pentry;
+	swp_entry_t entry;
 	struct swap_slots_cache *cache;
 
 	entry.val = 0;

@@ -336,13 +336,11 @@ swp_entry_t get_swap_page(struct page *p
 		if (cache->slots) {
 repeat:
 			if (cache->nr) {
-				pentry = &cache->slots[cache->cur++];
-				entry = *pentry;
-				pentry->val = 0;
+				entry = cache->slots[cache->cur];
+				cache->slots[cache->cur++].val = 0;
 				cache->nr--;
-			} else {
-				if (refill_swap_slots_cache(cache))
-					goto repeat;
+			} else if (refill_swap_slots_cache(cache)) {
+				goto repeat;
 			}
 		}
 		mutex_unlock(&cache->alloc_lock);

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #64 lore raw

PATCH 57/155 mm: swap: make page_evictable() inline

From: Yang Shi <[email protected]>
Subject: mm: swap: make page_evictable() inline

When backporting commit 9c4e6b1a7027 ("mm, mlock, vmscan: no more skipping pagevecs") to our 4.9 kernel, our test bench noticed around 10% down with a couple of vm-scalability's test cases (lru-file-readonce, lru-file-readtwice and lru-file-mmap-read). I didn't see that much down on my VM (32c-64g-2nodes). It might be caused by the test configuration, which is 32c-256g with NUMA disabled and the tests were run in root memcg, so the tests actually stress only one inactive and active lru. It sounds not very usual in mordern production environment.

That commit did two major changes:
1. Call page_evictable()
2. Use smp_mb to force the PG_lru set visible

It looks they contribute the most overhead. The page_evictable() is a function which does function prologue and epilogue, and that was used by page reclaim path only. However, lru add is a very hot path, so it sounds better to make it inline. However, it calls page_mapping() which is not inlined either, but the disassemble shows it doesn't do push and pop operations and it sounds not very straightforward to inline it.

Other than this, it sounds smp_mb() is not necessary for x86 since SetPageLRU is atomic which enforces memory barrier already, replace it with smp_mb__after_atomic() in the following patch.

With the two fixes applied, the tests can get back around 5% on that test bench and get back normal on my VM. Since the test bench configuration is not that usual and I also saw around 6% up on the latest upstream, so it sounds good enough IMHO.

The below is test data (lru-file-readtwice throughput) against the v5.6-rc4:

	mainline	w/ inline fix
          150MB            154MB

With this patch the throughput gets 2.67% up. The data with using smp_mb__after_atomic() is showed in the following patch.

Shakeel Butt did the below test:

On a real machine with limiting the 'dd' on a single node and reading 100 GiB sparse file (less than a single node). Just ran a single instance to not cause the lru lock contention. The cmdline used is "dd if=file-100GiB of=/dev/null bs=4k". Ran the cmd 10 times with drop_caches in between and measured the time it took.

Without patch: 56.64143 +- 0.672 sec

With patches: 56.10 +- 0.21 sec

[[email protected]: move page_evictable() to internal.h]

Link http://lkml.kernel.org/r/[email protected] open here

Fixes 9c4e6b1a7027 ("mm, mlock, vmscan: no more skipping pagevecs")

Signed-off-by Yang Shi <[email protected]>

Tested-by Shakeel Butt <[email protected]>

Reviewed-by Matthew Wilcox (Oracle) <[email protected]>

---

 include/linux/pagemap.h |    6 ++----
 include/linux/swap.h    |    1 -
 mm/internal.h           |   23 +++++++++++++++++++++++
 mm/vmscan.c             |   23 -----------------------
 4 files changed, 25 insertions(+), 28 deletions(-)

include/linux/pagemap.h renamed from include/linux/pagemap.h~mm-swap-make-page_evictable-inline +2 -4

@@ -70,11 +70,9 @@ static inline void mapping_clear_unevict
 	clear_bit(AS_UNEVICTABLE, &mapping->flags);
 }
 
-static inline int mapping_unevictable(struct address_space *mapping)
+static inline bool mapping_unevictable(struct address_space *mapping)
 {
-	if (mapping)
-		return test_bit(AS_UNEVICTABLE, &mapping->flags);
-	return !!mapping;
+	return mapping && test_bit(AS_UNEVICTABLE, &mapping->flags);
 }
 
 static inline void mapping_set_exiting(struct address_space *mapping)

include/linux/swap.h renamed from include/linux/swap.h~mm-swap-make-page_evictable-inline +0 -1

@@ -374,7 +374,6 @@ extern int sysctl_min_slab_ratio;
 #define node_reclaim_mode 0
 #endif
 
-extern int page_evictable(struct page *page);
 extern void check_move_unevictable_pages(struct pagevec *pvec);
 
 extern int kswapd_run(int nid);

mm/internal.h renamed from mm/internal.h~mm-swap-make-page_evictable-inline +23 -0

@@ -63,6 +63,29 @@ static inline unsigned long ra_submit(st
 					ra->start, ra->size, ra->async_size);
 }
 
+/**
+ * page_evictable - test whether a page is evictable
+ * @page: the page to test
+ *
+ * Test whether page is evictable--i.e., should be placed on active/inactive
+ * lists vs unevictable list.
+ *
+ * Reasons page might not be evictable:
+ * (1) page's mapping marked unevictable
+ * (2) page is part of an mlocked VMA
+ *
+ */
+static inline bool page_evictable(struct page *page)
+{
+	bool ret;
+
+	/* Prevent address_space of inode and swap cache from being freed */
+	rcu_read_lock();
+	ret = !mapping_unevictable(page_mapping(page)) && !PageMlocked(page);
+	rcu_read_unlock();
+	return ret;
+}
+
 /*
  * Turn a non-refcounted page (->_refcount == 0) into refcounted with
  * a count of one.

mm/vmscan.c renamed from mm/vmscan.c~mm-swap-make-page_evictable-inline +0 ~~-22~~

@@ -4277,29 +4277,6 @@ int node_reclaim(struct pglist_data *pgd
 }
 #endif
 
-/*
- * page_evictable - test whether a page is evictable
- * @page: the page to test
- *
- * Test whether page is evictable--i.e., should be placed on active/inactive
- * lists vs unevictable list.
- *
- * Reasons page might not be evictable:
- * (1) page's mapping marked unevictable
- * (2) page is part of an mlocked VMA
- *
- */
-int page_evictable(struct page *page)
-{
-	int ret;
-
-	/* Prevent address_space of inode and swap cache from being freed */
-	rcu_read_lock();
-	ret = !mapping_unevictable(page_mapping(page)) && !PageMlocked(page);
-	rcu_read_unlock();
-	return ret;
-}

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #65 lore raw

PATCH 58/155 mm: swap: use smp_mb__after_atomic() to order LRU bit set

From: Yang Shi <[email protected]>
Subject: mm: swap: use smp_mb__after_atomic() to order LRU bit set

Memory barrier is needed after setting LRU bit, but smp_mb() is too strong. Some architectures, i.e. x86, imply memory barrier with atomic operations, so replacing it with smp_mb__after_atomic() sounds better, which is nop on strong ordered machines, and full memory barriers on others. With this change the vm-scalability cases would perform better on x86, I saw total 6% improvement with this patch and previous inline fix.

The test data (lru-file-readtwice throughput) against v5.6-rc4:

	mainline	w/ inline fix	w/ both (adding this)
	150MB		154MB		159MB

Link http://lkml.kernel.org/r/[email protected] open here

Fixes 9c4e6b1a7027 ("mm, mlock, vmscan: no more skipping pagevecs")

Signed-off-by Yang Shi <[email protected]>

Tested-by Shakeel Butt <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

---

 mm/swap.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

mm/swap.c renamed from mm/swap.c~mm-swap-use-smp_mb__after_atomic-to-order-lru-bit-set +2 -2

@@ -931,7 +931,6 @@ static void __pagevec_lru_add_fn(struct
 
 	VM_BUG_ON_PAGE(PageLRU(page), page);
 
-	SetPageLRU(page);
 	/*
 	 * Page becomes evictable in two ways:
 	 * 1) Within LRU lock [munlock_vma_page() and __munlock_pagevec()].

@@ -958,7 +957,8 @@ static void __pagevec_lru_add_fn(struct
 	 * looking at the same page) and the evictable page will be stranded
 	 * in an unevictable LRU.
 	 */
-	smp_mb();
+	SetPageLRU(page);
+	smp_mb__after_atomic();
 
 	if (page_evictable(page)) {
 		lru = page_lru(page);

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #66 lore raw

PATCH 59/155 mm/swap_state.c: use the same way to count page in [add_to|delete_from]_swap_cache

From: Wei Yang <[email protected]>
Subject: mm/swap_state.c: use the same way to count page in [add_to|delete_from]_swap_cache

add_to_swap_cache() and delete_from_swap_cache() are counterparts, while currently they use different ways to count pages.

It doesn't break anything because we only have two sizes for PageAnon, but this is confusing and not good practice.

This patch corrects it by making both functions use hpage_nr_pages().

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Wei Yang <[email protected]>

Reviewed-by Matthew Wilcox (Oracle) <[email protected]>

---

 mm/swap_state.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/swap_state.c renamed from mm/swap_state.c~mm-swap_statec-use-the-same-way-to-count-page-in-_swap_cache +1 -1

@@ -116,7 +116,7 @@ int add_to_swap_cache(struct page *page,
 	struct address_space *address_space = swap_address_space(entry);
 	pgoff_t idx = swp_offset(entry);
 	XA_STATE_ORDER(xas, &address_space->i_pages, idx, compound_order(page));
-	unsigned long i, nr = compound_nr(page);
+	unsigned long i, nr = hpage_nr_pages(page);
 
 	VM_BUG_ON_PAGE(!PageLocked(page), page);
 	VM_BUG_ON_PAGE(PageSwapCache(page), page);

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #67 lore raw

PATCH 60/155 mm, memcg: fix build error around the usage of kmem_caches

From: Yafang Shao <[email protected]>
Subject: mm, memcg: fix build error around the usage of kmem_caches

When I manually set default n to MEMCG_KMEM in init/Kconfig, bellow error occurs,

mm/slab_common.c: In function 'memcg_slab_start':
mm/slab_common.c:1530:30: error: 'struct mem_cgroup' has no member named 'kmem_caches'

  return seq_list_start(&memcg->kmem_caches, *pos);
                              ^
mm/slab_common.c: In function 'memcg_slab_next':
mm/slab_common.c:1537:32: error: 'struct mem_cgroup' has no member named
'kmem_caches'
  return seq_list_next(p, &memcg->kmem_caches, pos);
                                ^

mm/slab_common.c: In function 'memcg_slab_show':
mm/slab_common.c:1551:16: error: 'struct mem_cgroup' has no member named 'kmem_caches'
if (p == memcg->kmem_caches.next)

CC arch/x86/xen/smp.o
mm/slab_common.c: In function 'memcg_slab_start':
mm/slab_common.c:1531:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
mm/slab_common.c: In function 'memcg_slab_next':
mm/slab_common.c:1538:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^

That's because kmem_caches is defined only when CONFIG_MEMCG_KMEM is set, while memcg_slab_start() will use it no matter CONFIG_MEMCG_KMEM is defined or not.

By the way, the reason I mannuly undefined CONFIG_MEMCG_KMEM is to verify whether my some other code change is still stable when CONFIG_MEMCG_KMEM is not set. Unfortunately, the existing code has been already unstable since v4.11.

Link http://lkml.kernel.org/r/[email protected] open here

Fixes bc2791f857e1 ("slab: link memcg kmem_caches on their associated memory cgroup")

Signed-off-by Yafang Shao <[email protected]>

Acked-by Andrew Morton <[email protected]>

Cc Tejun Heo <[email protected]>

Cc Vladimir Davydov <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Michal Hocko <[email protected]>

---

 mm/memcontrol.c  |    3 ++-
 mm/slab_common.c |    2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-fix-build-error-around-the-usage-of-kmem_caches +2 -1

@@ -4792,7 +4792,8 @@ static struct cftype mem_cgroup_legacy_f
 		.write = mem_cgroup_reset,
 		.read_u64 = mem_cgroup_read_u64,
 	},
-#if defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG)
+#if defined(CONFIG_MEMCG_KMEM) && \
+	(defined(CONFIG_SLAB) || defined(CONFIG_SLUB_DEBUG))
 	{
 		.name = "kmem.slabinfo",
 		.seq_start = memcg_slab_start,

mm/slab_common.c renamed from mm/slab_common.c~mm-memcg-fix-build-error-around-the-usage-of-kmem_caches +1 -1

@@ -1521,7 +1521,7 @@ void dump_unreclaimable_slab(void)
 	mutex_unlock(&slab_mutex);
 }
 
-#if defined(CONFIG_MEMCG)
+#if defined(CONFIG_MEMCG_KMEM)
 void *memcg_slab_start(struct seq_file *m, loff_t *pos)
 {
 	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #68 lore raw

PATCH 61/155 mm/memcontrol.c: allocate shrinker_map on appropriate NUMA node

From: Kirill Tkhai <[email protected]>
Subject: mm/memcontrol.c: allocate shrinker_map on appropriate NUMA node

The shrinker_map may be touched from any cpu (e.g., a bit there may be set by a task running everywhere) but kswapd is always bound to specific node. So allocate shrinker_map from the related NUMA node to respect its NUMA locality. Also, this follows generic way we use for allocation of memcg's per-node data.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Kirill Tkhai <[email protected]>

Reviewed-by Roman Gushchin <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 mm/memcontrol.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-allocate-shrinker_map-on-appropriate-numa-node +2 -2

@@ -334,7 +334,7 @@ static int memcg_expand_one_shrinker_map
 		if (!old)
 			return 0;
 
-		new = kvmalloc(sizeof(*new) + size, GFP_KERNEL);
+		new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid);
 		if (!new)
 			return -ENOMEM;

@@ -378,7 +378,7 @@ static int memcg_alloc_shrinker_maps(str
 	mutex_lock(&memcg_shrinker_map_mutex);
 	size = memcg_shrinker_map_size;
 	for_each_node(nid) {
-		map = kvzalloc(sizeof(*map) + size, GFP_KERNEL);
+		map = kvzalloc_node(sizeof(*map) + size, GFP_KERNEL, nid);
 		if (!map) {
 			memcg_free_shrinker_maps(memcg);
 			ret = -ENOMEM;

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #69 lore raw

PATCH 62/155 mm: memcg/slab: use mem_cgroup_from_obj()

From: Roman Gushchin <[email protected]>
Subject: mm: memcg/slab: use mem_cgroup_from_obj()

Sometimes we need to get a memcg pointer from a charged kernel object. The right way to get it depends on whether it's a proper slab object or it's backed by raw pages (e.g. it's a vmalloc alloction). In the first case the kmem_cache->memcg_params.memcg indirection should be used; in other cases it's just page->mem_cgroup.

To simplify this task and hide the implementation details let's use the mem_cgroup_from_obj() helper, which takes a pointer to any kernel object and returns a valid memcg pointer or NULL.

Passing a kernel address rather than a pointer to a page will allow to use this helper for per-object (rather than per-page) tracked objects in the future.

The caller is still responsible to ensure that the returned memcg isn't going away underneath: take the rcu read lock, cgroup mutex etc; depending on the context.

mem_cgroup_from_kmem() defined in mm/list_lru.c is now obsolete and can be removed.

Link http://lkml.kernel.org/r/[email protected] open here

Acked-by Yafang Shao <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 mm/list_lru.c   |   12 +-----------
 mm/memcontrol.c |    5 ++---
 2 files changed, 3 insertions(+), 14 deletions(-)

mm/list_lru.c renamed from mm/list_lru.c~mm-memcg-slab-introduce-mem_cgroup_from_obj +1 ~~-11~~

@@ -57,16 +57,6 @@ list_lru_from_memcg_idx(struct list_lru_
 	return &nlru->lru;
 }
 
-static __always_inline struct mem_cgroup *mem_cgroup_from_kmem(void *ptr)
-{
-	struct page *page;
-
-	if (!memcg_kmem_enabled())
-		return NULL;
-	page = virt_to_head_page(ptr);
-	return memcg_from_slab_page(page);
-}
-
 static inline struct list_lru_one *
 list_lru_from_kmem(struct list_lru_node *nlru, void *ptr,
 		   struct mem_cgroup **memcg_ptr)

@@ -77,7 +67,7 @@ list_lru_from_kmem(struct list_lru_node
 	if (!nlru->memcg_lrus)
 		goto out;
 
-	memcg = mem_cgroup_from_kmem(ptr);
+	memcg = mem_cgroup_from_obj(ptr);
 	if (!memcg)
 		goto out;

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-slab-introduce-mem_cgroup_from_obj +2 -3

@@ -759,13 +759,12 @@ void __mod_lruvec_state(struct lruvec *l
 
 void __mod_lruvec_slab_state(void *p, enum node_stat_item idx, int val)
 {
-	struct page *page = virt_to_head_page(p);
-	pg_data_t *pgdat = page_pgdat(page);
+	pg_data_t *pgdat = page_pgdat(virt_to_page(p));
 	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
 
 	rcu_read_lock();
-	memcg = memcg_from_slab_page(page);
+	memcg = mem_cgroup_from_obj(p);
 
 	/* Untracked pages have no memcg, no lruvec. Update only the node */
 	if (!memcg || memcg == root_mem_cgroup) {

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #70 lore raw

PATCH 63/155 mm: kmem: cleanup (__)memcg_kmem_charge_memcg() arguments

From: Roman Gushchin <[email protected]>
Subject: mm: kmem: cleanup (__)memcg_kmem_charge_memcg() arguments

Patch series "mm: memcg: kmem API cleanup", v2.

This patchset aims to clean up the kernel memory charging API. It doesn't bring any functional changes, just removes unused arguments, renames some functions and fixes some comments.

Currently it's not obvious which functions are most basic (memcg_kmem_(un)charge_memcg()) and which are based on them (memcg_kmem_(un)charge()). The patchset renames these functions and removes unused arguments:

TL;DR:
was:
memcg_kmem_charge_memcg(page, gfp, order, memcg) memcg_kmem_uncharge_memcg(memcg, nr_pages) memcg_kmem_charge(page, gfp, order) memcg_kmem_uncharge(page, order)

now:
memcg_kmem_charge(memcg, gfp, nr_pages)
memcg_kmem_uncharge(memcg, nr_pages)
memcg_kmem_charge_page(page, gfp, order)
memcg_kmem_uncharge_page(page, order)

This patch (of 6):

The first argument of memcg_kmem_charge_memcg() and __memcg_kmem_charge_memcg() is the page pointer and it's not used. Let's drop it.

Memcg pointer is passed as the last argument. Move it to the first place for consistency with other memcg functions, e.g. __memcg_kmem_uncharge_memcg() or try_charge().

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 include/linux/memcontrol.h |    9 ++++-----
 mm/memcontrol.c            |    8 +++-----
 mm/slab.h                  |    2 +-
 3 files changed, 8 insertions(+), 11 deletions(-)

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-kmem-cleanup-__memcg_kmem_charge_memcg-arguments +4 -5

@@ -1369,8 +1369,7 @@ void memcg_kmem_put_cache(struct kmem_ca
 #ifdef CONFIG_MEMCG_KMEM
 int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order);
 void __memcg_kmem_uncharge(struct page *page, int order);
-int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order,
-			      struct mem_cgroup *memcg);
+int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp, int order);
 void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
 				 unsigned int nr_pages);

@@ -1407,11 +1406,11 @@ static inline void memcg_kmem_uncharge(s
 		__memcg_kmem_uncharge(page, order);
 }
 
-static inline int memcg_kmem_charge_memcg(struct page *page, gfp_t gfp,
-					  int order, struct mem_cgroup *memcg)
+static inline int memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
+					  int order)
 {
 	if (memcg_kmem_enabled())
-		return __memcg_kmem_charge_memcg(page, gfp, order, memcg);
+		return __memcg_kmem_charge_memcg(memcg, gfp, order);
 	return 0;
 }

mm/memcontrol.c renamed from mm/memcontrol.c~mm-kmem-cleanup-__memcg_kmem_charge_memcg-arguments +3 -5

@@ -2882,15 +2882,13 @@ void memcg_kmem_put_cache(struct kmem_ca
 
 /**
  * __memcg_kmem_charge_memcg: charge a kmem page
- * @page: page to charge
+ * @memcg: memory cgroup to charge
  * @gfp: reclaim mode
  * @order: allocation order
- * @memcg: memory cgroup to charge
  *
  * Returns 0 on success, an error code on failure.
  */
-int __memcg_kmem_charge_memcg(struct page *page, gfp_t gfp, int order,
-			    struct mem_cgroup *memcg)
+int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp, int order)
 {
 	unsigned int nr_pages = 1 << order;
 	struct page_counter *counter;

@@ -2936,7 +2934,7 @@ int __memcg_kmem_charge(struct page *pag
 
 	memcg = get_mem_cgroup_from_current();
 	if (!mem_cgroup_is_root(memcg)) {
-		ret = __memcg_kmem_charge_memcg(page, gfp, order, memcg);
+		ret = __memcg_kmem_charge_memcg(memcg, gfp, order);
 		if (!ret) {
 			page->mem_cgroup = memcg;
 			__SetPageKmemcg(page);

mm/slab.h renamed from mm/slab.h~mm-kmem-cleanup-__memcg_kmem_charge_memcg-arguments +1 -1

@@ -365,7 +365,7 @@ static __always_inline int memcg_charge_
 		return 0;
 	}
 
-	ret = memcg_kmem_charge_memcg(page, gfp, order, memcg);
+	ret = memcg_kmem_charge_memcg(memcg, gfp, order);
 	if (ret)
 		goto out;

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #71 lore raw

PATCH 64/155 mm: kmem: cleanup memcg_kmem_uncharge_memcg() arguments

From: Roman Gushchin <[email protected]>
Subject: mm: kmem: cleanup memcg_kmem_uncharge_memcg() arguments

Drop the unused page argument and put the memcg pointer at the first place. This make the function consistent with its peers:
__memcg_kmem_uncharge_memcg(), memcg_kmem_charge_memcg(), etc.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 include/linux/memcontrol.h |    4 ++--
 mm/slab.h                  |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-kmem-cleanup-memcg_kmem_uncharge_memcg-arguments +2 -2

@@ -1414,8 +1414,8 @@ static inline int memcg_kmem_charge_memc
 	return 0;
 }
 
-static inline void memcg_kmem_uncharge_memcg(struct page *page, int order,
-					     struct mem_cgroup *memcg)
+static inline void memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
+					     int order)
 {
 	if (memcg_kmem_enabled())
 		__memcg_kmem_uncharge_memcg(memcg, 1 << order);

mm/slab.h renamed from mm/slab.h~mm-kmem-cleanup-memcg_kmem_uncharge_memcg-arguments +1 -1

@@ -395,7 +395,7 @@ static __always_inline void memcg_unchar
 	if (likely(!mem_cgroup_is_root(memcg))) {
 		lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page));
 		mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order));
-		memcg_kmem_uncharge_memcg(page, order, memcg);
+		memcg_kmem_uncharge_memcg(memcg, order);
 	} else {
 		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
 				    -(1 << order));

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #72 lore raw

PATCH 65/155 mm: kmem: rename memcg_kmem_(un)charge() into memcg_kmem_(un)charge_page()

From: Roman Gushchin <[email protected]>
Subject: mm: kmem: rename memcg_kmem_(un)charge() into memcg_kmem_(un)charge_page()

Rename (__)memcg_kmem_(un)charge() into (__)memcg_kmem_(un)charge_page() to better reflect what they are actually doing:

1) call __memcg_kmem_(un)charge_memcg() to actually charge or uncharge the current memcg

2) set or clear the PageKmemcg flag

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 fs/pipe.c                  |    2 +-
 include/linux/memcontrol.h |   23 +++++++++++++----------
 kernel/fork.c              |    9 +++++----
 mm/memcontrol.c            |    8 ++++----
 mm/page_alloc.c            |    4 ++--
 5 files changed, 25 insertions(+), 21 deletions(-)

fs/pipe.c renamed from fs/pipe.c~mm-kmem-rename-memcg_kmem_uncharge-into-memcg_kmem_uncharge_page +1 -1

@@ -146,7 +146,7 @@ static int anon_pipe_buf_steal(struct pi
 	struct page *page = buf->page;
 
 	if (page_count(page) == 1) {
-		memcg_kmem_uncharge(page, 0);
+		memcg_kmem_uncharge_page(page, 0);
 		__SetPageLocked(page);
 		return 0;
 	}

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-kmem-rename-memcg_kmem_uncharge-into-memcg_kmem_uncharge_page +13 ~~-10~~

@@ -1367,8 +1367,8 @@ struct kmem_cache *memcg_kmem_get_cache(
 void memcg_kmem_put_cache(struct kmem_cache *cachep);
 
 #ifdef CONFIG_MEMCG_KMEM
-int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order);
-void __memcg_kmem_uncharge(struct page *page, int order);
+int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
+void __memcg_kmem_uncharge_page(struct page *page, int order);
 int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp, int order);
 void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
 				 unsigned int nr_pages);

@@ -1393,17 +1393,18 @@ static inline bool memcg_kmem_enabled(vo
 	return static_branch_unlikely(&memcg_kmem_enabled_key);
 }
 
-static inline int memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
+static inline int memcg_kmem_charge_page(struct page *page, gfp_t gfp,
+					 int order)
 {
 	if (memcg_kmem_enabled())
-		return __memcg_kmem_charge(page, gfp, order);
+		return __memcg_kmem_charge_page(page, gfp, order);
 	return 0;
 }
 
-static inline void memcg_kmem_uncharge(struct page *page, int order)
+static inline void memcg_kmem_uncharge_page(struct page *page, int order)
 {
 	if (memcg_kmem_enabled())
-		__memcg_kmem_uncharge(page, order);
+		__memcg_kmem_uncharge_page(page, order);
 }
 
 static inline int memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,

@@ -1435,21 +1436,23 @@ struct mem_cgroup *mem_cgroup_from_obj(v
 
 #else
 
-static inline int memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
+static inline int memcg_kmem_charge_page(struct page *page, gfp_t gfp,
+					 int order)
 {
 	return 0;
 }
 
-static inline void memcg_kmem_uncharge(struct page *page, int order)
+static inline void memcg_kmem_uncharge_page(struct page *page, int order)
 {
 }
 
-static inline int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
+static inline int __memcg_kmem_charge_page(struct page *page, gfp_t gfp,
+					   int order)
 {
 	return 0;
 }
 
-static inline void __memcg_kmem_uncharge(struct page *page, int order)
+static inline void __memcg_kmem_uncharge_page(struct page *page, int order)
 {
 }

kernel/fork.c renamed from kernel/fork.c~mm-kmem-rename-memcg_kmem_uncharge-into-memcg_kmem_uncharge_page +5 -4

@@ -281,7 +281,7 @@ static inline void free_thread_stack(str
 					     MEMCG_KERNEL_STACK_KB,
 					     -(int)(PAGE_SIZE / 1024));
 
-			memcg_kmem_uncharge(vm->pages[i], 0);
+			memcg_kmem_uncharge_page(vm->pages[i], 0);
 		}
 
 		for (i = 0; i < NR_CACHED_STACKS; i++) {

@@ -413,12 +413,13 @@ static int memcg_charge_kernel_stack(str
 
 		for (i = 0; i < THREAD_SIZE / PAGE_SIZE; i++) {
 			/*
-			 * If memcg_kmem_charge() fails, page->mem_cgroup
-			 * pointer is NULL, and both memcg_kmem_uncharge()
+			 * If memcg_kmem_charge_page() fails, page->mem_cgroup
+			 * pointer is NULL, and both memcg_kmem_uncharge_page()
 			 * and mod_memcg_page_state() in free_thread_stack()
 			 * will ignore this page. So it's safe.
 			 */
-			ret = memcg_kmem_charge(vm->pages[i], GFP_KERNEL, 0);
+			ret = memcg_kmem_charge_page(vm->pages[i], GFP_KERNEL,
+						     0);
 			if (ret)
 				return ret;

mm/memcontrol.c renamed from mm/memcontrol.c~mm-kmem-rename-memcg_kmem_uncharge-into-memcg_kmem_uncharge_page +4 -4

@@ -2917,14 +2917,14 @@ int __memcg_kmem_charge_memcg(struct mem
 }
 
 /**
- * __memcg_kmem_charge: charge a kmem page to the current memory cgroup
+ * __memcg_kmem_charge_page: charge a kmem page to the current memory cgroup
  * @page: page to charge
  * @gfp: reclaim mode
  * @order: allocation order
  *
  * Returns 0 on success, an error code on failure.
  */
-int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order)
+int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order)
 {
 	struct mem_cgroup *memcg;
 	int ret = 0;

@@ -2960,11 +2960,11 @@ void __memcg_kmem_uncharge_memcg(struct
 		page_counter_uncharge(&memcg->memsw, nr_pages);
 }
 /**
- * __memcg_kmem_uncharge: uncharge a kmem page
+ * __memcg_kmem_uncharge_page: uncharge a kmem page
  * @page: page to uncharge
  * @order: allocation order
  */
-void __memcg_kmem_uncharge(struct page *page, int order)
+void __memcg_kmem_uncharge_page(struct page *page, int order)
 {
 	struct mem_cgroup *memcg = page->mem_cgroup;
 	unsigned int nr_pages = 1 << order;

mm/page_alloc.c renamed from mm/page_alloc.c~mm-kmem-rename-memcg_kmem_uncharge-into-memcg_kmem_uncharge_page +2 -2

@@ -1153,7 +1153,7 @@ static __always_inline bool free_pages_p
 	if (PageMappingFlags(page))
 		page->mapping = NULL;
 	if (memcg_kmem_enabled() && PageKmemcg(page))
-		__memcg_kmem_uncharge(page, order);
+		__memcg_kmem_uncharge_page(page, order);
 	if (check_free)
 		bad += free_pages_check(page);
 	if (bad)

@@ -4753,7 +4753,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, u
 
 out:
 	if (memcg_kmem_enabled() && (gfp_mask & __GFP_ACCOUNT) && page &&
-	    unlikely(__memcg_kmem_charge(page, gfp_mask, order) != 0)) {
+	    unlikely(__memcg_kmem_charge_page(page, gfp_mask, order) != 0)) {
 		__free_pages(page, order);
 		page = NULL;
 	}

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #73 lore raw

PATCH 66/155 mm: kmem: switch to nr_pages in (__)memcg_kmem_charge_memcg()

From: Roman Gushchin <[email protected]>
Subject: mm: kmem: switch to nr_pages in (__)memcg_kmem_charge_memcg()

These functions are charging the given number of kernel pages to the given memory cgroup. The number doesn't have to be a power of two. Let's make them to take the unsigned int nr_pages as an argument instead of the page order.

It makes them look consistent with the corresponding uncharge functions and functions like: mem_cgroup_charge_skmem(memcg, nr_pages).

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 include/linux/memcontrol.h |   11 ++++++-----
 mm/memcontrol.c            |    8 ++++----
 mm/slab.h                  |    2 +-
 3 files changed, 11 insertions(+), 10 deletions(-)

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-kmem-switch-to-nr_pages-in-__memcg_kmem_charge_memcg +6 -5

@@ -1369,7 +1369,8 @@ void memcg_kmem_put_cache(struct kmem_ca
 #ifdef CONFIG_MEMCG_KMEM
 int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
 void __memcg_kmem_uncharge_page(struct page *page, int order);
-int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp, int order);
+int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
+			      unsigned int nr_pages);
 void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
 				 unsigned int nr_pages);

@@ -1408,18 +1409,18 @@ static inline void memcg_kmem_uncharge_p
 }
 
 static inline int memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
-					  int order)
+					  unsigned int nr_pages)
 {
 	if (memcg_kmem_enabled())
-		return __memcg_kmem_charge_memcg(memcg, gfp, order);
+		return __memcg_kmem_charge_memcg(memcg, gfp, nr_pages);
 	return 0;
 }
 
 static inline void memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
-					     int order)
+					     unsigned int nr_pages)
 {
 	if (memcg_kmem_enabled())
-		__memcg_kmem_uncharge_memcg(memcg, 1 << order);
+		__memcg_kmem_uncharge_memcg(memcg, nr_pages);
 }
 
 /*

mm/memcontrol.c renamed from mm/memcontrol.c~mm-kmem-switch-to-nr_pages-in-__memcg_kmem_charge_memcg +4 -4

@@ -2884,13 +2884,13 @@ void memcg_kmem_put_cache(struct kmem_ca
  * __memcg_kmem_charge_memcg: charge a kmem page
  * @memcg: memory cgroup to charge
  * @gfp: reclaim mode
- * @order: allocation order
+ * @nr_pages: number of pages to charge
  *
  * Returns 0 on success, an error code on failure.
  */
-int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp, int order)
+int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
+			      unsigned int nr_pages)
 {
-	unsigned int nr_pages = 1 << order;
 	struct page_counter *counter;
 	int ret;

@@ -2934,7 +2934,7 @@ int __memcg_kmem_charge_page(struct page
 
 	memcg = get_mem_cgroup_from_current();
 	if (!mem_cgroup_is_root(memcg)) {
-		ret = __memcg_kmem_charge_memcg(memcg, gfp, order);
+		ret = __memcg_kmem_charge_memcg(memcg, gfp, 1 << order);
 		if (!ret) {
 			page->mem_cgroup = memcg;
 			__SetPageKmemcg(page);

mm/slab.h renamed from mm/slab.h~mm-kmem-switch-to-nr_pages-in-__memcg_kmem_charge_memcg +1 -1

@@ -365,7 +365,7 @@ static __always_inline int memcg_charge_
 		return 0;
 	}
 
-	ret = memcg_kmem_charge_memcg(memcg, gfp, order);
+	ret = memcg_kmem_charge_memcg(memcg, gfp, 1 << order);
 	if (ret)
 		goto out;

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #74 lore raw

PATCH 67/155 mm: memcg/slab: cache page number in memcg_(un)charge_slab()

From: Roman Gushchin <[email protected]>
Subject: mm: memcg/slab: cache page number in memcg_(un)charge_slab()

There are many places in memcg_charge_slab() and memcg_uncharge_slab() which are calculating the number of pages to charge, css references to grab etc depending on the order of the slab page.

Let's simplify the code by calculating it once and caching in the local variable.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 mm/slab.h |   22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

mm/slab.h renamed from mm/slab.h~mm-memcg-slab-cache-page-number-in-memcg_uncharge_slab +12 ~~-10~~

@@ -348,6 +348,7 @@ static __always_inline int memcg_charge_
 					     gfp_t gfp, int order,
 					     struct kmem_cache *s)
 {
+	unsigned int nr_pages = 1 << order;
 	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;
 	int ret;

@@ -360,21 +361,21 @@ static __always_inline int memcg_charge_
 
 	if (unlikely(!memcg || mem_cgroup_is_root(memcg))) {
 		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
-				    (1 << order));
-		percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order);
+				    nr_pages);
+		percpu_ref_get_many(&s->memcg_params.refcnt, nr_pages);
 		return 0;
 	}
 
-	ret = memcg_kmem_charge_memcg(memcg, gfp, 1 << order);
+	ret = memcg_kmem_charge_memcg(memcg, gfp, nr_pages);
 	if (ret)
 		goto out;
 
 	lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page));
-	mod_lruvec_state(lruvec, cache_vmstat_idx(s), 1 << order);
+	mod_lruvec_state(lruvec, cache_vmstat_idx(s), nr_pages);
 
 	/* transer try_charge() page references to kmem_cache */
-	percpu_ref_get_many(&s->memcg_params.refcnt, 1 << order);
-	css_put_many(&memcg->css, 1 << order);
+	percpu_ref_get_many(&s->memcg_params.refcnt, nr_pages);
+	css_put_many(&memcg->css, nr_pages);
 out:
 	css_put(&memcg->css);
 	return ret;

@@ -387,6 +388,7 @@ out:
 static __always_inline void memcg_uncharge_slab(struct page *page, int order,
 						struct kmem_cache *s)
 {
+	unsigned int nr_pages = 1 << order;
 	struct mem_cgroup *memcg;
 	struct lruvec *lruvec;

@@ -394,15 +396,15 @@ static __always_inline void memcg_unchar
 	memcg = READ_ONCE(s->memcg_params.memcg);
 	if (likely(!mem_cgroup_is_root(memcg))) {
 		lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page));
-		mod_lruvec_state(lruvec, cache_vmstat_idx(s), -(1 << order));
-		memcg_kmem_uncharge_memcg(memcg, order);
+		mod_lruvec_state(lruvec, cache_vmstat_idx(s), -nr_pages);
+		memcg_kmem_uncharge_memcg(memcg, nr_pages);
 	} else {
 		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
-				    -(1 << order));
+				    -nr_pages);
 	}
 	rcu_read_unlock();
 
-	percpu_ref_put_many(&s->memcg_params.refcnt, 1 << order);
+	percpu_ref_put_many(&s->memcg_params.refcnt, nr_pages);
 }
 
 extern void slab_init_memcg_params(struct kmem_cache *);

Andrew Morton

author maintainer

2020-04-02 04:06 UTC #75 lore raw

PATCH 68/155 mm: kmem: rename (__)memcg_kmem_(un)charge_memcg() to __memcg_kmem_(un)charge()

From: Roman Gushchin <[email protected]>
Subject: mm: kmem: rename (__)memcg_kmem_(un)charge_memcg() to __memcg_kmem_(un)charge()

Drop the _memcg suffix from (__)memcg_kmem_(un)charge functions. It's shorter and more obvious.

These are the most basic functions which are just (un)charging the given cgroup with the given amount of pages.

Also fix up the corresponding comments.

Link http://lkml.kernel.org/r/[email protected] open here

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 include/linux/memcontrol.h |   19 +++++++---------
 mm/memcontrol.c            |   40 +++++++++++++++++------------------
 mm/slab.h                  |    4 +--
 3 files changed, 31 insertions(+), 32 deletions(-)

include/linux/memcontrol.h renamed from include/linux/memcontrol.h~mm-kmem-rename-__memcg_kmem_uncharge_memcg-to-__memcg_kmem_uncharge +9 ~~-10~~

@@ -1367,12 +1367,11 @@ struct kmem_cache *memcg_kmem_get_cache(
 void memcg_kmem_put_cache(struct kmem_cache *cachep);
 
 #ifdef CONFIG_MEMCG_KMEM
+int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp,
+			unsigned int nr_pages);
+void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages);
 int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order);
 void __memcg_kmem_uncharge_page(struct page *page, int order);
-int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
-			      unsigned int nr_pages);
-void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
-				 unsigned int nr_pages);
 
 extern struct static_key_false memcg_kmem_enabled_key;
 extern struct workqueue_struct *memcg_kmem_cache_wq;

@@ -1408,19 +1407,19 @@ static inline void memcg_kmem_uncharge_p
 		__memcg_kmem_uncharge_page(page, order);
 }
 
-static inline int memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
-					  unsigned int nr_pages)
+static inline int memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp,
+				    unsigned int nr_pages)
 {
 	if (memcg_kmem_enabled())
-		return __memcg_kmem_charge_memcg(memcg, gfp, nr_pages);
+		return __memcg_kmem_charge(memcg, gfp, nr_pages);
 	return 0;
 }
 
-static inline void memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
-					     unsigned int nr_pages)
+static inline void memcg_kmem_uncharge(struct mem_cgroup *memcg,
+				       unsigned int nr_pages)
 {
 	if (memcg_kmem_enabled())
-		__memcg_kmem_uncharge_memcg(memcg, nr_pages);
+		__memcg_kmem_uncharge(memcg, nr_pages);
 }
 
 /*

mm/memcontrol.c renamed from mm/memcontrol.c~mm-kmem-rename-__memcg_kmem_uncharge_memcg-to-__memcg_kmem_uncharge +20 ~~-20~~

@@ -2881,15 +2881,15 @@ void memcg_kmem_put_cache(struct kmem_ca
 }
 
 /**
- * __memcg_kmem_charge_memcg: charge a kmem page
+ * __memcg_kmem_charge: charge a number of kernel pages to a memcg
  * @memcg: memory cgroup to charge
  * @gfp: reclaim mode
  * @nr_pages: number of pages to charge
  *
  * Returns 0 on success, an error code on failure.
  */
-int __memcg_kmem_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp,
-			      unsigned int nr_pages)
+int __memcg_kmem_charge(struct mem_cgroup *memcg, gfp_t gfp,
+			unsigned int nr_pages)
 {
 	struct page_counter *counter;
 	int ret;

@@ -2917,6 +2917,21 @@ int __memcg_kmem_charge_memcg(struct mem
 }
 
 /**
+ * __memcg_kmem_uncharge: uncharge a number of kernel pages from a memcg
+ * @memcg: memcg to uncharge
+ * @nr_pages: number of pages to uncharge
+ */
+void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
+{
+	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		page_counter_uncharge(&memcg->kmem, nr_pages);
+
+	page_counter_uncharge(&memcg->memory, nr_pages);
+	if (do_memsw_account())
+		page_counter_uncharge(&memcg->memsw, nr_pages);
+}
+
+/**
  * __memcg_kmem_charge_page: charge a kmem page to the current memory cgroup
  * @page: page to charge
  * @gfp: reclaim mode

@@ -2934,7 +2949,7 @@ int __memcg_kmem_charge_page(struct page
 
 	memcg = get_mem_cgroup_from_current();
 	if (!mem_cgroup_is_root(memcg)) {
-		ret = __memcg_kmem_charge_memcg(memcg, gfp, 1 << order);
+		ret = __memcg_kmem_charge(memcg, gfp, 1 << order);
 		if (!ret) {
 			page->mem_cgroup = memcg;
 			__SetPageKmemcg(page);

@@ -2945,21 +2960,6 @@ int __memcg_kmem_charge_page(struct page
 }
 
 /**
- * __memcg_kmem_uncharge_memcg: uncharge a kmem page
- * @memcg: memcg to uncharge
- * @nr_pages: number of pages to uncharge
- */
-void __memcg_kmem_uncharge_memcg(struct mem_cgroup *memcg,
-				 unsigned int nr_pages)
-{
-	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
-		page_counter_uncharge(&memcg->kmem, nr_pages);
-
-	page_counter_uncharge(&memcg->memory, nr_pages);
-	if (do_memsw_account())
-		page_counter_uncharge(&memcg->memsw, nr_pages);
-}
-/**
  * __memcg_kmem_uncharge_page: uncharge a kmem page
  * @page: page to uncharge
  * @order: allocation order

@@ -2973,7 +2973,7 @@ void __memcg_kmem_uncharge_page(struct p
 		return;
 
 	VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page);
-	__memcg_kmem_uncharge_memcg(memcg, nr_pages);
+	__memcg_kmem_uncharge(memcg, nr_pages);
 	page->mem_cgroup = NULL;
 
 	/* slab pages do not have PageKmemcg flag set */

mm/slab.h renamed from mm/slab.h~mm-kmem-rename-__memcg_kmem_uncharge_memcg-to-__memcg_kmem_uncharge +2 -2

@@ -366,7 +366,7 @@ static __always_inline int memcg_charge_
 		return 0;
 	}
 
-	ret = memcg_kmem_charge_memcg(memcg, gfp, nr_pages);
+	ret = memcg_kmem_charge(memcg, gfp, nr_pages);
 	if (ret)
 		goto out;

@@ -397,7 +397,7 @@ static __always_inline void memcg_unchar
 	if (likely(!mem_cgroup_is_root(memcg))) {
 		lruvec = mem_cgroup_lruvec(memcg, page_pgdat(page));
 		mod_lruvec_state(lruvec, cache_vmstat_idx(s), -nr_pages);
-		memcg_kmem_uncharge_memcg(memcg, nr_pages);
+		memcg_kmem_uncharge(memcg, nr_pages);
 	} else {
 		mod_node_page_state(page_pgdat(page), cache_vmstat_idx(s),
 				    -nr_pages);

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #76 lore raw

PATCH 69/155 mm: memcontrol: fix memory.low proportional distribution

From: Johannes Weiner <[email protected]>
Subject: mm: memcontrol: fix memory.low proportional distribution

Patch series "mm: memcontrol: recursive memory.low protection", v3.

The current memory.low (and memory.min) semantics require protection to be assigned to a cgroup in an untinterrupted chain from the top-level cgroup all the way to the leaf.

In practice, we want to protect entire cgroup subtrees from each other (system management software vs. workload), but we would like the VM to balance memory optimally *within* each subtree, without having to make explicit weight allocations among individual components. The current semantics make that impossible.

They also introduce unmanageable complexity into more advanced resource trees. For example:

host root
`- system.slice
`- rpm upgrades
`- logging
`- workload.slice
`- a container
`- system.slice
`- workload.slice
`- job A
`- component 1
`- component 2
`- job B

At a host-level perspective, we would like to protect the outer workload.slice subtree as a whole from rpm upgrades, logging etc. But for that to be effective, right now we'd have to propagate it down through the container, the inner workload.slice, into the job cgroup and ultimately the component cgroups where memory is actually, physically allocated. This may cross several tree delegation points and namespace boundaries, which make such a setup near impossible.

CPU and IO on the other hand are already distributed recursively. The user would simply configure allowances at the host level, and they would apply to the entire subtree without any downward propagation.

To enable the above-mentioned usecases and bring memory in line with other resource controllers, this patch series extends memory.low/min such that settings apply recursively to the entire subtree. Users can still assign explicit shares in subgroups, but if they don't, any ancestral protection will be distributed such that children compete freely amongst each other - as if no memory control were enabled inside the subtree - but enjoy protection from neighboring trees.

In the above example, the user would then be able to configure shares of CPU, IO and memory at the host level to comprehensively protect and isolate the workload.slice as a whole from system.slice activity.

Patch #1 fixes an existing bug that can give a cgroup tree more protection than it should receive as per ancestor configuration.

Patch #2 simplifies and documents the existing code to make it easier to reason about the changes in the next patch.

Patch #3 finally implements recursive memory protection semantics.

Because of a risk of regressing legacy setups, the new semantics are hidden behind a cgroup2 mount option, 'memory_recursiveprot'.

More details in patch #3.

This patch (of 3):

When memory.low is overcommitted - i.e. the children claim more protection than their shared ancestor grants them - the allowance is distributed in proportion to how much each sibling uses their own declared protection:

low_usage = min(memory.low, memory.current) elow = parent_elow * (low_usage / siblings_low_usage)

However, siblings_low_usage is not the sum of all low_usages. It sums up the usages of *only those cgroups that are within their memory.low* That means that low_usage can be *bigger* than siblings_low_usage, and consequently the total protection afforded to the children can be bigger than what the ancestor grants the subtree.

Consider three groups where two are in excess of their protection:

A/memory.low = 10G
A/A1/memory.low = 10G, memory.current = 20G
A/A2/memory.low = 10G, memory.current = 20G
A/A3/memory.low = 10G, memory.current = 8G
siblings_low_usage = 8G (only A3 contributes)

A1/elow = parent_elow(10G) * low_usage(10G) / siblings_low_usage(8G) = 12.5G -> 10G A2/elow = parent_elow(10G) * low_usage(10G) / siblings_low_usage(8G) = 12.5G -> 10G A3/elow = parent_elow(10G) * low_usage(8G) / siblings_low_usage(8G) = 10.0G

(the 12.5G are capped to the explicit memory.low setting of 10G)

With that, the sum of all awarded protection below A is 30G, when A only grants 10G for the entire subtree.

What does this mean in practice? A1 and A2 would still be in excess of their 10G allowance and would be reclaimed, whereas A3 would not. As they eventually drop below their protection setting, they would be counted in siblings_low_usage again and the error would right itself.

When reclaim was applied in a binary fashion (cgroup is reclaimed when it's above its protection, otherwise it's skipped) this would actually work out just fine. However, since 1bc63fb1272b ("mm, memcg: make scan aggression always exclude protection"), reclaim pressure is scaled to how much a cgroup is above its protection. As a result this calculation error unduly skews pressure away from A1 and A2 toward the rest of the system.

But why did we do it like this in the first place?

The reasoning behind exempting groups in excess from siblings_low_usage was to go after them first during reclaim in an overcommitted subtree:

A/memory.low = 2G, memory.current = 4G
A/A1/memory.low = 3G, memory.current = 2G
A/A2/memory.low = 1G, memory.current = 2G

siblings_low_usage = 2G (only A1 contributes)
A1/elow = parent_elow(2G) * low_usage(2G) / siblings_low_usage(2G) = 2G A2/elow = parent_elow(2G) * low_usage(1G) / siblings_low_usage(2G) = 1G

While the children combined are overcomitting A and are technically both at fault, A2 is actively declaring unprotected memory and we would like to reclaim that first.

However, while this sounds like a noble goal on the face of it, it doesn't make much difference in actual memory distribution: Because A is overcommitted, reclaim will not stop once A2 gets pushed back to within its allowance; we'll have to reclaim A1 either way. The end result is still that protection is distributed proportionally, with A1 getting 3/4 (1.5G) and A2 getting 1/4 (0.5G) of A's allowance.

[ If A weren't overcommitted, it wouldn't make a difference since each
cgroup would just get the protection it declares:

A/memory.low = 2G, memory.current = 3G
A/A1/memory.low = 1G, memory.current = 1G
A/A2/memory.low = 1G, memory.current = 2G

With the current calculation:

siblings_low_usage = 1G (only A1 contributes)
A1/elow = parent_elow(2G) * low_usage(1G) / siblings_low_usage(1G) = 2G -> 1G A2/elow = parent_elow(2G) * low_usage(1G) / siblings_low_usage(1G) = 2G -> 1G

Including excess groups in siblings_low_usage:

siblings_low_usage = 2G
A1/elow = parent_elow(2G) * low_usage(1G) / siblings_low_usage(2G) = 1G -> 1G A2/elow = parent_elow(2G) * low_usage(1G) / siblings_low_usage(2G) = 1G -> 1G ]

Simplify the calculation and fix the proportional reclaim bug by including excess cgroups in siblings_low_usage.

After this patch, the effective memory.low distribution from the example above would be as follows:

A/memory.low = 10G
A/A1/memory.low = 10G, memory.current = 20G
A/A2/memory.low = 10G, memory.current = 20G
A/A3/memory.low = 10G, memory.current = 8G
siblings_low_usage = 28G

A1/elow = parent_elow(10G) * low_usage(10G) / siblings_low_usage(28G) = 3.5G A2/elow = parent_elow(10G) * low_usage(10G) / siblings_low_usage(28G) = 3.5G A3/elow = parent_elow(10G) * low_usage(8G) / siblings_low_usage(28G) = 2.8G

Link http://lkml.kernel.org/r/[email protected] open here

Fixes 1bc63fb1272b ("mm, memcg: make scan aggression always exclude protection")

Fixes 230671533d64 ("mm: memory.low hierarchical behavior")

Signed-off-by Johannes Weiner <[email protected]>

Acked-by Roman Gushchin <[email protected]>

Cc Michal Koutný <[email protected]>

---

 mm/memcontrol.c   |    4 +---
 mm/page_counter.c |   12 ++----------
 2 files changed, 3 insertions(+), 13 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-fix-memorylow-proportional-distribution +1 -3

@@ -6266,9 +6266,7 @@ struct cgroup_subsys memory_cgrp_subsys
  * elow = min( memory.low, parent->elow * ------------------ ),
  *                                        siblings_low_usage
  *
- *             | memory.current, if memory.current < memory.low
- * low_usage = |
- *	       | 0, otherwise.
+ * low_usage = min(memory.low, memory.current)
  *
  *
  * Such definition of the effective memory.low provides the expected

mm/page_counter.c renamed from mm/page_counter.c~mm-memcontrol-fix-memorylow-proportional-distribution +2 ~~-10~~

@@ -23,11 +23,7 @@ static void propagate_protected_usage(st
 		return;
 
 	if (c->min || atomic_long_read(&c->min_usage)) {
-		if (usage <= c->min)
-			protected = usage;
-		else
-			protected = 0;
-
+		protected = min(usage, c->min);
 		old_protected = atomic_long_xchg(&c->min_usage, protected);
 		delta = protected - old_protected;
 		if (delta)

@@ -35,11 +31,7 @@ static void propagate_protected_usage(st
 	}
 
 	if (c->low || atomic_long_read(&c->low_usage)) {
-		if (usage <= c->low)
-			protected = usage;
-		else
-			protected = 0;
-
+		protected = min(usage, c->low);
 		old_protected = atomic_long_xchg(&c->low_usage, protected);
 		delta = protected - old_protected;
 		if (delta)

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #77 lore raw

PATCH 70/155 mm: memcontrol: clean up and document effective low/min calculations

From: Johannes Weiner <[email protected]>
Subject: mm: memcontrol: clean up and document effective low/min calculations

The effective protection of any given cgroup is a somewhat complicated construct that depends on the ancestor's configuration, siblings' configurations, as well as current memory utilization in all these groups. It's done this way to satisfy hierarchical delegation requirements while also making the configuration semantics flexible and expressive in complex real life scenarios.

Unfortunately, all the rules and requirements are sparsely documented, and the code is a little too clever in merging different scenarios into a single min() expression. This makes it hard to reason about the implementation and avoid breaking semantics when making changes to it.

This patch documents each semantic rule individually and splits out the handling of the overcommit case from the regular case.

Michal Koutný also points out that the points of equilibrium as described in the existing example scenarios aren't actually accurate. Delete these examples for now to avoid confusion.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Johannes Weiner <[email protected]>

Acked-by Roman Gushchin <[email protected]>

Cc Michal Koutný <[email protected]>

---

 mm/memcontrol.c |  177 +++++++++++++++++++++-------------------------
 1 file changed, 84 insertions(+), 93 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-clean-up-and-document-effective-low-min-calculations +84 ~~-93~~

@@ -6234,6 +6234,76 @@ struct cgroup_subsys memory_cgrp_subsys
 	.early_init = 0,
 };
 
+/*
+ * This function calculates an individual cgroup's effective
+ * protection which is derived from its own memory.min/low, its
+ * parent's and siblings' settings, as well as the actual memory
+ * distribution in the tree.
+ *
+ * The following rules apply to the effective protection values:
+ *
+ * 1. At the first level of reclaim, effective protection is equal to
+ *    the declared protection in memory.min and memory.low.
+ *
+ * 2. To enable safe delegation of the protection configuration, at
+ *    subsequent levels the effective protection is capped to the
+ *    parent's effective protection.
+ *
+ * 3. To make complex and dynamic subtrees easier to configure, the
+ *    user is allowed to overcommit the declared protection at a given
+ *    level. If that is the case, the parent's effective protection is
+ *    distributed to the children in proportion to how much protection
+ *    they have declared and how much of it they are utilizing.
+ *
+ *    This makes distribution proportional, but also work-conserving:
+ *    if one cgroup claims much more protection than it uses memory,
+ *    the unused remainder is available to its siblings.
+ *
+ * 4. Conversely, when the declared protection is undercommitted at a
+ *    given level, the distribution of the larger parental protection
+ *    budget is NOT proportional. A cgroup's protection from a sibling
+ *    is capped to its own memory.min/low setting.
+ *
+ */
+static unsigned long effective_protection(unsigned long usage,
+					  unsigned long setting,
+					  unsigned long parent_effective,
+					  unsigned long siblings_protected)
+{
+	unsigned long protected;
+
+	protected = min(usage, setting);
+	/*
+	 * If all cgroups at this level combined claim and use more
+	 * protection then what the parent affords them, distribute
+	 * shares in proportion to utilization.
+	 *
+	 * We are using actual utilization rather than the statically
+	 * claimed protection in order to be work-conserving: claimed
+	 * but unused protection is available to siblings that would
+	 * otherwise get a smaller chunk than what they claimed.
+	 */
+	if (siblings_protected > parent_effective)
+		return protected * parent_effective / siblings_protected;
+
+	/*
+	 * Ok, utilized protection of all children is within what the
+	 * parent affords them, so we know whatever this child claims
+	 * and utilizes is effectively protected.
+	 *
+	 * If there is unprotected usage beyond this value, reclaim
+	 * will apply pressure in proportion to that amount.
+	 *
+	 * If there is unutilized protection, the cgroup will be fully
+	 * shielded from reclaim, but we do return a smaller value for
+	 * protection than what the group could enjoy in theory. This
+	 * is okay. With the overcommit distribution above, effective
+	 * protection is always dependent on how memory is actually
+	 * consumed among the siblings anyway.
+	 */
+	return protected;
+}
+
 /**
  * mem_cgroup_protected - check if memory consumption is in the normal range
  * @root: the top ancestor of the sub-tree being checked

@@ -6247,67 +6317,11 @@ struct cgroup_subsys memory_cgrp_subsys
  *   MEMCG_PROT_LOW: cgroup memory is protected as long there is
  *     an unprotected supply of reclaimable memory from other cgroups.
  *   MEMCG_PROT_MIN: cgroup memory is protected
- *
- * @root is exclusive; it is never protected when looked at directly
- *
- * To provide a proper hierarchical behavior, effective memory.min/low values
- * are used. Below is the description of how effective memory.low is calculated.
- * Effective memory.min values is calculated in the same way.
- *
- * Effective memory.low is always equal or less than the original memory.low.
- * If there is no memory.low overcommittment (which is always true for
- * top-level memory cgroups), these two values are equal.
- * Otherwise, it's a part of parent's effective memory.low,
- * calculated as a cgroup's memory.low usage divided by sum of sibling's
- * memory.low usages, where memory.low usage is the size of actually
- * protected memory.
- *
- *                                             low_usage
- * elow = min( memory.low, parent->elow * ------------------ ),
- *                                        siblings_low_usage
- *
- * low_usage = min(memory.low, memory.current)
- *
- *
- * Such definition of the effective memory.low provides the expected
- * hierarchical behavior: parent's memory.low value is limiting
- * children, unprotected memory is reclaimed first and cgroups,
- * which are not using their guarantee do not affect actual memory
- * distribution.
- *
- * For example, if there are memcgs A, A/B, A/C, A/D and A/E:
- *
- *     A      A/memory.low = 2G, A/memory.current = 6G
- *    //\\
- *   BC  DE   B/memory.low = 3G  B/memory.current = 2G
- *            C/memory.low = 1G  C/memory.current = 2G
- *            D/memory.low = 0   D/memory.current = 2G
- *            E/memory.low = 10G E/memory.current = 0
- *
- * and the memory pressure is applied, the following memory distribution
- * is expected (approximately):
- *
- *     A/memory.current = 2G
- *
- *     B/memory.current = 1.3G
- *     C/memory.current = 0.6G
- *     D/memory.current = 0
- *     E/memory.current = 0
- *
- * These calculations require constant tracking of the actual low usages
- * (see propagate_protected_usage()), as well as recursive calculation of
- * effective memory.low values. But as we do call mem_cgroup_protected()
- * path for each memory cgroup top-down from the reclaim,
- * it's possible to optimize this part, and save calculated elow
- * for next usage. This part is intentionally racy, but it's ok,
- * as memory.low is a best-effort mechanism.
  */
 enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root,
 						struct mem_cgroup *memcg)
 {
 	struct mem_cgroup *parent;
-	unsigned long emin, parent_emin;
-	unsigned long elow, parent_elow;
 	unsigned long usage;
 
 	if (mem_cgroup_disabled())

@@ -6322,52 +6336,29 @@ enum mem_cgroup_protection mem_cgroup_pr
 	if (!usage)
 		return MEMCG_PROT_NONE;
 
-	emin = memcg->memory.min;
-	elow = memcg->memory.low;
-
 	parent = parent_mem_cgroup(memcg);
 	/* No parent means a non-hierarchical mode on v1 memcg */
 	if (!parent)
 		return MEMCG_PROT_NONE;
 
-	if (parent == root)
-		goto exit;
-
-	parent_emin = READ_ONCE(parent->memory.emin);
-	emin = min(emin, parent_emin);
-	if (emin && parent_emin) {
-		unsigned long min_usage, siblings_min_usage;
-
-		min_usage = min(usage, memcg->memory.min);
-		siblings_min_usage = atomic_long_read(
-			&parent->memory.children_min_usage);
-
-		if (min_usage && siblings_min_usage)
-			emin = min(emin, parent_emin * min_usage /
-				   siblings_min_usage);
-	}
-
-	parent_elow = READ_ONCE(parent->memory.elow);
-	elow = min(elow, parent_elow);
-	if (elow && parent_elow) {
-		unsigned long low_usage, siblings_low_usage;
-
-		low_usage = min(usage, memcg->memory.low);
-		siblings_low_usage = atomic_long_read(
-			&parent->memory.children_low_usage);
-
-		if (low_usage && siblings_low_usage)
-			elow = min(elow, parent_elow * low_usage /
-				   siblings_low_usage);
+	if (parent == root) {
+		memcg->memory.emin = memcg->memory.min;
+		memcg->memory.elow = memcg->memory.low;
+		goto out;
 	}
 
-exit:
-	memcg->memory.emin = emin;
-	memcg->memory.elow = elow;
+	memcg->memory.emin = effective_protection(usage,
+			memcg->memory.min, READ_ONCE(parent->memory.emin),
+			atomic_long_read(&parent->memory.children_min_usage));
+
+	memcg->memory.elow = effective_protection(usage,
+			memcg->memory.low, READ_ONCE(parent->memory.elow),
+			atomic_long_read(&parent->memory.children_low_usage));
 
-	if (usage <= emin)
+out:
+	if (usage <= memcg->memory.emin)
 		return MEMCG_PROT_MIN;
-	else if (usage <= elow)
+	else if (usage <= memcg->memory.elow)
 		return MEMCG_PROT_LOW;
 	else
 		return MEMCG_PROT_NONE;

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #78 lore raw

PATCH 71/155 mm: memcontrol: recursive memory.low protection

From: Johannes Weiner <[email protected]>
Subject: mm: memcontrol: recursive memory.low protection

Right now, the effective protection of any given cgroup is capped by its own explicit memory.low setting, regardless of what the parent says. The reasons for this are mostly historical and ease of implementation: to make delegation of memory.low safe, effective protection is the min() of all memory.low up the tree.

Unfortunately, this limitation makes it impossible to protect an entire subtree from another without forcing the user to make explicit protection allocations all the way to the leaf cgroups - something that is highly undesirable in real life scenarios.

Consider memory in a data center host. At the cgroup top level, we have a distinction between system management software and the actual workload the system is executing. Both branches are further subdivided into individual services, job components etc.

We want to protect the workload as a whole from the system management software, but that doesn't mean we want to protect and prioritize individual workload wrt each other. Their memory demand can vary over time, and we'd want the VM to simply cache the hottest data within the workload subtree. Yet, the current memory.low limitations force us to allocate a fixed amount of protection to each workload component in order to get protection from system management software in general. This results in very inefficient resource distribution.

Another concern with mandating downward allocation is that, as the complexity of the cgroup tree grows, it gets harder for the lower levels to be informed about decisions made at the host-level. Consider a container inside a namespace that in turn creates its own nested tree of cgroups to run multiple workloads. It'd be extremely difficult to configure memory.low parameters in those leaf cgroups that on one hand balance pressure among siblings as the container desires, while also reflecting the host-level protection from e.g. rpm upgrades, that lie beyond one or more delegation and namespacing points in the tree.

It's highly unusual from a cgroup interface POV that nested levels have to be aware of and reflect decisions made at higher levels for them to be effective.

To enable such use cases and scale configurability for complex trees, this patch implements a resource inheritance model for memory that is similar to how the CPU and the IO controller implement work-conserving resource
allocations: a share of a resource allocated to a subree always applies to the entire subtree recursively, while allowing, but not mandating, children to further specify distribution rules.

That means that if protection is explicitly allocated among siblings, those configured shares are being followed during page reclaim just like they are now. However, if the memory.low set at a higher level is not fully claimed by the children in that subtree, the "floating" remainder is applied to each cgroup in the tree in proportion to its size. Since reclaim pressure is applied in proportion to size as well, each child in that tree gets the same boost, and the effect is neutral among siblings - with respect to each other, they behave as if no memory control was enabled at all, and the VM simply balances the memory demands optimally within the subtree. But collectively those cgroups enjoy a boost over the cgroups in neighboring trees.

E.g. a leaf cgroup with a memory.low setting of 0 no longer means that it's not getting a share of the hierarchically assigned resource, just that it doesn't claim a fixed amount of it to protect from its siblings.

This allows us to recursively protect one subtree (workload) from another (system management), while letting subgroups compete freely among each other - without having to assign fixed shares to each leaf, and without nested groups having to echo higher-level settings.

The floating protection composes naturally with fixed protection. Consider the following example tree:

A A: low = 2G
/ \ A1: low = 1G
A1 A2 A2: low = 0G

As outside pressure is applied to this tree, A1 will enjoy a fixed protection from A2 of 1G, but the remaining, unclaimed 1G from A is split evenly among A1 and A2, coming out to 1.5G and 0.5G.

There is a slight risk of regressing theoretical setups where the top-level cgroups don't know about the true budgeting and set bogusly high "bypass" values that are meaningfully allocated down the tree. Such setups would rely on unclaimed protection to be discarded, and distributing it would change the intended behavior. Be safe and hide the new behavior behind a mount option, 'memory_recursiveprot'.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Johannes Weiner <[email protected]>

Acked-by Roman Gushchin <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Michal Koutný <[email protected]>

---

 Documentation/admin-guide/cgroup-v2.rst |   11 ++++
 include/linux/cgroup-defs.h             |    5 ++
 kernel/cgroup/cgroup.c                  |   17 ++++++-
 mm/memcontrol.c                         |   51 ++++++++++++++++++++--
 4 files changed, 79 insertions(+), 5 deletions(-)

Documentation/admin-guide/cgroup-v2.rst renamed from Documentation/admin-guide/cgroup-v2.rst~mm-memcontrol-recursive-memorylow-protection +11 -0

@@ -188,6 +188,17 @@ cgroup v2 currently supports the followi
         modified through remount from the init namespace. The mount
         option is ignored on non-init namespace mounts.
 
+  memory_recursiveprot
+
+        Recursively apply memory.min and memory.low protection to
+        entire subtrees, without requiring explicit downward
+        propagation into leaf cgroups.  This allows protecting entire
+        subtrees from one another, while retaining free competition
+        within those subtrees.  This should have been the default
+        behavior but is a mount-option to avoid regressing setups
+        relying on the original semantics (e.g. specifying bogusly
+        high 'bypass' protection values at higher tree levels).
+
 
 Organizing Processes and Threads
 --------------------------------

include/linux/cgroup-defs.h renamed from include/linux/cgroup-defs.h~mm-memcontrol-recursive-memorylow-protection +5 -0

@@ -94,6 +94,11 @@ enum {
 	 * Enable legacy local memory.events.
 	 */
 	CGRP_ROOT_MEMORY_LOCAL_EVENTS = (1 << 5),
+
+	/*
+	 * Enable recursive subtree protection
+	 */
+	CGRP_ROOT_MEMORY_RECURSIVE_PROT = (1 << 6),
 };
 
 /* cftype->flags */

kernel/cgroup/cgroup.c renamed from kernel/cgroup/cgroup.c~mm-memcontrol-recursive-memorylow-protection +16 -1

@@ -1813,12 +1813,14 @@ int cgroup_show_path(struct seq_file *sf
 enum cgroup2_param {
 	Opt_nsdelegate,
 	Opt_memory_localevents,
+	Opt_memory_recursiveprot,
 	nr__cgroup2_params
 };
 
 static const struct fs_parameter_spec cgroup2_fs_parameters[] = {
 	fsparam_flag("nsdelegate",		Opt_nsdelegate),
 	fsparam_flag("memory_localevents",	Opt_memory_localevents),
+	fsparam_flag("memory_recursiveprot",	Opt_memory_recursiveprot),
 	{}
 };

@@ -1839,6 +1841,9 @@ static int cgroup2_parse_param(struct fs
 	case Opt_memory_localevents:
 		ctx->flags |= CGRP_ROOT_MEMORY_LOCAL_EVENTS;
 		return 0;
+	case Opt_memory_recursiveprot:
+		ctx->flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
+		return 0;
 	}
 	return -EINVAL;
 }

@@ -1855,6 +1860,11 @@ static void apply_cgroup_root_flags(unsi
 			cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_LOCAL_EVENTS;
 		else
 			cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_LOCAL_EVENTS;
+
+		if (root_flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT)
+			cgrp_dfl_root.flags |= CGRP_ROOT_MEMORY_RECURSIVE_PROT;
+		else
+			cgrp_dfl_root.flags &= ~CGRP_ROOT_MEMORY_RECURSIVE_PROT;
 	}
 }

@@ -1864,6 +1874,8 @@ static int cgroup_show_options(struct se
 		seq_puts(seq, ",nsdelegate");
 	if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_LOCAL_EVENTS)
 		seq_puts(seq, ",memory_localevents");
+	if (cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT)
+		seq_puts(seq, ",memory_recursiveprot");
 	return 0;
 }

@@ -6412,7 +6424,10 @@ static struct kobj_attribute cgroup_dele
 static ssize_t features_show(struct kobject *kobj, struct kobj_attribute *attr,
 			     char *buf)
 {
-	return snprintf(buf, PAGE_SIZE, "nsdelegate\nmemory_localevents\n");
+	return snprintf(buf, PAGE_SIZE,
+			"nsdelegate\n"
+			"memory_localevents\n"
+			"memory_recursiveprot\n");
 }
 static struct kobj_attribute cgroup_features_attr = __ATTR_RO(features);

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcontrol-recursive-memorylow-protection +47 -4

86 lines not on this page · raw message

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #79 lore raw

PATCH 72/155 memcg: css_tryget_online cleanups

From: Shakeel Butt <[email protected]>
Subject: memcg: css_tryget_online cleanups

Currently multiple locations in memcg code, css_tryget_online() is being used. However it doesn't matter whether the cgroup is online for the callers. Online used to matter when we had reparenting on offlining and we needed a way to prevent new ones from showing up.

The failure case for couple of these css_tryget_online usage is to fallback to root_mem_cgroup which kind of make bypassing the memcg limits possible for some workloads. For example creating an inotify group in a subcontainer and then deleting that container after moving the process to a different container will make all the event objects allocated for that group to the root_mem_cgroup. So, using css_tryget_online() is dangerous for such cases.

Two locations still use the online version. The swapin of offlined memcg's pages and the memcg kmem cache creation. The kmem cache indeed needs the online version as the kernel does the reparenting of memcg kmem caches. For the swapin case, it has been left for later as the fallback is not really that concerning.

With swap accounting enabled, if the memcg of the swapped out page is not online then the memcg extracted from the given 'mm' will be charged and if 'mm' is NULL then root memcg will be charged. However I could not find a code path where the given 'mm' will be NULL for swap-in case.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Shakeel Butt <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Vladimir Davydov <[email protected]>

Cc Roman Gushchin <[email protected]>

---

 mm/memcontrol.c |   14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~memcg-css_tryget_online-cleanups +9 -5

@@ -656,7 +656,7 @@ retry:
 	 */
 	__mem_cgroup_remove_exceeded(mz, mctz);
 	if (!soft_limit_excess(mz->memcg) ||
-	    !css_tryget_online(&mz->memcg->css))
+	    !css_tryget(&mz->memcg->css))
 		goto retry;
 done:
 	return mz;

@@ -972,7 +972,8 @@ struct mem_cgroup *get_mem_cgroup_from_p
 		return NULL;
 
 	rcu_read_lock();
-	if (!memcg || !css_tryget_online(&memcg->css))
+	/* Page should not get uncharged and freed memcg under us. */
+	if (!memcg || WARN_ON_ONCE(!css_tryget(&memcg->css)))
 		memcg = root_mem_cgroup;
 	rcu_read_unlock();
 	return memcg;

@@ -985,10 +986,13 @@ EXPORT_SYMBOL(get_mem_cgroup_from_page);
 static __always_inline struct mem_cgroup *get_mem_cgroup_from_current(void)
 {
 	if (unlikely(current->active_memcg)) {
-		struct mem_cgroup *memcg = root_mem_cgroup;
+		struct mem_cgroup *memcg;
 
 		rcu_read_lock();
-		if (css_tryget_online(&current->active_memcg->css))
+		/* current->active_memcg must hold a ref. */
+		if (WARN_ON_ONCE(!css_tryget(&current->active_memcg->css)))
+			memcg = root_mem_cgroup;
+		else
 			memcg = current->active_memcg;
 		rcu_read_unlock();
 		return memcg;

@@ -6789,7 +6793,7 @@ void mem_cgroup_sk_alloc(struct sock *sk
 		goto out;
 	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && !memcg->tcpmem_active)
 		goto out;
-	if (css_tryget_online(&memcg->css))
+	if (css_tryget(&memcg->css))
 		sk->sk_memcg = memcg;
 out:
 	rcu_read_unlock();

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #80 lore raw

PATCH 73/155 mm/memcontrol.c: make mem_cgroup_id_get_many() __maybe_unused

From: Vincenzo Frascino <[email protected]>
Subject: mm/memcontrol.c: make mem_cgroup_id_get_many() __maybe_unused

mem_cgroup_id_get_many() is currently used only when MMU or MEMCG_SWAP configuration options are enabled. Having them disabled triggers the following warning at compile time:

linux/mm/memcontrol.c:4797:13: warning: `mem_cgroup_id_get_many' defined but not used [-Wunused-function]
static void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n)

Make mem_cgroup_id_get_many() __maybe_unused to address the issue.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Vincenzo Frascino <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Vladimir Davydov <[email protected]>

---

 mm/memcontrol.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-make-mem_cgroup_id_get_many-__maybe_unused +2 -1

@@ -4863,7 +4863,8 @@ static void mem_cgroup_id_remove(struct
 	}
 }
 
-static void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n)
+static void __maybe_unused mem_cgroup_id_get_many(struct mem_cgroup *memcg,
+						  unsigned int n)
 {
 	refcount_add(n, &memcg->id.ref);
 }

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #81 lore raw

PATCH 74/155 mm, memcg: prevent memory.high load/store tearing

Link http://lkml.kernel.org/r/2f66f7038ed1d4688e59de72b627ae0ea52efa83.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent memory.high load/store tearing

A mem_cgroup's high attribute can be concurrently set at the same time as we are trying to read it -- for example, if we are in memory_high_write at the same time as we are trying to do high reclaim.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/memcontrol.c |   13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-prevent-memoryhigh-load-store-tearing +7 -6

@@ -2242,7 +2242,7 @@ static void reclaim_high(struct mem_cgro
 			 gfp_t gfp_mask)
 {
 	do {
-		if (page_counter_read(&memcg->memory) <= memcg->high)
+		if (page_counter_read(&memcg->memory) <= READ_ONCE(memcg->high))
 			continue;
 		memcg_memory_event(memcg, MEMCG_HIGH);
 		try_to_free_mem_cgroup_pages(memcg, nr_pages, gfp_mask, true);

@@ -2582,7 +2582,7 @@ done_restock:
 	 * reclaim, the cost of mismatch is negligible.
 	 */
 	do {
-		if (page_counter_read(&memcg->memory) > memcg->high) {
+		if (page_counter_read(&memcg->memory) > READ_ONCE(memcg->high)) {
 			/* Don't bother a random interrupted task */
 			if (in_interrupt()) {
 				schedule_work(&memcg->high_work);

@@ -4325,7 +4325,8 @@ void mem_cgroup_wb_stats(struct bdi_writ
 	*pheadroom = PAGE_COUNTER_MAX;
 
 	while ((parent = parent_mem_cgroup(memcg))) {
-		unsigned long ceiling = min(memcg->memory.max, memcg->high);
+		unsigned long ceiling = min(memcg->memory.max,
+					    READ_ONCE(memcg->high));
 		unsigned long used = page_counter_read(&memcg->memory);
 
 		*pheadroom = min(*pheadroom, ceiling - min(ceiling, used));

@@ -5047,7 +5048,7 @@ mem_cgroup_css_alloc(struct cgroup_subsy
 	if (!memcg)
 		return ERR_PTR(error);
 
-	memcg->high = PAGE_COUNTER_MAX;
+	WRITE_ONCE(memcg->high, PAGE_COUNTER_MAX);
 	memcg->soft_limit = PAGE_COUNTER_MAX;
 	if (parent) {
 		memcg->swappiness = mem_cgroup_swappiness(parent);

@@ -5200,7 +5201,7 @@ static void mem_cgroup_css_reset(struct
 	page_counter_set_max(&memcg->tcpmem, PAGE_COUNTER_MAX);
 	page_counter_set_min(&memcg->memory, 0);
 	page_counter_set_low(&memcg->memory, 0);
-	memcg->high = PAGE_COUNTER_MAX;
+	WRITE_ONCE(memcg->high, PAGE_COUNTER_MAX);
 	memcg->soft_limit = PAGE_COUNTER_MAX;
 	memcg_wb_domain_size_changed(memcg);
 }

@@ -6016,7 +6017,7 @@ static ssize_t memory_high_write(struct
 	if (err)
 		return err;
 
-	memcg->high = high;
+	WRITE_ONCE(memcg->high, high);
 
 	for (;;) {
 		unsigned long nr_pages = page_counter_read(&memcg->memory);

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #82 lore raw

PATCH 75/155 mm, memcg: prevent memory.max load tearing

Link http://lkml.kernel.org/r/50a31e5f39f8ae6c8fb73966ba1455f0924e8f44.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent memory.max load tearing

This one is a bit more nuanced because we have memcg_max_mutex, which is mostly just used for enforcing invariants, but we still need to READ_ONCE since (despite its name) it doesn't really protect memory.max access.

On write (page_counter_set_max() and memory_max_write()) we use xchg(), which uses smp_mb(), so that's already fine.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/memcontrol.c |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-prevent-memorymax-load-tearing +6 -6

@@ -1521,7 +1521,7 @@ void mem_cgroup_print_oom_meminfo(struct
 
 	pr_info("memory: usage %llukB, limit %llukB, failcnt %lu\n",
 		K((u64)page_counter_read(&memcg->memory)),
-		K((u64)memcg->memory.max), memcg->memory.failcnt);
+		K((u64)READ_ONCE(memcg->memory.max)), memcg->memory.failcnt);
 	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->swap)),

@@ -1552,7 +1552,7 @@ unsigned long mem_cgroup_get_max(struct
 {
 	unsigned long max;
 
-	max = memcg->memory.max;
+	max = READ_ONCE(memcg->memory.max);
 	if (mem_cgroup_swappiness(memcg)) {
 		unsigned long memsw_max;
 		unsigned long swap_max;

@@ -3068,7 +3068,7 @@ static int mem_cgroup_resize_max(struct
 		 * Make sure that the new limit (memsw or memory limit) doesn't
 		 * break our basic invariant rule memory.max <= memsw.max.
 		 */
-		limits_invariant = memsw ? max >= memcg->memory.max :
+		limits_invariant = memsw ? max >= READ_ONCE(memcg->memory.max) :
 					   max <= memcg->memsw.max;
 		if (!limits_invariant) {
 			mutex_unlock(&memcg_max_mutex);

@@ -3815,8 +3815,8 @@ static int memcg_stat_show(struct seq_fi
 	/* Hierarchical information */
 	memory = memsw = PAGE_COUNTER_MAX;
 	for (mi = memcg; mi; mi = parent_mem_cgroup(mi)) {
-		memory = min(memory, mi->memory.max);
-		memsw = min(memsw, mi->memsw.max);
+		memory = min(memory, READ_ONCE(mi->memory.max));
+		memsw = min(memsw, READ_ONCE(mi->memsw.max));
 	}
 	seq_printf(m, "hierarchical_memory_limit %llu\n",
 		   (u64)memory * PAGE_SIZE);

@@ -4325,7 +4325,7 @@ void mem_cgroup_wb_stats(struct bdi_writ
 	*pheadroom = PAGE_COUNTER_MAX;
 
 	while ((parent = parent_mem_cgroup(memcg))) {
-		unsigned long ceiling = min(memcg->memory.max,
+		unsigned long ceiling = min(READ_ONCE(memcg->memory.max),
 					    READ_ONCE(memcg->high));
 		unsigned long used = page_counter_read(&memcg->memory);

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #83 lore raw

PATCH 76/155 mm, memcg: prevent memory.low load/store tearing

Link http://lkml.kernel.org/r/448206f44b0fa7be9dad2ca2601d2bcb2c0b7844.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent memory.low load/store tearing

This can be set concurrently with reads, which may cause the wrong value to be propagated.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/page_counter.c |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

mm/page_counter.c renamed from mm/page_counter.c~mm-memcg-prevent-memorylow-load-store-tearing +5 -3

@@ -17,6 +17,7 @@ static void propagate_protected_usage(st
 				      unsigned long usage)
 {
 	unsigned long protected, old_protected;
+	unsigned long low;
 	long delta;
 
 	if (!c->parent)

@@ -30,8 +31,9 @@ static void propagate_protected_usage(st
 			atomic_long_add(delta, &c->parent->children_min_usage);
 	}
 
-	if (c->low || atomic_long_read(&c->low_usage)) {
-		protected = min(usage, c->low);
+	low = READ_ONCE(c->low);
+	if (low || atomic_long_read(&c->low_usage)) {
+		protected = min(usage, low);
 		old_protected = atomic_long_xchg(&c->low_usage, protected);
 		delta = protected - old_protected;
 		if (delta)

@@ -222,7 +224,7 @@ void page_counter_set_low(struct page_co
 {
 	struct page_counter *c;
 
-	counter->low = nr_pages;
+	WRITE_ONCE(counter->low, nr_pages);
 
 	for (c = counter; c; c = c->parent)
 		propagate_protected_usage(c, atomic_long_read(&c->usage));

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #84 lore raw

PATCH 77/155 mm, memcg: prevent memory.min load/store tearing

Link http://lkml.kernel.org/r/e809b4e6b0c1626dac6945970de06409a180ee65.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent memory.min load/store tearing

This can be set concurrently with reads, which may cause the wrong value to be propagated.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/memcontrol.c   |    5 +++--
 mm/page_counter.c |    9 +++++----
 2 files changed, 8 insertions(+), 6 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-prevent-memorymin-load-store-tearing +3 -2

@@ -6389,7 +6389,7 @@ enum mem_cgroup_protection mem_cgroup_pr
 		return MEMCG_PROT_NONE;
 
 	if (parent == root) {
-		memcg->memory.emin = memcg->memory.min;
+		memcg->memory.emin = READ_ONCE(memcg->memory.min);
 		memcg->memory.elow = memcg->memory.low;
 		goto out;
 	}

@@ -6397,7 +6397,8 @@ enum mem_cgroup_protection mem_cgroup_pr
 	parent_usage = page_counter_read(&parent->memory);
 
 	memcg->memory.emin = effective_protection(usage, parent_usage,
-			memcg->memory.min, READ_ONCE(parent->memory.emin),
+			READ_ONCE(memcg->memory.min),
+			READ_ONCE(parent->memory.emin),
 			atomic_long_read(&parent->memory.children_min_usage));
 
 	memcg->memory.elow = effective_protection(usage, parent_usage,

mm/page_counter.c renamed from mm/page_counter.c~mm-memcg-prevent-memorymin-load-store-tearing +5 -4

@@ -17,14 +17,15 @@ static void propagate_protected_usage(st
 				      unsigned long usage)
 {
 	unsigned long protected, old_protected;
-	unsigned long low;
+	unsigned long low, min;
 	long delta;
 
 	if (!c->parent)
 		return;
 
-	if (c->min || atomic_long_read(&c->min_usage)) {
-		protected = min(usage, c->min);
+	min = READ_ONCE(c->min);
+	if (min || atomic_long_read(&c->min_usage)) {
+		protected = min(usage, min);
 		old_protected = atomic_long_xchg(&c->min_usage, protected);
 		delta = protected - old_protected;
 		if (delta)

@@ -207,7 +208,7 @@ void page_counter_set_min(struct page_co
 {
 	struct page_counter *c;
 
-	counter->min = nr_pages;
+	WRITE_ONCE(counter->min, nr_pages);
 
 	for (c = counter; c; c = c->parent)
 		propagate_protected_usage(c, atomic_long_read(&c->usage));

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #85 lore raw

PATCH 78/155 mm, memcg: prevent memory.swap.max load tearing

Link http://lkml.kernel.org/r/bbec2c3d822217334855c8877a9d28b2a6d395fb.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent memory.swap.max load tearing

The write side of this is xchg()/smp_mb(), so that's all good. Just a few sites missing a READ_ONCE.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/memcontrol.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-prevent-memoryswapmax-load-tearing +4 -3

@@ -1525,7 +1525,7 @@ void mem_cgroup_print_oom_meminfo(struct
 	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->swap)),
-			K((u64)memcg->swap.max), memcg->swap.failcnt);
+			K((u64)READ_ONCE(memcg->swap.max)), memcg->swap.failcnt);
 	else {
 		pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->memsw)),

@@ -1558,7 +1558,7 @@ unsigned long mem_cgroup_get_max(struct
 		unsigned long swap_max;
 
 		memsw_max = memcg->memsw.max;
-		swap_max = memcg->swap.max;
+		swap_max = READ_ONCE(memcg->swap.max);
 		swap_max = min(swap_max, (unsigned long)total_swap_pages);
 		max = min(max + swap_max, memsw_max);
 	}

@@ -7117,7 +7117,8 @@ bool mem_cgroup_swap_full(struct page *p
 		return false;
 
 	for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg))
-		if (page_counter_read(&memcg->swap) * 2 >= memcg->swap.max)
+		if (page_counter_read(&memcg->swap) * 2 >=
+		    READ_ONCE(memcg->swap.max))
 			return true;
 
 	return false;

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #86 lore raw

PATCH 79/155 mm, memcg: prevent mem_cgroup_protected store tearing

Link http://lkml.kernel.org/r/d1e9fbc0379fe8db475d82c8b6fbe048876e12ae.1584034301.git.chris@chrisdown.name open here

From: Chris Down <[email protected]>
Subject: mm, memcg: prevent mem_cgroup_protected store tearing

The read side of this is all protected, but we can still tear if multiple iterations of mem_cgroup_protected are going at the same time.

There's some intentional racing in mem_cgroup_protected which is ok, but load/store tearing should be avoided.

Signed-off-by Chris Down <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc Roman Gushchin <[email protected]>

Cc Tejun Heo <[email protected]>

---

 mm/memcontrol.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-prevent-mem_cgroup_protected-store-tearing +4 -4

@@ -6396,14 +6396,14 @@ enum mem_cgroup_protection mem_cgroup_pr
 
 	parent_usage = page_counter_read(&parent->memory);
 
-	memcg->memory.emin = effective_protection(usage, parent_usage,
+	WRITE_ONCE(memcg->memory.emin, effective_protection(usage, parent_usage,
 			READ_ONCE(memcg->memory.min),
 			READ_ONCE(parent->memory.emin),
-			atomic_long_read(&parent->memory.children_min_usage));
+			atomic_long_read(&parent->memory.children_min_usage)));
 
-	memcg->memory.elow = effective_protection(usage, parent_usage,
+	WRITE_ONCE(memcg->memory.elow, effective_protection(usage, parent_usage,
 			memcg->memory.low, READ_ONCE(parent->memory.elow),
-			atomic_long_read(&parent->memory.children_low_usage));
+			atomic_long_read(&parent->memory.children_low_usage)));
 
 out:
 	if (usage <= memcg->memory.emin)

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #87 lore raw

PATCH 80/155 mm: memcg: make memory.oom.group tolerable to task migration

From: Roman Gushchin <[email protected]>
Subject: mm: memcg: make memory.oom.group tolerable to task migration

If a task is getting moved out of the OOMing cgroup, it might result in unexpected OOM killings if memory.oom.group is used anywhere in the cgroup tree.

Imagine the following example:

          A (oom.group = 1)
         / \

(OOM) B C

Let's say B's memory.max is exceeded and it's OOMing. The OOM killer selects a task in B as a victim, but someone asynchronously moves the task into C. mem_cgroup_get_oom_group() will iterate over all ancestors of C up to the root cgroup. In theory it had to stop at the oom_domain level - the memory cgroup which is OOMing. But because B is not an ancestor of C, it's not happening. Instead it chooses A (because it's oom.group is set), and kills all tasks in A. This behavior is wrong because the OOM happened in B, so there is no reason to kill anything outside.

Fix this by checking it the memory cgroup to which the task belongs is a descendant of the oom_domain. If not, memory.oom.group should be ignored, and the OOM killer should kill only the victim task.

Link http://lkml.kernel.org/r/[email protected] open here

Reported-by Dan Schatzberg <[email protected]>

---

 mm/memcontrol.c |    8 ++++++++
 1 file changed, 8 insertions(+)

mm/memcontrol.c renamed from mm/memcontrol.c~mm-memcg-make-memoryoomgroup-tolerable-to-task-migration +8 -0

@@ -1931,6 +1931,14 @@ struct mem_cgroup *mem_cgroup_get_oom_gr
 		goto out;
 
 	/*
+	 * If the victim task has been asynchronously moved to a different
+	 * memory cgroup, we might end up killing tasks outside oom_domain.
+	 * In this case it's better to ignore memory.group.oom.
+	 */
+	if (unlikely(!mem_cgroup_is_descendant(memcg, oom_domain)))
+		goto out;
+
+	/*
 	 * Traverse the memory cgroup hierarchy from the victim task's
 	 * cgroup up to the OOMing cgroup (or root) to find the
 	 * highest-level memory cgroup with oom.group set.

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #88 lore raw

PATCH 81/155 mm/mapping_dirty_helpers: update huge page-table entry callbacks

From: Thomas Hellstrom <[email protected]>
Subject: mm/mapping_dirty_helpers: update huge page-table entry callbacks

Following the update of pagewalk code commit a07984d48146 ("mm: pagewalk:
add p4d_entry() and pgd_entry()") we can modify the mapping_dirty_helpers' huge page-table entry callbacks to avoid splitting when a huge pud or -pmd is encountered.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Thomas Hellstrom <[email protected]>

Reviewed-by Steven Price <[email protected]>

Cc Andrew Morton <[email protected]>

---

 mm/mapping_dirty_helpers.c |   42 +++++++++++++++++++++++++++++++----
 1 file changed, 38 insertions(+), 4 deletions(-)

mm/mapping_dirty_helpers.c renamed from mm/mapping_dirty_helpers.c~mm-mapping_dirty_helpers-update-huge-page-table-entry-callbacks +38 -4

@@ -111,26 +111,60 @@ static int clean_record_pte(pte_t *pte,
 	return 0;
 }
 
-/* wp_clean_pmd_entry - The pagewalk pmd callback. */
+/*
+ * wp_clean_pmd_entry - The pagewalk pmd callback.
+ *
+ * Dirty-tracking should take place on the PTE level, so
+ * WARN() if encountering a dirty huge pmd.
+ * Furthermore, never split huge pmds, since that currently
+ * causes dirty info loss. The pagefault handler should do
+ * that if needed.
+ */
 static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end,
 			      struct mm_walk *walk)
 {
-	/* Dirty-tracking should be handled on the pte level */
 	pmd_t pmdval = pmd_read_atomic(pmd);
 
+	if (!pmd_trans_unstable(&pmdval))
+		return 0;
+
+	if (pmd_none(pmdval)) {
+		walk->action = ACTION_AGAIN;
+		return 0;
+	}
+
+	/* Huge pmd, present or migrated */
+	walk->action = ACTION_CONTINUE;
 	if (pmd_trans_huge(pmdval) || pmd_devmap(pmdval))
 		WARN_ON(pmd_write(pmdval) || pmd_dirty(pmdval));
 
 	return 0;
 }
 
-/* wp_clean_pud_entry - The pagewalk pud callback. */
+/*
+ * wp_clean_pud_entry - The pagewalk pud callback.
+ *
+ * Dirty-tracking should take place on the PTE level, so
+ * WARN() if encountering a dirty huge puds.
+ * Furthermore, never split huge puds, since that currently
+ * causes dirty info loss. The pagefault handler should do
+ * that if needed.
+ */
 static int wp_clean_pud_entry(pud_t *pud, unsigned long addr, unsigned long end,
 			      struct mm_walk *walk)
 {
-	/* Dirty-tracking should be handled on the pte level */
 	pud_t pudval = READ_ONCE(*pud);
 
+	if (!pud_trans_unstable(&pudval))
+		return 0;
+
+	if (pud_none(pudval)) {
+		walk->action = ACTION_AGAIN;
+		return 0;
+	}
+
+	/* Huge pud */
+	walk->action = ACTION_CONTINUE;
 	if (pud_trans_huge(pudval) || pud_devmap(pudval))
 		WARN_ON(pud_write(pudval) || pud_dirty(pudval));

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #89 lore raw

PATCH 82/155 mm/vma: move VM_NO_KHUGEPAGED into generic header

From: Anshuman Khandual <[email protected]>
Subject: mm/vma: move VM_NO_KHUGEPAGED into generic header

Patch series "mm/vma: some more minor changes", v2.

The motivation here is to consolidate VMA flags and helpers in generic memory header and reduce code duplication when ever applicable. If there are other possible similar instances which might be missing here, please do let me me know. I will be happy to incorporate them.

This patch (of 3):

Move VM_NO_KHUGEPAGED into generic header (include/linux/mm.h). This just makes sure that no VMA flag is scattered in individual function files any longer. While at this, fix an old comment which is no longer valid. This should not cause any functional change.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Anshuman Khandual <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Michael Ellerman <[email protected]>

Cc Paul Mackerras <[email protected]>

Cc Thomas Gleixner <[email protected]>

---

 include/linux/mm.h |    4 +++-
 mm/khugepaged.c    |    2 --
 2 files changed, 3 insertions(+), 3 deletions(-)

include/linux/mm.h renamed from include/linux/mm.h~mm-vma-move-vm_no_khugepaged-into-generic-header +3 -1

@@ -356,10 +356,12 @@ extern unsigned int kobjsize(const void
 
 /*
  * Special vmas that are non-mergable, non-mlock()able.
- * Note: mm/huge_memory.c VM_NO_THP depends on this definition.
  */
 #define VM_SPECIAL (VM_IO | VM_DONTEXPAND | VM_PFNMAP | VM_MIXEDMAP)
 
+/* This mask prevents VMA from being scanned with khugepaged */
+#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB)
+
 /* This mask defines which mm->def_flags a process can inherit its parent */
 #define VM_INIT_DEF_MASK	VM_NOHUGEPAGE

mm/khugepaged.c renamed from mm/khugepaged.c~mm-vma-move-vm_no_khugepaged-into-generic-header +0 -2

@@ -308,8 +308,6 @@ struct attribute_group khugepaged_attr_g
 };
 #endif /* CONFIG_SYSFS */
 
-#define VM_NO_KHUGEPAGED (VM_SPECIAL | VM_HUGETLB)
-
 int hugepage_madvise(struct vm_area_struct *vma,
 		     unsigned long *vm_flags, int advice)
 {

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #90 lore raw

PATCH 83/155 mm/vma: make vma_is_foreign() available for general use

From: Anshuman Khandual <[email protected]>
Subject: mm/vma: make vma_is_foreign() available for general use

Idea of a foreign VMA with respect to the present context is very generic. But currently there are two identical definitions for this in powerpc and x86 platforms. Lets consolidate those redundant definitions while making vma_is_foreign() available for general use later. This should not cause any functional change.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Anshuman Khandual <[email protected]>

Cc Paul Mackerras <[email protected]>

Cc Michael Ellerman <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc Ingo Molnar <[email protected]>

---

 arch/powerpc/mm/book3s64/pkeys.c   |   12 ------------
 arch/x86/include/asm/mmu_context.h |   15 ---------------
 include/linux/mm.h                 |   11 +++++++++++
 3 files changed, 11 insertions(+), 27 deletions(-)

arch/powerpc/mm/book3s64/pkeys.c renamed from arch/powerpc/mm/book3s64/pkeys.c~mm-vma-make-vma_is_foreign-available-for-general-use +0 ~~-12~~

@@ -381,18 +381,6 @@ bool arch_pte_access_permitted(u64 pte,
  * So do not enforce things if the VMA is not from the current mm, or if we are
  * in a kernel thread.
  */
-static inline bool vma_is_foreign(struct vm_area_struct *vma)
-{
-	if (!current->mm)
-		return true;
-
-	/* if it is not our ->mm, it has to be foreign */
-	if (current->mm != vma->vm_mm)
-		return true;
-
-	return false;
-}
-
 bool arch_vma_access_permitted(struct vm_area_struct *vma, bool write,
 			       bool execute, bool foreign)
 {

arch/x86/include/asm/mmu_context.h renamed from arch/x86/include/asm/mmu_context.h~mm-vma-make-vma_is_foreign-available-for-general-use +0 ~~-15~~

@@ -213,21 +213,6 @@ static inline void arch_unmap(struct mm_
  * So do not enforce things if the VMA is not from the current
  * mm, or if we are in a kernel thread.
  */
-static inline bool vma_is_foreign(struct vm_area_struct *vma)
-{
-	if (!current->mm)
-		return true;
-	/*
-	 * Should PKRU be enforced on the access to this VMA?  If
-	 * the VMA is from another process, then PKRU has no
-	 * relevance and should not be enforced.
-	 */
-	if (current->mm != vma->vm_mm)
-		return true;
-
-	return false;
-}
-
 static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,
 		bool write, bool execute, bool foreign)
 {

include/linux/mm.h renamed from include/linux/mm.h~mm-vma-make-vma_is_foreign-available-for-general-use +11 -0

@@ -27,6 +27,7 @@
 #include <linux/memremap.h>
 #include <linux/overflow.h>
 #include <linux/sizes.h>
+#include <linux/sched.h>
 
 struct mempolicy;
 struct anon_vma;

@@ -543,6 +544,16 @@ static inline bool vma_is_anonymous(stru
 	return !vma->vm_ops;
 }
 
+static inline bool vma_is_foreign(struct vm_area_struct *vma)
+{
+	if (!current->mm)
+		return true;
+
+	if (current->mm != vma->vm_mm)
+		return true;
+
+	return false;
+}
 #ifdef CONFIG_SHMEM
 /*
  * The vma_is_shmem is not inline because it is used only by slow

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #91 lore raw

PATCH 84/155 mm/vma: make is_vma_temporary_stack() available for general use

From: Anshuman Khandual <[email protected]>
Subject: mm/vma: make is_vma_temporary_stack() available for general use

Currently the declaration and definition for is_vma_temporary_stack() are scattered. Lets make is_vma_temporary_stack() helper available for general use and also drop the declaration from (include/linux/huge_mm.h) which is no longer required. While at this, rename this as vma_is_temporary_stack() in line with existing helpers. This should not cause any functional change.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Anshuman Khandual <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Michael Ellerman <[email protected]>

Cc Paul Mackerras <[email protected]>

Cc Thomas Gleixner <[email protected]>

---

 include/linux/huge_mm.h |    4 +---
 include/linux/mm.h      |   14 ++++++++++++++
 mm/khugepaged.c         |    2 +-
 mm/mremap.c             |    2 +-
 mm/rmap.c               |   16 +---------------
 5 files changed, 18 insertions(+), 20 deletions(-)

include/linux/huge_mm.h renamed from include/linux/huge_mm.h~mm-vma-make-is_vma_temporary_stack-available-for-general-use +1 -3

@@ -87,8 +87,6 @@ extern struct kobj_attribute shmem_enabl
 #define HPAGE_PUD_SIZE	((1UL) << HPAGE_PUD_SHIFT)
 #define HPAGE_PUD_MASK	(~(HPAGE_PUD_SIZE - 1))
 
-extern bool is_vma_temporary_stack(struct vm_area_struct *vma);
-
 extern unsigned long transparent_hugepage_flags;
 
 /*

@@ -100,7 +98,7 @@ static inline bool __transparent_hugepag
 	if (vma->vm_flags & VM_NOHUGEPAGE)
 		return false;
 
-	if (is_vma_temporary_stack(vma))
+	if (vma_is_temporary_stack(vma))
 		return false;
 
 	if (test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags))

include/linux/mm.h renamed from include/linux/mm.h~mm-vma-make-is_vma_temporary_stack-available-for-general-use +14 -0

@@ -544,6 +544,20 @@ static inline bool vma_is_anonymous(stru
 	return !vma->vm_ops;
 }
 
+static inline bool vma_is_temporary_stack(struct vm_area_struct *vma)
+{
+	int maybe_stack = vma->vm_flags & (VM_GROWSDOWN | VM_GROWSUP);
+
+	if (!maybe_stack)
+		return false;
+
+	if ((vma->vm_flags & VM_STACK_INCOMPLETE_SETUP) ==
+						VM_STACK_INCOMPLETE_SETUP)
+		return true;
+
+	return false;
+}
+
 static inline bool vma_is_foreign(struct vm_area_struct *vma)
 {
 	if (!current->mm)

mm/khugepaged.c renamed from mm/khugepaged.c~mm-vma-make-is_vma_temporary_stack-available-for-general-use +1 -1

@@ -421,7 +421,7 @@ static bool hugepage_vma_check(struct vm
 	}
 	if (!vma->anon_vma || vma->vm_ops)
 		return false;
-	if (is_vma_temporary_stack(vma))
+	if (vma_is_temporary_stack(vma))
 		return false;
 	return !(vm_flags & VM_NO_KHUGEPAGED);
 }

mm/mremap.c renamed from mm/mremap.c~mm-vma-make-is_vma_temporary_stack-available-for-general-use +1 -1

@@ -133,7 +133,7 @@ static void move_ptes(struct vm_area_str
 	 * such races:
 	 *
 	 * - During exec() shift_arg_pages(), we use a specially tagged vma
-	 *   which rmap call sites look for using is_vma_temporary_stack().
+	 *   which rmap call sites look for using vma_is_temporary_stack().
 	 *
 	 * - During mremap(), new_vma is often known to be placed after vma
 	 *   in rmap traversal order. This ensures rmap will always observe

mm/rmap.c renamed from mm/rmap.c~mm-vma-make-is_vma_temporary_stack-available-for-general-use +0 ~~-13~~

@@ -1699,23 +1699,9 @@ discard:
 	return ret;
 }
 
-bool is_vma_temporary_stack(struct vm_area_struct *vma)
-{
-	int maybe_stack = vma->vm_flags & (VM_GROWSDOWN | VM_GROWSUP);
-
-	if (!maybe_stack)
-		return false;
-
-	if ((vma->vm_flags & VM_STACK_INCOMPLETE_SETUP) ==
-						VM_STACK_INCOMPLETE_SETUP)
-		return true;
-
-	return false;
-}

Andrew Morton

author maintainer

2020-04-02 04:07 UTC #92 lore raw

PATCH 85/155 mm: add pagemap.h to the fine documentation

From: "Matthew Wilcox (Oracle)" <[email protected]>
Subject: mm: add pagemap.h to the fine documentation

The documentation currently does not include the deathless prose written to describe functions in pagemap.h because it's not included in any rst file. Fix up the mismatches between parameter names and the documentation and add the file to mm-api.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Matthew Wilcox (Oracle) <[email protected]>

Reviewed-by Zi Yan <[email protected]>

Reviewed-by John Hubbard <[email protected]>

Cc Jonathan Corbet <[email protected]>

---

 Documentation/core-api/mm-api.rst |    3 +++
 include/linux/pagemap.h           |    8 ++++----
 2 files changed, 7 insertions(+), 4 deletions(-)

Documentation/core-api/mm-api.rst renamed from Documentation/core-api/mm-api.rst~mm-add-pagemaph-to-the-fine-documentation +3 -0

@@ -73,6 +73,9 @@ File Mapping and Page Cache
 .. kernel-doc:: mm/truncate.c
    :export:
 
+.. kernel-doc:: include/linux/pagemap.h
+   :internal:
+
 Memory pools
 ============

include/linux/pagemap.h renamed from include/linux/pagemap.h~mm-add-pagemaph-to-the-fine-documentation +4 -4

@@ -33,8 +33,8 @@ enum mapping_flags {
 
 /**
  * mapping_set_error - record a writeback error in the address_space
- * @mapping - the mapping in which an error should be set
- * @error - the error to set in the mapping
+ * @mapping: the mapping in which an error should be set
+ * @error: the error to set in the mapping
  *
  * When writeback fails in some way, we must record that error so that
  * userspace can be informed when fsync and the like are called.  We endeavor

@@ -303,9 +303,9 @@ static inline struct page *find_lock_pag
  * atomic allocation!
  */
 static inline struct page *find_or_create_page(struct address_space *mapping,
-					pgoff_t offset, gfp_t gfp_mask)
+					pgoff_t index, gfp_t gfp_mask)
 {
-	return pagecache_get_page(mapping, offset,
+	return pagecache_get_page(mapping, index,
 					FGP_LOCK|FGP_ACCESSED|FGP_CREAT,
 					gfp_mask);
 }

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #93 lore raw

PATCH 86/155 mm/gup: rename "nonblocking" to "locked" where proper

https://lkml.org/lkml/2017/11/2/833

From: Peter Xu <[email protected]>
Subject: mm/gup: rename "nonblocking" to "locked" where proper

Patch series "mm: Page fault enhancements", v6.

This series contains cleanups and enhancements to current page fault logic. The whole idea comes from the discussion between Andrea and Linus on the bug reported by syzbot here:

Basically it does two things:

(a) Allows the page fault logic to be more interactive on not only SIGKILL, but also the rest of userspace signals, and,

(b) Allows the page fault retry (VM_FAULT_RETRY) to happen for more than once.

For (a): with the changes we should be able to react faster when page faults are working in parallel with userspace signals like SIGSTOP and SIGCONT (and more), and with that we can remove the buggy part in userfaultfd and benefit the whole page fault mechanism on faster signal processing to reach the userspace.

For (b), we should be able to allow the page fault handler to loop for even more than twice. Some context: for now since we have FAULT_FLAG_ALLOW_RETRY we can allow to retry the page fault once with the same interrupt context, however never more than twice. This can be not only a potential cleanup to remove this assumption since AFAIU the code itself doesn't really have this twice-only limitation (though that should be a protective approach in the past), at the same time it'll greatly simplify future works like userfaultfd write-protect where it's possible to retry for more than twice (please have a look at [1] below for a possible user that might require the page fault to be handled for a third time; if we can remove the retry limitation we can simply drop that patch and those complexity).

This patch (of 16):

There's plenty of places around __get_user_pages() that has a parameter "nonblocking" which does not really mean that "it won't block" (because it can really block) but instead it shows whether the mmap_sem is released by up_read() during the page fault handling mostly when VM_FAULT_RETRY is returned.

We have the correct naming in e.g. get_user_pages_locked() or get_user_pages_remote() as "locked", however there're still many places that are using the "nonblocking" as name.

Renaming the places to "locked" where proper to better suite the functionality of the variable. While at it, fixing up some of the comments accordingly.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Reviewed-by Mike Rapoport <[email protected]>

Reviewed-by Jerome Glisse <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Bobby Powers <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 mm/gup.c     |   44 +++++++++++++++++++++-----------------------
 mm/hugetlb.c |    8 ++++----
 2 files changed, 25 insertions(+), 27 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-rename-nonblocking-to-locked-where-proper +21 ~~-23~~

@@ -846,12 +846,12 @@ unmap:
 }
 
 /*
- * mmap_sem must be held on entry.  If @nonblocking != NULL and
- * *@flags does not include FOLL_NOWAIT, the mmap_sem may be released.
- * If it is, *@nonblocking will be set to 0 and -EBUSY returned.
+ * mmap_sem must be held on entry.  If @locked != NULL and *@flags
+ * does not include FOLL_NOWAIT, the mmap_sem may be released.  If it
+ * is, *@locked will be set to 0 and -EBUSY returned.
  */
 static int faultin_page(struct task_struct *tsk, struct vm_area_struct *vma,
-		unsigned long address, unsigned int *flags, int *nonblocking)
+		unsigned long address, unsigned int *flags, int *locked)
 {
 	unsigned int fault_flags = 0;
 	vm_fault_t ret;

@@ -863,7 +863,7 @@ static int faultin_page(struct task_stru
 		fault_flags |= FAULT_FLAG_WRITE;
 	if (*flags & FOLL_REMOTE)
 		fault_flags |= FAULT_FLAG_REMOTE;
-	if (nonblocking)
+	if (locked)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
 	if (*flags & FOLL_NOWAIT)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT;

@@ -889,8 +889,8 @@ static int faultin_page(struct task_stru
 	}
 
 	if (ret & VM_FAULT_RETRY) {
-		if (nonblocking && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
-			*nonblocking = 0;
+		if (locked && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
+			*locked = 0;
 		return -EBUSY;
 	}

@@ -967,7 +967,7 @@ static int check_vma_flags(struct vm_are
  *		only intends to ensure the pages are faulted in.
  * @vmas:	array of pointers to vmas corresponding to each page.
  *		Or NULL if the caller does not require them.
- * @nonblocking: whether waiting for disk IO or mmap_sem contention
+ * @locked:     whether we're still with the mmap_sem held
  *
  * Returns either number of pages pinned (which may be less than the
  * number requested), or an error. Details about the return value:

@@ -1002,13 +1002,11 @@ static int check_vma_flags(struct vm_are
  * appropriate) must be called after the page is finished with, and
  * before put_page is called.
  *
- * If @nonblocking != NULL, __get_user_pages will not wait for disk IO
- * or mmap_sem contention, and if waiting is needed to pin all pages,
- * *@nonblocking will be set to 0.  Further, if @gup_flags does not
- * include FOLL_NOWAIT, the mmap_sem will be released via up_read() in
- * this case.
+ * If @locked != NULL, *@locked will be set to 0 when mmap_sem is
+ * released by an up_read().  That can happen if @gup_flags does not
+ * have FOLL_NOWAIT.
  *
- * A caller using such a combination of @nonblocking and @gup_flags
+ * A caller using such a combination of @locked and @gup_flags
  * must therefore hold the mmap_sem for reading only, and recognize
  * when it's been released.  Otherwise, it must be held for either
  * reading or writing and will not be released.

@@ -1020,7 +1018,7 @@ static int check_vma_flags(struct vm_are
 static long __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 		unsigned long start, unsigned long nr_pages,
 		unsigned int gup_flags, struct page **pages,
-		struct vm_area_struct **vmas, int *nonblocking)
+		struct vm_area_struct **vmas, int *locked)
 {
 	long ret = 0, i = 0;
 	struct vm_area_struct *vma = NULL;

@@ -1066,7 +1064,7 @@ static long __get_user_pages(struct task
 			if (is_vm_hugetlb_page(vma)) {
 				i = follow_hugetlb_page(mm, vma, pages, vmas,
 						&start, &nr_pages, i,
-						gup_flags, nonblocking);
+						gup_flags, locked);
 				continue;
 			}
 		}

@@ -1084,7 +1082,7 @@ retry:
 		page = follow_page_mask(vma, start, foll_flags, &ctx);
 		if (!page) {
 			ret = faultin_page(tsk, vma, start, &foll_flags,
-					nonblocking);
+					   locked);
 			switch (ret) {
 			case 0:
 				goto retry;

@@ -1345,7 +1343,7 @@ static __always_inline long __get_user_p
  * @vma:   target vma
  * @start: start address
  * @end:   end address
- * @nonblocking:
+ * @locked: whether the mmap_sem is still held
  *
  * This takes care of mlocking the pages too if VM_LOCKED is set.
  *

@@ -1353,14 +1351,14 @@ static __always_inline long __get_user_p
  *
  * vma->vm_mm->mmap_sem must be held.
  *
- * If @nonblocking is NULL, it may be held for read or write and will
+ * If @locked is NULL, it may be held for read or write and will
  * be unperturbed.
  *
- * If @nonblocking is non-NULL, it must held for read only and may be
- * released.  If it's released, *@nonblocking will be set to 0.
+ * If @locked is non-NULL, it must held for read only and may be
+ * released.  If it's released, *@locked will be set to 0.
  */
 long populate_vma_page_range(struct vm_area_struct *vma,
-		unsigned long start, unsigned long end, int *nonblocking)
+		unsigned long start, unsigned long end, int *locked)
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long nr_pages = (end - start) / PAGE_SIZE;

@@ -1395,7 +1393,7 @@ long populate_vma_page_range(struct vm_a
 	 * not result in a stack expansion that recurses back here.
 	 */
 	return __get_user_pages(current, mm, start, nr_pages, gup_flags,
-				NULL, NULL, nonblocking);
+				NULL, NULL, locked);
 }
 
 /*

mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-rename-nonblocking-to-locked-where-proper +4 -4

@@ -4272,7 +4272,7 @@ out_release_nounlock:
 long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
 			 struct page **pages, struct vm_area_struct **vmas,
 			 unsigned long *position, unsigned long *nr_pages,
-			 long i, unsigned int flags, int *nonblocking)
+			 long i, unsigned int flags, int *locked)
 {
 	unsigned long pfn_offset;
 	unsigned long vaddr = *position;

@@ -4343,7 +4343,7 @@ long follow_hugetlb_page(struct mm_struc
 				spin_unlock(ptl);
 			if (flags & FOLL_WRITE)
 				fault_flags |= FAULT_FLAG_WRITE;
-			if (nonblocking)
+			if (locked)
 				fault_flags |= FAULT_FLAG_ALLOW_RETRY;
 			if (flags & FOLL_NOWAIT)
 				fault_flags |= FAULT_FLAG_ALLOW_RETRY |

@@ -4360,9 +4360,9 @@ long follow_hugetlb_page(struct mm_struc
 				break;
 			}
 			if (ret & VM_FAULT_RETRY) {
-				if (nonblocking &&
+				if (locked &&
 				    !(fault_flags & FAULT_FLAG_RETRY_NOWAIT))
-					*nonblocking = 0;
+					*locked = 0;
 				*nr_pages = 0;
 				/*
 				 * VM_FAULT_RETRY must not return an

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #94 lore raw

PATCH 87/155 mm/gup: fix __get_user_pages() on fault retry of hugetlb

From: Peter Xu <[email protected]>
Subject: mm/gup: fix __get_user_pages() on fault retry of hugetlb

When follow_hugetlb_page() returns with *locked==0, it means we've got a VM_FAULT_RETRY within the fauling process and we've released the mmap_sem. When that happens, we should stop and bail out.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 mm/gup.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

mm/gup.c renamed from mm/gup.c~mm-gup-fix-__get_user_pages-on-fault-retry-of-hugetlb +10 -0

@@ -1065,6 +1065,16 @@ static long __get_user_pages(struct task
 				i = follow_hugetlb_page(mm, vma, pages, vmas,
 						&start, &nr_pages, i,
 						gup_flags, locked);
+				if (locked && *locked == 0) {
+					/*
+					 * We've got a VM_FAULT_RETRY
+					 * and we've lost mmap_sem.
+					 * We must stop here.
+					 */
+					BUG_ON(gup_flags & FOLL_NOWAIT);
+					BUG_ON(ret != 0);
+					goto out;
+				}
 				continue;
 			}
 		}

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #95 lore raw

PATCH 88/155 mm: introduce fault_signal_pending()

Link http://lkml.kernel.org/r/20200311145921.GD479302@xz-x1 open here

From: Peter Xu <[email protected]>
Subject: mm: introduce fault_signal_pending()

For most architectures, we've got a quick path to detect fatal signal after a handle_mm_fault(). Introduce a helper for that quick path.

It cleans the current codes a bit so we don't need to duplicate the same check across archs. More importantly, this will be an unified place that we handle the signal immediately right after an interrupted page fault, so it'll be much easier for us if we want to change the behavior of handling signals later on for all the archs.

Note that currently only part of the archs are using this new helper, because some archs have their own way to handle signals. In the follow up patches, we'll try to apply this helper to all the rest of archs.

Another note is that the "regs" parameter in the new helper is not used yet. It'll be used very soon. Now we kept it in this patch only to avoid touching all the archs again in the follow up patches.

[[email protected]: fix sparse warnings]

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/alpha/mm/fault.c        |    2 +-
 arch/arm/mm/fault.c          |    2 +-
 arch/hexagon/mm/vm_fault.c   |    2 +-
 arch/ia64/mm/fault.c         |    2 +-
 arch/m68k/mm/fault.c         |    2 +-
 arch/microblaze/mm/fault.c   |    2 +-
 arch/mips/mm/fault.c         |    2 +-
 arch/nds32/mm/fault.c        |    2 +-
 arch/nios2/mm/fault.c        |    2 +-
 arch/openrisc/mm/fault.c     |    2 +-
 arch/parisc/mm/fault.c       |    2 +-
 arch/riscv/mm/fault.c        |    2 +-
 arch/s390/mm/fault.c         |    3 +--
 arch/sparc/mm/fault_32.c     |    2 +-
 arch/sparc/mm/fault_64.c     |    2 +-
 arch/unicore32/mm/fault.c    |    2 +-
 arch/xtensa/mm/fault.c       |    2 +-
 include/linux/sched/signal.h |   15 +++++++++++++++
 18 files changed, 32 insertions(+), 18 deletions(-)

arch/alpha/mm/fault.c renamed from arch/alpha/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -150,7 +150,7 @@ retry:
 	   the fault.  */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/arm/mm/fault.c renamed from arch/arm/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -295,7 +295,7 @@ retry:
 	 * signal first. We do not need to release the mmap_sem because
 	 * it would already be released in __lock_page_or_retry in
 	 * mm/filemap.c. */
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)) {
+	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			goto no_context;
 		return 0;

arch/hexagon/mm/vm_fault.c renamed from arch/hexagon/mm/vm_fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -91,7 +91,7 @@ good_area:
 
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	/* The most common case -- we are done. */

arch/ia64/mm/fault.c renamed from arch/ia64/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -141,7 +141,7 @@ retry:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/m68k/mm/fault.c renamed from arch/m68k/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -138,7 +138,7 @@ good_area:
 	fault = handle_mm_fault(vma, address, flags);
 	pr_debug("handle_mm_fault returns %x\n", fault);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return 0;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/microblaze/mm/fault.c renamed from arch/microblaze/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -217,7 +217,7 @@ good_area:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/mips/mm/fault.c renamed from arch/mips/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -154,7 +154,7 @@ good_area:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);

arch/nds32/mm/fault.c renamed from arch/nds32/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -214,7 +214,7 @@ good_area:
 	 * signal first. We do not need to release the mmap_sem because it
 	 * would already be released in __lock_page_or_retry in mm/filemap.c.
 	 */
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)) {
+	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			goto no_context;
 		return;

arch/nios2/mm/fault.c renamed from arch/nios2/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -133,7 +133,7 @@ good_area:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/openrisc/mm/fault.c renamed from arch/openrisc/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -161,7 +161,7 @@ good_area:
 
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/parisc/mm/fault.c renamed from arch/parisc/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -304,7 +304,7 @@ good_area:
 
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/riscv/mm/fault.c renamed from arch/riscv/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -117,7 +117,7 @@ good_area:
 	 * signal first. We do not need to release the mmap_sem because it
 	 * would already be released in __lock_page_or_retry in mm/filemap.c.
 	 */
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(tsk))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/s390/mm/fault.c renamed from arch/s390/mm/fault.c~mm-introduce-fault_signal_pending +1 -2

@@ -480,8 +480,7 @@ retry:
 	 * the fault.
 	 */
 	fault = handle_mm_fault(vma, address, flags);
-	/* No reason to continue if interrupted by SIGKILL. */
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)) {
+	if (fault_signal_pending(fault, regs)) {
 		fault = VM_FAULT_SIGNAL;
 		if (flags & FAULT_FLAG_RETRY_NOWAIT)
 			goto out_up;

arch/sparc/mm/fault_32.c renamed from arch/sparc/mm/fault_32.c~mm-introduce-fault_signal_pending +1 -1

@@ -237,7 +237,7 @@ good_area:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/sparc/mm/fault_64.c renamed from arch/sparc/mm/fault_64.c~mm-introduce-fault_signal_pending +1 -1

@@ -425,7 +425,7 @@ good_area:
 
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		goto exit_exception;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

arch/unicore32/mm/fault.c renamed from arch/unicore32/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -250,7 +250,7 @@ retry:
 	 * signal first. We do not need to release the mmap_sem because
 	 * it would already be released in __lock_page_or_retry in
 	 * mm/filemap.c. */
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return 0;
 
 	if (!(fault & VM_FAULT_ERROR) && (flags & FAULT_FLAG_ALLOW_RETRY)) {

arch/xtensa/mm/fault.c renamed from arch/xtensa/mm/fault.c~mm-introduce-fault_signal_pending +1 -1

@@ -110,7 +110,7 @@ good_area:
 	 */
 	fault = handle_mm_fault(vma, address, flags);
 
-	if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current))
+	if (fault_signal_pending(fault, regs))
 		return;
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {

include/linux/sched/signal.h renamed from include/linux/sched/signal.h~mm-introduce-fault_signal_pending +15 -0

@@ -10,6 +10,8 @@
 #include <linux/cred.h>
 #include <linux/refcount.h>
 #include <linux/posix-timers.h>
+#include <linux/mm_types.h>
+#include <asm/ptrace.h>
 
 /*
  * Types defining task->signal and task->sighand and APIs using them:

@@ -370,6 +372,19 @@ static inline int signal_pending_state(l
 }
 
 /*
+ * This should only be used in fault handlers to decide whether we
+ * should stop the current fault routine to handle the signals
+ * instead, especially with the case where we've got interrupted with
+ * a VM_FAULT_RETRY.
+ */
+static inline bool fault_signal_pending(vm_fault_t fault_flags,
+					struct pt_regs *regs)
+{
+	return unlikely((fault_flags & VM_FAULT_RETRY) &&
+			fatal_signal_pending(current));
+}
+
+/*
  * Reevaluate whether the task has signals pending delivery.
  * Wake the task if so.
  * This is required every time the blocked sigset_t changes.

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #96 lore raw

PATCH 89/155 x86/mm: use helper fault_signal_pending()

From: Peter Xu <[email protected]>
Subject: x86/mm: use helper fault_signal_pending()

Let's move the fatal signal check even earlier so that we can directly use the new fault_signal_pending() in x86 mm code.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/x86/mm/fault.c |   28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~x86-mm-use-helper-fault_signal_pending +10 ~~-11~~

@@ -1464,27 +1464,25 @@ good_area:
 	fault = handle_mm_fault(vma, address, flags);
 	major |= fault & VM_FAULT_MAJOR;
 
+	/* Quick path to respond to signals */
+	if (fault_signal_pending(fault, regs)) {
+		if (!user_mode(regs))
+			no_context(regs, hw_error_code, address, SIGBUS,
+				   BUS_ADRERR);
+		return;
+	}
+
 	/*
 	 * If we need to retry the mmap_sem has already been released,
 	 * and if there is a fatal signal pending there is no guarantee
 	 * that we made any progress. Handle this case first.
 	 */
-	if (unlikely(fault & VM_FAULT_RETRY)) {
+	if (unlikely((fault & VM_FAULT_RETRY) &&
+		     (flags & FAULT_FLAG_ALLOW_RETRY))) {
 		/* Retry at most once */
-		if (flags & FAULT_FLAG_ALLOW_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
-			flags |= FAULT_FLAG_TRIED;
-			if (!fatal_signal_pending(tsk))
-				goto retry;
-		}
-
-		/* User mode? Just return to handle the fatal exception */
-		if (flags & FAULT_FLAG_USER)
-			return;

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #97 lore raw

PATCH 90/155 arc/mm: use helper fault_signal_pending()

From: Peter Xu <[email protected]>
Subject: arc/mm: use helper fault_signal_pending()

Let ARC to use the new helper fault_signal_pending() by moving the signal check out of the retry logic as standalone. This should also helps to simplify the code a bit.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/arc/mm/fault.c |   34 +++++++++++++---------------------
 1 file changed, 13 insertions(+), 21 deletions(-)

arch/arc/mm/fault.c renamed from arch/arc/mm/fault.c~arc-mm-use-helper-fault_signal_pending +8 -2

@@ -133,29 +133,21 @@ retry:
 
 	fault = handle_mm_fault(vma, address, flags);
 
+	/* Quick path to respond to signals */
+	if (fault_signal_pending(fault, regs)) {
+		if (!user_mode(regs))
+			goto no_context;
+		return;
+	}
+
 	/*
-	 * Fault retry nuances
+	 * Fault retry nuances, mmap_sem already relinquished by core mm
 	 */
-	if (unlikely(fault & VM_FAULT_RETRY)) {

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #98 lore raw

PATCH 91/155 arm64/mm: use helper fault_signal_pending()

From: Peter Xu <[email protected]>
Subject: arm64/mm: use helper fault_signal_pending()

Let the arm64 fault handling to use the new fault_signal_pending() helper, by moving the signal handling out of the retry logic.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/arm64/mm/fault.c |   19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

arch/arm64/mm/fault.c renamed from arch/arm64/mm/fault.c~arm64-mm-use-helper-fault_signal_pending +7 ~~-12~~

@@ -513,19 +513,14 @@ retry:
 	fault = __do_page_fault(mm, addr, mm_flags, vm_flags);
 	major |= fault & VM_FAULT_MAJOR;
 
-	if (fault & VM_FAULT_RETRY) {
-		/*
-		 * If we need to retry but a fatal signal is pending,
-		 * handle the signal first. We do not need to release
-		 * the mmap_sem because it would already be released
-		 * in __lock_page_or_retry in mm/filemap.c.
-		 */
-		if (fatal_signal_pending(current)) {
-			if (!user_mode(regs))
-				goto no_context;
-			return 0;
-		}
+	/* Quick path to respond to signals */
+	if (fault_signal_pending(fault, regs)) {
+		if (!user_mode(regs))
+			goto no_context;
+		return 0;
+	}
 
+	if (fault & VM_FAULT_RETRY) {
 		/*
 		 * Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk of
 		 * starvation.

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #99 lore raw

PATCH 92/155 powerpc/mm: use helper fault_signal_pending()

From: Peter Xu <[email protected]>
Subject: powerpc/mm: use helper fault_signal_pending()

Let powerpc code to use the new helper, by moving the signal handling earlier before the retry logic.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/powerpc/mm/fault.c |   12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

arch/powerpc/mm/fault.c renamed from arch/powerpc/mm/fault.c~powerpc-mm-use-helper-fault_signal_pending +4 -2

@@ -582,6 +582,9 @@ good_area:
 
 	major |= fault & VM_FAULT_MAJOR;
 
+	if (fault_signal_pending(fault, regs))
+		return user_mode(regs) ? 0 : SIGBUS;
+
 	/*
 	 * Handle the retry right now, the mmap_sem has been released in that
 	 * case.

@@ -595,15 +598,8 @@ good_area:
 			 */
 			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
-			if (!fatal_signal_pending(current))
-				goto retry;
+			goto retry;
 		}

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #100 lore raw

PATCH 93/155 sh/mm: use helper fault_signal_pending()

From: Peter Xu <[email protected]>
Subject: sh/mm: use helper fault_signal_pending()

Let SH to use the new fault_signal_pending() helper. Here we'll need to move the up_read() out because that's actually needed as long as !RETRY cases. At the meantime we can drop all the rest of up_read()s now (which seems to be cleaner).

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/sh/mm/fault.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

arch/sh/mm/fault.c renamed from arch/sh/mm/fault.c~sh-mm-use-helper-fault_signal_pending +5 -5

@@ -302,25 +302,25 @@ mm_fault_error(struct pt_regs *regs, uns
 	 * Pagefault was interrupted by SIGKILL. We have no reason to
 	 * continue pagefault.
 	 */
-	if (fatal_signal_pending(current)) {
-		if (!(fault & VM_FAULT_RETRY))
-			up_read(&current->mm->mmap_sem);
+	if (fault_signal_pending(fault, regs)) {
 		if (!user_mode(regs))
 			no_context(regs, error_code, address);
 		return 1;
 	}
 
+	/* Release mmap_sem first if necessary */
+	if (!(fault & VM_FAULT_RETRY))
+		up_read(&current->mm->mmap_sem);
+
 	if (!(fault & VM_FAULT_ERROR))
 		return 0;
 
 	if (fault & VM_FAULT_OOM) {
 		/* Kernel mode? Handle exceptions or die: */
 		if (!user_mode(regs)) {
-			up_read(&current->mm->mmap_sem);
 			no_context(regs, error_code, address);
 			return 1;
 		}
-		up_read(&current->mm->mmap_sem);
 
 		/*
 		 * We ran out of memory, call the OOM killer, and return the

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #101 lore raw

PATCH 94/155 mm: return faster for non-fatal signals in user mode faults

https://lore.kernel.org/lkml/[email protected]/

From: Peter Xu <[email protected]>
Subject: mm: return faster for non-fatal signals in user mode faults

The idea comes from the upstream discussion between Linus and Andrea:

A summary to the issue: there was a special path in handle_userfault() in the past that we'll return a VM_FAULT_NOPAGE when we detected non-fatal signals when waiting for userfault handling. We did that by reacquiring the mmap_sem before returning. However that brings a risk in that the vmas might have changed when we retake the mmap_sem and even we could be holding an invalid vma structure.

This patch is a preparation of removing that special path by allowing the page fault to return even faster if we were interrupted by a non-fatal signal during a user-mode page fault handling routine.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 include/linux/sched/signal.h |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

include/linux/sched/signal.h renamed from include/linux/sched/signal.h~mm-return-faster-for-non-fatal-signals-in-user-mode-faults +2 -1

@@ -381,7 +381,8 @@ static inline bool fault_signal_pending(
 					struct pt_regs *regs)
 {
 	return unlikely((fault_flags & VM_FAULT_RETRY) &&
-			fatal_signal_pending(current));
+			(fatal_signal_pending(current) ||
+			 (user_mode(regs) && signal_pending(current))));
 }
 
 /*

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #102 lore raw

PATCH 95/155 userfaultfd: don't retake mmap_sem to emulate NOPAGE

From: Peter Xu <[email protected]>
Subject: userfaultfd: don't retake mmap_sem to emulate NOPAGE

This patch removes the risk path in handle_userfault() then we will be sure that the callers of handle_mm_fault() will know that the VMAs might have changed. Meanwhile with previous patch we don't lose responsiveness as well since the core mm code now can handle the nonfatal userspace signals even if we return VM_FAULT_RETRY.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Reviewed-by Jerome Glisse <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 fs/userfaultfd.c |   24 ------------------------
 1 file changed, 24 deletions(-)

fs/userfaultfd.c renamed from fs/userfaultfd.c~userfaultfd-dont-retake-mmap_sem-to-emulate-nopage +0 ~~-24~~

@@ -524,30 +524,6 @@ vm_fault_t handle_userfault(struct vm_fa
 
 	__set_current_state(TASK_RUNNING);
 
-	if (return_to_userland) {
-		if (signal_pending(current) &&
-		    !fatal_signal_pending(current)) {
-			/*
-			 * If we got a SIGSTOP or SIGCONT and this is
-			 * a normal userland page fault, just let
-			 * userland return so the signal will be
-			 * handled and gdb debugging works.  The page
-			 * fault code immediately after we return from
-			 * this function is going to release the
-			 * mmap_sem and it's not depending on it
-			 * (unlike gup would if we were not to return
-			 * VM_FAULT_RETRY).
-			 *
-			 * If a fatal signal is pending we still take
-			 * the streamlined VM_FAULT_RETRY failure path
-			 * and there's no need to retake the mmap_sem
-			 * in such case.
-			 */
-			down_read(&mm->mmap_sem);
-			ret = VM_FAULT_NOPAGE;
-		}
-	}
-
 	/*
 	 * Here we race with the list_del; list_add in
 	 * userfaultfd_ctx_read(), however because we don't ever run

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #103 lore raw

PATCH 96/155 mm: introduce FAULT_FLAG_DEFAULT

From: Peter Xu <[email protected]>
Subject: mm: introduce FAULT_FLAG_DEFAULT

Although there're tons of arch-specific page fault handlers, most of them are still sharing the same initial value of the page fault flags. Say, merely all of the page fault handlers would allow the fault to be retried, and they also allow the fault to respond to SIGKILL.

Let's define a default value for the fault flags to replace those initial page fault flags that were copied over. With this, it'll be far easier to introduce new fault flag that can be used by all the architectures instead of touching all the archs.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/alpha/mm/fault.c      |    2 +-
 arch/arc/mm/fault.c        |    2 +-
 arch/arm/mm/fault.c        |    2 +-
 arch/arm64/mm/fault.c      |    2 +-
 arch/hexagon/mm/vm_fault.c |    2 +-
 arch/ia64/mm/fault.c       |    2 +-
 arch/m68k/mm/fault.c       |    2 +-
 arch/microblaze/mm/fault.c |    2 +-
 arch/mips/mm/fault.c       |    2 +-
 arch/nds32/mm/fault.c      |    2 +-
 arch/nios2/mm/fault.c      |    2 +-
 arch/openrisc/mm/fault.c   |    2 +-
 arch/parisc/mm/fault.c     |    2 +-
 arch/powerpc/mm/fault.c    |    2 +-
 arch/riscv/mm/fault.c      |    2 +-
 arch/s390/mm/fault.c       |    2 +-
 arch/sh/mm/fault.c         |    2 +-
 arch/sparc/mm/fault_32.c   |    2 +-
 arch/sparc/mm/fault_64.c   |    2 +-
 arch/um/kernel/trap.c      |    2 +-
 arch/unicore32/mm/fault.c  |    2 +-
 arch/x86/mm/fault.c        |    2 +-
 arch/xtensa/mm/fault.c     |    2 +-
 include/linux/mm.h         |    7 +++++++
 24 files changed, 30 insertions(+), 23 deletions(-)

arch/alpha/mm/fault.c renamed from arch/alpha/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -89,7 +89,7 @@ do_page_fault(unsigned long address, uns
 	const struct exception_table_entry *fixup;
 	int si_code = SEGV_MAPERR;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	/* As of EV6, a load into $31/$f31 is a prefetch, and never faults
 	   (or is suppressed by the PALcode).  Support that for older CPUs

arch/arc/mm/fault.c renamed from arch/arc/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -100,7 +100,7 @@ void do_page_fault(unsigned long address
 	         (regs->ecr_cause == ECR_C_PROTV_INST_FETCH))
 		exec = 1;
 
-	flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	flags = FAULT_FLAG_DEFAULT;
 	if (user_mode(regs))
 		flags |= FAULT_FLAG_USER;
 	if (write)

arch/arm64/mm/fault.c renamed from arch/arm64/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -446,7 +446,7 @@ static int __kprobes do_page_fault(unsig
 	struct mm_struct *mm = current->mm;
 	vm_fault_t fault, major = 0;
 	unsigned long vm_flags = VM_READ | VM_WRITE | VM_EXEC;
-	unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int mm_flags = FAULT_FLAG_DEFAULT;
 
 	if (kprobe_page_fault(regs, esr))
 		return 0;

arch/arm/mm/fault.c renamed from arch/arm/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -241,7 +241,7 @@ do_page_fault(unsigned long addr, unsign
 	struct mm_struct *mm;
 	int sig, code;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	if (kprobe_page_fault(regs, fsr))
 		return 0;

arch/hexagon/mm/vm_fault.c renamed from arch/hexagon/mm/vm_fault.c~mm-introduce-fault_flag_default +1 -1

@@ -41,7 +41,7 @@ void do_page_fault(unsigned long address
 	int si_code = SEGV_MAPERR;
 	vm_fault_t fault;
 	const struct exception_table_entry *fixup;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	/*
 	 * If we're in an interrupt or have no user context,

arch/ia64/mm/fault.c renamed from arch/ia64/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -65,7 +65,7 @@ ia64_do_page_fault (unsigned long addres
 	struct mm_struct *mm = current->mm;
 	unsigned long mask;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	mask = ((((isr >> IA64_ISR_X_BIT) & 1UL) << VM_EXEC_BIT)
 		| (((isr >> IA64_ISR_W_BIT) & 1UL) << VM_WRITE_BIT));

arch/m68k/mm/fault.c renamed from arch/m68k/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -71,7 +71,7 @@ int do_page_fault(struct pt_regs *regs,
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct * vma;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	pr_debug("do page fault:\nregs->sr=%#x, regs->pc=%#lx, address=%#lx, %ld, %p\n",
 		regs->sr, regs->pc, address, error_code, mm ? mm->pgd : NULL);

arch/microblaze/mm/fault.c renamed from arch/microblaze/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -91,7 +91,7 @@ void do_page_fault(struct pt_regs *regs,
 	int code = SEGV_MAPERR;
 	int is_write = error_code & ESR_S;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	regs->ear = address;
 	regs->esr = error_code;

arch/mips/mm/fault.c renamed from arch/mips/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -44,7 +44,7 @@ static void __kprobes __do_page_fault(st
 	const int field = sizeof(unsigned long) * 2;
 	int si_code;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 10);

arch/nds32/mm/fault.c renamed from arch/nds32/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -80,7 +80,7 @@ void do_page_fault(unsigned long entry,
 	int si_code;
 	vm_fault_t fault;
 	unsigned int mask = VM_READ | VM_WRITE | VM_EXEC;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	error_code = error_code & (ITYPE_mskINST | ITYPE_mskETYPE);
 	tsk = current;

arch/nios2/mm/fault.c renamed from arch/nios2/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -47,7 +47,7 @@ asmlinkage void do_page_fault(struct pt_
 	struct mm_struct *mm = tsk->mm;
 	int code = SEGV_MAPERR;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	cause >>= 2;

arch/openrisc/mm/fault.c renamed from arch/openrisc/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -50,7 +50,7 @@ asmlinkage void do_page_fault(struct pt_
 	struct vm_area_struct *vma;
 	int si_code;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	tsk = current;

arch/parisc/mm/fault.c renamed from arch/parisc/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -274,7 +274,7 @@ void do_page_fault(struct pt_regs *regs,
 	if (!mm)
 		goto no_context;
 
-	flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	flags = FAULT_FLAG_DEFAULT;
 	if (user_mode(regs))
 		flags |= FAULT_FLAG_USER;

arch/powerpc/mm/fault.c renamed from arch/powerpc/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -434,7 +434,7 @@ static int __do_page_fault(struct pt_reg
 {
 	struct vm_area_struct * vma;
 	struct mm_struct *mm = current->mm;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
  	int is_exec = TRAP(regs) == 0x400;
 	int is_user = user_mode(regs);
 	int is_write = page_fault_is_write(error_code);

arch/riscv/mm/fault.c renamed from arch/riscv/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -30,7 +30,7 @@ asmlinkage void do_page_fault(struct pt_
 	struct vm_area_struct *vma;
 	struct mm_struct *mm;
 	unsigned long addr, cause;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 	int code = SEGV_MAPERR;
 	vm_fault_t fault;

arch/s390/mm/fault.c renamed from arch/s390/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -429,7 +429,7 @@ static inline vm_fault_t do_exception(st
 
 	address = trans_exc_code & __FAIL_ADDR_MASK;
 	perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address);
-	flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	flags = FAULT_FLAG_DEFAULT;
 	if (user_mode(regs))
 		flags |= FAULT_FLAG_USER;
 	if (access == VM_WRITE || (trans_exc_code & store_indication) == 0x400)

arch/sh/mm/fault.c renamed from arch/sh/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -380,7 +380,7 @@ asmlinkage void __kprobes do_page_fault(
 	struct mm_struct *mm;
 	struct vm_area_struct * vma;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	tsk = current;
 	mm = tsk->mm;

arch/sparc/mm/fault_32.c renamed from arch/sparc/mm/fault_32.c~mm-introduce-fault_flag_default +1 -1

@@ -168,7 +168,7 @@ asmlinkage void do_sparc_fault(struct pt
 	int from_user = !(regs->psr & PSR_PS);
 	int code;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	if (text_fault)
 		address = regs->pc;

arch/sparc/mm/fault_64.c renamed from arch/sparc/mm/fault_64.c~mm-introduce-fault_flag_default +1 -1

@@ -271,7 +271,7 @@ asmlinkage void __kprobes do_sparc64_fau
 	int si_code, fault_code;
 	vm_fault_t fault;
 	unsigned long address, mm_rss;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	fault_code = get_thread_fault_code();

arch/um/kernel/trap.c renamed from arch/um/kernel/trap.c~mm-introduce-fault_flag_default +1 -1

@@ -33,7 +33,7 @@ int handle_page_fault(unsigned long addr
 	pmd_t *pmd;
 	pte_t *pte;
 	int err = -EFAULT;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	*code_out = SEGV_MAPERR;

arch/unicore32/mm/fault.c renamed from arch/unicore32/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -202,7 +202,7 @@ static int do_pf(unsigned long addr, uns
 	struct mm_struct *mm;
 	int sig, code;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	tsk = current;
 	mm = tsk->mm;

arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -1310,7 +1310,7 @@ void do_user_addr_fault(struct pt_regs *
 	struct task_struct *tsk;
 	struct mm_struct *mm;
 	vm_fault_t fault, major = 0;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	tsk = current;
 	mm = tsk->mm;

arch/xtensa/mm/fault.c renamed from arch/xtensa/mm/fault.c~mm-introduce-fault_flag_default +1 -1

@@ -43,7 +43,7 @@ void do_page_fault(struct pt_regs *regs)
 
 	int is_write, is_exec;
 	vm_fault_t fault;
-	unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+	unsigned int flags = FAULT_FLAG_DEFAULT;
 
 	code = SEGV_MAPERR;

include/linux/mm.h renamed from include/linux/mm.h~mm-introduce-fault_flag_default +7 -0

@@ -391,6 +391,13 @@ extern pgprot_t protection_map[16];
 #define FAULT_FLAG_REMOTE	0x80	/* faulting for non current tsk/mm */
 #define FAULT_FLAG_INSTRUCTION  0x100	/* The fault was during an instruction fetch */
 
+/*
+ * The default fault flags that should be used by most of the
+ * arch-specific page fault handlers.
+ */
+#define FAULT_FLAG_DEFAULT  (FAULT_FLAG_ALLOW_RETRY | \
+			     FAULT_FLAG_KILLABLE)
+
 #define FAULT_FLAG_TRACE \
 	{ FAULT_FLAG_WRITE,		"WRITE" }, \
 	{ FAULT_FLAG_MKWRITE,		"MKWRITE" }, \

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #104 lore raw

PATCH 97/155 mm: introduce FAULT_FLAG_INTERRUPTIBLE

From: Peter Xu <[email protected]>
Subject: mm: introduce FAULT_FLAG_INTERRUPTIBLE

handle_userfaultfd() is currently the only one place in the kernel page fault procedures that can respond to non-fatal userspace signals. It was trying to detect such an allowance by checking against USER & KILLABLE flags, which was "un-official".

In this patch, we introduced a new flag (FAULT_FLAG_INTERRUPTIBLE) to show that the fault handler allows the fault procedure to respond even to non-fatal signals. Meanwhile, add this new flag to the default fault flags so that all the page fault handlers can benefit from the new flag. With that, replacing the userfault check to this one.

Since the line is getting even longer, clean up the fault flags a bit too to ease TTY users.

Although we've got a new flag and applied it, we shouldn't have any functional change with this patch so far.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 fs/userfaultfd.c   |    4 +---
 include/linux/mm.h |   39 ++++++++++++++++++++++++++++-----------
 2 files changed, 29 insertions(+), 14 deletions(-)

fs/userfaultfd.c renamed from fs/userfaultfd.c~mm-introduce-fault_flag_interruptible +1 -3

@@ -462,9 +462,7 @@ vm_fault_t handle_userfault(struct vm_fa
 	uwq.ctx = ctx;
 	uwq.waken = false;
 
-	return_to_userland =
-		(vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
-		(FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
+	return_to_userland = vmf->flags & FAULT_FLAG_INTERRUPTIBLE;
 	blocking_state = return_to_userland ? TASK_INTERRUPTIBLE :
 			 TASK_KILLABLE;

include/linux/mm.h renamed from include/linux/mm.h~mm-introduce-fault_flag_interruptible +28 ~~-11~~

@@ -381,22 +381,38 @@ extern unsigned int kobjsize(const void
  */
 extern pgprot_t protection_map[16];
 
-#define FAULT_FLAG_WRITE	0x01	/* Fault was a write access */
-#define FAULT_FLAG_MKWRITE	0x02	/* Fault was mkwrite of existing pte */
-#define FAULT_FLAG_ALLOW_RETRY	0x04	/* Retry fault if blocking */
-#define FAULT_FLAG_RETRY_NOWAIT	0x08	/* Don't drop mmap_sem and wait when retrying */
-#define FAULT_FLAG_KILLABLE	0x10	/* The fault task is in SIGKILL killable region */
-#define FAULT_FLAG_TRIED	0x20	/* Second try */
-#define FAULT_FLAG_USER		0x40	/* The fault originated in userspace */
-#define FAULT_FLAG_REMOTE	0x80	/* faulting for non current tsk/mm */
-#define FAULT_FLAG_INSTRUCTION  0x100	/* The fault was during an instruction fetch */
+/**
+ * Fault flag definitions.
+ *
+ * @FAULT_FLAG_WRITE: Fault was a write fault.
+ * @FAULT_FLAG_MKWRITE: Fault was mkwrite of existing PTE.
+ * @FAULT_FLAG_ALLOW_RETRY: Allow to retry the fault if blocked.
+ * @FAULT_FLAG_RETRY_NOWAIT: Don't drop mmap_sem and wait when retrying.
+ * @FAULT_FLAG_KILLABLE: The fault task is in SIGKILL killable region.
+ * @FAULT_FLAG_TRIED: The fault has been tried once.
+ * @FAULT_FLAG_USER: The fault originated in userspace.
+ * @FAULT_FLAG_REMOTE: The fault is not for current task/mm.
+ * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch.
+ * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals.
+ */
+#define FAULT_FLAG_WRITE			0x01
+#define FAULT_FLAG_MKWRITE			0x02
+#define FAULT_FLAG_ALLOW_RETRY			0x04
+#define FAULT_FLAG_RETRY_NOWAIT			0x08
+#define FAULT_FLAG_KILLABLE			0x10
+#define FAULT_FLAG_TRIED			0x20
+#define FAULT_FLAG_USER				0x40
+#define FAULT_FLAG_REMOTE			0x80
+#define FAULT_FLAG_INSTRUCTION  		0x100
+#define FAULT_FLAG_INTERRUPTIBLE		0x200
 
 /*
  * The default fault flags that should be used by most of the
  * arch-specific page fault handlers.
  */
 #define FAULT_FLAG_DEFAULT  (FAULT_FLAG_ALLOW_RETRY | \
-			     FAULT_FLAG_KILLABLE)
+			     FAULT_FLAG_KILLABLE | \
+			     FAULT_FLAG_INTERRUPTIBLE)
 
 #define FAULT_FLAG_TRACE \
 	{ FAULT_FLAG_WRITE,		"WRITE" }, \

@@ -407,7 +423,8 @@ extern pgprot_t protection_map[16];
 	{ FAULT_FLAG_TRIED,		"TRIED" }, \
 	{ FAULT_FLAG_USER,		"USER" }, \
 	{ FAULT_FLAG_REMOTE,		"REMOTE" }, \
-	{ FAULT_FLAG_INSTRUCTION,	"INSTRUCTION" }
+	{ FAULT_FLAG_INSTRUCTION,	"INSTRUCTION" }, \
+	{ FAULT_FLAG_INTERRUPTIBLE,	"INTERRUPTIBLE" }
 
 /*
  * vm_fault is filled by the the pagefault handler and passed to the vma's

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #105 lore raw

PATCH 98/155 mm: allow VM_FAULT_RETRY for multiple times

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

From: Peter Xu <[email protected]>
Subject: mm: allow VM_FAULT_RETRY for multiple times

The idea comes from a discussion between Linus and Andrea [1].

Before this patch we only allow a page fault to retry once. We achieved this by clearing the FAULT_FLAG_ALLOW_RETRY flag when doing handle_mm_fault() the second time. This was majorly used to avoid unexpected starvation of the system by looping over forever to handle the page fault on a single page. However that should hardly happen, and after all for each code path to return a VM_FAULT_RETRY we'll first wait for a condition (during which time we should possibly yield the cpu) to happen before VM_FAULT_RETRY is really returned.

This patch removes the restriction by keeping the FAULT_FLAG_ALLOW_RETRY flag when we receive VM_FAULT_RETRY. It means that the page fault handler now can retry the page fault for multiple times if necessary without the need to generate another page fault event. Meanwhile we still keep the FAULT_FLAG_TRIED flag so page fault handler can still identify whether a page fault is the first attempt or not.

Then we'll have these combinations of fault flags (only considering ALLOW_RETRY flag and TRIED flag):

- ALLOW_RETRY and !TRIED: this means the page fault allows to retry, and this is the first try

- ALLOW_RETRY and TRIED: this means the page fault allows to
retry, and this is not the first try

- !ALLOW_RETRY and !TRIED: this means the page fault does not allow to retry at all

- !ALLOW_RETRY and TRIED: this is forbidden and should never be used

In existing code we have multiple places that has taken special care of the first condition above by checking against (fault_flags & FAULT_FLAG_ALLOW_RETRY). This patch introduces a simple helper to detect the first retry of a page fault by checking against both (fault_flags & FAULT_FLAG_ALLOW_RETRY) and !(fault_flag & FAULT_FLAG_TRIED) because now even the 2nd try will have the ALLOW_RETRY set, then use that helper in all existing special paths. One example is in __lock_page_or_retry(), now we'll drop the mmap_sem only in the first attempt of page fault and we'll keep it in follow up retries, so old locking behavior will be retained.

This will be a nice enhancement for current code [2] at the same time a supporting material for the future userfaultfd-writeprotect work, since in that work there will always be an explicit userfault writeprotect retry for protected pages, and if that cannot resolve the page fault (e.g., when userfaultfd-writeprotect is used in conjunction with swapped pages) then we'll possibly need a 3rd retry of the page fault. It might also benefit other potential users who will have similar requirement like userfault write-protection.

GUP code is not touched yet and will be covered in follow up patch.

Please read the thread below for more information.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 arch/alpha/mm/fault.c           |    2 -
 arch/arc/mm/fault.c             |    1 
 arch/arm/mm/fault.c             |    3 --
 arch/arm64/mm/fault.c           |    5 ----
 arch/hexagon/mm/vm_fault.c      |    1 
 arch/ia64/mm/fault.c            |    1 
 arch/m68k/mm/fault.c            |    3 --
 arch/microblaze/mm/fault.c      |    1 
 arch/mips/mm/fault.c            |    1 
 arch/nds32/mm/fault.c           |    1 
 arch/nios2/mm/fault.c           |    3 --
 arch/openrisc/mm/fault.c        |    1 
 arch/parisc/mm/fault.c          |    4 ---
 arch/powerpc/mm/fault.c         |    6 ----
 arch/riscv/mm/fault.c           |    5 ----
 arch/s390/mm/fault.c            |    5 ----
 arch/sh/mm/fault.c              |    1 
 arch/sparc/mm/fault_32.c        |    1 
 arch/sparc/mm/fault_64.c        |    1 
 arch/um/kernel/trap.c           |    1 
 arch/unicore32/mm/fault.c       |    4 ---
 arch/x86/mm/fault.c             |    2 -
 arch/xtensa/mm/fault.c          |    1 
 drivers/gpu/drm/ttm/ttm_bo_vm.c |   12 +++++++--
 include/linux/mm.h              |   37 ++++++++++++++++++++++++++++++
 mm/filemap.c                    |    2 -
 mm/internal.h                   |    6 ++--
 27 files changed, 54 insertions(+), 57 deletions(-)

arch/alpha/mm/fault.c renamed from arch/alpha/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +1 -1

@@ -169,7 +169,7 @@ retry:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
+			flags |= FAULT_FLAG_TRIED;
 
 			 /* No need to up_read(&mm->mmap_sem) as we would
 			 * have already released it in __lock_page_or_retry

arch/arc/mm/fault.c renamed from arch/arc/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -145,7 +145,6 @@ retry:
 	 */
 	if (unlikely((fault & VM_FAULT_RETRY) &&
 		     (flags & FAULT_FLAG_ALLOW_RETRY))) {
-		flags &= ~FAULT_FLAG_ALLOW_RETRY;
 		flags |= FAULT_FLAG_TRIED;
 		goto retry;
 	}

arch/arm64/mm/fault.c renamed from arch/arm64/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -5

@@ -521,12 +521,7 @@ retry:
 	}
 
 	if (fault & VM_FAULT_RETRY) {
-		/*
-		 * Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk of
-		 * starvation.
-		 */
 		if (mm_flags & FAULT_FLAG_ALLOW_RETRY) {
-			mm_flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			mm_flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}

arch/arm/mm/fault.c renamed from arch/arm/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -3

@@ -319,9 +319,6 @@ retry:
 					regs, addr);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			* of starvation. */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}

arch/hexagon/mm/vm_fault.c renamed from arch/hexagon/mm/vm_fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -102,7 +102,6 @@ good_area:
 			else
 				current->min_flt++;
 			if (fault & VM_FAULT_RETRY) {
-				flags &= ~FAULT_FLAG_ALLOW_RETRY;
 				flags |= FAULT_FLAG_TRIED;
 				goto retry;
 			}

arch/ia64/mm/fault.c renamed from arch/ia64/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -167,7 +167,6 @@ retry:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			 /* No need to up_read(&mm->mmap_sem) as we would

arch/m68k/mm/fault.c renamed from arch/m68k/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -3

@@ -162,9 +162,6 @@ good_area:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation. */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/microblaze/mm/fault.c renamed from arch/microblaze/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -236,7 +236,6 @@ good_area:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/mips/mm/fault.c renamed from arch/mips/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -178,7 +178,6 @@ good_area:
 			tsk->min_flt++;
 		}
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/nds32/mm/fault.c renamed from arch/nds32/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -246,7 +246,6 @@ good_area:
 				      1, regs, addr);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/* No need to up_read(&mm->mmap_sem) as we would

arch/nios2/mm/fault.c renamed from arch/nios2/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -3

@@ -157,9 +157,6 @@ good_area:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation. */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/openrisc/mm/fault.c renamed from arch/openrisc/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -181,7 +181,6 @@ good_area:
 		else
 			tsk->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			 /* No need to up_read(&mm->mmap_sem) as we would

arch/parisc/mm/fault.c renamed from arch/parisc/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +1 -3

@@ -328,14 +328,12 @@ good_area:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
-
 			/*
 			 * No need to up_read(&mm->mmap_sem) as we would
 			 * have already released it in __lock_page_or_retry
 			 * in mm/filemap.c.
 			 */
-
+			flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}
 	}

arch/powerpc/mm/fault.c renamed from arch/powerpc/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -6

@@ -590,13 +590,7 @@ good_area:
 	 * case.
 	 */
 	if (unlikely(fault & VM_FAULT_RETRY)) {
-		/* We retry only once */
 		if (flags & FAULT_FLAG_ALLOW_RETRY) {
-			/*
-			 * Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation.
-			 */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}

arch/riscv/mm/fault.c renamed from arch/riscv/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -5

@@ -144,11 +144,6 @@ good_area:
 				      1, regs, addr);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			/*
-			 * Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation.
-			 */
-			flags &= ~(FAULT_FLAG_ALLOW_RETRY);
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/s390/mm/fault.c renamed from arch/s390/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +1 -4

@@ -513,10 +513,7 @@ retry:
 				fault = VM_FAULT_PFAULT;
 				goto out_up;
 			}
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			 * of starvation. */
-			flags &= ~(FAULT_FLAG_ALLOW_RETRY |
-				   FAULT_FLAG_RETRY_NOWAIT);
+			flags &= ~FAULT_FLAG_RETRY_NOWAIT;
 			flags |= FAULT_FLAG_TRIED;
 			down_read(&mm->mmap_sem);
 			goto retry;

arch/sh/mm/fault.c renamed from arch/sh/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -481,7 +481,6 @@ good_area:
 				      regs, address);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/*

arch/sparc/mm/fault_32.c renamed from arch/sparc/mm/fault_32.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -261,7 +261,6 @@ good_area:
 				      1, regs, address);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/* No need to up_read(&mm->mmap_sem) as we would

arch/sparc/mm/fault_64.c renamed from arch/sparc/mm/fault_64.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -449,7 +449,6 @@ good_area:
 				      1, regs, address);
 		}
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			/* No need to up_read(&mm->mmap_sem) as we would

arch/um/kernel/trap.c renamed from arch/um/kernel/trap.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -97,7 +97,6 @@ good_area:
 			else
 				current->min_flt++;
 			if (fault & VM_FAULT_RETRY) {
-				flags &= ~FAULT_FLAG_ALLOW_RETRY;
 				flags |= FAULT_FLAG_TRIED;
 
 				goto retry;

arch/unicore32/mm/fault.c renamed from arch/unicore32/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +1 -3

@@ -259,9 +259,7 @@ retry:
 		else
 			tsk->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			/* Clear FAULT_FLAG_ALLOW_RETRY to avoid any risk
-			* of starvation. */
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
+			flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}
 	}

arch/x86/mm/fault.c renamed from arch/x86/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -2

@@ -1479,8 +1479,6 @@ good_area:
 	 */
 	if (unlikely((fault & VM_FAULT_RETRY) &&
 		     (flags & FAULT_FLAG_ALLOW_RETRY))) {
-		/* Retry at most once */
-		flags &= ~FAULT_FLAG_ALLOW_RETRY;
 		flags |= FAULT_FLAG_TRIED;
 		goto retry;
 	}

arch/xtensa/mm/fault.c renamed from arch/xtensa/mm/fault.c~mm-allow-vm_fault_retry-for-multiple-times +0 -1

@@ -128,7 +128,6 @@ good_area:
 		else
 			current->min_flt++;
 		if (fault & VM_FAULT_RETRY) {
-			flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			flags |= FAULT_FLAG_TRIED;
 
 			 /* No need to up_read(&mm->mmap_sem) as we would

drivers/gpu/drm/ttm/ttm_bo_vm.c renamed from drivers/gpu/drm/ttm/ttm_bo_vm.c~mm-allow-vm_fault_retry-for-multiple-times +9 -3

@@ -59,9 +59,10 @@ static vm_fault_t ttm_bo_vm_fault_idle(s
 
 	/*
 	 * If possible, avoid waiting for GPU with mmap_sem
-	 * held.
+	 * held.  We only do this if the fault allows retry and this
+	 * is the first attempt.
 	 */
-	if (vmf->flags & FAULT_FLAG_ALLOW_RETRY) {
+	if (fault_flag_allow_retry_first(vmf->flags)) {
 		ret = VM_FAULT_RETRY;
 		if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
 			goto out_unlock;

@@ -135,7 +136,12 @@ vm_fault_t ttm_bo_vm_reserve(struct ttm_
 	 * for the buffer to become unreserved.
 	 */
 	if (unlikely(!dma_resv_trylock(bo->base.resv))) {
-		if (vmf->flags & FAULT_FLAG_ALLOW_RETRY) {
+		/*
+		 * If the fault allows retry and this is the first
+		 * fault attempt, we try to release the mmap_sem
+		 * before waiting
+		 */
+		if (fault_flag_allow_retry_first(vmf->flags)) {
 			if (!(vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) {
 				ttm_bo_get(bo);
 				up_read(&vmf->vma->vm_mm->mmap_sem);

include/linux/mm.h renamed from include/linux/mm.h~mm-allow-vm_fault_retry-for-multiple-times +37 -0

@@ -394,6 +394,25 @@ extern pgprot_t protection_map[16];
  * @FAULT_FLAG_REMOTE: The fault is not for current task/mm.
  * @FAULT_FLAG_INSTRUCTION: The fault was during an instruction fetch.
  * @FAULT_FLAG_INTERRUPTIBLE: The fault can be interrupted by non-fatal signals.
+ *
+ * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify
+ * whether we would allow page faults to retry by specifying these two
+ * fault flags correctly.  Currently there can be three legal combinations:
+ *
+ * (a) ALLOW_RETRY and !TRIED:  this means the page fault allows retry, and
+ *                              this is the first try
+ *
+ * (b) ALLOW_RETRY and TRIED:   this means the page fault allows retry, and
+ *                              we've already tried at least once
+ *
+ * (c) !ALLOW_RETRY and !TRIED: this means the page fault does not allow retry
+ *
+ * The unlisted combination (!ALLOW_RETRY && TRIED) is illegal and should never
+ * be used.  Note that page faults can be allowed to retry for multiple times,
+ * in which case we'll have an initial fault with flags (a) then later on
+ * continuous faults with flags (b).  We should always try to detect pending
+ * signals before a retry to make sure the continuous page faults can still be
+ * interrupted if necessary.
  */
 #define FAULT_FLAG_WRITE			0x01
 #define FAULT_FLAG_MKWRITE			0x02

@@ -414,6 +433,24 @@ extern pgprot_t protection_map[16];
 			     FAULT_FLAG_KILLABLE | \
 			     FAULT_FLAG_INTERRUPTIBLE)
 
+/**
+ * fault_flag_allow_retry_first - check ALLOW_RETRY the first time
+ *
+ * This is mostly used for places where we want to try to avoid taking
+ * the mmap_sem for too long a time when waiting for another condition
+ * to change, in which case we can try to be polite to release the
+ * mmap_sem in the first round to avoid potential starvation of other
+ * processes that would also want the mmap_sem.
+ *
+ * Return: true if the page fault allows retry and this is the first
+ * attempt of the fault handling; false otherwise.
+ */
+static inline bool fault_flag_allow_retry_first(unsigned int flags)
+{
+	return (flags & FAULT_FLAG_ALLOW_RETRY) &&
+	    (!(flags & FAULT_FLAG_TRIED));
+}
+
 #define FAULT_FLAG_TRACE \
 	{ FAULT_FLAG_WRITE,		"WRITE" }, \
 	{ FAULT_FLAG_MKWRITE,		"MKWRITE" }, \

mm/filemap.c renamed from mm/filemap.c~mm-allow-vm_fault_retry-for-multiple-times +1 -1

@@ -1386,7 +1386,7 @@ EXPORT_SYMBOL_GPL(__lock_page_killable);
 int __lock_page_or_retry(struct page *page, struct mm_struct *mm,
 			 unsigned int flags)
 {
-	if (flags & FAULT_FLAG_ALLOW_RETRY) {
+	if (fault_flag_allow_retry_first(flags)) {
 		/*
 		 * CAUTION! In this case, mmap_sem is not released
 		 * even though return 0.

mm/internal.h renamed from mm/internal.h~mm-allow-vm_fault_retry-for-multiple-times +3 -3

@@ -400,10 +400,10 @@ static inline struct file *maybe_unlock_
 	/*
 	 * FAULT_FLAG_RETRY_NOWAIT means we don't want to wait on page locks or
 	 * anything, so we only pin the file and drop the mmap_sem if only
-	 * FAULT_FLAG_ALLOW_RETRY is set.
+	 * FAULT_FLAG_ALLOW_RETRY is set, while this is the first attempt.
 	 */
-	if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) ==
-	    FAULT_FLAG_ALLOW_RETRY) {
+	if (fault_flag_allow_retry_first(flags) &&
+	    !(flags & FAULT_FLAG_RETRY_NOWAIT)) {
 		fpin = get_file(vmf->vma->vm_file);
 		up_read(&vmf->vma->vm_mm->mmap_sem);
 	}

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #106 lore raw

PATCH 99/155 mm/gup: allow VM_FAULT_RETRY for multiple times

From: Peter Xu <[email protected]>
Subject: mm/gup: allow VM_FAULT_RETRY for multiple times

This is the gup counterpart of the change that allows the VM_FAULT_RETRY to happen for more than once. One thing to mention is that we must check the fatal signal here before retry because the GUP can be interrupted by that, otherwise we can loop forever.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 mm/gup.c     |   27 +++++++++++++++++++++------
 mm/hugetlb.c |    6 ++++--
 2 files changed, 25 insertions(+), 8 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-allow-vm_fault_retry-for-multiple-times +21 -6

@@ -868,7 +868,10 @@ static int faultin_page(struct task_stru
 	if (*flags & FOLL_NOWAIT)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT;
 	if (*flags & FOLL_TRIED) {
-		VM_WARN_ON_ONCE(fault_flags & FAULT_FLAG_ALLOW_RETRY);
+		/*
+		 * Note: FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_TRIED
+		 * can co-exist
+		 */
 		fault_flags |= FAULT_FLAG_TRIED;
 	}

@@ -1228,7 +1231,6 @@ retry:
 		down_read(&mm->mmap_sem);
 		if (!(fault_flags & FAULT_FLAG_TRIED)) {
 			*unlocked = true;
-			fault_flags &= ~FAULT_FLAG_ALLOW_RETRY;
 			fault_flags |= FAULT_FLAG_TRIED;
 			goto retry;
 		}

@@ -1312,17 +1314,30 @@ static __always_inline long __get_user_p
 		if (likely(pages))
 			pages += ret;
 		start += ret << PAGE_SHIFT;
+		lock_dropped = true;
 
+retry:
 		/*
 		 * Repeat on the address that fired VM_FAULT_RETRY
-		 * without FAULT_FLAG_ALLOW_RETRY but with
-		 * FAULT_FLAG_TRIED.
+		 * with both FAULT_FLAG_ALLOW_RETRY and
+		 * FAULT_FLAG_TRIED.  Note that GUP can be interrupted
+		 * by fatal signals, so we need to check it before we
+		 * start trying again otherwise it can loop forever.
 		 */
+
+		if (fatal_signal_pending(current))
+			break;
+
 		*locked = 1;
-		lock_dropped = true;
 		down_read(&mm->mmap_sem);
+
 		ret = __get_user_pages(tsk, mm, start, 1, flags | FOLL_TRIED,
-				       pages, NULL, NULL);
+				       pages, NULL, locked);
+		if (!*locked) {
+			/* Continue to retry until we succeeded */
+			BUG_ON(ret != 0);
+			goto retry;
+		}
 		if (ret != 1) {
 			BUG_ON(ret > 1);
 			if (!pages_done)

mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-allow-vm_fault_retry-for-multiple-times +4 -2

@@ -4349,8 +4349,10 @@ long follow_hugetlb_page(struct mm_struc
 				fault_flags |= FAULT_FLAG_ALLOW_RETRY |
 					FAULT_FLAG_RETRY_NOWAIT;
 			if (flags & FOLL_TRIED) {
-				VM_WARN_ON_ONCE(fault_flags &
-						FAULT_FLAG_ALLOW_RETRY);
+				/*
+				 * Note: FAULT_FLAG_ALLOW_RETRY and
+				 * FAULT_FLAG_TRIED can co-exist
+				 */
 				fault_flags |= FAULT_FLAG_TRIED;
 			}
 			ret = hugetlb_fault(mm, vma, vaddr, fault_flags);

Andrew Morton

author maintainer

2020-04-02 04:08 UTC #107 lore raw

PATCH 100/155 mm/gup: allow to react to fatal signals

From: Peter Xu <[email protected]>
Subject: mm/gup: allow to react to fatal signals

The existing gup code does not react to the fatal signals in many code paths. For example, in one retry path of gup we're still using down_read() rather than down_read_killable(). Also, when doing page faults we don't pass in FAULT_FLAG_KILLABLE as well, which means that within the faulting process we'll wait in non-killable way as well. These were spotted by Linus during the code review of some other patches.

Let's allow the gup code to react to fatal signals to improve the responsiveness of threads when during gup and being killed.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 mm/gup.c     |   12 +++++++++---
 mm/hugetlb.c |    3 ++-
 2 files changed, 11 insertions(+), 4 deletions(-)

mm/gup.c renamed from mm/gup.c~mm-gup-allow-to-react-to-fatal-signals +9 -3

@@ -864,7 +864,7 @@ static int faultin_page(struct task_stru
 	if (*flags & FOLL_REMOTE)
 		fault_flags |= FAULT_FLAG_REMOTE;
 	if (locked)
-		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
+		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 	if (*flags & FOLL_NOWAIT)
 		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT;
 	if (*flags & FOLL_TRIED) {

@@ -1207,7 +1207,7 @@ int fixup_user_fault(struct task_struct
 	address = untagged_addr(address);
 
 	if (unlocked)
-		fault_flags |= FAULT_FLAG_ALLOW_RETRY;
+		fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
 
 retry:
 	vma = find_extend_vma(mm, address);

@@ -1329,7 +1329,13 @@ retry:
 			break;
 
 		*locked = 1;
-		down_read(&mm->mmap_sem);
+		ret = down_read_killable(&mm->mmap_sem);
+		if (ret) {
+			BUG_ON(ret > 0);
+			if (!pages_done)
+				pages_done = ret;
+			break;
+		}
 
 		ret = __get_user_pages(tsk, mm, start, 1, flags | FOLL_TRIED,
 				       pages, NULL, locked);

mm/hugetlb.c renamed from mm/hugetlb.c~mm-gup-allow-to-react-to-fatal-signals +2 -1

@@ -4344,7 +4344,8 @@ long follow_hugetlb_page(struct mm_struc
 			if (flags & FOLL_WRITE)
 				fault_flags |= FAULT_FLAG_WRITE;
 			if (locked)
-				fault_flags |= FAULT_FLAG_ALLOW_RETRY;
+				fault_flags |= FAULT_FLAG_ALLOW_RETRY |
+					FAULT_FLAG_KILLABLE;
 			if (flags & FOLL_NOWAIT)
 				fault_flags |= FAULT_FLAG_ALLOW_RETRY |
 					FAULT_FLAG_RETRY_NOWAIT;

Andrew Morton

author maintainer

2020-04-02 04:09 UTC #108 lore raw

PATCH 101/155 mm/userfaultfd: honor FAULT_FLAG_KILLABLE in fault path

From: Peter Xu <[email protected]>
Subject: mm/userfaultfd: honor FAULT_FLAG_KILLABLE in fault path

Userfaultfd fault path was by default killable even if the caller does not have FAULT_FLAG_KILLABLE. That makes sense before in that when with gup we don't have FAULT_FLAG_KILLABLE properly set before. Now after previous patch we've got FAULT_FLAG_KILLABLE applied even for gup code so it should also make sense to let userfaultfd to honor the FAULT_FLAG_KILLABLE.

Because we're unconditionally setting FAULT_FLAG_KILLABLE in gup code right now, this patch should have no functional change. It also cleaned the code a little bit by introducing some helpers.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Peter Xu <[email protected]>

Cc Andrea Arcangeli <[email protected]>

Cc Bobby Powers <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Denis Plotnikov <[email protected]>

Cc "Dr . David Alan Gilbert" <[email protected]>

Cc Hugh Dickins <[email protected]>

Cc Jerome Glisse <[email protected]>

Cc Johannes Weiner <[email protected]>

Cc "Kirill A . Shutemov" <[email protected]>

Cc Linus Torvalds <[email protected]>

Cc Martin Cracauer <[email protected]>

Cc Marty McFadden <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Maya Gokhale <[email protected]>

Cc Mel Gorman <[email protected]>

Cc Mike Kravetz <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Pavel Emelyanov <[email protected]>

---

 fs/userfaultfd.c |   36 ++++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

fs/userfaultfd.c renamed from fs/userfaultfd.c~mm-userfaultfd-honor-fault_flag_killable-in-fault-path +28 -8

@@ -334,6 +334,30 @@ out:
 	return ret;
 }
 
+/* Should pair with userfaultfd_signal_pending() */
+static inline long userfaultfd_get_blocking_state(unsigned int flags)
+{
+	if (flags & FAULT_FLAG_INTERRUPTIBLE)
+		return TASK_INTERRUPTIBLE;
+
+	if (flags & FAULT_FLAG_KILLABLE)
+		return TASK_KILLABLE;
+
+	return TASK_UNINTERRUPTIBLE;
+}
+
+/* Should pair with userfaultfd_get_blocking_state() */
+static inline bool userfaultfd_signal_pending(unsigned int flags)
+{
+	if (flags & FAULT_FLAG_INTERRUPTIBLE)
+		return signal_pending(current);
+
+	if (flags & FAULT_FLAG_KILLABLE)
+		return fatal_signal_pending(current);
+
+	return false;
+}
+
 /*
  * The locking rules involved in returning VM_FAULT_RETRY depending on
  * FAULT_FLAG_ALLOW_RETRY, FAULT_FLAG_RETRY_NOWAIT and

@@ -355,7 +379,7 @@ vm_fault_t handle_userfault(struct vm_fa
 	struct userfaultfd_ctx *ctx;
 	struct userfaultfd_wait_queue uwq;
 	vm_fault_t ret = VM_FAULT_SIGBUS;
-	bool must_wait, return_to_userland;
+	bool must_wait;
 	long blocking_state;
 
 	/*

@@ -462,9 +486,7 @@ vm_fault_t handle_userfault(struct vm_fa
 	uwq.ctx = ctx;
 	uwq.waken = false;
 
-	return_to_userland = vmf->flags & FAULT_FLAG_INTERRUPTIBLE;
-	blocking_state = return_to_userland ? TASK_INTERRUPTIBLE :
-			 TASK_KILLABLE;
+	blocking_state = userfaultfd_get_blocking_state(vmf->flags);
 
 	spin_lock_irq(&ctx->fault_pending_wqh.lock);
 	/*

@@ -490,8 +512,7 @@ vm_fault_t handle_userfault(struct vm_fa
 	up_read(&mm->mmap_sem);
 
 	if (likely(must_wait && !READ_ONCE(ctx->released) &&
-		   (return_to_userland ? !signal_pending(current) :
-		    !fatal_signal_pending(current)))) {
+		   !userfaultfd_signal_pending(vmf->flags))) {
 		wake_up_poll(&ctx->fd_wqh, EPOLLIN);
 		schedule();
 		ret |= VM_FAULT_MAJOR;

@@ -513,8 +534,7 @@ vm_fault_t handle_userfault(struct vm_fa
 			set_current_state(blocking_state);
 			if (READ_ONCE(uwq.waken) ||
 			    READ_ONCE(ctx->released) ||
-			    (return_to_userland ? signal_pending(current) :
-			     fatal_signal_pending(current)))
+			    userfaultfd_signal_pending(vmf->flags))
 				break;
 			schedule();
 		}

Andrew Morton

author maintainer

2020-04-02 04:09 UTC #109 lore raw

PATCH 102/155 mm: clarify a confusing comment for remap_pfn_range()

From: WANG Wenhu <[email protected]>
Subject: mm: clarify a confusing comment for remap_pfn_range()

It really made me scratch my head. Replace the comment with an accurate and consistent description.

The parameter pfn actually refers to the page frame number which is right-shifted by PAGE_SHIFT from the physical address.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by WANG Wenhu <[email protected]>

---

 mm/memory.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/memory.c renamed from mm/memory.c~mm-clarify-a-confusing-comment-of-remap_pfn_range +1 -1

@@ -1939,7 +1939,7 @@ static inline int remap_p4d_range(struct
  * remap_pfn_range - remap kernel memory to userspace
  * @vma: user vma to map to
  * @addr: target user address to start at
- * @pfn: physical address of kernel memory
+ * @pfn: page frame number of kernel physical memory address
  * @size: size of map area
  * @prot: page protection flags for this mapping
  *

Andrew Morton

author maintainer

2020-04-02 04:09 UTC #110 lore raw

PATCH 103/155 mm/memory.c: clarify a confusing comment for vm_iomap_memory

From: Wang Wenhu <[email protected]>
Subject: mm/memory.c: clarify a confusing comment for vm_iomap_memory

The param "start" actually referes to the physical memory start, which is to be mapped into virtual area vma. And it is the field vma->vm_start which stands for the start of the area.

Most of the time, we do not read through whole implementation of a function but only the definition and essential comments. Accurate comments are definitely the base stone.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Wang Wenhu <[email protected]>

---

 mm/memory.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

mm/memory.c renamed from mm/memory.c~mm-clarify-a-confusing-comment-for-vm_iomap_memory +1 -1

@@ -2009,7 +2009,7 @@ EXPORT_SYMBOL(remap_pfn_range);
 /**
  * vm_iomap_memory - remap memory to userspace
  * @vma: user vma to map to
- * @start: start of area
+ * @start: start of the physical memory to be mapped
  * @len: size of area
  *
  * This is a simplified io_remap_pfn_range() for common driver use. The

Andrew Morton

author maintainer

2020-04-02 04:09 UTC #111 lore raw

PATCH 104/155 mmap: remove inline of vm_unmapped_area

From: Jaewon Kim <[email protected]>
Subject: mmap: remove inline of vm_unmapped_area

Patch series "mm: mmap: add mmap trace point", v3.

Create mmap trace file and add trace point of vm_unmapped_area().

This patch (of 2):

In preparation for next patch remove inline of vm_unmapped_area and move code to mmap.c. There is no logical change.

Also remove unmapped_area[_topdown] out of mm.h, there is no code calling to them.

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Jaewon Kim <[email protected]>

Reviewed-by Vlastimil Babka <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Michel Lespinasse <[email protected]>

Cc Borislav Petkov <[email protected]>

---

 include/linux/mm.h |   21 +--------------------
 mm/mmap.c          |   20 ++++++++++++++++++--
 2 files changed, 19 insertions(+), 22 deletions(-)

include/linux/mm.h renamed from include/linux/mm.h~mmap-remove-inline-of-vm_unmapped_area +1 ~~-20~~

@@ -2530,26 +2530,7 @@ struct vm_unmapped_area_info {
 	unsigned long align_offset;
 };
 
-extern unsigned long unmapped_area(struct vm_unmapped_area_info *info);
-extern unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info);
-
-/*
- * Search for an unmapped address range.
- *
- * We are looking for a range that:
- * - does not intersect with any VMA;
- * - is contained within the [low_limit, high_limit) interval;
- * - is at least the desired size.
- * - satisfies (begin_addr & align_mask) == (align_offset & align_mask)
- */
-static inline unsigned long
-vm_unmapped_area(struct vm_unmapped_area_info *info)
-{
-	if (info->flags & VM_UNMAPPED_AREA_TOPDOWN)
-		return unmapped_area_topdown(info);
-	else
-		return unmapped_area(info);
-}
+extern unsigned long vm_unmapped_area(struct vm_unmapped_area_info *info);
 
 /* truncate.c */
 extern void truncate_inode_pages(struct address_space *, loff_t);

mm/mmap.c renamed from mm/mmap.c~mmap-remove-inline-of-vm_unmapped_area +18 -2

@@ -1848,7 +1848,7 @@ unacct_error:
 	return error;
 }
 
-unsigned long unmapped_area(struct vm_unmapped_area_info *info)
+static unsigned long unmapped_area(struct vm_unmapped_area_info *info)
 {
 	/*
 	 * We implement the search by looking for an rbtree node that

@@ -1951,7 +1951,7 @@ found:
 	return gap_start;
 }
 
-unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)
+static unsigned long unmapped_area_topdown(struct vm_unmapped_area_info *info)
 {
 	struct mm_struct *mm = current->mm;
 	struct vm_area_struct *vma;

@@ -2050,6 +2050,22 @@ found_highest:
 	return gap_end;
 }
 
+/*
+ * Search for an unmapped address range.
+ *
+ * We are looking for a range that:
+ * - does not intersect with any VMA;
+ * - is contained within the [low_limit, high_limit) interval;
+ * - is at least the desired size.
+ * - satisfies (begin_addr & align_mask) == (align_offset & align_mask)
+ */
+unsigned long vm_unmapped_area(struct vm_unmapped_area_info *info)
+{
+	if (info->flags & VM_UNMAPPED_AREA_TOPDOWN)
+		return unmapped_area_topdown(info);
+	else
+		return unmapped_area(info);
+}
 
 #ifndef arch_get_mmap_end
 #define arch_get_mmap_end(addr)	(TASK_SIZE)

Andrew Morton

author maintainer

2020-04-02 04:09 UTC #112 lore raw

PATCH 105/155 mm: mmap: add trace point of vm_unmapped_area

From: Jaewon Kim <[email protected]>
Subject: mm: mmap: add trace point of vm_unmapped_area

Even on 64 bit kernel, the mmap failure can happen for a 32 bit task. Virtual memory space shortage of a task on mmap is reported to userspace as -ENOMEM. It can be confused as physical memory shortage of overall system.

The vm_unmapped_area can be called to by some drivers or other kernel core system like filesystem. In my platform, GPU driver calls to vm_unmapped_area and the driver returns -ENOMEM even in GPU side shortage. It can be hard to distinguish which code layer returns the -ENOMEM.

Create mmap trace file and add trace point of vm_unmapped_area.

i.e.)
277.156599: vm_unmapped_area: addr=77e0d03000 err=0 total_vm=0x17014b flags=0x1 len=0x400000 lo=0x8000 hi=0x7878c27000 mask=0x0 ofs=0x1 342.838740: vm_unmapped_area: addr=0 err=-12 total_vm=0xffb08 flags=0x0 len=0x100000 lo=0x40000000 hi=0xfffff000 mask=0x0 ofs=0x22

[[email protected]: prefix address printk with 0x, per Matthew]

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Jaewon Kim <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Matthew Wilcox (Oracle) <[email protected]>

Cc Michel Lespinasse <[email protected]>

Cc Vlastimil Babka <[email protected]>