Crash in MM code in v4.4.y, v4.9.y with TRANSPARENT_HUGEPAGE enabled
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with CONFIG_TRANSPARENT_HUGEPAGE=y, CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.
[ 6.649970] random: crng init done [ 6.689002] BUG: unable to handle kernel paging request at ffffeafffa1a0020 [ 6.689082] IP: [<ffffffff8116ba10>] page_remove_rmap+0x10/0x230 [ 6.689082] PGD 0 [ 6.689082] [ 6.689082] Oops: 0000 [#1] SMP [ 6.689082] Modules linked in: [ 6.689082] CPU: 3 PID: 1132 Comm: mmtest Not tainted 4.9.121 #16 [ 6.689082] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/204 [ 6.689082] task: ffff88017a558c40 task.stack: ffffc900007a8000 [ 6.689082] RIP: 0010:[<ffffffff8116ba10>] [<ffffffff8116ba10>] page_remove_rmap+0x10/0x230 [ 6.689082] RSP: 0018:ffffc900007abc18 EFLAGS: 00000296 [ 6.689082] RAX: ffffea0005e58000 RBX: ffffeafffa1a0000 RCX: 0000000020200000 [ 6.689082] RDX: 00003fffffe00000 RSI: 0000000000000001 RDI: ffffeafffa1a0000 [ 6.689082] RBP: ffffc900007abc38 R08: 0000000000000000 R09: 0000000020800000 [ 6.689082] R10: ffffea0005e51ec0 R11: 0000000000000000 R12: ffffeafffa1a0000 [ 6.689082] R13: ffffc900007abdc0 R14: ffff880179426808 R15: ffffc900007abdc0 [ 6.689082] FS: 0000000000000000(0000) GS:ffff88017fd80000(0000) knlGS:0000000000000000 [ 6.689082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 6.689082] CR2: ffffeafffa1a0020 CR3: 000000017a3f8000 CR4: 0000000000340670 [ 6.689082] Stack: [ 6.689082] ffffffff81138517 ffffea0005e50980 ffffeafffa1a0000 ffffc900007abdc0 [ 6.689082] ffffc900007abc68 ffffffff8118d52c ffff880179426808 0000000020200000 [ 6.689082] ffffc900007abdc0 ffffc900007abdc0 ffffc900007abd40 ffffffff8115e270 [ 6.689082] Call Trace: [ 6.689082] [<ffffffff81138517>] ? __alloc_pages_nodemask+0xd7/0x210 [ 6.689082] [<ffffffff8118d52c>] zap_huge_pmd+0xec/0x2a0 [ 6.689082] [<ffffffff8115e270>] unmap_page_range+0x7d0/0x8d0 [ 6.689082] [<ffffffff8115e3c4>] unmap_single_vma+0x54/0xd0 [ 6.689082] [<ffffffff8115e6dc>] unmap_vmas+0x4c/0xa0 [ 6.689082] [<ffffffff81166297>] exit_mmap+0xa7/0x130 [ 6.689082] [<ffffffff81193d8f>] ? __khugepaged_exit+0x6f/0x100 [ 6.689082] [<ffffffff8105a078>] mmput+0x38/0x100 [ 6.689082] [<ffffffff8106080c>] do_exit+0x25c/0xb10 [ 6.689082] [<ffffffff81061f1e>] do_group_exit+0x3e/0xa0 [ 6.689082] [<ffffffff81061f8f>] SyS_exit_group+0xf/0x10 [ 6.689082] [<ffffffff81002a9c>] do_syscall_64+0x5c/0xc0 [ 6.689082] [<ffffffff819a62be>] entry_SYSCALL_64_after_swapgs+0x58/0xc6 [ 6.689082] Code: 77 ff ff ff eb b8 be 13 00 00 00 4c 89 e7 e8 d8 40 fe ff 48 63 d3 eb b3 0f 1f 00 55 48 89 e5 41 55 41 54 53 48 [ 6.689082] RIP [<ffffffff8116ba10>] page_remove_rmap+0x10/0x230 [ 6.689082] RSP <ffffc900007abc18> [ 6.689082] CR2: ffffeafffa1a0020 [ 6.689082] ---[ end trace 62ac9ace190510cd ]--- [ 6.689082] Fixing recursive fault but reboot is needed!
A test program to trigger the crash is attached, as are bisect results and some additional information.
Upstream commit 19f5c49bbc3 ("x86/speculation/l1tf: Exempt zeroed PTEs from inversion") does not fix the problem.
Many thanks to the syzcaller team for finding the problem and for providing a reproducer.
Any help to track down the problem would be appreciated. This is out of my league.
Thanks,
Guenter
---
#define _GNU_SOURCE
#include <endian.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
int main()
{
syscall(__NR_mmap, 0x20000000, 0x1000000, 3, 0x32, -1, 0); syscall(__NR_madvise, 0x20a93000, 0x4000, 0xe); syscall(__NR_mremap, 0x20a96000, 0x1000, 0x800000, 3, 0x20130000); syscall(__NR_sigaltstack, 0x20341000, 0x20ef9ff8); syscall(__NR_mprotect, 0x20000000, 0x800000, 0);
return 0;
}
---
# bad: [93e02ae4200184bab43ce29966e895826a756a37] Linux 4.9.120
# good: [8f21ecb4249a0914aea08bef1befca9019a3b44b] Linux 4.9.119
git bisect start 'v4.9.120' 'v4.9.119'
# bad: [a0695af3406ae2a08184bd47a9e948fe6f9858b9] x86/KVM: Warn user if KVM is loaded SMT and L1TF CPU bug being present git bisect bad a0695af3406ae2a08184bd47a9e948fe6f9858b9
# good: [1a4922e0f01d08a4789b1e17b195bc30bf234a3b] mm: x86: move _PAGE_SWP_SOFT_DIRTY from bit 7 to bit 1 git bisect good 1a4922e0f01d08a4789b1e17b195bc30bf234a3b
# bad: [e0439285c628dea71517a1e77cab805d9134f898] x86/cpu: Remove the pointless CPU printout
git bisect bad e0439285c628dea71517a1e77cab805d9134f898
# bad: [e3923475ebb1b503668dfdb3ba90e2ebd46931e6] x86/speculation/l1tf: Limit swap file size to MAX_PA/2 git bisect bad e3923475ebb1b503668dfdb3ba90e2ebd46931e6
# bad: [33182fe97add6e83c195e9d0f7297a6499563b52] x86/speculation/l1tf: Protect PROT_NONE PTEs against speculation git bisect bad 33182fe97add6e83c195e9d0f7297a6499563b52
# good: [60712274887fcd4ad5eb8e01796022b6b202143c] x86/speculation/l1tf: Protect swap entries against L1TF git bisect good 60712274887fcd4ad5eb8e01796022b6b202143c
# first bad commit: [33182fe97add6e83c195e9d0f7297a6499563b52] # x86/speculation/l1tf: Protect PROT_NONE PTEs against speculation
---
qemu command line:
qemu-system-x86_64 \ -kernel arch/x86/boot/bzImage -M q35 -cpu Broadwell-noTSX \ -no-reboot -m 4G -smp 4 \ -drive file=rootfs.ext2.l1tf,format=raw,if=ide \ -nographic -monitor none \ --append "root=/dev/sda rw console=ttyS0 console=tty"
CPU and machine don't really matter. I'll be happy to provide the root file system if needed.
---
Configuration file:
# CONFIG_LOCALVERSION_AUTO is not setCONFIG_SYSVIPC=yCONFIG_POSIX_MQUEUE=yCONFIG_AUDIT=yCONFIG_NO_HZ=yCONFIG_HIGH_RES_TIMERS=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_TASKSTATS=yCONFIG_TASK_DELAY_ACCT=yCONFIG_TASK_XACCT=yCONFIG_TASK_IO_ACCOUNTING=yCONFIG_LOG_BUF_SHIFT=18CONFIG_CGROUPS=yCONFIG_CGROUP_SCHED=yCONFIG_CGROUP_FREEZER=yCONFIG_CPUSETS=yCONFIG_CGROUP_CPUACCT=yCONFIG_BLK_DEV_INITRD=y
# CONFIG_COMPAT_BRK is not setCONFIG_PROFILING=yCONFIG_KPROBES=yCONFIG_JUMP_LABEL=yCONFIG_MODULES=yCONFIG_MODULE_UNLOAD=yCONFIG_MODULE_FORCE_UNLOAD=yCONFIG_PARTITION_ADVANCED=yCONFIG_OSF_PARTITION=yCONFIG_AMIGA_PARTITION=yCONFIG_MAC_PARTITION=yCONFIG_BSD_DISKLABEL=yCONFIG_MINIX_SUBPARTITION=yCONFIG_SOLARIS_X86_PARTITION=yCONFIG_UNIXWARE_DISKLABEL=yCONFIG_SGI_PARTITION=yCONFIG_SUN_PARTITION=yCONFIG_KARMA_PARTITION=yCONFIG_SMP=yCONFIG_CALGARY_IOMMU=yCONFIG_SCHED_SMT=yCONFIG_PREEMPT_VOLUNTARY=yCONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=yCONFIG_MICROCODE_AMD=yCONFIG_X86_MSR=yCONFIG_X86_CPUID=yCONFIG_NUMA=yCONFIG_TRANSPARENT_HUGEPAGE=yCONFIG_TRANSPARENT_HUGEPAGE_MADVISE=yCONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_MTRR_SANITIZER is not setCONFIG_EFI=yCONFIG_HZ_1000=yCONFIG_KEXEC=yCONFIG_CRASH_DUMP=yCONFIG_HIBERNATION=yCONFIG_PM_DEBUG=yCONFIG_PM_TRACE_RTC=yCONFIG_ACPI_DOCK=yCONFIG_CPU_FREQ=yCONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=yCONFIG_CPU_FREQ_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_ONDEMAND=yCONFIG_X86_ACPI_CPUFREQ=yCONFIG_PCI_MMCONFIG=yCONFIG_PCIEPORTBUS=yCONFIG_HOTPLUG_PCI=yCONFIG_PCCARD=yCONFIG_YENTA=yCONFIG_BINFMT_MISC=yCONFIG_IA32_EMULATION=yCONFIG_NET=yCONFIG_PACKET=yCONFIG_UNIX=yCONFIG_XFRM_USER=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_MULTIPLE_TABLES=yCONFIG_IP_ROUTE_MULTIPATH=yCONFIG_IP_ROUTE_VERBOSE=yCONFIG_IP_PNP=yCONFIG_IP_PNP_DHCP=yCONFIG_IP_PNP_BOOTP=yCONFIG_IP_PNP_RARP=yCONFIG_IP_MROUTE=yCONFIG_IP_PIMSM_V1=yCONFIG_IP_PIMSM_V2=yCONFIG_SYN_COOKIES=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_DIAG is not setCONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not setCONFIG_TCP_MD5SIG=yCONFIG_INET6_AH=yCONFIG_INET6_ESP=yCONFIG_NETLABEL=yCONFIG_NETFILTER=y
# CONFIG_NETFILTER_ADVANCED is not setCONFIG_NF_CONNTRACK=yCONFIG_NF_CONNTRACK_FTP=yCONFIG_NF_CONNTRACK_IRC=yCONFIG_NF_CONNTRACK_SIP=yCONFIG_NF_CT_NETLINK=yCONFIG_NETFILTER_XT_TARGET_CONNSECMARK=yCONFIG_NETFILTER_XT_TARGET_NFLOG=yCONFIG_NETFILTER_XT_TARGET_SECMARK=yCONFIG_NETFILTER_XT_TARGET_TCPMSS=yCONFIG_NETFILTER_XT_MATCH_CONNTRACK=yCONFIG_NETFILTER_XT_MATCH_POLICY=yCONFIG_NETFILTER_XT_MATCH_STATE=yCONFIG_NF_CONNTRACK_IPV4=yCONFIG_IP_NF_IPTABLES=yCONFIG_IP_NF_FILTER=yCONFIG_IP_NF_TARGET_REJECT=yCONFIG_IP_NF_MANGLE=yCONFIG_NF_CONNTRACK_IPV6=yCONFIG_IP6_NF_IPTABLES=yCONFIG_IP6_NF_MATCH_IPV6HEADER=yCONFIG_IP6_NF_FILTER=yCONFIG_IP6_NF_TARGET_REJECT=yCONFIG_IP6_NF_MANGLE=yCONFIG_NET_SCHED=yCONFIG_NET_EMATCH=yCONFIG_NET_CLS_ACT=yCONFIG_HAMRADIO=yCONFIG_CFG80211=yCONFIG_MAC80211=yCONFIG_MAC80211_LEDS=yCONFIG_RFKILL=yCONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"CONFIG_DEVTMPFS=yCONFIG_DEVTMPFS_MOUNT=yCONFIG_DEBUG_DEVRES=yCONFIG_CONNECTOR=yCONFIG_BLK_DEV_LOOP=yCONFIG_VIRTIO_BLK=yCONFIG_BLK_DEV_NVME=yCONFIG_BLK_DEV_SD=yCONFIG_BLK_DEV_SR=yCONFIG_BLK_DEV_SR_VENDOR=yCONFIG_CHR_DEV_SG=yCONFIG_SCSI_CONSTANTS=yCONFIG_MEGARAID_SAS=yCONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_DC395x=yCONFIG_SCSI_AM53C974=yCONFIG_SCSI_VIRTIO=yCONFIG_ATA=yCONFIG_SATA_AHCI=yCONFIG_ATA_PIIX=yCONFIG_PATA_AMD=yCONFIG_PATA_OLDPIIX=yCONFIG_PATA_SCH=yCONFIG_MD=yCONFIG_BLK_DEV_MD=yCONFIG_BLK_DEV_DM=yCONFIG_DM_MIRROR=yCONFIG_DM_ZERO=yCONFIG_FUSION=yCONFIG_FUSION_SAS=yCONFIG_MACINTOSH_DRIVERS=yCONFIG_MAC_EMUMOUSEBTN=yCONFIG_NETDEVICES=yCONFIG_NETCONSOLE=yCONFIG_TIGON3=yCONFIG_NET_TULIP=yCONFIG_E100=yCONFIG_E1000=yCONFIG_E1000E=yCONFIG_SKY2=yCONFIG_FORCEDETH=yCONFIG_8139TOO=yCONFIG_FDDI=yCONFIG_INPUT_POLLDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not setCONFIG_INPUT_EVDEV=yCONFIG_INPUT_JOYSTICK=yCONFIG_INPUT_TABLET=yCONFIG_INPUT_TOUCHSCREEN=yCONFIG_INPUT_MISC=y
# CONFIG_LEGACY_PTYS is not setCONFIG_SERIAL_NONSTANDARD=yCONFIG_SERIAL_8250=yCONFIG_SERIAL_8250_CONSOLE=yCONFIG_SERIAL_8250_NR_UARTS=32CONFIG_SERIAL_8250_EXTENDED=yCONFIG_SERIAL_8250_MANY_PORTS=yCONFIG_SERIAL_8250_SHARE_IRQ=yCONFIG_SERIAL_8250_DETECT_IRQ=yCONFIG_SERIAL_8250_RSA=yCONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_INTEL is not set
# CONFIG_HW_RANDOM_AMD is not setCONFIG_NVRAM=yCONFIG_HPET=y
# CONFIG_HPET_MMAP is not setCONFIG_I2C_I801=yCONFIG_WATCHDOG=yCONFIG_AGP=yCONFIG_AGP_AMD64=yCONFIG_AGP_INTEL=yCONFIG_DRM=yCONFIG_DRM_I915=yCONFIG_FB_MODE_HELPERS=yCONFIG_FB_TILEBLITTING=yCONFIG_FB_EFI=y
# CONFIG_LCD_CLASS_DEVICE is not setCONFIG_VGACON_SOFT_SCROLLBACK=yCONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not setCONFIG_SOUND=yCONFIG_SND=yCONFIG_SND_SEQUENCER=yCONFIG_SND_SEQ_DUMMY=yCONFIG_SND_MIXER_OSS=yCONFIG_SND_PCM_OSS=yCONFIG_SND_SEQUENCER_OSS=yCONFIG_SND_HRTIMER=yCONFIG_SND_HDA_INTEL=yCONFIG_SND_HDA_HWDEP=yCONFIG_HIDRAW=yCONFIG_HID_GYRATION=yCONFIG_LOGITECH_FF=yCONFIG_HID_NTRIG=yCONFIG_HID_PANTHERLORD=yCONFIG_PANTHERLORD_FF=yCONFIG_HID_PETALYNX=yCONFIG_HID_SAMSUNG=yCONFIG_HID_SONY=yCONFIG_HID_SUNPLUS=yCONFIG_HID_TOPSEED=yCONFIG_HID_PID=yCONFIG_USB_HIDDEV=yCONFIG_USB=yCONFIG_USB_ANNOUNCE_NEW_DEVICES=yCONFIG_USB_MON=yCONFIG_USB_XHCI_HCD=yCONFIG_USB_EHCI_HCD=yCONFIG_USB_OHCI_HCD=yCONFIG_USB_UHCI_HCD=yCONFIG_USB_PRINTER=yCONFIG_USB_STORAGE=yCONFIG_USB_UAS=yCONFIG_MMC=yCONFIG_MMC_SDHCI=yCONFIG_MMC_SDHCI_PCI=yCONFIG_EDAC=yCONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not setCONFIG_DMADEVICES=yCONFIG_VIRTIO_PCI=yCONFIG_VIRTIO_BALLOON=yCONFIG_VIRTIO_MMIO=yCONFIG_EEEPC_LAPTOP=yCONFIG_AMD_IOMMU=yCONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not setCONFIG_EFI_VARS=yCONFIG_EXT4_FS=yCONFIG_EXT4_FS_POSIX_ACL=yCONFIG_EXT4_FS_SECURITY=yCONFIG_QUOTA=yCONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not setCONFIG_QFMT_V2=yCONFIG_AUTOFS4_FS=yCONFIG_ISO9660_FS=yCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_MSDOS_FS=yCONFIG_VFAT_FS=yCONFIG_PROC_KCORE=yCONFIG_TMPFS_POSIX_ACL=yCONFIG_HUGETLBFS=yCONFIG_NFS_FS=yCONFIG_NFS_V3_ACL=yCONFIG_NFS_V4=yCONFIG_ROOT_NFS=yCONFIG_NLS_DEFAULT="utf8"CONFIG_NLS_CODEPAGE_437=yCONFIG_NLS_ASCII=yCONFIG_NLS_ISO8859_1=yCONFIG_NLS_UTF8=yCONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_UNUSED_SYMBOLS is not setCONFIG_MAGIC_SYSRQ=yCONFIG_DEBUG_KERNEL=yCONFIG_DEBUG_STACK_USAGE=yCONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_SCHED_DEBUG is not setCONFIG_SCHEDSTATS=yCONFIG_TIMER_STATS=yCONFIG_BLK_DEV_IO_TRACE=yCONFIG_PROVIDE_OHCI1394_DMA_INIT=yCONFIG_EARLY_PRINTK_DBGP=y
# CONFIG_DEBUG_RODATA_TEST is not setCONFIG_DEBUG_BOOT_PARAMS=yCONFIG_OPTIMIZE_INLINING=yCONFIG_SECURITY=yCONFIG_SECURITY_NETWORK=yCONFIG_SECURITY_SELINUX=yCONFIG_SECURITY_SELINUX_BOOTPARAM=yCONFIG_SECURITY_SELINUX_DISABLE=y
Re: Crash in MM code in v4.4.y, v4.9.y with TRANSPARENT_HUGEPAGE enabledg
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.
Just to confirm, it's not in mainline right? So only in backports?
There's also a report for 4.17 in
https://bugzilla.redhat.com/show_bug.cgi?id=1618792
(was trying to reproduce that, but not successfull so far)
-Andi
Re: Crash in MM code in v4.4.y, v4.9.y with TRANSPARENT_HUGEPAGE enabledg
Quoted 8 lines · Andi Kleen
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.Just to confirm, it's not in mainline right? So only in backports?
I have only seen it in v.4.4.y and v4.9.y. v4.14.y does not seem to be affected.
Quoted 4 lines
There's also a report for 4.17 in
This is a different traceback, so it is most likely a different problem.
I did not test v4.17.y. I'll do that (and v4.18.y, and mainline) for completeness tonight.
Guenter
Quoted 3 lines · Guenter Roeck
[ 6.649970] random: crng init done [ 6.689002] BUG: unable to handle kernel paging request at ffffeafffa1a0020
Hmm. Lots of bits set.
Quoted 4 lines
[ 6.689082] RIP: 0010:[<ffffffff8116ba10>] [<ffffffff8116ba10>] page_remove_rmap+0x10/0x230 [ 6.689082] RSP: 0018:ffffc900007abc18 EFLAGS: 00000296 [ 6.689082] RAX: ffffea0005e58000 RBX: ffffeafffa1a0000 RCX: 0000000020200000 [ 6.689082] RDX: 00003fffffe00000 RSI: 0000000000000001 RDI: ffffeafffa1a0000
Is that RDX value the same value as PHYSICAL_PMD_PAGE_MASK?
If I did my math right, it would be, if your CPU has 46 bits of physical memory. Might that be the case?
The reason I mention that is because we had the bug with spurious inversion of the zero pte/pmd, fixed by
f19f5c49bbc3 ("x86/speculation/l1tf: Exempt zeroed PTEs from inversion")
and that would make a zeroed pmd entry be inverted by PHYSICAL_PMD_PAGE_MASK, and then you get odd garbage page pointers etc.
Maybe. I could have gotten the math wrong too, but it sounds like the register contents _potentially_ might match up with something like this, and then we'd zap a bogus hugepage because of some confusion.
Although then I'd have expected the bisection to hit "x86/speculation/l1tf: Invert all not present mappings" instead of the one you hit, so I don't know.
Plus I'd have expected the problem to have been in mainline too, and apparently it's just the 4.4 and 4.9 backports.
Your test-case does have mprotect with PROT_NONE. Which together with that mask that *might* be PHYSICAL_PMD_PAGE_MASK makes me think it might be related.
Linus
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Guenter Roeck
[ 6.649970] random: crng init done [ 6.689002] BUG: unable to handle kernel paging request at ffffeafffa1a0020Hmm. Lots of bits set.
Quoted 4 lines
[ 6.689082] RIP: 0010:[<ffffffff8116ba10>] [<ffffffff8116ba10>] page_remove_rmap+0x10/0x230 [ 6.689082] RSP: 0018:ffffc900007abc18 EFLAGS: 00000296 [ 6.689082] RAX: ffffea0005e58000 RBX: ffffeafffa1a0000 RCX: 0000000020200000 [ 6.689082] RDX: 00003fffffe00000 RSI: 0000000000000001 RDI: ffffeafffa1a0000
Quoted 5 lines
Is that RDX value the same value as
PHYSICAL_PMD_PAGE_MASK?If I did my math right, it would be, if your CPU has 46 bits of physical memory. Might that be the case?
Yes.
Quoted 5 lines
The reason I mention that is because we had the bug with spurious inversion of the zero pte/pmd, fixed by
f19f5c49bbc3("x86/speculation/l1tf: Exempt zeroed PTEs from inversion")
I applied that patch, but it didn't help. I get exactly the same crash and traceback.
Quoted 8 lines
and that would make a zeroed pmd entry be inverted by
PHYSICAL_PMD_PAGE_MASK, and then you get odd garbage page pointers etc.Maybe. I could have gotten the math wrong too, but it sounds like the register contents _potentially_ might match up with something like this, and then we'd zap a bogus hugepage because of some confusion.
Quoted 7 lines
Although then I'd have expected the bisection to hit "x86/speculation/l1tf: Invert all not present mappings" instead of the one you hit, so I don't know.
Plus I'd have expected the problem to have been in mainline too, and apparently it's just the 4.4 and 4.9 backports.
Personally I suspect that something went wrong or is missing in the backport from 4.14 to 4.9. 5-level paging was introduced in between, and thp support was extended to support additional architectures. With all those changes, it is easy to miss something. Only I have no idea what that might be.
Guenter
Quoted 2 lines
Plus I'd have expected the problem to have been in mainline too, and apparently it's just the 4.4 and 4.9 backports.
There's another problem in 4.17, but not 4.18, see
https://bugzilla.redhat.com/show_bug.cgi?id=1618792
Could be the same or different.
-Andi
Quoted 6 lines
Your test-case does have mprotect with
PROT_NONE. Which together with that mask that *might* bePHYSICAL_PMD_PAGE_MASKmakes me think it might be related.Linus
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.
Could you try to apply fd7e315988b7 ("x86/mm: Simplify p[g4um]d_page() macros"). I do not see it in stable 4.4 tree and it has been introduced much later in 4.14. This one gave us quite some headache because it is soooo easy to overlook.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Michal Hocko
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.Could you try to apply
fd7e315988b7("x86/mm: Simplify p[g4um]d_page() macros"). I do not see it in stable 4.4 tree and it has been introduced much later in 4.14. This one gave us quite some headache because it is soooo easy to overlook.
Good catch!
I tested that with 4.9 and backporting the patch indeed fixes the syzcaller test case running in a KVM VM. Backported patch appended.
Should probably go into 4.4 and 4.9.
Cannot explain the 4.17 report unfortunately.
I'll resend it as an email too
---
x86/mm: Simplify p[g4um]d_page() macros
Create a pgd_pfn() macro similar to the p[4um]d_pfn() macros and then use the p[g4um]d_pfn() macros in the p[g4um]d_page() macros instead of duplicating the code.
[Needed to fix crashes caused by earlier backports in 4.9 stable, likely 4.4 too]
arch/x86/include/asm/pgtable.h +8 -5
@@ -173,6 +173,11 @@ static inline unsigned long pud_pfn(pud_t pud)return (pfn &pud_pfn_mask (pud)) >>PAGE_SHIFT ; } +static inline unsigned long pgd_pfn (pgd_t pgd) +{ +return (pgd_val (pgd) &PTE_PFN_MASK ) >>PAGE_SHIFT ; +} +#define pte_page (pte) pfn_to_page(pte_pfn(pte))static inline int pmd_large (pmd_t pte)
@@ -578,8 +583,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)* Currently stuck as a macro due to indirect forward reference to * linux/mmzone.h's __section_mem_map_addr() definition: */ -#define pmd_page (pmd) \ - pfn_to_page((pmd_val(pmd) & pmd_pfn_mask(pmd)) >> PAGE_SHIFT) +#define pmd_page (pmd) pfn_to_page(pmd_pfn(pmd))/* * the pmd page can be thought of an array like this: pmd_t[PTRS_PER_PMD]
@@ -647,8 +651,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud)* Currently stuck as a macro due to indirect forward reference to * linux/mmzone.h's __section_mem_map_addr() definition: */ -#define pud_page (pud) \ - pfn_to_page((pud_val(pud) & pud_pfn_mask(pud)) >> PAGE_SHIFT) +#define pud_page (pud) pfn_to_page(pud_pfn(pud))/* Find an entry in the second-level page table.. */ static inline pmd_t *pmd_offset (pud_t *pud,unsigned long address)
@@ -688,7 +691,7 @@ static inline unsigned long pgd_page_vaddr(pgd_t pgd)* Currently stuck as a macro due to indirect forward reference to * linux/mmzone.h's __section_mem_map_addr() definition: */ -#define pgd_page (pgd) pfn_to_page(pgd_val(pgd) >> PAGE_SHIFT) +#define pgd_page (pgd) pfn_to_page(pgd_pfn(pgd))/* to find an entry in a page-table-directory. */ static inline unsigned long pud_index (unsigned long address)
Quoted 12 lines · Andi Kleen
Quoted 10 lines · Michal Hocko
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.Could you try to apply
fd7e315988b7("x86/mm: Simplify p[g4um]d_page() macros"). I do not see it in stable 4.4 tree and it has been introduced much later in 4.14. This one gave us quite some headache because it is soooo easy to overlook.
Quoted 8 lines
Good catch!
I tested that with 4.9 and backporting the patch indeed fixes the syzcaller test case running in a KVM VM. Backported patch appended.
Should probably go into 4.4 and 4.9.
Cannot explain the 4.17 report unfortunately.
I haven't seen that one yet and likely won't get to it tomorrow as well but I would start looking for a direct pte_val usage. We have had som out of tree xen code which was doing exactly this. Not really easy to find by a code inspection.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 12 lines · Andi Kleen
Quoted 10 lines · Michal Hocko
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.Could you try to apply
fd7e315988b7("x86/mm: Simplify p[g4um]d_page() macros"). I do not see it in stable 4.4 tree and it has been introduced much later in 4.14. This one gave us quite some headache because it is soooo easy to overlook.
Quoted 4 lines
Good catch!
I tested that with 4.9 and backporting the patch indeed fixes the syzcaller test case running in a KVM VM. Backported patch appended.
Tested on 4.4 too and it fixes the syzkaller test case there too.
-Andi
Quoted 13 lines · Andi Kleen
Quoted 12 lines · Andi Kleen
Quoted 10 lines · Michal Hocko
Quoted 4 lines · Guenter Roeck
Hi,
the following crash is seen in v4.4.148, v4.4.149, v4.9.120, and v4.9.121 with
CONFIG_TRANSPARENT_HUGEPAGE=y,CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y.Could you try to apply
fd7e315988b7("x86/mm: Simplify p[g4um]d_page() macros"). I do not see it in stable 4.4 tree and it has been introduced much later in 4.14. This one gave us quite some headache because it is soooo easy to overlook.
Quoted 7 lines
Quoted 4 lines
Good catch!
I tested that with 4.9 and backporting the patch indeed fixes the syzcaller test case running in a KVM VM. Backported patch appended.
Tested on 4.4 too and it fixes the syzkaller test case there too.
Confirmed that the problem is fixed in v4.4.151-rc1 and v4.9.123-rc1.
Thanks a lot for tracking this down and backporting the fix!
Guenter