Skip to contentSkip to conversation

x86/mm changes for v4.14: PCID support, 5-level paging support, Secure Memory Encryption support

GIT PULL Ingo Molnar 9 participants 19 messages Sep 4, 2017 – Sep 15, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest x86-mm-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-mm-for-linus

# HEAD: 9e52fc2b50de3a1c08b44f94c610fbe998c0031a x86/mm: Enable RCU based page table freeing (CONFIG_HAVE_RCU_TABLE_FREE=y)

[ NOTE: this tree depends on you having merged x86-boot-for-linus successfully.
If that tree could not be merged for whatever reason then please disregard this pull request. ]

The main changes in this cycle are support for three new, complex hardware features of x86 CPUs:

- Add 5-level paging support, which is a new hardware feature on upcoming Intel CPUs allowing up to 128 PB of virtual address space and 4 PB of physical RAM space - a 256-fold increase over the old limits. (Supercomputers of the future forecasting hurricanes on an ever warming planet can certainly make good use of more RAM.)

Many of the necessary changes went upstream in previous cycles, v4.14 is the first kernel that can enable 5-level paging.

This feature is activated via CONFIG_X86_5LEVEL=y - disabled by default.

(By Kirill A. Shutemov)

- Add 'encrypted memory' support, which is a new hardware feature on upcoming AMD CPUs ('Secure Memory Encryption', SME) allowing system RAM to be encrypted and decrypted (mostly) transparently by the CPU, with a little help from the kernel to transition to/from encrypted RAM. Such RAM should be more secure against various attacks like RAM access via the memory bus and should make the radio signature of memory bus traffic harder to intercept (and decrypt) as well.

This feature is activated via CONFIG_AMD_MEM_ENCRYPT=y - disabled by default.

(By Tom Lendacky)

- Enable PCID optimized TLB flushing on newer Intel CPUs: PCID is a hardware feature that attaches an address space tag to TLB entries and thus allows to skip TLB flushing in many cases, even if we switch mm's.

(By Andy Lutomirski)

All three of these features were in the works for a long time, and it's coincidence of the three independent development paths that they are all enabled in v4.14 at once.

out-of-topic modifications in x86-mm-for-linus:
-------------------------------------------------

  arch/ia64/include/asm/acpi.h       # 43858b4f25cf: x86/mm: Stop calling leave_m
  arch/ia64/kernel/efi.c             # f99afd08a45f: efi: Update efi_mem_type() t
  drivers/acpi/processor_idle.c      # 43858b4f25cf: x86/mm: Stop calling leave_m
  drivers/firmware/dmi-sysfs.c       # f7750a795687: x86, mpparse, x86/acpi, x86/
  drivers/firmware/efi/efi.c         # a19d66c56af1: efi: Add an EFI table addres
  drivers/firmware/pcdp.c            # f7750a795687: x86, mpparse, x86/acpi, x86/
  drivers/gpu/drm/drm_gem.c          # 95cf9264d5f3: x86, drm, fbdev: Do not spec
  drivers/gpu/drm/drm_vm.c           # 95cf9264d5f3: x86, drm, fbdev: Do not spec
  drivers/gpu/drm/ttm/ttm_bo_vm.c    # 95cf9264d5f3: x86, drm, fbdev: Do not spec
  drivers/gpu/drm/udl/udl_fb.c       # 95cf9264d5f3: x86, drm, fbdev: Do not spec
  drivers/idle/intel_idle.c          # 43858b4f25cf: x86/mm: Stop calling leave_m
  drivers/iommu/amd_iommu.c          # 2543a786aa25: iommu/amd: Allow the AMD IOM
  drivers/iommu/amd_iommu_init.c     # 2543a786aa25: iommu/amd: Allow the AMD IOM
  drivers/iommu/amd_iommu_proto.h    # 2543a786aa25: iommu/amd: Allow the AMD IOM
  drivers/iommu/amd_iommu_types.h    # 2543a786aa25: iommu/amd: Allow the AMD IOM
  drivers/sfi/sfi_core.c             # 693bf0aa01b7: x86/boot: Fix memremap() rel
                                   # f7750a795687: x86, mpparse, x86/acpi, x86/
  drivers/video/fbdev/core/fbmem.c   # 95cf9264d5f3: x86, drm, fbdev: Do not spec
  include/asm-generic/early_ioremap.h# f88a68facd9a: x86/mm: Extend early_memrema
  include/asm-generic/pgtable.h      # 21729f81ce8a: x86/mm: Provide general kern
  include/linux/compiler.h           # 7375ae3a0b79: compiler-gcc.h: Introduce __
  include/linux/dma-mapping.h        # 648babb7078c: swiotlb: Add warnings for us
  include/linux/io.h                 # 8f716c9b5feb: x86/mm: Add support to acces
  include/linux/kexec.h              # bba4ed011a52: x86/mm, kexec: Allow kexec t
  include/linux/mem_encrypt.h        # 21729f81ce8a: x86/mm: Provide general kern
                                   # 5868f3651fa0: x86/mm: Add support to enabl
                                   # 7744ccdbc16f: x86/mm: Add Secure Memory En
  include/linux/mm_inline.h          # ce0fa3e56ad2: x86/mm, mm/hwpoison: Clear P
  include/linux/swiotlb.h            # c7753208a94c: x86, swiotlb: Add memory enc
  kernel/kexec_core.c                # bba4ed011a52: x86/mm, kexec: Allow kexec t
  kernel/memremap.c                  # 8f716c9b5feb: x86/mm: Add support to acces
  lib/swiotlb.c                      # 648babb7078c: swiotlb: Add warnings for us
                                   # c7753208a94c: x86, swiotlb: Add memory enc
  mm/early_ioremap.c                 # 8f716c9b5feb: x86/mm: Add support to acces
                                   # f88a68facd9a: x86/mm: Extend early_memrema

mm/memory-failure.c # ce0fa3e56ad2: x86/mm, mm/hwpoison: Clear P

Thanks,

Ingo

------------------>
Andrey Ryabinin (1):
x86/mm/dump_pagetables: Speed up page tables dump for CONFIG_KASAN=y

Andy Lutomirski (8):
x86/mm: Give each mm TLB flush generation a unique ID
x86/mm: Track the TLB's tlb_gen and update the flushing algorithm x86/mm: Rework lazy TLB mode and TLB freshness tracking
x86/mm: Stop calling leave_mm() in idle code
x86/mm: Disable PCID on 32-bit kernels
x86/mm: Add the 'nopcid' boot option to turn off PCID
x86/mm: Enable CR4.PCIDE on supported systems
x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID

Baoquan He (3):
x86/boot/KASLR: Wrap e820 entries walking code into new function process_e820_entries() x86/boot/KASLR: Switch to pass struct mem_vector to process_e820_entry() x86/boot/KASLR: Rename process_e820_entry() into process_mem_region()

Borislav Petkov (2):
x86/CPU: Align CR3 defines
x86/mm: Fix SME encryption stack ptr handling

Brijesh Singh (1):
kvm/x86: Avoid clearing the C-bit in rsvd_bits()

Ingo Molnar (1):
x86/boot: Fix memremap() related build failure

Jan Beulich (1):
x86/mm: Use pr_cont() in dump_pagetable()

Kirill A. Shutemov (8):
x86/mm/dump_pagetables: Generalize address normalization x86/mm/dump_pagetables: Fix printout of p4d level x86/xen: Redefine XEN_ELFNOTE_INIT_P2M using PUD_SIZE * PTRS_PER_PUD x86/mm: Rename tasksize_32bit/64bit to task_size_32bit/64bit() x86/mpx: Do not allow MPX if we have mappings above 47-bit x86/mm: Prepare to expose larger address space to userspace x86/mm: Allow userspace have mappings above 47-bit
x86: Enable 5-level paging support via CONFIG_X86_5LEVEL=y

Tom Lendacky (40):
x86/cpu/AMD: Document AMD Secure Memory Encryption (SME) x86/mm/pat: Set write-protect cache mode for full PAT support x86, mpparse, x86/acpi, x86/PCI, x86/dmi, SFI: Use memremap() for RAM mappings x86/cpu/AMD: Add the Secure Memory Encryption CPU feature x86/cpu/AMD: Handle SME reduction in physical address size x86/mm: Add Secure Memory Encryption (SME) support
x86/mm: Remove phys_to_virt() usage in ioremap()
x86/mm: Add support to enable SME in early boot processing x86/mm: Simplify p[g4um]d_page() macros
x86/mm: Provide general kernel support for memory encryption x86/mm: Add SME support for read_cr3_pa()
x86/mm: Extend early_memremap() support with additional attrs x86/mm: Add support for early encryption/decryption of memory x86/mm: Insure that boot memory areas are mapped properly x86/boot/e820: Add support to determine the E820 type of an address
efi: Add an EFI table address match function
efi: Update efi_mem_type() to return an error rather than 0 x86/efi: Update EFI pagetable creation to work with SME
x86/mm: Add support to access boot related data in the clear x86/boot: Use memremap() to map the MPF and MPC data
x86/mm: Add support to access persistent memory in the clear x86/mm: Add support for changing the memory encryption attribute x86/realmode: Decrypt trampoline area if memory encryption is active x86, swiotlb: Add memory encryption support
swiotlb: Add warnings for use of bounce buffers with SME x86/cpu/AMD: Make the microcode level available earlier in the boot iommu/amd: Allow the AMD IOMMU to work with memory encryption x86/boot/realmode: Check for memory encryption on the APs
x86, drm, fbdev: Do not specify encrypted memory for video mappings kvm/x86/svm: Support Secure Memory Encryption within KVM
x86/mm, kexec: Allow kexec to be used with SME
xen/x86: Remove SME feature in PV guests
x86/mm: Use proper encryption attributes with /dev/mem
x86/mm: Create native_make_p4d() for PGTABLE_LEVELS <= 4
x86/mm: Add support to encrypt the kernel in-place
x86/boot: Add early cmdline parsing for options with arguments compiler-gcc.h: Introduce __nostackprotector function attribute x86/mm: Add support to make use of Secure Memory Encryption x86/mm, kexec: Fix memory corruption with SME on successive kexecs acpi, x86/mm: Remove encryption mask from ACPI page protection type

Tony Luck (1):
x86/mm, mm/hwpoison: Clear PRESENT bit for kernel 1:1 mappings of poison pages

Vitaly Kuznetsov (1):
x86/mm: Enable RCU based page table freeing (CONFIG_HAVE_RCU_TABLE_FREE=y)

Wang Kai (1):
x86/mm/pkeys: Fix typo in Documentation/x86/protection-keys.txt

 Documentation/admin-guide/kernel-parameters.txt |  13 +
 Documentation/x86/amd-memory-encryption.txt     |  68 +++
 Documentation/x86/protection-keys.txt           |   6 +-
 Documentation/x86/x86_64/5level-paging.txt      |  64 +++
 arch/ia64/include/asm/acpi.h                    |   2 -
 arch/ia64/kernel/efi.c                          |   4 +-
 arch/x86/Kconfig                                |  49 ++
 arch/x86/boot/compressed/kaslr.c                |  63 +--
 arch/x86/boot/compressed/pagetable.c            |   7 +
 arch/x86/include/asm/acpi.h                     |  13 +-
 arch/x86/include/asm/cmdline.h                  |   2 +
 arch/x86/include/asm/cpufeatures.h              |   1 +
 arch/x86/include/asm/disabled-features.h        |   4 +-
 arch/x86/include/asm/dma-mapping.h              |   5 +-
 arch/x86/include/asm/dmi.h                      |   8 +-
 arch/x86/include/asm/e820/api.h                 |   2 +
 arch/x86/include/asm/elf.h                      |   4 +-
 arch/x86/include/asm/fixmap.h                   |  20 +
 arch/x86/include/asm/init.h                     |   1 +
 arch/x86/include/asm/io.h                       |   8 +
 arch/x86/include/asm/kexec.h                    |  11 +-
 arch/x86/include/asm/kvm_host.h                 |   2 +-
 arch/x86/include/asm/mem_encrypt.h              |  80 ++++
 arch/x86/include/asm/mmu.h                      |  25 +-
 arch/x86/include/asm/mmu_context.h              |  15 +-
 arch/x86/include/asm/mpx.h                      |   9 +
 arch/x86/include/asm/msr-index.h                |   2 +
 arch/x86/include/asm/page_64.h                  |   4 +
 arch/x86/include/asm/page_types.h               |   3 +-
 arch/x86/include/asm/pgtable.h                  |  28 +-
 arch/x86/include/asm/pgtable_types.h            |  58 ++-
 arch/x86/include/asm/processor-flags.h          |  13 +-
 arch/x86/include/asm/processor.h                |  20 +-
 arch/x86/include/asm/realmode.h                 |  12 +
 arch/x86/include/asm/set_memory.h               |   3 +
 arch/x86/include/asm/tlb.h                      |  14 +
 arch/x86/include/asm/tlbflush.h                 |  87 +++-
 arch/x86/include/asm/vga.h                      |  14 +-
 arch/x86/kernel/acpi/boot.c                     |   6 +-
 arch/x86/kernel/cpu/amd.c                       |  29 +-
 arch/x86/kernel/cpu/bugs.c                      |   8 +
 arch/x86/kernel/cpu/common.c                    |  40 ++
 arch/x86/kernel/cpu/mcheck/mce.c                |  43 ++
 arch/x86/kernel/cpu/scattered.c                 |   1 +
 arch/x86/kernel/e820.c                          |  26 +-
 arch/x86/kernel/espfix_64.c                     |   2 +-
 arch/x86/kernel/head64.c                        |  95 +++-
 arch/x86/kernel/head_64.S                       |  40 +-
 arch/x86/kernel/kdebugfs.c                      |  34 +-
 arch/x86/kernel/ksysfs.c                        |  28 +-
 arch/x86/kernel/machine_kexec_64.c              |  25 +-
 arch/x86/kernel/mpparse.c                       | 108 +++--
 arch/x86/kernel/pci-dma.c                       |  11 +-
 arch/x86/kernel/pci-nommu.c                     |   2 +-
 arch/x86/kernel/pci-swiotlb.c                   |  15 +-
 arch/x86/kernel/process.c                       |  17 +-
 arch/x86/kernel/relocate_kernel_64.S            |  14 +
 arch/x86/kernel/setup.c                         |   9 +
 arch/x86/kernel/sys_x86_64.c                    |  30 +-
 arch/x86/kvm/mmu.c                              |  41 +-
 arch/x86/kvm/svm.c                              |  35 +-
 arch/x86/kvm/vmx.c                              |   2 +-
 arch/x86/kvm/x86.c                              |   3 +-
 arch/x86/lib/cmdline.c                          | 105 +++++
 arch/x86/mm/Makefile                            |   2 +
 arch/x86/mm/dump_pagetables.c                   |  93 ++--
 arch/x86/mm/fault.c                             |  26 +-
 arch/x86/mm/hugetlbpage.c                       |  27 +-
 arch/x86/mm/ident_map.c                         |  12 +-
 arch/x86/mm/init.c                              |   2 +-
 arch/x86/mm/ioremap.c                           | 287 +++++++++++-
 arch/x86/mm/kasan_init_64.c                     |   6 +-
 arch/x86/mm/mem_encrypt.c                       | 593 ++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt_boot.S                  | 149 ++++++
 arch/x86/mm/mmap.c                              |  12 +-
 arch/x86/mm/mpx.c                               |  33 +-
 arch/x86/mm/pageattr.c                          |  67 +++
 arch/x86/mm/pat.c                               |   9 +-
 arch/x86/mm/pgtable.c                           |   8 +-
 arch/x86/mm/tlb.c                               | 331 +++++++++----
 arch/x86/pci/common.c                           |   4 +-
 arch/x86/platform/efi/efi.c                     |   6 +-
 arch/x86/platform/efi/efi_64.c                  |  15 +-
 arch/x86/realmode/init.c                        |  12 +
 arch/x86/realmode/rm/trampoline_64.S            |  24 +
 arch/x86/xen/Kconfig                            |   5 +
 arch/x86/xen/enlighten_pv.c                     |   7 +
 arch/x86/xen/mmu_pv.c                           |   5 +-
 arch/x86/xen/xen-head.S                         |   2 +-
 drivers/acpi/processor_idle.c                   |   2 -
 drivers/firmware/dmi-sysfs.c                    |   5 +-
 drivers/firmware/efi/efi.c                      |  33 ++
 drivers/firmware/pcdp.c                         |   4 +-
 drivers/gpu/drm/drm_gem.c                       |   2 +
 drivers/gpu/drm/drm_vm.c                        |   4 +
 drivers/gpu/drm/ttm/ttm_bo_vm.c                 |   7 +-
 drivers/gpu/drm/udl/udl_fb.c                    |   4 +
 drivers/idle/intel_idle.c                       |   9 +-
 drivers/iommu/amd_iommu.c                       |  30 +-
 drivers/iommu/amd_iommu_init.c                  |  34 +-
 drivers/iommu/amd_iommu_proto.h                 |  10 +
 drivers/iommu/amd_iommu_types.h                 |   2 +-
 drivers/sfi/sfi_core.c                          |  23 +-
 drivers/video/fbdev/core/fbmem.c                |  12 +
 include/asm-generic/early_ioremap.h             |   2 +
 include/asm-generic/pgtable.h                   |  12 +
 include/linux/compiler-gcc.h                    |   2 +
 include/linux/compiler.h                        |   4 +
 include/linux/dma-mapping.h                     |  13 +
 include/linux/efi.h                             |   9 +-
 include/linux/io.h                              |   2 +
 include/linux/kexec.h                           |   8 +
 include/linux/mem_encrypt.h                     |  48 ++
 include/linux/mm_inline.h                       |   6 +
 include/linux/swiotlb.h                         |   1 +
 init/main.c                                     |  10 +
 kernel/kexec_core.c                             |  12 +-
 kernel/memremap.c                               |  20 +-
 lib/swiotlb.c                                   |  57 ++-
 mm/early_ioremap.c                              |  28 +-
 mm/memory-failure.c                             |   2 +
 121 files changed, 3169 insertions(+), 498 deletions(-)
 create mode 100644 Documentation/x86/amd-memory-encryption.txt
 create mode 100644 Documentation/x86/x86_64/5level-paging.txt
 create mode 100644 arch/x86/include/asm/mem_encrypt.h
 create mode 100644 arch/x86/mm/mem_encrypt.c
 create mode 100644 arch/x86/mm/mem_encrypt_boot.S
 create mode 100644 include/linux/mem_encrypt.h
#1 lore raw
Quoted 6 lines · Ingo Molnar

The main changes in this cycle are support for three new, complex hardware features of x86 CPUs:

- Add 5-level paging support, which is a new hardware feature on upcoming Intel CPUs allowing up to 128 PB of virtual address space and 4 PB of physical RAM space - a 256-fold increase over the old limits.

Minor nitpick: I don't see where "256-fold" comes from.

Virtual address space increased from 256 TB to 128 PB -- 512 times.
Physical: 64 TB -> 4 PB -- 64 times.

Signature · 1 line
 Kirill A. Shutemov
#2 lore raw
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Ingo Molnar

The main changes in this cycle are support for three new, complex hardware features of x86 CPUs:

Quoted 8 lines
Quoted 3 lines

- Add 5-level paging support, which is a new hardware feature on upcoming Intel CPUs allowing up to 128 PB of virtual address space and 4 PB of physical RAM space - a 256-fold increase over the old limits.

Minor nitpick: I don't see where "256-fold" comes from.

Virtual address space increased from 256 TB to 128 PB -- 512 times.
Physical: 64 TB -> 4 PB -- 64 times.

See arch/x86/kernel/tsc.c:

[email protected] "math is hard, lets go shopping!"

#3 lore raw
Quoted 9 lines · Thomas Gleixner
Quoted 8 lines · Kirill A. Shutemov
Quoted 6 lines · Ingo Molnar

The main changes in this cycle are support for three new, complex hardware features of x86 CPUs:

- Add 5-level paging support, which is a new hardware feature on upcoming Intel CPUs allowing up to 128 PB of virtual address space and 4 PB of physical RAM space - a 256-fold increase over the old limits.

Quoted 8 lines
Quoted 4 lines

Minor nitpick: I don't see where "256-fold" comes from.

Virtual address space increased from 256 TB to 128 PB -- 512 times.
Physical: 64 TB -> 4 PB -- 64 times.

See arch/x86/kernel/tsc.c:

[email protected] "math is hard, lets go shopping!"

Yeah, I mis-remembered the increase in number of bits, should have double checked ...

/me goes shopping

Thanks,

Ingo

#4 lore raw
In reply to Ingo Molnar
Quoted 4 lines · Ingo Molnar

Please pull the latest ... git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git ...

Hmm. My laptop (XPS 13) doesn't resume any more. It suspends, but doesn't come back from resume.

I immediately assumed it was the power management pulls I just did, but then I started bisecting, and now it's actually pointing into the various x86 pulls I did yesterday instead.

Now, I'm reasonably early in my bisection (so literally "somewhere between the 'docs-next' and the 'x86-mm-for-linus' pull), and maybe the problem isn't even entirely repeatable and my bisection has already gone off the rails, but I thought I'd give at least an early heads-up about this thing.

I'll have more as it bisects deeper into the merge window, but it might be a while.

Linus

#5 lore raw
Quoted 13 lines · Linus Torvalds
Quoted 6 lines · Ingo Molnar

Please pull the latest ... git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git ...

Hmm. My laptop (XPS 13) doesn't resume any more. It suspends, but doesn't come back from resume.

You could try booting with nopcid to rule out CR4 issues. I can also imagine SME's very clever CR3 masking causing a problem.

Unfortunately, if it's a PCID problem involving wrong ordering of CR4 initialization, you might get lucky if you suspend with ASID 0 active, causing unfortunate bisection results.

I will test on my XPS 13 after lunch.

Quoted 14 lines

I immediately assumed it was the power management pulls I just did, but then I started bisecting, and now it's actually pointing into the various x86 pulls I did yesterday instead.

Now, I'm reasonably early in my bisection (so literally "somewhere between the 'docs-next' and the 'x86-mm-for-linus' pull), and maybe the problem isn't even entirely repeatable and my bisection has already gone off the rails, but I thought I'd give at least an early heads-up about this thing.

I'll have more as it bisects deeper into the merge window, but it might be a while.

Linus

maintainer
#6 lore raw
Quoted 1 line · Andy Lutomirski

I can also imagine SME's very clever CR3 masking causing a problem.

So that mask should be 0 on an !SME system. One of the main goals of the SME code was to have no effect when SME is disabled/not available.

Signature · 4 lines
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.
#7 lore raw
In reply to Linus Torvalds
Quoted 13 lines · Linus Torvalds
Quoted 4 lines · Ingo Molnar

Please pull the latest ... git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git ...

Hmm. My laptop (XPS 13) doesn't resume any more. It suspends, but doesn't come back from resume.

I immediately assumed it was the power management pulls I just did, but then I started bisecting, and now it's actually pointing into the various x86 pulls I did yesterday instead.

Quoted 8 lines

Now, I'm reasonably early in my bisection (so literally "somewhere between the 'docs-next' and the 'x86-mm-for-linus' pull), and maybe the problem isn't even entirely repeatable and my bisection has already gone off the rails, but I thought I'd give at least an early heads-up about this thing.

I'll have more as it bisects deeper into the merge window, but it might be a while.

Hm, just as background, there are no regression reports I'm aware of against any of these trees, plus most of the dangerous commits have been in linux-next for at least two weeks - the majority of them even longer. The last 2-4 commits of x86/mm are fresher.

But it could be a regression in an older commit in x86/mm just as well, triggered by you for the first time :-/

Does your laptop CPU have PCID support by any chance? The CPU coming out of resume with PCID disabled and us not properly re-enabling it might be a possible failure mode.

Thanks,

Ingo

#8 lore raw
Quoted 4 lines · Ingo Molnar

Does your laptop CPU have PCID support by any chance? The CPU coming out of resume with PCID disabled and us not properly re-enabling it might be a possible failure mode.

It does have pcid. It's a -7-6560U in the prev-gen XPS 13 (aka "9350").

I have bisected a bit deeper, and the pcid code is definitely in that bisection window still. But are a fair number of other commits (about 160 right now)

Linus

#9 lore raw
Quoted 4 lines · Linus Torvalds

I have bisected a bit deeper, and the pcid code is definitely in that bisection window still. But are a fair number of other commits (about 160 right now)

Now down to 18.

And one of those 18 is commit 10af6235e0d3 ("x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID"), which I guess is where the problem might actually start showing up if it is pcid.

I'll continue to bisect rather than just test "nopcid", because I want to get that bisection result regardless.

Linus

maintainer
#10 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Linus Torvalds

I have bisected a bit deeper, and the pcid code is definitely in that bisection window still. But are a fair number of other commits (about 160 right now)

Now down to 18.

Quoted 7 lines

And one of those 18 is commit 10af6235e0d3 ("x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID"), which I guess is where the problem might actually start showing up if it is pcid.

I'll continue to bisect rather than just test "nopcid", because I want to get that bisection result regardless.

The thing that would surprise me about this is that I dogfooded my PCID tree on my XPS 13 9250 for quite a while and it worked just fine. Maybe something got merged oddly or had an unpleasant interaction somewhere?

Anyway, all two cores of that fancy Skylake are compiling your tree as we speak :). It'll finish eventually.

--Andy

#11 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

And one of those 18 is commit 10af6235e0d3 ("x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID"), which I guess is where the problem might actually start showing up if it is pcid.

Yup, that's what it bisected down to in the end.

And then rebooting once more into that kernel, but with "nopcid" on the command line, and it all works.

I'll go back to top-of-tree just to verify that 'nopcid' thing there too, but it does seem pretty clear-cut.

Linus

#12 lore raw
Quoted 5 lines · Linus Torvalds
Quoted 4 lines

And one of those 18 is commit 10af6235e0d3 ("x86/mm: Implement PCID based optimization: try to preserve old TLB entries using PCID"), which I guess is where the problem might actually start showing up if it is pcid.

Quoted 7 lines

Yup, that's what it bisected down to in the end.

And then rebooting once more into that kernel, but with "nopcid" on the command line, and it all works.

I'll go back to top-of-tree just to verify that 'nopcid' thing there too, but it does seem pretty clear-cut.

This is a "me too", observed on my Lenovo thinkpad x270 (so it's not specific to that XPS 13 system at all).

The symptom I observe is that an attempt to resume from hibernation proceeds up to reading 100% of the hibernation image, and then reboot happens (IOW looks like triple fault).

nopcid cures it, I haven't tried to revert 10af6235e0d3 yet, but looks like it's the same thing.

Signature · 2 lines
Jiri Kosina
SUSE Labs
#13 lore raw
Quoted 9 lines · Jiri Kosina

This is a "me too", observed on my Lenovo thinkpad x270 (so it's not specific to that XPS 13 system at all).

The symptom I observe is that an attempt to resume from hibernation proceeds up to reading 100% of the hibernation image, and then reboot happens (IOW looks like triple fault).

nopcid cures it, I haven't tried to revert 10af6235e0d3 yet, but looks like it's the same thing.

[ reposting the information again with LKML re-introduced to CC ]

As suggested by Andy off-list, I tested with this change to always force ASID 0

arch/x86/mm/tlb.c +1 -1
index 5ca71d1..c3b0811 100644
@@ -35,7 +35,7 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen,
 {
 	u16 asid;
 
-	if (!static_cpu_has(X86_FEATURE_PCID)) {
+	if (true || !static_cpu_has(X86_FEATURE_PCID)) {
 		*new_asid = 0;
 		*need_flush = true;
 		return;

and that fixes the issue on my system.

Signature · 2 lines
Jiri Kosina
SUSE Labs
maintainer
#14 lore raw
Quoted 31 lines · Jiri Kosina
Quoted 9 lines · Jiri Kosina

This is a "me too", observed on my Lenovo thinkpad x270 (so it's not specific to that XPS 13 system at all).

The symptom I observe is that an attempt to resume from hibernation proceeds up to reading 100% of the hibernation image, and then reboot happens (IOW looks like triple fault).

nopcid cures it, I haven't tried to revert 10af6235e0d3 yet, but looks like it's the same thing.

[ reposting the information again with LKML re-introduced to CC ]

As suggested by Andy off-list, I tested with this change to always force ASID 0

arch/x86/mm/tlb.c +1 -1
index 5ca71d1..c3b0811 100644
@@ -35,7 +35,7 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen,
 {
        u16 asid;
 
-       if (!static_cpu_has(X86_FEATURE_PCID)) {
+       if (true || !static_cpu_has(X86_FEATURE_PCID)) {
                *new_asid = 0;
                *need_flush = true;
                return;
Quoted 1 line

and that fixes the issue on my system.

I got Linus' config to boot. The problem was that I ended up with a root-owned file (not sure which) in my tree that cause an incorrect build but didn't generate errors. I don't know how this happened, but an ill-timed sudo make -j4 modules_install install was probably involved. git clean -ffxxxd , did *not* fix it or even notice it in any obvious way.

Anyway, the problem appears to depend on kernel config because it's dying here on resume on secondary cpus:

    VM_BUG_ON(__read_cr3() != (__sme_pa(real_prev->pgd) | prev_asid));

in switch_mm_irqs_off().

What seems to be going on is that the wakeup CPU is exactly restoring original state. All other CPUs are restoring swapper_pg_dir but are failing to restore the PCID tag bits, which trips the assertion w.p. 5/6 per non-boot CPU. So, if you have that debug option set, you die w.p. 1 - (1/6)^(cpus - 1), which is pretty large.

I'll come up with a clean fix this evening, I hope.

#15 lore raw
In reply to Ingo Molnar
Quoted 5 lines · Ingo Molnar

Hm, just as background, there are no regression reports I'm aware of against any of these trees, plus most of the dangerous commits have been in linux-next for at least two weeks - the majority of them even longer. The last 2-4 commits of x86/mm are fresher.

Side note: I do not believe a lot of people actually run linux-next on laptops, so suspend/resume likely doesn't get a lot of testing in next.

I think most people who run linux-next tend to be automation things on farms.

Don't get me wrong - I love linux-next and your tip testing, but I think linux-next is best for finding build errors etc big integration issues, with some very rudimentary actual boot checking.

Maybe I'm wrong.

I _wish_ I am wrong.

But honestly, I see problems on my machines most merge windows. Last release was actually unusually calm, in that I don't think I had to bisect anything at all.

Which really says to me: "very few people actually _run_ those next trees".

Linus

#16 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Ingo Molnar

Hm, just as background, there are no regression reports I'm aware of against any of these trees, plus most of the dangerous commits have been in linux-next for at least two weeks - the majority of them even longer. The last 2-4 commits of x86/mm are fresher.

Quoted 5 lines

Side note: I do not believe a lot of people actually run linux-next on laptops, so suspend/resume likely doesn't get a lot of testing in next.

I think most people who run linux-next tend to be automation things on farms.

Yeah, so 10af6235e0d3 was in linux-next for over a month, yet no-one reported the bug.

Quoted 5 lines

Don't get me wrong - I love linux-next and your tip testing, but I think linux-next is best for finding build errors etc big integration issues, with some very rudimentary actual boot checking.

Maybe I'm wrong.

I don't think you are wrong - most boot tests don't involve laptops. linux-next is mostly server oriented - and servers are often more debuggable than laptops. (Have actual serial ports or physical network connections with serial emulation, etc.)

I tried to maintain a laptop testbox in -tip testing with netconsole for a time - but it was quite a bit of pain so I eventually dropped it. (Not that the simple boot + kernel build test that -tip does would have uncovered this particular bug.)

Maybe a tester or two saw the 'dead on resume' bug and didn't bother reporting it, because it's a very difficult category of bug to debug short of a full bisection?

Thanks,

Ingo

#17 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Ingo Molnar

Hm, just as background, there are no regression reports I'm aware of against any of these trees, plus most of the dangerous commits have been in linux-next for at least two weeks - the majority of them even longer. The last 2-4 commits of x86/mm are fresher.

Quoted 8 lines
Quoted 5 lines

Side note: I do not believe a lot of people actually run linux-next on laptops, so suspend/resume likely doesn't get a lot of testing in next.

I think most people who run linux-next tend to be automation things on farms.

Yeah, so 10af6235e0d3 was in linux-next for over a month, yet no-one reported the bug.

That was also smack in the middle of the vacation season on the northern hemisphere, which didn't help testing coverage either I suspect ...

In hindsight it was perhaps not the smartest thing from me to send three major hw-enablement features to you - although only PCID was the one that should have real widespread effects, and I did stage those changes pretty conservatively over several months. Hindsight is 20/20 ...

Thanks,

Ingo

#18 lore raw
In reply to Ingo Molnar
Quoted 25 lines · Ingo Molnar
Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Ingo Molnar

Hm, just as background, there are no regression reports I'm aware of against any of these trees, plus most of the dangerous commits have been in linux-next for at least two weeks - the majority of them even longer. The last 2-4 commits of x86/mm are fresher.

Quoted 5 lines

Side note: I do not believe a lot of people actually run linux-next on laptops, so suspend/resume likely doesn't get a lot of testing in next.

I think most people who run linux-next tend to be automation things on farms.

Yeah, so 10af6235e0d3 was in linux-next for over a month, yet no-one reported the bug.

Quoted 5 lines

Don't get me wrong - I love linux-next and your tip testing, but I think linux-next is best for finding build errors etc big integration issues, with some very rudimentary actual boot checking.

Maybe I'm wrong.

Quoted 7 lines

I don't think you are wrong - most boot tests don't involve laptops. linux-next is mostly server oriented - and servers are often more debuggable than laptops. (Have actual serial ports or physical network connections with serial emulation, etc.)

I tried to maintain a laptop testbox in -tip testing with netconsole for a time - but it was quite a bit of pain so I eventually dropped it. (Not that the simple boot + kernel build test that -tip does would have uncovered this particular bug.)

Some time ago, Tony Lindgren asked me to test Nokia N900 on -next from time to time. I do, and it uncovers problems from time to time.

Perhaps if Linus or you asks for a volunteer, they can get one? We have people submitting patches due to various challenges, perhaps "run -next for a month" would be suitable challenge?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html