Redoing eXclusive Page Frame Ownership (XPFO) with isolated CPUs in mind (for KVM to isolate its guests per CPU)

Hi!

See eXclusive Page Frame Ownership (https://lwn.net/Articles/700606/) which was posted way back in in 2016..

In the last couple of months there has been a slew of CPU issues that have complicated a lot of things. The latest - L1TF - is still fresh in folks's mind and it is especially acute to virtualization workloads.

As such a bunch of various folks from different cloud companies (CCed) are looking at a way to make Linux kernel be more resistant to hardware having these sort of bugs.

In particular we are looking at a way to "remove as many mappings from the global kernel address space as possible. Specifically, while being in the context of process A, memory of process B should not be visible in the kernel." (email from Julian Stecklina). That is the high-level view and how this could get done, well, that is why posting this on LKML/linux-hardening/kvm-devel/linux-mm to start the discussion.

Usually I would start with a draft of RFC patches so folks can rip it apart, but thanks to other people (Juerg thank you!) it already exists:

(see https://www.mail-archive.com/[email protected]/msg1222756.html)

The idea would be to extend this to:

1) Only do it for processes that run under CPUS which are in isolcpus list.

2) Expand this to be a per-cpu page tables. That is each CPU has its own unique set of pagetables - naturally _START_KERNEL -> __end would be mapped but the rest would not.

Thoughts? Is this possible? Crazy? Better ideas?

Ok, so my gut feel is that the above was reasonable within the context of 2016, but that the XPFO model is completely pointless and wrong in the post-Meltdown world we now live in.

Why?

Because with the Meltdown patches, we ALREADY HAVE the isolated page tables that XPFO tries to do.

They are just the normal user page tables.

So don't go around doing other crazy things.

All you need to do is to literally:

- before you enter VMX mode, switch to the user page tables

- when you exit, switch back to the kernel page tables

don't do anything else. You're done.

Now, this is complicated a bit by the fact that in order to enter VMX mode with the user page tables, you do need to add the VMX state itself to those user page tables (and add the actual trampoline code to the vmenter too).

So it does imply we need to slightly extend the user mapping with a few new patches, but that doesn't sound bad.

In fact, it sounds absolutely trivial to me.

The other thing you want to do is is the trivial optimization of "hey. we exited VMX mode due to a host interrupt", which would look like this:

* switch to user page tables in order to do vmenter
* vmenter
* host interrupt happens
- switch to kernel page tables to handle irq
- do_IRQ etc
- switch back to user page tables
- iret
* switch to kernel page tables because the vmenter returned

so you want to have some trivial short-circuiting of that last "switch to user page tables and back" dance. It may actually be that we don't even need it, because the irq code may just be looking at what *mode* we were in, not what page tables we were in. I looked at that code back in the meltdown days, but that's already so last-year now that we have all these _other_ CPU bugs we handled.

But other than small details like that, doesn't this "use our Meltdown user page table" sound like the right thing to do?

And note: no new VM code or complexity. None. We already have the "isolated KVM context with only pages for the KVM process" case handled.

Of course, after the long (and entirely unrelated) discussion about the TLB flushing bug we had, I'm starting to worry about my own competence, and maybe I'm missing something really fundamental, and the XPFO patches do something else than what I think they do, or my "hey, let's use our Meltdown code" idea has some fundamental weakness that I'm missing.

Linus

The interesting part is taking the user (and other) pages out of the kernel's 1:1 physmap.

It's the *kernel* we don't want being able to access those pages, because of the multitude of unfixable cache load gadgets.

Amazon Development Centre (London) Ltd. Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom.

Right. And even before Meltdown, it was desirable to remove those from the physmap to avoid SMAP (and in some cases SMEP) bypasses (as detailed in the mentioned paper:
http://www.cs.columbia.edu/~vpk/papers/ret2dir.sec14.pdf).

-Kees

Kees Cook
Pixel Security

Ahh.

I guess the proof is in the pudding. Did somebody try to forward-port that patch set and see what the performance is like?

It used to be just 500 LOC. Was that because they took horrible shortcuts? Are the performance numbers for the 32-bit case that already had the kmap() overhead?

Linus

The last version I worked on was a bit before Meltdown was public:
https://lkml.org/lkml/2017/9/7/445

The overhead was a lot, but Dave Hansen gave some ideas about how to speed things up in this thread: https://lkml.org/lkml/2017/9/20/828

Since meltdown hit, I haven't worked seriously on understand and implementing his suggestions, in part because it wasn't clear to me what pieces of the infrastructure we might be able to re-use. Someone who knows more about mm/ might be able to suggest an approach, though.

Tycho

Unfortunately, I'm not sure there's much of KPTI we can reuse. KPTI still has a very static kernel map (well, two static kernel maps) and XPFO really needs a much more dynamic map.

We do have a bit of infrastructure now to do TLB flushes near the kernel exit point, but it's entirely for the user address space, which isn't affected by XPFO.

One option is to have separate kernel address spaces, both with and without the full physmap.

If you need the physmap, then rather than manually mapping with 4KiB pages, you just switch. Having first ensured that no malicious guest or userspace is running on a sibling, of course.

I'm not sure it's a win, but it might be worth looking at.

The problem is determining when "you need the physmap". Tycho's patches, as I remember them basically classify pages between being "user" pages which are accessed only via kmap() and friends and "kernel" pages which need to be mapped all the time because they might have a 'task_struct' or a page table or a 'struct file'.

You're right that we could have a full physmap that we switch to for kmap()-like access to user pages. But, the real problem is transitioning pages from kernel to user usage since it requires shooting down the old kernel mappings for those pages in some way.

You might decide that you simply don't care enough, and are willing to leave possible stale TLB entries rather than shoot things down.

Then you'd still possibly see user pages in the kernel map, but only for a fairly limited time, and only until the TLB entry gets re-used for other reasons.

Even with kernel page table entries being marked global, their lifetime in the TLB is likely not very long, and definitely not long enough for some user that tries to scan for pages.

Linus

I hadn't actually seen the XPFO patch set before; we're going to take a serious look.

Of course, this is only really something that a select few people (with quite a lot of machines) would turn on. And they might be willing to tolerate a significant performance cost if the alternative way to be safe is to disable hyperthreading entirely — which is Intel's best recommendation so far, it seems.

Another alternative... I'm told POWER8 does an interesting thing with hyperthreading and gang scheduling for KVM. The host kernel doesn't actually *see* the hyperthreads at all, and KVM just launches the full set of siblings when it enters a guest, and gathers them again when any of them exits. That's definitely worth investigating as an option for x86, too.

I actually think that such scheduling mechanism which prevents leaking cache entries to sibling hyperthreads should co-exist together with the KVM address space isolation to fully mitigate L1TF and other similar vulnerabilities. The address space isolation should prevent VMExit handlers code gadgets from loading arbitrary host memory to the cache. Once VMExit code path switches to full host address space, then we should also make sure that no other sibling hyprethread is running in the guest.

Focusing on the scheduling mechanism, we must make sure that when a logical processor runs guest code, all siblings logical processors must run code which do not populate L1D cache with information unrelated to this VM. This includes forbidding one logical processor to run guest code while sibling is running a host task such as a NIC interrupt handler. Thus, when a vCPU thread exits the guest into the host and VMExit handler reaches code flow which could populate L1D cache with this information, we should force an exit from the guest of the siblings logical processors, such that they will be allowed to resume only on a core which we can promise that the L1D cache is free from information unrelated to this VM.

At first, I have created a patch series which attempts to implement such mechanism in KVM. However, it became clear to me that this may need to be implemented in the scheduler itself. This is because:
1. It is difficult to handle all new scheduling contrains only in KVM.
2. This mechanism should be relevant for any Type-2 hypervisor which runs inside Linux besides KVM (Such as VMware Workstation or VirtualBox).
3. This mechanism could also be used to prevent future “core-cache-leaking” vulnerabilities to be exploited between processes of different security domains which run as siblings on the same core.

The main idea is a mechanism which is very similar to Microsoft's "core scheduler" which they implemented to mitigate this vulnerability. The mechanism should work as follows:
1. Each CPU core will now be tagged with a "security domain id".
2. The scheduler will provide a mechanism to tag a task with a security domain id.
3. Tasks will inherit their security domain id from their parent task.
3.1. First task in system will have security domain id of 0. Thus, if nothing special is done, all tasks will be assigned with security domain id of 0.
4. Tasks will be able to allocate a new security domain id from the scheduler and assign it to another task dynamically.
5. Linux scheduler will prevent scheduling tasks on a core with a different security domain id:
5.0. CPU core security domain id will be set to the security domain id of the tasks which currently run on it.
5.1. The scheduler will attempt to first schedule a task on a core with required security domain id if such exists.
5.2. Otherwise, will need to decide if it wishes to kick all tasks running on some core to run the task with a different security domain id on that core.

The above mechanism can be used to mitigate the L1TF HT variant by just assigning vCPU tasks with a security domain id which is unique per VM and also different than the security domain id of the host which is 0.

I would be glad to hear feedback on the above suggestion.
If this should better be discussed on a separate email thread, please say so and I will open a new thread.

Thanks,
-Liran

The KVM POWER8 solution (see arch/powerpc/kvm/book3s_hv.c) does that. The siblings are *never* running host kernel code; they're all torn down when any of them exits the guest. And it's always the *same* guest.

I'm not sure I agree. If KVM is handling "only let siblings run the *same* guest" and the siblings aren't visible to the host at all, that's quite simple. Any other hypervisor can also do it.

Now, the down-side of this is that the siblings aren't visible to the host. They can't be used to run multiple threads of the same userspace processes; only multiple threads of the same KVM guest. A truly generic core scheduler would cope with userspace threads too.

BUT I strongly suspect there's a huge correlation between the set of people who care enough about the KVM/L1TF issue to enable a costly XFPO-like solution, and the set of people who mostly don't give a shit about having sibling CPUs available to run the host's userspace anyway.

This is not the "I happen to run a Windows VM on my Linux desktop" use case...

I wasn’t aware of this KVM Power8 mechanism. Thanks for the pointer. (371fefd6f2dc ("KVM: PPC: Allow book3s_hv guests to use SMT processor modes”))

Note though that my point regarding the co-existence of the isolated address space together with such scheduling mechanism is still valid. The scheduling mechanism should not be seen as an alternative to the isolated address space if we wish to reduce the frequency of events in which we need to kick sibling hyperthreads from guest.

If I understand your proposal correctly, you suggest to do something similar to the KVM Power8 solution:
1. Disable HyperThreading for use by host user space.
2. Use sibling hyperthreads only in KVM and schedule group of vCPUs that run on a single core as a “gang” to enter and exit guest together.

This solution may work well for KVM-based cloud providers that match the following criteria:
1. All compute instances run with SR-IOV and IOMMU Posted-Interrupts.
2. Configure affinity such that host dedicate distinct set of physical cores per guest. No physical core is able to run vCPUs from multiple guests.

However, this may not necessarily be the case: Some cloud providers have compute instances which all their devices are emulated or ParaVirtualized. In the proposed scheduling mechanism, all the IOThreads of these guests will not be able to utilize HyperThreading which can be a significant performance hit. So Oracle Cloud (OCI) are folks who do care enough about the KVM/L1TF issue but gives a shit about having sibling CPUs available to run host userspace. :) Unless I’m missing something of course...

In addition, desktop users who run VMs today, expect a security boundary to exist between the guest and the host. Besides the L1TF HyperThreading variant, we were able to preserve such a security boundary.
It seems a bit weird that we will implement a mechanism in x86 KVM that it’s message to users is basically:
“If you want to have a security boundary between a VM and the host, you need to enable this knob which will also cause the rest of your host to see half the amount of logical processors”.

Furthermore, I think it is important to think about a mechanism which may help us to mitigate future similar “core-cache-leak” vulnerabilities. As I previously mentioned, the “core scheduler” could help us mitigate these vulnerabilities on OS-level by disallowing userspace tasks of different “security domain” to run as siblings on the same core.

-Liran

(Cc Paolo who probably have good feedback on the entire email thread as-well)

Hey everyone,

I've been spending some cycles on the XPFO patch set this week. For the patch set as it was posted for v4.13, the performance overhead of compiling a Linux kernel is ~40% on x86_64[1]. The overhead comes almost completely from TLB flushing. If we can live with stale TLB entries allowing temporary access (which I think is reasonable), we can remove all TLB flushing (on x86). This reduces the overhead to 2-3% for kernel compile.

There were no problems in forward-porting the patch set to master. You can find the result here, including a patch makes the TLB flushing configurable:
http://git.infradead.org/users/jsteckli/linux-xpfo.git/shortlog/refs/heads/xpfo-master

It survived some casual stress-ng runs. I can rerun the benchmarks on this version, but I doubt there is any change.

The patch is still fairly small. As for the horrible shortcuts, I let others comment on that.

HTH,
Julian

[1] Measured on my quad-core (8 hyperthreads) Kaby Lake desktop building Linux 4.18 with the Phoronix Test Suite.

--
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

Cool, thanks for doing this! Do you have any thoughts about what the 2-3% is? It seems to me like if we're not doing the TLB flushes, the rest of this should be *really* cheap, even cheaper than 2-3%. Dave Hansen had suggested coalescing things on a per mapping basis vs. doing it per page, which might help?

Heh, things like xpfo_temp_map() aren't awesome, but that can hopefully be fixed by keeping a little bit of memory around for use where we are mapping things and can't fail. I remember some discussion about hopefully not having to sprinkle xpfo mapping calls everywhere in the kernel, so perhaps we could get rid of it entirely?

Anyway, I'm working on some other stuff for the kernel right now, but I hope (:D) that it should be close to done, and I'll have more cycles to work on this soon too.

Tycho

I have to say, even 2-3% for a kernel compile sounds absolutely horrendous.

Kernel bullds are 90% user space at least for me, so a 2-3% slowdown from a kernel is not some small unnoticeable thing.

Linus

Speaking of pages and slowdowns,
is there a better place to ask this question:

"""
'New "Turning Tables" Technique Bypasses All Windows Kernel Mitigations'
https://www.bleepingcomputer.com/news/security/new-turning-tables-technique-bypasses-all-windows-kernel-mitigations/

the Linux project, macOS and Linux are, in theory, also vulnerable to this technique, albeit the researchers have not verified such attacks, as of yet.

Slides:
https://cdn2.hubspot.net/hubfs/487909/Turning%20(Page)%20Tables_Slides.pdf

Naturally, I took notice and decided to forward the latest scary headline to this list to see if this is already being addressed?
"""

This essentially just says that if you can change page tables you can subvert kernels. That's always been the case, always will be, I'm sure has been used forever by root kits, and I don't know why anybody would pass it off as a "new attack".

-Andi

Well, it's at least in a range where it doesn't look hopeless.

The overhead seems to come from the hooks that XPFO adds to alloc/free_pages. These hooks add a couple of atomic operations per allocated (4K) page for book keeping. Some of these atomic ops are only for debugging and could be removed. There is also some opportunity to streamline the per-page space overhead of XPFO.

I'll do some more in-depth profiling later this week.

Julian
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

I've updated my XPFO branch[1] to make some of the debugging optional and also integrated the XPFO bookkeeping with struct page, instead of requiring CONFIG_PAGE_EXTENSION, which removes some checks in the hot path. These changes push the overhead down to somewhere between 1.5 and 2% for my quad core box in kernel compile. This is close to the measurement noise, so I take suggestions for a better benchmark here.

Of course, if you hit contention on the xpfo spinlock then performance will suffer. I guess this is what happened on Khalid's large box.

I'll try to remove the spinlocks and add fixup code to the pagefault handler to see whether this improves the situation on large boxes. This might turn out to be ugly, though.

Julian

[1] http://git.infradead.org/users/jsteckli/linux-xpfo.git/shortlog/refs/heads/xpfo-master
--
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

FWIW, that was my original design but there was some resistance to adding more to the page struct and page extension was suggested instead.

I'm wondering how much performance we're loosing by having to split hugepages. Any chance this can be quantified somehow? Maybe we can have a pool of some sorts reserved for userpages and group allocations so that we can track the XPFO state at the hugepage level instead of at the 4k level to prevent/reduce page splitting. Not sure if that causes issues or has any unwanted side effects though...

...Juerg

struct page makes the code easier to understand and removes some special cases and bookkeeping.

Optimizing the allocation/deallocation path might be worthwhile, because that's where most of the overhead goes. I haven't looked into how to do this yet. I'd appreciate if someone has pointers to code that tries to achieve similar functionality to get me started.

That being said, I'm wondering whether we have unrealistic expectations about the overhead here and whether it's worth turning this patch into something far more complicated. Opinions?

Julian
--
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

I think that implementing Dave Hansen's suggestions of not doing flushes/other work on every map/unmap, but only when pages are added to the various free lists will probably help out a lot. That's where I got stuck last time when I was trying to do it, though :)

Cheers,

Tycho

Btw, can xpfo_lock be replaced with a bit spinlock in the page? Growing struct page too much might cause performance issues. Then again going beyong the 64 byte cache line might already cause that, and even then it propbably is still way better than the page extensions.

OTOH if you keep the spinlock it might be worth to use atomic_dec_and_lock on the count. Maybe the answer is an hash of spinlock, as we obviously can't take all that many of them at the same time anyway.

Also for your trasitions froms zero it might be worth at looking at atomic_inc_unless_zero.

Hi Julian,

I ran tests with your updated code and gathered lock statistics. Change in system time for "make -j60" was in the noise margin (It actually went up by about 2%). There is some contention on xpfo_lock. Average wait time does not look high compared to other locks. Max hold time looks a little long. From /proc/lock_stat:

              &(&page->xpfo_lock)->rlock:         29698          29897           0.06         134.39       15345.58           0.51      422474670      960222532           0.05       30362.05   195807002.62           0.20

Nevertheless even a smaller average wait time can add up.

--
Khalid

Thanks for doing this!

I've spent some time optimizing spinlock usage in the code. See the two last commits in my xpfo-master branch[1]. The optimization in xpfo_kunmap is pretty safe. The last commit that optimizes locking in xpfo_kmap is tricky, though, and I'm not sure this is the right approach. FWIW, I've modeled this locking strategy in Spin and it doesn't find any problems with it.

I've tested the result on a box with 72 hardware threads and I didn't see a meaningful difference in kernel compile performance. It's still hovering around 2%. So the question is, whether it's actually useful to do these optimizations.

Khalid, you mentioned 5% overhead. Can you give the new code a spin and see whether anything changes?

Julian

[1] http://git.infradead.org/users/jsteckli/linux-xpfo.git/shortlog/refs/heads/xpfo-master

--
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

Hi Julian,

I tested the kernel with this new code. When booted without "xpfotlbflush", there is no meaningful change in system time with kernel compile. Kernel locks up during bootup when booted with xpfotlbflush:

[   52.967060] RIP: 0010:queued_spin_lock_slowpath+0xf6/0x1e0
[   52.967061] Code: 48 03 34 c5 80 97 12 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 0f 84 93 00 00
[   52.967061] RSP: 0018:ffffc9001cc83a00 EFLAGS: 00000002
[   52.967062] RAX: 0000000000340101 RBX: ffffea06c16292e8 RCX: 0000000000580000
[   52.967062] RDX: ffff88603c9e3980 RSI: 0000000000000000 RDI: ffffea06c16292e8
[   52.967063] RBP: ffffea06c1629300 R08: 0000000000000001 R09: 0000000000000000
[   52.967063] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88c02765a000
[   52.967063] R13: 0000000000000000 R14: ffff8860152a0d00 R15: 0000000000000000
[   52.967064] FS:  00007f41ad1658c0(0000) GS:ffff88603c800000(0000) knlGS:0000000000000000
[   52.967064] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   52.967064] CR2: ffff88c02765a000 CR3: 00000060252e4003 CR4: 00000000007606e0
[   52.967065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   52.967065] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   52.967065] PKRU: 55555554
[   52.967066] Call Trace:
[   52.967066]  do_raw_spin_lock+0x6d/0xa0
[   52.967066]  _raw_spin_lock+0x53/0x70
[   52.967067]  ? xpfo_do_map+0x1b/0x52
[   52.967067]  xpfo_do_map+0x1b/0x52
[   52.967067]  xpfo_spurious_fault+0xac/0xae
[   52.967068]  __do_page_fault+0x3cc/0x4e0
[   52.967068]  ? __lock_acquire.isra.31+0x165/0x710
[   52.967068]  do_page_fault+0x32/0x180
[   52.967068]  page_fault+0x1e/0x30
[   52.967069] RIP: 0010:memcpy_erms+0x6/0x10
[   52.967069] Code: 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   52.967070] RSP: 0018:ffffc9001cc83bb8 EFLAGS: 00010246
[   52.967070] RAX: ffff8860299d0f00 RBX: ffffc9001cc83dc8 RCX: 0000000000000080
[   52.967071] RDX: 0000000000000080 RSI: ffff88c02765a000 RDI: ffff8860299d0f00
[   52.967071] RBP: 0000000000000080 R08: ffffc9001cc83d90 R09: 0000000000000001
[   52.967071] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000080
[   52.967072] R13: 0000000000000080 R14: 0000000000000000 R15: ffff88c02765a080
[   52.967072]  _copy_to_iter+0x3b6/0x430
[   52.967072]  copy_page_to_iter+0x1cf/0x390
[   52.967073]  ? pagecache_get_page+0x26/0x200
[   52.967073]  generic_file_read_iter+0x620/0xaf0
[   52.967073]  ? avc_has_perm+0x12e/0x200
[   52.967074]  ? avc_has_perm+0x34/0x200
[   52.967074]  ? sched_clock+0x5/0x10
[   52.967074]  __vfs_read+0x112/0x190
[   52.967074]  vfs_read+0x8c/0x140
[   52.967075]  kernel_read+0x2c/0x40
[   52.967075]  prepare_binprm+0x121/0x230
[   52.967075]  __do_execve_file.isra.32+0x56f/0x930
[   52.967076]  ? __do_execve_file.isra.32+0x140/0x930
[   52.967076]  __x64_sys_execve+0x44/0x50
[   52.967076]  do_syscall_64+0x5b/0x190
[   52.967077]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   52.967077] RIP: 0033:0x7f41abd898c7
[   52.967078] Code: ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de 64 41 89 32 eb db 66 2e 0f 1f 84 00 00 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 98 05 30 00 f7 d8 64 89 02
[   52.967078] RSP: 002b:00007ffc34b18f48 EFLAGS: 00000207 ORIG_RAX: 000000000000003b
[   52.967078] RAX: ffffffffffffffda RBX: 00007ffc34b190a0 RCX: 00007f41abd898c7
[   52.967079] RDX: 00005573e1da99d0 RSI: 00007ffc34b190a0 RDI: 00007ffc34b194a0
[   52.967079] RBP: 00005573e0895140 R08: 0000000000000008 R09: 0000000000000383
[   52.967080] R10: 0000000000000008 R11: 0000000000000207 R12: 00005573e1da99d0
[   52.967080] R13: 0000000000000007 R14: 000000000000000c R15: 00007ffc34b1ad10
[   52.967080] Kernel panic - not syncing: Hard LOCKUP
[   52.967081] CPU: 21 PID: 1127 Comm: systemd-udevd Not tainted 4.19.0-rc3-xpfo+ #3
[   52.967081] Hardware name: Oracle Corporation ORACLE SERVER X7-2/ASM, MB, X7-2, BIOS 41017600 10/06/2017
[   52.967081] Call Trace:
[   52.967082]  <NMI>
[   52.967082]  dump_stack+0x5a/0x73
[   52.967082]  panic+0xe8/0x25c
[   52.967082]  nmi_panic+0x37/0x40
[   52.967083]  watchdog_overflow_callback+0xef/0x110
[   52.967083]  __perf_event_overflow+0x51/0xe0
[   52.967083]  intel_pmu_handle_irq+0x222/0x4c0
[   52.967084]  ? _raw_spin_unlock+0x24/0x30
[   52.967084]  ? ghes_copy_tofrom_phys+0xf2/0x1a0
[   52.967084]  ? ghes_read_estatus+0x91/0x160
[   52.967085]  perf_event_nmi_handler+0x2e/0x50
[   52.967085]  nmi_handle+0x9a/0x180
[   52.967085]  ? nmi_handle+0x5/0x180
[   52.967086]  default_do_nmi+0xca/0x120
[   52.967086]  do_nmi+0x100/0x160
[   52.967086]  end_repeat_nmi+0x16/0x50
[   52.967086] RIP: 0010:queued_spin_lock_slowpath+0xf6/0x1e0
[   52.967087] Code: 48 03 34 c5 80 97 12 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 0f 84 93 00 00
[   52.967087] RSP: 0018:ffffc9001cc83a00 EFLAGS: 00000002
[   52.967088] RAX: 0000000000340101 RBX: ffffea06c16292e8 RCX: 0000000000580000
[   52.967088] RDX: ffff88603c9e3980 RSI: 0000000000000000 RDI: ffffea06c16292e8
[   52.967089] RBP: ffffea06c1629300 R08: 0000000000000001 R09: 0000000000000000
[   52.967089] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88c02765a000
[   52.967089] R13: 0000000000000000 R14: ffff8860152a0d00 R15: 0000000000000000
[   52.967090]  ? queued_spin_lock_slowpath+0xf6/0x1e0
[   52.967090]  ? queued_spin_lock_slowpath+0xf6/0x1e0
[   52.967090]  </NMI>
[   52.967091]  do_raw_spin_lock+0x6d/0xa0
[   52.967091]  _raw_spin_lock+0x53/0x70
[   52.967091]  ? xpfo_do_map+0x1b/0x52
[   52.967092]  xpfo_do_map+0x1b/0x52
[   52.967092]  xpfo_spurious_fault+0xac/0xae
[   52.967092]  __do_page_fault+0x3cc/0x4e0
[   52.967092]  ? __lock_acquire.isra.31+0x165/0x710
[   52.967093]  do_page_fault+0x32/0x180
[   52.967093]  page_fault+0x1e/0x30
[   52.967093] RIP: 0010:memcpy_erms+0x6/0x10
[   52.967094] Code: 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   52.967094] RSP: 0018:ffffc9001cc83bb8 EFLAGS: 00010246
[   52.967095] RAX: ffff8860299d0f00 RBX: ffffc9001cc83dc8 RCX: 0000000000000080
[   52.967095] RDX: 0000000000000080 RSI: ffff88c02765a000 RDI: ffff8860299d0f00
[   52.967096] RBP: 0000000000000080 R08: ffffc9001cc83d90 R09: 0000000000000001
[   52.967096] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000080
[   52.967096] R13: 0000000000000080 R14: 0000000000000000 R15: ffff88c02765a080
[   52.967097]  _copy_to_iter+0x3b6/0x430
[   52.967097]  copy_page_to_iter+0x1cf/0x390
[   52.967097]  ? pagecache_get_page+0x26/0x200
[   52.967098]  generic_file_read_iter+0x620/0xaf0
[   52.967098]  ? avc_has_perm+0x12e/0x200
[   52.967098]  ? avc_has_perm+0x34/0x200
[   52.967098]  ? sched_clock+0x5/0x10
[   52.967099]  __vfs_read+0x112/0x190
[   52.967099]  vfs_read+0x8c/0x140
[   52.967099]  kernel_read+0x2c/0x40
[   52.967100]  prepare_binprm+0x121/0x230
[   52.967100]  __do_execve_file.isra.32+0x56f/0x930
[   52.967100]  ? __do_execve_file.isra.32+0x140/0x930
[   52.967101]  __x64_sys_execve+0x44/0x50
[   52.967101]  do_syscall_64+0x5b/0x190
[   52.967101]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   52.967102] RIP: 0033:0x7f41abd898c7
[   52.967102] Code: ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de 64 41 89 32 eb db 66 2e 0f 1f 84 00 00 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 98 05 30 00 f7 d8 64 89 02
[   52.967102] RSP: 002b:00007ffc34b18f48 EFLAGS: 00000207 ORIG_RAX: 000000000000003b
[   52.967103] RAX: ffffffffffffffda RBX: 00007ffc34b190a0 RCX: 00007f41abd898c7
[   52.967103] RDX: 00005573e1da99d0 RSI: 00007ffc34b190a0 RDI: 00007ffc34b194a0
[   52.967104] RBP: 00005573e0895140 R08: 0000000000000008 R09: 0000000000000383
[   52.967104] R10: 0000000000000008 R11: 0000000000000207 R12: 00005573e1da99d0
[   52.967104] R13: 0000000000000007 R14: 000000000000000c R15: 00007ffc34b1ad10
[   54.001888] Shutting down cpus with NMI
[   54.001889] Kernel Offset: disabled
[   54.860701] ---[ end Kernel panic - not syncing: Hard LOCKUP ]---
[   54.867733] ------------[ cut here ]------------
[   54.867734] unchecked MSR access error: WRMSR to 0x83f (tried to write 0x00000000000000f6) at rIP: 0xffffffff81055864 (native_write_msr+0x4/0x20)
[   54.867734] Call Trace:
[   54.867734]  <IRQ>
[   54.867735]  native_apic_msr_write+0x2e/0x40
[   54.867735]  arch_irq_work_raise+0x28/0x40
[   54.867735]  irq_work_queue+0x69/0x70
[   54.867736]  printk_safe_log_store+0xd0/0xf0
[   54.867736]  printk+0x58/0x6f
[   54.867736]  __warn_printk+0x46/0x90
[   54.867737]  ? enqueue_task_fair+0x8e/0x760
[   54.867737]  native_smp_send_reschedule+0x39/0x40
[   54.867737]  check_preempt_curr+0x75/0xb0
[   54.867738]  ttwu_do_wakeup+0x19/0x190
[   54.867738]  try_to_wake_up+0x21e/0x4f0
[   54.867738]  __wake_up_common+0x9d/0x190
[   54.867738]  ep_poll_callback+0xd5/0x370
[   54.867739]  ? ep_poll_callback+0x2b5/0x370
[   54.867739]  __wake_up_common+0x9d/0x190
[   54.867739]  __wake_up_common_lock+0x7a/0xc0
[   54.867740]  irq_work_run_list+0x4c/0x70
[   54.867740]  smp_call_function_interrupt+0x59/0x110
[   54.867740]  call_function_interrupt+0xf/0x20
[   54.867741]  </IRQ>
[   54.867741]  <NMI>
[   54.867741] RIP: 0010:panic+0x206/0x25c
[   54.867742] Code: 83 3d 11 83 c9 01 00 74 05 e8 d2 87 02 00 48 c7 c6 80 15 d1 82 48 c7 c7 80 fe 06 82 31 c0 e8 71 ed 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 de 11 0e 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 ce 82
[   54.867742] RSP: 0018:fffffe00003a4b58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff03
[   54.867743] RAX: 0000000000000038 RBX: fffffe00003a4e00 RCX: 0000000000000000
[   54.867743] RDX: 0000000000000000 RSI: 0000000000000038 RDI: ffff88603c9d5d08
[   54.867744] RBP: fffffe00003a4bc8 R08: 0000000000000000 R09: ffff88603c9d5d47
[   54.867744] R10: 000000000000000b R11: 0000000000000000 R12: ffffffff8207b979
[   54.867744] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88603c80f560
[   54.867745]  ? panic+0x1ff/0x25c
[   54.867745]  nmi_panic+0x37/0x40
[   54.867745]  watchdog_overflow_callback+0xef/0x110
[   54.867746]  __perf_event_overflow+0x51/0xe0
[   54.867746]  intel_pmu_handle_irq+0x222/0x4c0
[   54.867746]  ? _raw_spin_unlock+0x24/0x30
[   54.867747]  ? ghes_copy_tofrom_phys+0xf2/0x1a0
[   54.867747]  ? ghes_read_estatus+0x91/0x160
[   54.867747]  perf_event_nmi_handler+0x2e/0x50
[   54.867748]  nmi_handle+0x9a/0x180
[   54.867748]  ? nmi_handle+0x5/0x180
[   54.867748]  default_do_nmi+0xca/0x120
[   54.867748]  do_nmi+0x100/0x160
[   54.867749]  end_repeat_nmi+0x16/0x50
[   54.867749] RIP: 0010:queued_spin_lock_slowpath+0xf6/0x1e0
[   54.867750] Code: 48 03 34 c5 80 97 12 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 0f 84 93 00 00
[   54.867750] RSP: 0018:ffffc9001cc83a00 EFLAGS: 00000002
[   54.867751] RAX: 0000000000340101 RBX: ffffea06c16292e8 RCX: 0000000000580000
[   54.867751] RDX: ffff88603c9e3980 RSI: 0000000000000000 RDI: ffffea06c16292e8
[   54.867751] RBP: ffffea06c1629300 R08: 0000000000000001 R09: 0000000000000000
[   54.867752] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88c02765a000
[   54.867752] R13: 0000000000000000 R14: ffff8860152a0d00 R15: 0000000000000000
[   54.867752]  ? queued_spin_lock_slowpath+0xf6/0x1e0
[   54.867753]  ? queued_spin_lock_slowpath+0xf6/0x1e0
[   54.867753]  </NMI>
[   54.867753]  do_raw_spin_lock+0x6d/0xa0
[   54.867754]  _raw_spin_lock+0x53/0x70
[   54.867754]  ? xpfo_do_map+0x1b/0x52
[   54.867754]  xpfo_do_map+0x1b/0x52
[   54.867754]  xpfo_spurious_fault+0xac/0xae
[   54.867755]  __do_page_fault+0x3cc/0x4e0
[   54.867755]  ? __lock_acquire.isra.31+0x165/0x710
[   54.867755]  do_page_fault+0x32/0x180
[   54.867756]  page_fault+0x1e/0x30
[   54.867756] RIP: 0010:memcpy_erms+0x6/0x10
[   54.867757] Code: 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   54.867757] RSP: 0018:ffffc9001cc83bb8 EFLAGS: 00010246
[   54.867758] RAX: ffff8860299d0f00 RBX: ffffc9001cc83dc8 RCX: 0000000000000080
[   54.867758] RDX: 0000000000000080 RSI: ffff88c02765a000 RDI: ffff8860299d0f00
[   54.867758] RBP: 0000000000000080 R08: ffffc9001cc83d90 R09: 0000000000000001
[   54.867759] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000080
[   54.867759] R13: 0000000000000080 R14: 0000000000000000 R15: ffff88c02765a080
[   54.867759]  _copy_to_iter+0x3b6/0x430
[   54.867760]  copy_page_to_iter+0x1cf/0x390
[   54.867760]  ? pagecache_get_page+0x26/0x200
[   54.867760]  generic_file_read_iter+0x620/0xaf0
[   54.867761]  ? avc_has_perm+0x12e/0x200
[   54.867761]  ? avc_has_perm+0x34/0x200
[   54.867761]  ? sched_clock+0x5/0x10
[   54.867761]  __vfs_read+0x112/0x190
[   54.867762]  vfs_read+0x8c/0x140
[   54.867762]  kernel_read+0x2c/0x40
[   54.867762]  prepare_binprm+0x121/0x230
[   54.867763]  __do_execve_file.isra.32+0x56f/0x930
[   54.867763]  ? __do_execve_file.isra.32+0x140/0x930
[   54.867763]  __x64_sys_execve+0x44/0x50
[   54.867764]  do_syscall_64+0x5b/0x190
[   54.867764]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   54.867764] RIP: 0033:0x7f41abd898c7
[   54.867765] Code: ff ff 76 df 89 c6 f7 de 64 41 89 32 eb d5 89 c6 f7 de 64 41 89 32 eb db 66 2e 0f 1f 84 00 00 00 00 00 90 b8 3b 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 02 f3 c3 48 8b 15 98 05 30 00 f7 d8 64 89 02
[   54.867765] RSP: 002b:00007ffc34b18f48 EFLAGS: 00000207 ORIG_RAX: 000000000000003b
[   54.867766] RAX: ffffffffffffffda RBX: 00007ffc34b190a0 RCX: 00007f41abd898c7
[   54.867766] RDX: 00005573e1da99d0 RSI: 00007ffc34b190a0 RDI: 00007ffc34b194a0
[   54.867767] RBP: 00005573e0895140 R08: 0000000000000008 R09: 0000000000000383
[   54.867767] R10: 0000000000000008 R11: 0000000000000207 R12: 00005573e1da99d0
[   54.867767] R13: 0000000000000007 R14: 000000000000000c R15: 00007ffc34b1ad10
[   54.867768] sched: Unexpected reschedule of offline CPU#4!
[   54.867768] WARNING: CPU: 21 PID: 1127 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x39/0x40
[   54.867768] Modules linked in: crc32c_intel nvme nvme_core igb megaraid_sas ahci i2c_algo_bit bnxt_en libahci i2c_core libata dca dm_mirror dm_region_hash dm_log dm_mod
[   54.867773] CPU: 21 PID: 1127 Comm: systemd-udevd Not tainted 4.19.0-rc3-xpfo+ #3
[   54.867773] Hardware name: Oracle Corporation ORACLE SERVER X7-2/ASM, MB, X7-2, BIOS 41017600 10/06/2017
[   54.867773] RIP: 0010:native_smp_send_reschedule+0x39/0x40
[   54.867774] Code: 0f 92 c0 84 c0 74 15 48 8b 05 13 84 0f 01 be fd 00 00 00 48 8b 40 30 e9 e5 16 bc 00 89 fe 48 c7 c7 d8 48 06 82 e8 67 71 03 00 <0f> 0b c3 0f 1f 40 00 0f 1f 44 00 00 53 be 20 00 48 00 48 89 fb 48
[   54.867774] RSP: 0018:ffff88603c803db8 EFLAGS: 00010086
[   54.867775] RAX: 0000000000000000 RBX: ffff88603a7e2c80 RCX: 0000000000000000
[   54.867775] RDX: 0000000000000000 RSI: 0000000000001277 RDI: ffff88603c9d5d08
[   54.867776] RBP: ffff88603a7e2c80 R08: 0000000000000000 R09: 0000000000000000
[   54.867776] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8860250a8000
[   54.867776] R13: ffff88603c803e00 R14: 0000000000000000 R15: ffff88603a7e2c98
[   54.867777] FS:  00007f41ad1658c0(0000) GS:ffff88603c800000(0000) knlGS:0000000000000000
[   54.867777] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   54.867778] CR2: ffff88c02765a000 CR3: 00000060252e4003 CR4: 00000000007606e0
[   54.867778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   54.867778] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   54.867779] PKRU: 55555554
[   54.867779] Call Trace:
[   54.867779]  <IRQ>
[   54.867779]  check_preempt_curr+0x75/0xb0
[   54.867780]  ttwu_do_wakeup+0x19/0x190
[   54.867780]  try_to_wake_up+0x21e/0x4f0
[   54.867780]  __wake_up_common+0x9d/0x190
[   54.867781]  ep_poll_callback+0xd5/0x370
[   54.867781]  ? ep_poll_callback+0x2b5/0x370
[   54.867781]  __wake_up_common+0x9d/0x190
[   54.867782]  __wake_up_common_lock+0x7a/0xc0
[   54.867782]  irq_work_run_list+0x4c/0x70
[   54.867782]  smp_call_function_interrupt+0x59/0x110
[   54.867782]  call_function_interrupt+0xf/0x20
[   54.867783]  </IRQ>
[   54.867783]  <NMI>
[   54.867783] RIP: 0010:panic+0x206/0x25c
[   54.867784] Code: 83 3d 11 83 c9 01 00 74 05 e8 d2 87 02 00 48 c7 c6 80 15 d1 82 48 c7 c7 80 fe 06 82 31 c0 e8 71 ed 06 00 fb 66 0f 1f 44 00 00 <45> 31 e4 e8 de 11 0e 00 4d 39 ec 7c 1e 41 83 f6 01 48 8b 05 ce 82
[   54.867784] RSP: 0018:fffffe00003a4b58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff03
[   54.867785] RAX: 0000000000000038 RBX: fffffe00003a4e00 RCX: 0000000000000000
[   54.867785] RDX: 0000000000000000 RSI: 0000000000000038 RDI: ffff88603c9d5d08
[   54.867786] RBP: fffffe00003a4bc8 R08: 0000000000000000 R09: ffff88603c9d5d47
[   54.867786] R10: 000000000000000b R11: 0000000000000000 R12: ffffffff8207b979
[   54.867786] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88603c80f560
[   54.867787]  ? panic+0x1ff/0x25c
[   54.867787]  nmi_panic+0x37/0x40
[   54.867787]  watchdog_overflow_callback+0xef/0x110
[   54.867787]  __perf_event_overflow+0x51/0xe0
[   54.867788]  intel_pmu_handle_irq+0x222/0x4c0
[   54.867788]  ? _raw_spin_unlock+0x24/0x30
[   54.867788]  ? ghes_copy_tofrom_phys+0xf2/0x1a0
[   54.867789]  ? ghes_read_estatus+0x91/0x160
[   54.867789]  perf_event_nmi_handler+0x2e/0x50
[   54.867789]  nmi_handle+0x9a/0x180
[   54.867790]  ? nmi_handle+0x5/0x180
[   54.867790]  default_do_nmi+0xca/0x120
[   54.867790]  do_nmi+0x100/0x160
[   54.867791]  end_repeat_nmi+0x16/0x50
[   54.867791] RIP: 0010:queued_spin_lock_slowpath+0xf6/0x1e0
[   54.867791] Code: 48 03 34 c5 80 97 12 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 <8b> 07 66 85 c0 75 f7 41 89 c0 66 45 31 c0 41 39 c8 0f 84 93 00 00
[   54.867792] RSP: 0018:ffffc9001cc83a00 EFLAGS: 00000002
[   54.867792] RAX: 0000000000340101 RBX: ffffea06c16292e8 RCX: 0000000000580000
[   54.867793] RDX: ffff88603c9e3980 RSI: 0000000000000000 RDI: ffffea06c16292e8
[   54.867793] RBP: ffffea06c1629300 R08: 0000000000000001 R09: 0000000000000000
[   54.867793] R1
[   54.867794] Lost 48 message(s)!

--
Khalid

That's good news! So the lock optimizations seem to help.

I didn't test the version with TLB flushes, because it's clear that the overhead is so bad that no one wants to use this.

It shouldn't lock up though, so maybe there is still a race condition somewhere. I'll give this a spin on my end later this week.

Thanks for trying this out!

Julian
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

I don't think we can ignore the vulnerability caused by not flushing stale TLB entries. On a mostly idle system, TLB entries hang around long enough to make it fairly easy to exploit this. I was able to use the additional test in lkdtm module added by this patch series to successfully read pages unmapped from physmap by just waiting for system to become idle. A rogue program can simply monitor system load and mount its attack using ret2dir exploit when system is mostly idle. This brings us back to the prohibitive cost of TLB flushes. If we are unmapping a page from physmap every time the page is allocated to userspace, we are forced to incur the cost of TLB flushes in some way. Work Tycho was doing to implement Dave's suggestion can help here. Once Tycho has something working, I can measure overhead on my test machine. Tycho, I can help with your implementation if you need.

--
Khalid

I looked at Tycho's last patch with batch update from <https://lkml.org/lkml/2017/11/9/951>. I ported it on top of Julian's patches and got it working well enough to gather performance numbers. Here is what I see for system times on a machine with dual Xeon E5-2630 and 256GB of memory when running "make -j30 all" on 4.18.6 kernel (percentages are relative to base 4.19-rc8 kernel without xpfo):

Base 4.19-rc8 913.84s
4.19-rc8 + xpfo, no TLB flush 1027.985s (+12.5%)
4.19-rc8 + batch update, no TLB flush 970.39s (+6.2%)
4.19-rc8 + xpfo, TLB flush 8458.449s (+825.6%)
4.19-rc8 + batch update, TLB flush 4665.659s (+410.6%)

Batch update is significant improvement but we are starting so far behind baseline, it is still a huge slow down.

--
Khalid

There's some other stuff that Dave suggested that I didn't do; in particular coalesce xpfo bits instead of setting things once per page when mappings are shared, etc.

Perhaps that will help more?

I'm still stuck working on something else for now, but I hope to be able to participate more on this Soon (TM). Thanks for the testing!

Tycho

Also the problem is that depending on the workload everything may fit into the TLBs, so the temporary stale TLB entries may be around for a long time. Modern CPUs have very large TLBs, and good LRU policies. For the kernel entries with global bit set and which are used for something there may be no reason ever to evict.

Julian, I think you would need at least some quantitative perfmon data about TLB replacement rates in the kernel to show that it's "reasonable" instead of hand waving.

Most likely I suspect you would need a low frequency regular TLB flush for the global entries at least, which will increase the overhead again.

-Andi

That's a fair point. It definitely depends on the workload. My idle laptop gnome GUI session still causes ~40k dtlb-load-misses per second per core. My idle server (some shells, IRC client) still has ~8k dTLB load misses per second per core. Compiling something pushes this to millions of misses per second.

For comparison according to https://www.7-cpu.com/cpu/Skylake_X.html SKX can fit 1536 entries into its L2 dTLB.

Given the tiny experiment above, I don't think this is necessary except for highly special usecases. If stale TLB entries are a concern, the better intermediate step is to do INVLPG on the core that modified the page table.

And even with these shortcomings, XPFO severely limits the data an attacker can leak from the kernel.

Julian
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B

Looks like the performance impact can be whole lot worse. On my test system with 2 Xeon Platinum 8160 (HT enabled) CPUs and 768 GB of memory, I am seeing very high penalty with XPFO when building 4.18.6 kernel sources with "make -j60":

No XPFO patch XPFO patch(No TLB flush) XPFO(TLB Flush)
sys time 52m 54.036s 55m 47.897s 434m 8.645s

That is ~8% worse with TLB flush disabled and ~720% worse with TLB flush enabled. This test was with kernel sources being compiled on an ext4 filesystem. XPFO seems to affect ext2 even more. With ext2 filesystem, impact was ~18.6% and ~900%.

--
Khalid

A long time ago, I gave a talk about precisely this at OLS (2005 I think). On PA-RISC we have a problem with inequivalent aliasing in the
page cache (same physical page with two different virtual addresses
modulo 4MB) which causes a machine check if it occurs. Architecturally, PA can move into the cache any page for which it has a mapping and the kernel offset map of every page causes an inequivalency if the same page is in use in user space. Of course, practically the caching machinery is too busy moving in and out pages we reference to have an interest in speculating on other pages it has a mapping for, so it almost never (the almost being a set of machine checks we see very occasionally in the latest and most aggressively cached and speculating CPUs). If this were implemented, we'd be interested in using it.

James

I am missing why we need this since the kernel can't access (SMAP) unless we go through to the copy/to/from interface or execute any of the user pages. Is it because of the dependency on the availability of those features?

Balbir Singh.

mappings, but that isn't relevant to what's being discussed here.

Davis is talking about the kernel Direct Map, which is a PRIV (i.e. kernel) mapping of all physical memory on the system, at VA = (base + PA). Since this mapping exists for all physical addresses, speculative load gadgets (and the processor's prefetch mechanism, etc.) can load arbitrary data even if it is only otherwise mapped into user space.

XPFO fixes this by unmapping the Direct Map translations when the page is allocated as a user page. The mapping is only restored:
1. temporarily re-if the kernel needs direct access to the page (i.e. to
zero it, access it from a device driver, etc),
2. when the page is freed

And in to doing, significantly reduces the amount of non-kernel data vulnerable to speculative execution attacks against the kernel. (and reduces what data can be loaded into the L1 data cache while in kernel mode, to be peeked at by the recent L1 Terminal Fault vulnerability).

Does that make sense?

Cheers,
- jonathan

(resending due to formatting issues)

SMAP protects against kernel accesses to non-PRIV (i.e. userspace) mappings, but that isn't relevant to what's being discussed here.

Davis is talking about the kernel Direct Map, which is a PRIV (i.e. kernel) mapping of all physical memory on the system, at
VA = (base + PA).
Since this mapping exists for all physical addresses, speculative load gadgets (and the processor's prefetch mechanism, etc.) can load arbitrary data even if it is only otherwise mapped into user space.

XPFO fixes this by unmapping the Direct Map translations when the page is allocated as a user page. The mapping is only restored:
1. temporarily if the kernel needs direct access to the page (i.e. to zero it, access it from a device driver, etc),
2. when the page is freed

And in so doing, significantly reduces the amount of non-kernel data vulnerable to speculative execution attacks against the kernel. (and reduces what data can be loaded into the L1 data cache while in kernel mode, to be peeked at by the recent L1 Terminal Fault vulnerability).

Does that make sense?

Cheers,
- jonathan

(apologies again; resending due to formatting issues)

SMAP protects against kernel accesses to non-PRIV (i.e. userspace) mappings, but that isn't relevant to what's being discussed here.

Davis is talking about the kernel Direct Map, which is a PRIV (i.e. kernel) mapping of all physical memory on the system, at
VA = (base + PA).
Since this mapping exists for all physical addresses, speculative load gadgets (and the processor's prefetch mechanism, etc.) can load arbitrary data even if it is only otherwise mapped into user space.

XPFO fixes this by unmapping the Direct Map translations when the page is allocated as a user page. The mapping is only restored:
1. temporarily if the kernel needs direct access to the page (i.e. to zero it, access it from a device driver, etc),
2. when the page is freed

And in so doing, significantly reduces the amount of non-kernel data vulnerable to speculative execution attacks against the kernel. (and reduces what data can be loaded into the L1 data cache while in kernel mode, to be peeked at by the recent L1 Terminal Fault vulnerability).

Does that make sense?

Cheers,
- jonathan

Load aribtrary data with no permission checks (strict RWX).

I see and there is no way for gadgets to invoke this path from user space to make their speculation successful? We still have to flush L1, indepenedent of whether XPFO is enabled or not right?

Balbir Singh.

Yes. And even with XPFO and L1 cache flushing enabled, there are more steps that need to be taken to reliably guard against information leaks using speculative execution.

Specifically, I'm looking into making certain allocations in the Linux kernel process-local to hide even more memory from prefetching.

Another puzzle piece is co-scheduling support that is relevant for systems with enabled hyperthreading: https://lwn.net/Articles/764461/

Julian
Amazon Development Center Germany GmbH
Berlin - Dresden - Aachen
main office: Krausenstr. 38, 10117 Berlin
Geschaeftsfuehrer: Dr. Ralf Herbrich, Christian Schlaeger
Ust-ID: DE289237879
Eingetragen am Amtsgericht Charlottenburg HRB 149173 B