Skip to contentSkip to conversation

Process-local memory allocations

RFC PATCH Tycho Andersen 4 participants 10 messages Nov 20, 2018 – Dec 19, 2018 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 22 lines · Julian Stecklina

In a world with processor information leak vulnerabilities, having a treasure trove of information available for leaking in the global kernel address space is starting to be a liability. The biggest offender is the linear mapping of all physical memory and there are already efforts (XPFO) to start addressing this. In this patch series, I'd like to propose breaking up the kernel address space further and introduce process-local mappings in the kernel.

The rationale is that there are allocations in the kernel containing data that should only be accessible when the kernel is executing in the context of a specific process. A prime example is KVM vCPU state. This patch series introduces process-local memory in the kernel address space by claiming a PGD entry for this specific purpose. Then it converts KVM on x86 to use these new primitives to store GPR and FPU registers of vCPUs. KVM is a good testing ground, because it makes sure userspace can only interact with a VM from a single process.

Process-local allocations in the kernel can be part of a robust L1TF mitigation strategy that works even with SMT enabled. The specific goal here is to make it harder for a random thread using cache load gadget (usually a bounds check of a system call argument plus array access suffices) to prefetch interesting data into the L1 cache and use L1TF to leak this data.

Quoted 3 lines

The patch set applies to kvm/next [1]. Feedback is very welcome, both about the general approach and the actual implementation. As far as testing goes, the KVM unit tests seem happy on Intel. AMD is only compile tested at the moment.

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

type thing? Could we share some infrastructure then? (I also didn't follow what happened to the patches Nadav was going to send that might replace prmem somehow.)

Tycho

#2 lore raw

Hi,

Quoted 23 lines · Tycho Andersen
Quoted 22 lines · Julian Stecklina

In a world with processor information leak vulnerabilities, having a treasure trove of information available for leaking in the global kernel address space is starting to be a liability. The biggest offender is the linear mapping of all physical memory and there are already efforts (XPFO) to start addressing this. In this patch series, I'd like to propose breaking up the kernel address space further and introduce process-local mappings in the kernel.

The rationale is that there are allocations in the kernel containing data that should only be accessible when the kernel is executing in the context of a specific process. A prime example is KVM vCPU state. This patch series introduces process-local memory in the kernel address space by claiming a PGD entry for this specific purpose. Then it converts KVM on x86 to use these new primitives to store GPR and FPU registers of vCPUs. KVM is a good testing ground, because it makes sure userspace can only interact with a VM from a single process.

Process-local allocations in the kernel can be part of a robust L1TF mitigation strategy that works even with SMT enabled. The specific goal here is to make it harder for a random thread using cache load gadget (usually a bounds check of a system call argument plus array access suffices) to prefetch interesting data into the L1 cache and use L1TF to leak this data.

Quoted 3 lines
Quoted 3 lines

The patch set applies to kvm/next [1]. Feedback is very welcome, both about the general approach and the actual implementation. As far as testing goes, the KVM unit tests seem happy on Intel. AMD is only compile tested at the moment.

Where is the full set of patches?
I'm sorry, I searched both KVM and LKML archives, but I couldn't find it.

Quoted 10 lines

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

type thing? Could we share some infrastructure then?

For what I can understand from the intro only, this seems to focus on "local" information, related to processes, while prmem was mostly aimed, at least at this stage, at system-level features, like LSM or SELinux. Those have probably little value, for protection from reading.
And they are used quite often, typically on a critical path.
The main thing is to prevent rogue writes, but reads are not a problem.
Hiding/unhiding them even from read operations might not be so useful.

However other components, like the kernel keyring, are used less frequently and might be worth protecting them even from read operations.

Quoted 3 lines

(I also didn't
follow what happened to the patches Nadav was going to send that might replace prmem somehow.)

I just replied to the old prmem thread - I stil lhave some doubts about the implementation, however my understanding is that I could replicate or at least base on those patches the very low level part of the write rare mechanism.

--
igor

#3 lore raw
Quoted 26 lines · Igor Stoppa

Hi,

Quoted 23 lines · Tycho Andersen
Quoted 22 lines · Julian Stecklina

In a world with processor information leak vulnerabilities, having a treasure trove of information available for leaking in the global kernel address space is starting to be a liability. The biggest offender is the linear mapping of all physical memory and there are already efforts (XPFO) to start addressing this. In this patch series, I'd like to propose breaking up the kernel address space further and introduce process-local mappings in the kernel.

The rationale is that there are allocations in the kernel containing data that should only be accessible when the kernel is executing in the context of a specific process. A prime example is KVM vCPU state. This patch series introduces process-local memory in the kernel address space by claiming a PGD entry for this specific purpose. Then it converts KVM on x86 to use these new primitives to store GPR and FPU registers of vCPUs. KVM is a good testing ground, because it makes sure userspace can only interact with a VM from a single process.

Process-local allocations in the kernel can be part of a robust L1TF mitigation strategy that works even with SMT enabled. The specific goal here is to make it harder for a random thread using cache load gadget (usually a bounds check of a system call argument plus array access suffices) to prefetch interesting data into the L1 cache and use L1TF to leak this data.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

The patch set applies to kvm/next [1]. Feedback is very welcome, both about the general approach and the actual implementation. As far as testing goes, the KVM unit tests seem happy on Intel. AMD is only compile tested at the moment.

Where is the full set of patches?
I'm sorry, I searched both KVM and LKML archives, but I couldn't find it.

It looks like they were only sent to kernel hardening, but not sent to the archives? I only see our replies here:

https://www.openwall.com/lists/kernel-hardening/

Julian, perhaps you can re-send with a CC to lkml as well?

Quoted 19 lines
Quoted 10 lines

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

type thing? Could we share some infrastructure then?

For what I can understand from the intro only, this seems to focus on "local" information, related to processes, while prmem was mostly aimed, at least at this stage, at system-level features, like LSM or SELinux. Those have probably little value, for protection from reading. And they are used quite often, typically on a critical path. The main thing is to prevent rogue writes, but reads are not a problem. Hiding/unhiding them even from read operations might not be so useful.

Quoted 2 lines

However other components, like the kernel keyring, are used less frequently and might be worth protecting them even from read operations.

Right, the goals are different, but the idea is basically the same. We allocate memory in some "special" way. I'm just wondering if we'll be adding more of these special ways in the future, and if it's worth synchronizing the APIs so that it's easy for people to use.

Quoted 8 lines
Quoted 3 lines

(I also didn't
follow what happened to the patches Nadav was going to send that might replace prmem somehow.)

I just replied to the old prmem thread - I stil lhave some doubts about the implementation, however my understanding is that I could replicate or at least base on those patches the very low level part of the write rare mechanism.

Cool, thanks.

Tycho

#4 lore raw
Quoted 4 lines · Tycho Andersen
Quoted 3 lines · Igor Stoppa

Hi,

On 21/11/2018 01:26, Tycho Andersen wrote:

[...]

Quoted 19 lines
Quoted 19 lines
Quoted 10 lines

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

type thing? Could we share some infrastructure then?

For what I can understand from the intro only, this seems to focus on "local" information, related to processes, while prmem was mostly aimed, at least at this stage, at system-level features, like LSM or SELinux. Those have probably little value, for protection from reading. And they are used quite often, typically on a critical path. The main thing is to prevent rogue writes, but reads are not a problem. Hiding/unhiding them even from read operations might not be so useful.

Quoted 7 lines
Quoted 2 lines

However other components, like the kernel keyring, are used less frequently and might be worth protecting them even from read operations.

Right, the goals are different, but the idea is basically the same. We allocate memory in some "special" way. I'm just wondering if we'll be adding more of these special ways in the future, and if it's worth synchronizing the APIs so that it's easy for people to use.

Yes, I agree. I do not see much use for this "locality" in most of the use cases that I have looked into, apart from maybe the kernel keyring, but it might be possible to add a "scope" property to a memory pool, if it is associated to some very specific code.

Doing it for system-level components, however, might introduce too big an overhead. In these cases, the scope would stay global.

But I really need to see the code, before I can say more.

--
igor

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 7 lines · Julian
Quoted 6 lines · Tycho Andersen
Quoted 2 lines

Where is the full set of patches?
I'm sorry, I searched both KVM and LKML archives, but I couldn't find it.

It looks like they were only sent to kernel hardening, but not sent to the archives? I only see our replies here:

Quoted 6 lines
Quoted 3 lines

https://www.openwall.com/lists/kernel-hardening/

Julian, perhaps you can re-send with a CC to lkml as well?

Will do. I messed up the threading, due to git send-email being hostile to me. Sorry for the confusion.

If you are not subscribed to the hardening ml, it will neither archive, nor forward to subscribers your emails.

This very mail you wrote is not archived either.

--
igor

#7 lore raw
Quoted 10 lines · Igor Stoppa
Quoted 9 lines · Julian
Quoted 8 lines · Tycho Andersen
Quoted 2 lines

Where is the full set of patches?
I'm sorry, I searched both KVM and LKML archives, but I couldn't find it.

It looks like they were only sent to kernel hardening, but not sent to the archives? I only see our replies here:

https://www.openwall.com/lists/kernel-hardening/

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Julian, perhaps you can re-send with a CC to lkml as well?

Will do. I messed up the threading, due to git send-email being hostile to me.
Sorry for the confusion.

If you are not subscribed to the hardening ml, it will neither archive, nor forward to subscribers your emails.

No. There's no requirement to be subscribed to kernel-hardening in order to be able to post, nor for messages to be archived.

Quoted 1 line

This very mail you wrote is not archived either.

Julian's message in fact did not appear on kernel-hardening, and the reason is that Amazon's mail servers - for reasons unknown to me (and to an Amazoner with whom I tried discussing it before) sometimes(?!) add the "Precedence: junk" header on messages. ezmlm-idx drops such messages by default (and this behavior is documented). I'm not eager to patch this out. ezmlm-idx also drops messages with "Precedence: bulk", and it sets that header on messages it sends to list subscribers, which helps against loops. Another reason is to avoid vacation auto-replies, which I think can use either header. It'd be weird to continue dropping "bulk", yet start accepting "junk", wouldn't it? But if this is unfixable on Amazon's end, I'll have to.

Alexander

#8 lore raw
Quoted 25 lines · Solar Designer
Quoted 10 lines · Igor Stoppa
Quoted 9 lines · Julian
Quoted 8 lines · Tycho Andersen
Quoted 2 lines

Where is the full set of patches?
I'm sorry, I searched both KVM and LKML archives, but I couldn't find it.

It looks like they were only sent to kernel hardening, but not sent to the archives? I only see our replies here:

https://www.openwall.com/lists/kernel-hardening/

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Julian, perhaps you can re-send with a CC to lkml as well?

Will do. I messed up the threading, due to git send-email being hostile to me.
Sorry for the confusion.

If you are not subscribed to the hardening ml, it will neither archive, nor forward to subscribers your emails.

No. There's no requirement to be subscribed to kernel-hardening in order to be able to post, nor for messages to be archived.

Quoted 1 line

This very mail you wrote is not archived either.

Quoted 11 lines

Julian's message in fact did not appear on kernel-hardening, and the reason is that Amazon's mail servers - for reasons unknown to me (and to an Amazoner with whom I tried discussing it before) sometimes(?!) add the "Precedence: junk" header on messages. ezmlm-idx drops such messages by default (and this behavior is documented). I'm not eager to patch this out. ezmlm-idx also drops messages with "Precedence: bulk", and it sets that header on messages it sends to list subscribers, which helps against loops. Another reason is to avoid vacation auto-replies, which I think can use either header. It'd be weird to continue dropping "bulk", yet start accepting "junk", wouldn't it? But if this is unfixable on Amazon's end, I'll have to.

OK, I've just patched the "junk" check out of ezmlm-reject (processing of messages to the posting addresses), but kept it in ezmlm-weed (processing of messages to the bounce addresses). I hope Julian's messages will be getting through now.

Alexander

#9 lore raw
In reply to Tycho Andersen

Tycho,

sorry for the late response, I just returned from vacation.

Quoted 17 lines · Tycho Andersen
Quoted 16 lines · Julian Stecklina

In a world with processor information leak vulnerabilities, having a treasure trove of information available for leaking in the global kernel address space is
starting to be a liability. The biggest offender is the linear mapping of all physical memory and there are already efforts (XPFO) to start addressing this. In this patch series, I'd like to propose breaking up the kernel address space further and introduce process-local mappings in the kernel.

The rationale is that there are allocations in the kernel containing data that should only be accessible when the kernel is executing in the context of a specific process. A prime example is KVM vCPU state. This patch series introduces process-local memory in the kernel address space by claiming a PGD entry for this specific purpose. Then it converts KVM on x86 to use these new primitives to store GPR and FPU registers of vCPUs. KVM is a good testing ground, because it makes sure userspace can only interact with a VM from a single process.

[...]

Quoted 2 lines

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

It's similar in the sense that it adds a new way to allocate memory with interesting properties. In that sense, it would be useful to have a common allocation function.

As far as usage is concerned, prmem and the process local memory is very different. From a quick glance at the prmem patchset, I see that it works for statically allocated memory. Process-local memory is by definition bound to the lifetime of a process (or address space to be more precise) and thus cannot be statically allocated. It's breaking up the global address space of the kernel and starts binding some allocations in the kernel to specific processes.

My goal is specifically L1TF mitigation and preventing random parts of the kernel from prefetching user data, but this approach is equally effective for ordinary info leak vulnerabilities.

Quoted 2 lines

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I

Unmapped should really be unmapped, i.e. not present in the page table. Having it read-only defeats the purpose.

Quoted 7 lines

wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

type thing? Could we share some infrastructure then? (I also didn't follow what happened to the patches Nadav was going to send that might replace prmem somehow.)

When writing the patch series, I had the feeling that the whole bookkeeping of what is allocated where is something that could be abstracted and re-used. Maybe there is already such an abstraction layer in the kernel and I just didn't find it.

Julian

#10 lore raw
Quoted 43 lines · Julian Stecklina

Tycho,

sorry for the late response, I just returned from vacation.

Quoted 17 lines · Tycho Andersen
Quoted 16 lines · Julian Stecklina

In a world with processor information leak vulnerabilities, having a treasure trove of information available for leaking in the global kernel address space is
starting to be a liability. The biggest offender is the linear mapping of all physical memory and there are already efforts (XPFO) to start addressing this. In this patch series, I'd like to propose breaking up the kernel address space further and introduce process-local mappings in the kernel.

The rationale is that there are allocations in the kernel containing data that should only be accessible when the kernel is executing in the context of a specific process. A prime example is KVM vCPU state. This patch series introduces process-local memory in the kernel address space by claiming a PGD entry for this specific purpose. Then it converts KVM on x86 to use these new primitives to store GPR and FPU registers of vCPUs. KVM is a good testing ground, because it makes sure userspace can only interact with a VM from a single process.

[...]

Quoted 2 lines

This seems similar in spirit to prmem:
https://lore.kernel.org/lkml/[email protected]/T/#u

It's similar in the sense that it adds a new way to allocate memory with interesting properties. In that sense, it would be useful to have a common allocation function.

As far as usage is concerned, prmem and the process local memory is very different. From a quick glance at the prmem patchset, I see that it works for statically allocated memory. Process-local memory is by definition bound to the lifetime of a process (or address space to be more precise) and thus cannot be statically allocated. It's breaking up the global address space of the kernel and starts binding some allocations in the kernel to specific processes.

My goal is specifically L1TF mitigation and preventing random parts of the kernel from prefetching user data, but this approach is equally effective for ordinary info leak vulnerabilities.

Quoted 5 lines
Quoted 2 lines

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes. I

Unmapped should really be unmapped, i.e. not present in the page table. Having it read-only defeats the purpose.

Yeah, sorry about the crappy notation, the parens were supposed to indicate the prmem bits, and the non-parens were this patch's bits.

Quoted 4 lines
Quoted 4 lines

wonder if we should merge the APIs in to one

spmemalloc(size, flags, PRLOCAL)

Quoted 8 lines
Quoted 3 lines

type thing? Could we share some infrastructure then? (I also didn't follow what happened to the patches Nadav was going to send that might replace prmem somehow.)

When writing the patch series, I had the feeling that the whole bookkeeping of what is allocated where is something that could be abstracted and re-used. Maybe there is already such an abstraction layer in the kernel and I just didn't find it.

Yeah, page_ext isn't a good fit, since you don't want to keep track of this metadata for most pages, just ones that have this special property. Hence the comparison to prmem, that series also wants to track special properties of some subset of pages.

Tycho

#11 lore raw
In reply to Julian Stecklina

Hi,

Quoted 3 lines · Julian Stecklina

As far as usage is concerned, prmem and the process local memory is very different. From a quick glance at the prmem patchset, I see that it works for statically allocated memory.

I suspect you have looked only at the latest patchset, which is a subset of the whole feature-set. It was agreed that I would first address the integration of write-rare for statically allocated memory, and so I did.

But the origin of this work actually resides in protecting the SELinux policyDB, which is entirely dynamically allocated.

Quoted 5 lines

Process-local memory is by
definition bound to the lifetime of a process (or address space to be more precise) and thus cannot be statically allocated. It's breaking up the global address space of the kernel and starts binding some allocations in the kernel to specific processes.

yes, I agree with the explanation you give, however the starting assumption (prmem is only static) is not correct.

Quoted 7 lines

My goal is specifically L1TF mitigation and preventing random parts of the kernel from prefetching user data, but this approach is equally effective for ordinary info leak vulnerabilities.

Quoted 2 lines

Basically, we have some special memory that we want to leave unmapped (read only) most of the time, but map it (writable) sometimes.

This would apply nicely also to kernel data that is particularly sensitive and one would not want to expose to random (possibly buggy) reads.

Quoted 2 lines

Unmapped should really be unmapped, i.e. not present in the page table. Having it read-only defeats the purpose.

Given the current implementation (for x86_64) which uses an alternate mapping for writable, it should not be too difficult to use a similar mechanism for implementing a sort of alternative to a TEE data-vault.

Instead of using a special function to alter normally readable data, it would use a special function to read/write normally unaccessible data.

But, before getting into that, I'd prefer to first get the write rare static merged or at least ACKed.

--
igor