Skip to contentSkip to conversation

Restricted kernel address spaces

LSF/MM/BPF TOPIC Mike Rapoport 3 participants 7 messages Feb 6, 2020 – Feb 18, 2020 Memory management

lore
Share & tools
lore
#0 lore raw

Restricted mappings in the kernel mode may improve mitigation of hardware speculation vulnerabilities and minimize the damage exploitable kernel bugs can cause.

There are several ongoing efforts to use restricted address spaces in Linux kernel for various use cases:
* speculation vulnerabilities mitigation in KVM [1]
* support for memory areas visible only in a single owning context, or more generically, a memory areas with more restrictive protection that the defaults ("secret" memory) [2], [3], [4]
* hardening of the Linux containers [ no reference yet :) ]

Last year we had vague ideas and possible directions, this year we have several real challenges and design decisions we'd like to discuss:

* "Secret" memory userspace APIs

Should such API follow "native" MM interfaces like mmap(), mprotect(), madvise() or it would be better to use a file descriptor , e.g. like memfd-create does?

MM "native" APIs would require VM_something flag and probably a page flag or page_ext. With file-descriptor VM_SPECIAL and custom implementation of .mmap() and .fault() would suffice. On the other hand, mmap() and mprotect() seem better fit semantically and they could be more easily adopted by the userspace.

* Direct/linear map fragmentation

Whenever we want to drop some mappings from the direct map or even change the protection bits for some memory area, the gigantic and huge pages that comprise the direct map need to be broken and there's no THP for the kernel page tables to collapse them back. Moreover, the existing API defined in <asm/set_memory.h> by several architectures do not really presume it would be widely used.

For the "secret" memory use-case the fragmentation can be minimized by caching large pages, use them to satisfy smaller "secret" allocations and than collapse them back once the "secret" memory is freed. Another possibility is to pre-allocate physical memory at boot time.

Yet another idea is to make page allocator aware of the direct map layout.

* Kernel page table management

Currently we presume that only one kernel page table exists (well, mostly) and the page table abstraction is required only for the user page tables. As such, we presume that 'page table == struct mm_struct' and the mm_struct is used all over by the operations that manage the page tables.

The management of the restricted address space in the kernel requires ability to create, update and remove kernel contexts the same way we do for the userspace.

One way is to overload the mm_struct, like EFI and text poking did. But it is quite an overkill, because most of the mm_struct contains information required to manage user mappings.

My suggestion is to introduce a first class abstraction for the page table and then it could be used in the same way for user and kernel context management. For now I have a very basic POC that slitted several fields from the mm_struct into a new 'struct pg_table' [5]. This new abstraction can be used e.g. by PTI implementation of the page table cloning and the KVM ASI work.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/20200130162340.GA14232@rapoport-lnx/
[5] https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=pg_table/v0.0

Signature · 2 lines
Sincerely yours,
Mike.
#1 lore raw
Quoted 13 lines · Mike Rapoport

Restricted mappings in the kernel mode may improve mitigation of hardware speculation vulnerabilities and minimize the damage exploitable kernel bugs can cause.

There are several ongoing efforts to use restricted address spaces in Linux kernel for various use cases:
* speculation vulnerabilities mitigation in KVM [1]
* support for memory areas visible only in a single owning context, or more generically, a memory areas with more restrictive protection that the defaults ("secret" memory) [2], [3], [4]
* hardening of the Linux containers [ no reference yet :) ]

Quoted 8 lines

Last year we had vague ideas and possible directions, this year we have several real challenges and design decisions we'd like to discuss:

* "Secret" memory userspace APIs

Should such API follow "native" MM interfaces like mmap(), mprotect(), madvise() or it would be better to use a file descriptor , e.g. like memfd-create does?

I don't really see a point in such file-descriptor. It suppose to be very private secret data. What functionality that provide a file descriptor do you see valuable in this scenario?

File descriptor makes it easier to spill the secrets to other process: over fork(), UNIX socket or via /proc/PID/fd/.

Quoted 5 lines

MM "native" APIs would require VM_something flag and probably a page flag or page_ext. With file-descriptor VM_SPECIAL and custom implementation of .mmap() and .fault() would suffice. On the other hand, mmap() and mprotect() seem better fit semantically and they could be more easily adopted by the userspace.

You mix up implementation and interface. You can provide an interface which doesn't require a file descriptor, but still use a magic file internally to the VMA distinct.

Quoted 9 lines

* Direct/linear map fragmentation

Whenever we want to drop some mappings from the direct map or even change the protection bits for some memory area, the gigantic and huge pages that comprise the direct map need to be broken and there's no THP for the kernel page tables to collapse them back. Moreover, the existing API defined in <asm/set_memory.h> by several architectures do not really presume it would be widely used.

Quoted 4 lines

For the "secret" memory use-case the fragmentation can be minimized by caching large pages, use them to satisfy smaller "secret" allocations and than collapse them back once the "secret" memory is freed. Another possibility is to pre-allocate physical memory at boot time.

I would rather go with pre-allocation path. At least at first. We always can come up with more dynamic and complicated solution later if the interface would be wildly adopted.

Quoted 9 lines

Yet another idea is to make page allocator aware of the direct map layout.

* Kernel page table management

Currently we presume that only one kernel page table exists (well, mostly) and the page table abstraction is required only for the user page tables. As such, we presume that 'page table == struct mm_struct' and the mm_struct is used all over by the operations that manage the page tables.

Quoted 7 lines

The management of the restricted address space in the kernel requires ability to create, update and remove kernel contexts the same way we do for the userspace.

One way is to overload the mm_struct, like EFI and text poking did. But it is quite an overkill, because most of the mm_struct contains information required to manage user mappings.

In what way is it overkill? Just memory overhead? How many of such contexts do you expect to see in the system?

Quoted 19 lines

My suggestion is to introduce a first class abstraction for the page table and then it could be used in the same way for user and kernel context management. For now I have a very basic POC that slitted several fields from the mm_struct into a new 'struct pg_table' [5]. This new abstraction can be used e.g. by PTI implementation of the page table cloning and the KVM ASI work.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/20200130162340.GA14232@rapoport-lnx/
[5] https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=pg_table/v0.0

Signature · 2 lines
Sincerely yours,
Mike.
Signature · 1 line
 Kirill A. Shutemov
#2 lore raw
Quoted 23 lines · Kirill A. Shutemov
Quoted 13 lines · Mike Rapoport

Restricted mappings in the kernel mode may improve mitigation of hardware speculation vulnerabilities and minimize the damage exploitable kernel bugs can cause.

There are several ongoing efforts to use restricted address spaces in Linux kernel for various use cases:
* speculation vulnerabilities mitigation in KVM [1]
* support for memory areas visible only in a single owning context, or more generically, a memory areas with more restrictive protection that the defaults ("secret" memory) [2], [3], [4]
* hardening of the Linux containers [ no reference yet :) ]

Quoted 8 lines

Last year we had vague ideas and possible directions, this year we have several real challenges and design decisions we'd like to discuss:

* "Secret" memory userspace APIs

Should such API follow "native" MM interfaces like mmap(), mprotect(), madvise() or it would be better to use a file descriptor , e.g. like memfd-create does?

Quoted 6 lines

I don't really see a point in such file-descriptor. It suppose to be very private secret data. What functionality that provide a file descriptor do you see valuable in this scenario?

File descriptor makes it easier to spill the secrets to other process: over fork(), UNIX socket or via /proc/PID/fd/.

On the other hand it is may be desired to share a secret between several processes. Then UNIX socket or fork() actually become handy.

Quoted 9 lines
Quoted 5 lines

MM "native" APIs would require VM_something flag and probably a page flag or page_ext. With file-descriptor VM_SPECIAL and custom implementation of .mmap() and .fault() would suffice. On the other hand, mmap() and mprotect() seem better fit semantically and they could be more easily adopted by the userspace.

You mix up implementation and interface. You can provide an interface which doesn't require a file descriptor, but still use a magic file internally to the VMA distinct.

If I understand correctly, if we go with mmap(MAP_SECRET) example, the mmap() would implicitly create a magic file with its .mmap() and .fault() implementing the protection? That's a possibility. But then, if we already have a file, why not let user get a handle for it and allow fine grained control over its sharing between processes?

Quoted 9 lines
Quoted 9 lines

* Direct/linear map fragmentation

Whenever we want to drop some mappings from the direct map or even change the protection bits for some memory area, the gigantic and huge pages that comprise the direct map need to be broken and there's no THP for the kernel page tables to collapse them back. Moreover, the existing API defined in <asm/set_memory.h> by several architectures do not really presume it would be widely used.

Quoted 8 lines
Quoted 4 lines

For the "secret" memory use-case the fragmentation can be minimized by caching large pages, use them to satisfy smaller "secret" allocations and than collapse them back once the "secret" memory is freed. Another possibility is to pre-allocate physical memory at boot time.

I would rather go with pre-allocation path. At least at first. We always can come up with more dynamic and complicated solution later if the interface would be wildly adopted.

We still must manage the "secret" allocations, so I don't think that the dynamic solution will be much more complicated.

Quoted 13 lines
Quoted 13 lines

Yet another idea is to make page allocator aware of the direct map layout.

* Kernel page table management

Currently we presume that only one kernel page table exists (well, mostly) and the page table abstraction is required only for the user page tables. As such, we presume that 'page table == struct mm_struct' and the mm_struct is used all over by the operations that manage the page tables.

The management of the restricted address space in the kernel requires ability to create, update and remove kernel contexts the same way we do for the userspace.

Quoted 6 lines
Quoted 3 lines

One way is to overload the mm_struct, like EFI and text poking did. But it is quite an overkill, because most of the mm_struct contains information required to manage user mappings.

In what way is it overkill? Just memory overhead? How many of such contexts do you expect to see in the system?

Well, memory overhead is not that big, but it'd not negligible. For the KVM ASI usescase, for instance, there will be at least as much contexts as running VMs. We also have thoughts about how to make namespaces use restricted address spaces, for this usecase there will be quite a lot of such contexts.

Besides, it does not feel right to have the mm_struct to represent a page table.

Quoted 17 lines
Quoted 14 lines

My suggestion is to introduce a first class abstraction for the page table and then it could be used in the same way for user and kernel context management. For now I have a very basic POC that slitted several fields from the mm_struct into a new 'struct pg_table' [5]. This new abstraction can be used e.g. by PTI implementation of the page table cloning and the KVM ASI work.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/20200130162340.GA14232@rapoport-lnx/
[5] https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git/log/?h=pg_table/v0.0

Signature · 1 line
 Kirill A. Shutemov
Signature · 2 lines
Sincerely yours,
Mike.
#3 lore raw
Quoted 28 lines · Mike Rapoport
Quoted 27 lines · Kirill A. Shutemov
Quoted 13 lines · Mike Rapoport

Restricted mappings in the kernel mode may improve mitigation of hardware speculation vulnerabilities and minimize the damage exploitable kernel bugs can cause.

There are several ongoing efforts to use restricted address spaces in Linux kernel for various use cases:
* speculation vulnerabilities mitigation in KVM [1]
* support for memory areas visible only in a single owning context, or more generically, a memory areas with more restrictive protection that the defaults ("secret" memory) [2], [3], [4]
* hardening of the Linux containers [ no reference yet :) ]

Quoted 8 lines

Last year we had vague ideas and possible directions, this year we have several real challenges and design decisions we'd like to discuss:

* "Secret" memory userspace APIs

Should such API follow "native" MM interfaces like mmap(), mprotect(), madvise() or it would be better to use a file descriptor , e.g. like memfd-create does?

I don't really see a point in such file-descriptor. It suppose to be very private secret data. What functionality that provide a file descriptor do you see valuable in this scenario?

Quoted 5 lines
Quoted 2 lines

File descriptor makes it easier to spill the secrets to other process: over fork(), UNIX socket or via /proc/PID/fd/.

On the other hand it is may be desired to share a secret between several processes. Then UNIX socket or fork() actually become handy.

If more than one knows, it is secret no longer :P

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

MM "native" APIs would require VM_something flag and probably a page flag or page_ext. With file-descriptor VM_SPECIAL and custom implementation of .mmap() and .fault() would suffice. On the other hand, mmap() and mprotect() seem better fit semantically and they could be more easily adopted by the userspace.

You mix up implementation and interface. You can provide an interface which doesn't require a file descriptor, but still use a magic file internally to the VMA distinct.

Quoted 5 lines

If I understand correctly, if we go with mmap(MAP_SECRET) example, the mmap() would implicitly create a magic file with its .mmap() and .fault() implementing the protection? That's a possibility. But then, if we already have a file, why not let user get a handle for it and allow fine grained control over its sharing between processes?

A proper file descriptor would have wider exposer with security implications. It has to be at least scoped properly.

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

* Direct/linear map fragmentation

Whenever we want to drop some mappings from the direct map or even change the protection bits for some memory area, the gigantic and huge pages that comprise the direct map need to be broken and there's no THP for the kernel page tables to collapse them back. Moreover, the existing API defined in <asm/set_memory.h> by several architectures do not really presume it would be widely used.

For the "secret" memory use-case the fragmentation can be minimized by caching large pages, use them to satisfy smaller "secret" allocations and than collapse them back once the "secret" memory is freed. Another possibility is to pre-allocate physical memory at boot time.

Quoted 6 lines
Quoted 3 lines

I would rather go with pre-allocation path. At least at first. We always can come up with more dynamic and complicated solution later if the interface would be wildly adopted.

We still must manage the "secret" allocations, so I don't think that the dynamic solution will be much more complicated.

Okay.

BTW, with clarified scope of the AMD Erratum, I believe we can implement "collapse" for direct mapping. Willing to try?

Quoted 20 lines
Quoted 13 lines
Quoted 13 lines

Yet another idea is to make page allocator aware of the direct map layout.

* Kernel page table management

Currently we presume that only one kernel page table exists (well, mostly) and the page table abstraction is required only for the user page tables. As such, we presume that 'page table == struct mm_struct' and the mm_struct is used all over by the operations that manage the page tables.

The management of the restricted address space in the kernel requires ability to create, update and remove kernel contexts the same way we do for the userspace.

Quoted 6 lines
Quoted 3 lines

One way is to overload the mm_struct, like EFI and text poking did. But it is quite an overkill, because most of the mm_struct contains information required to manage user mappings.

In what way is it overkill? Just memory overhead? How many of such contexts do you expect to see in the system?

Quoted 8 lines

Well, memory overhead is not that big, but it'd not negligible. For the KVM ASI usescase, for instance, there will be at least as much contexts as running VMs. We also have thoughts about how to make namespaces use restricted address spaces, for this usecase there will be quite a lot of such contexts.

Besides, it does not feel right to have the mm_struct to represent a page table.

Fair enough. It might be interesting.

Signature · 1 line
 Kirill A. Shutemov
#4 lore raw
Quoted 14 lines · Kirill A. Shutemov
Quoted 13 lines · Mike Rapoport
Quoted 12 lines · Kirill A. Shutemov
Quoted 6 lines · Mike Rapoport

* "Secret" memory userspace APIs

Should such API follow "native" MM interfaces like mmap(), mprotect(), madvise() or it would be better to use a file descriptor , e.g. like memfd-create does?

I don't really see a point in such file-descriptor. It suppose to be very private secret data. What functionality that provide a file descriptor do you see valuable in this scenario?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

File descriptor makes it easier to spill the secrets to other process: over fork(), UNIX socket or via /proc/PID/fd/.

On the other hand it is may be desired to share a secret between several processes. Then UNIX socket or fork() actually become handy.

If more than one knows, it is secret no longer :P

But even cryptographers define "shared secret" ;-)

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

MM "native" APIs would require VM_something flag and probably a page flag or page_ext. With file-descriptor VM_SPECIAL and custom implementation of .mmap() and .fault() would suffice. On the other hand, mmap() and mprotect() seem better fit semantically and they could be more easily adopted by the userspace.

You mix up implementation and interface. You can provide an interface which doesn't require a file descriptor, but still use a magic file internally to the VMA distinct.

Quoted 8 lines
Quoted 5 lines

If I understand correctly, if we go with mmap(MAP_SECRET) example, the mmap() would implicitly create a magic file with its .mmap() and .fault() implementing the protection? That's a possibility. But then, if we already have a file, why not let user get a handle for it and allow fine grained control over its sharing between processes?

A proper file descriptor would have wider exposer with security implications. It has to be at least scoped properly.

Agree.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 9 lines

* Direct/linear map fragmentation

Whenever we want to drop some mappings from the direct map or even change the protection bits for some memory area, the gigantic and huge pages that comprise the direct map need to be broken and there's no THP for the kernel page tables to collapse them back. Moreover, the existing API defined in <asm/set_memory.h> by several architectures do not really presume it would be widely used.

Quoted 4 lines

For the "secret" memory use-case the fragmentation can be minimized by caching large pages, use them to satisfy smaller "secret" allocations and than collapse them back once the "secret" memory is freed. Another possibility is to pre-allocate physical memory at boot time.

I would rather go with pre-allocation path. At least at first. We always can come up with more dynamic and complicated solution later if the interface would be wildly adopted.

Quoted 7 lines
Quoted 2 lines

We still must manage the "secret" allocations, so I don't think that the dynamic solution will be much more complicated.

Okay.

BTW, with clarified scope of the AMD Erratum, I believe we can implement "collapse" for direct mapping. Willing to try?

My initial plan was to use a pool of large pages to satisfy "secret" allocation requests. Whenever a new large page is allocated for that pool, it's removed from the direct map without being split into small pages and then when it would be reinstated back there would be no need to collapse it.

Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Yet another idea is to make page allocator aware of the direct map layout.

Signature · 1 line
 Kirill A. Shutemov
Signature · 2 lines
Sincerely yours,
Mike.
#5 lore raw
Quoted 8 lines · Mike Rapoport
Quoted 2 lines

BTW, with clarified scope of the AMD Erratum, I believe we can implement "collapse" for direct mapping. Willing to try?

My initial plan was to use a pool of large pages to satisfy "secret" allocation requests. Whenever a new large page is allocated for that pool, it's removed from the direct map without being split into small pages and then when it would be reinstated back there would be no need to collapse it.

It might be okay. But you likely will have to split 1G pages in direct mapping into 2M. Being able to repare the direct mapping is more generally useful.

Signature · 1 line
 Kirill A. Shutemov
#6 lore raw
Quoted 4 lines · Kirill A. Shutemov
Quoted 3 lines · Mike Rapoport
Quoted 2 lines

BTW, with clarified scope of the AMD Erratum, I believe we can implement "collapse" for direct mapping. Willing to try?

Quoted 8 lines
Quoted 5 lines

My initial plan was to use a pool of large pages to satisfy "secret" allocation requests. Whenever a new large page is allocated for that pool, it's removed from the direct map without being split into small pages and then when it would be reinstated back there would be no need to collapse it.

It might be okay. But you likely will have to split 1G pages in direct mapping into 2M.

Right, it is quite likely, at least for the dynamic allocations.

Quoted 2 lines

Being able to repare the direct mapping is more generally useful.

And it's not strictly related to the "secret" memory. I'll try to have a look at it.

Quoted 2 lines
Signature · 1 line
 Kirill A. Shutemov
Signature · 2 lines
Sincerely yours,
Mike.