PKS write protected page tables
Subsystems (7)
cover PKS write protected page tables
This is a POC for write protecting page tables with PKS (Protection Keys for Supervisor) [1]. The basic idea is to make the page tables read only, except temporarily on a per-cpu basis when they need to be modified. I’m looking for opinions on whether people like the general direction of this in terms of value and implementation.
Why would people want this?
===========================
Page tables are the basis for many types of protections and as such, are a juicy target for attackers. Mapping them read-only will make them harder to use in attacks.
This protects against an attacker that has acquired the ability to write to the page tables. It's not foolproof because an attacker who can execute arbitrary code can either disable PKS directly, or simply call the same functions that the kernel uses for legitimate page table writes.
Why use PKS for this?
=====================
PKS is an upcoming CPU feature that allows supervisor virtual memory permissions to be changed without flushing the TLB, like PKU does for user memory. Protecting page tables would normally be really expensive because you would have to do it with paging itself. PKS helps by providing a way to toggle the writability of the page tables with just a per-cpu MSR.
Performance impacts
===================
Setting direct map permissions on whatever random page gets allocated for a page table would result in a lot of kernel range shootdowns and direct map large page shattering. So the way the PKS page table memory is created is similar to this module page clustering series[2], where a cache of pages is replenished from 2MB pages such that the direct map permissions and associated breakage is localized on the direct map. In the PKS page tables case, a PKS key is pre-applied to the direct map for pages in the cache.
There would be some costs of memory overhead in order to protect the direct map page tables. There would also be some extra kernel range shootdowns to replenish the cache on occasion, from setting the PKS key on the direct map of the new pages. I don’t have any actual performance data yet.
This is based on V6 [1] of the core PKS infrastructure patches. PKS infrastructure follow-on’s are planned to enable keys to be set to the same permissions globally. Since this usage needs a key to be set globally read-only by default, a small temporary solution is hacked up in patch 8. Long term, PKS protected page tables would use a better and more generic solution to achieve this.
[1]
https://lore.kernel.org/lkml/[email protected]/
[2]
https://lore.kernel.org/lkml/[email protected]
/
Thanks,
Rick
Rick Edgecombe (9):
list: Support getting most recent element in list_lru
list: Support list head not in object for list_lru x86/mm/cpa: Add grouped page allocations
mm: Explicitly zero page table lock ptr
x86, mm: Use cache of page tables x86/mm/cpa: Add set_memory_pks()
x86/mm/cpa: Add perm callbacks to grouped pages x86, mm: Protect page tables with PKS
x86, cpa: PKS protect direct map page tables
arch/x86/boot/compressed/ident_map_64.c | 5 + arch/x86/include/asm/pgalloc.h | 6 + arch/x86/include/asm/pgtable.h | 26 +- arch/x86/include/asm/pgtable_64.h | 33 ++- arch/x86/include/asm/pkeys_common.h | 8 +- arch/x86/include/asm/set_memory.h | 23 ++ arch/x86/mm/init.c | 40 +++ arch/x86/mm/pat/set_memory.c | 312 +++++++++++++++++++++++- arch/x86/mm/pgtable.c | 144 ++++++++++- include/asm-generic/pgalloc.h | 42 +++- include/linux/list_lru.h | 26 ++ include/linux/mm.h | 7 + mm/Kconfig | 6 +- mm/list_lru.c | 38 ++- mm/memory.c | 1 + mm/swap.c | 7 + mm/swap_state.c | 6 + 17 files changed, 705 insertions(+), 25 deletions(-)
Signature · 1 line
2.30.2
RFC PATCH 1/9 list: Support getting most recent element in list_lru
In future patches, some functionality will use list_lru that also needs to keep track of the most recently used element on a node. Since this information is already contained within list_lru, add a function to get it so that an additional list is not needed in the caller.
Do not support memcg aware list_lru's since it is not needed by the intended caller.
--- include/linux/list_lru.h | 13 +++++++++++++ mm/list_lru.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+)
include/linux/list_lru.h +13 -0
@@ -103,6 +103,19 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item);*/ bool list_lru_del (struct list_lru *lru,struct list_head *item); +/** +* list_lru_get_mru: gets and removes the tail from one of the node lists +* @list_lru: the lru pointer +* @nid: the node id +* +* This function removes the most recently added item from one of the node +* id specified. This function should not be used if the list_lru is memcg +* aware. +* +* Return value: The element removed +*/ +struct list_head *list_lru_get_mru (struct list_lru *lru,int nid); +/** * list_lru_count_one: return the number of objects currently held by @lru * @lru: the lru pointer.
mm/list_lru.c +28 -0
@@ -156,6 +156,34 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) }EXPORT_SYMBOL_GPL (list_lru_del); +struct list_head *list_lru_get_mru (struct list_lru *lru,int nid) +{ +struct list_lru_node *nlru = &lru->node[nid]; +struct list_lru_one *l = &nlru->lru; +struct list_head *ret; + +/* This function does not attempt to search through the memcg lists */ +if (list_lru_memcg_aware (lru)) { +WARN_ONCE (1 ,"list_lru: %s not supported on memcg aware list_lrus" , __func__); +return NULL ; + } + +spin_lock (&nlru->lock); +if (list_empty (&l->list)) { + ret =NULL ; + }else { +/* Get tail */ + ret = l->list.prev; +list_del_init (ret); + + l->nr_items--; + nlru->nr_items--; + } +spin_unlock (&nlru->lock); + +return ret; +} +void list_lru_isolate (struct list_lru_one *list,struct list_head *item) {list_del_init (item);
Signature · 1 line
2.30.2
RFC PATCH 2/9 list: Support list head not in object for list_lru
In reply to Rick EdgecombeIn future patches, there will be a need to keep track of objects with list_lru where the list_head is not in the object (will be in struct page). Since list_lru automatically determines the node id from the list_head, this will fail when using struct page.
So create a new function in list_lru, list_lru_add_node(), that allows the node id of the item to be passed in. Otherwise it behaves exactly like list_lru_add().
--- include/linux/list_lru.h | 13 +++++++++++++ mm/list_lru.c | 10 ++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/list_lru.h +13 -0
@@ -90,6 +90,19 @@ void memcg_drain_all_list_lrus(int src_idx, struct mem_cgroup *dst_memcg);*/ bool list_lru_add (struct list_lru *lru,struct list_head *item); +/** +* list_lru_add_node: add an element to the lru list's tail +* @list_lru: the lru pointer +* @item: the item to be added. +* @nid: the node id of the item +* +* Like list_lru_add, but takes the node id as parameter instead of +* calculating it from the list_head passed in. +* +* Return value: true if the list was updated, false otherwise +*/ +bool list_lru_add_node (struct list_lru *lru,struct list_head *item,int nid); +/** * list_lru_del: delete an element to the lru list * @list_lru: the lru pointer
mm/list_lru.c +8 -2
@@ -112,9 +112,8 @@ list_lru_from_kmem(struct list_lru_node *nlru, void *ptr, }#endif /* CONFIG_MEMCG_KMEM */ -bool list_lru_add (struct list_lru *lru,struct list_head *item) +bool list_lru_add_node (struct list_lru *lru,struct list_head *item,int nid) { -int nid =page_to_nid (virt_to_page (item));struct list_lru_node *nlru = &lru->node[nid];struct mem_cgroup *memcg;struct list_lru_one *l;
@@ -134,6 +133,13 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item)spin_unlock (&nlru->lock);return false; } + +bool list_lru_add (struct list_lru *lru,struct list_head *item) +{ +int nid =page_to_nid (virt_to_page (item)); + +return list_lru_add_node (lru, item, nid); +}EXPORT_SYMBOL_GPL (list_lru_add);bool list_lru_del (struct list_lru *lru,struct list_head *item)
Signature · 1 line
2.30.2
RFC PATCH 3/9 x86/mm/cpa: Add grouped page allocations
In reply to Rick EdgecombeFor x86, setting memory permissions on the direct map results in fracturing large pages. Direct map fracturing can be reduced by locating pages that will have their permissions set close together.
Create a simple page cache that allocates pages from huge page size blocks. Don't guarantee that a page will come from a huge page grouping, instead fallback to non-grouped pages to fulfill the allocation if needed. Also, register a shrinker such that the system can ask for the pages back if needed. Since this is only needed when there is a direct map, compile it out on highmem systems.
Free pages in the cache are kept track of in per-node list inside a list_lru. NUMA_NO_NODE requests are serviced by checking each per-node list in a round robin fashion. If pages are requested for a certain node but the cache is empty for that node, a whole additional huge page size page is allocated.
--- arch/x86/include/asm/set_memory.h | 14 +++ arch/x86/mm/pat/set_memory.c | 151 ++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+)
arch/x86/include/asm/set_memory.h +14 -0
@@ -4,6 +4,9 @@#include <asm/page.h> #include <asm-generic/set_memory.h> +#include <linux/gfp.h> +#include <linux/list_lru.h> +#include <linux/shrinker.h> /* * The set_memory_* API can be used to change various attributes of a virtual
@@ -135,4 +138,15 @@ static inline int clear_mce_nospec(unsigned long pfn)*/ #endif +struct grouped_page_cache { +struct shrinker shrinker; +struct list_lru lru; +gfp_t gfp; +atomic_t nid_round_robin; +}; + +int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp); +struct page *get_grouped_page (int node,struct grouped_page_cache *gpc); +void free_grouped_page (struct grouped_page_cache *gpc,struct page *page); +#endif /* _ASM_X86_SET_MEMORY_H */
arch/x86/mm/pat/set_memory.c +151 -0
157 lines not on this page · raw message
Signature · 1 line
2.30.2
Mike Rapoport
reviewerRe: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 10 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results in fracturing large pages. Direct map fracturing can be reduced by locating pages that will have their permissions set close together.
Create a simple page cache that allocates pages from huge page size blocks. Don't guarantee that a page will come from a huge page grouping, instead fallback to non-grouped pages to fulfill the allocation if needed. Also, register a shrinker such that the system can ask for the pages back if needed. Since this is only needed when there is a direct map, compile it out on highmem systems.
I only had time to skim through the patches, I like the idea of having a simple cache that allocates larger pages with a fallback to basic page size.
I just think it should be more generic and closer to the page allocator. I was thinking about adding a GFP flag that will tell that the allocated pages should be removed from the direct map. Then alloc_pages() could use such cache whenever this GFP flag is specified with a fallback for lower order allocations.
Quoted 207 lines
Free pages in the cache are kept track of in per-node list inside a
list_lru.NUMA_NO_NODErequests are serviced by checking each per-node list in a round robin fashion. If pages are requested for a certain node but the cache is empty for that node, a whole additional huge page size page is allocated.Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/include/asm/set_memory.h | 14 +++ arch/x86/mm/pat/set_memory.c | 151 ++++++++++++++++++++++++++++++ 2 files changed, 165 insertions(+)arch/x86/include/asm/set_memory.h +14
-0@@ -4,6 +4,9 @@#include <asm/page.h> #include <asm-generic/set_memory.h> +#include <linux/gfp.h> +#include <linux/list_lru.h> +#include <linux/shrinker.h> /* * The set_memory_* API can be used to change various attributes of a virtual @@ -135,4 +138,15 @@ static inline int clear_mce_nospec(unsigned long pfn)*/ #endif +struct grouped_page_cache { +struct shrinker shrinker; +struct list_lru lru; +gfp_t gfp; +atomic_t nid_round_robin; +}; + +int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp); +struct page *get_grouped_page (int node,struct grouped_page_cache *gpc); +void free_grouped_page (struct grouped_page_cache *gpc,struct page *page); +#endif /* _ASM_X86_SET_MEMORY_H */ arch/x86/mm/pat/set_memory.c +151
-0@@ -2306,6 +2306,157 @@ int __init kernel_unmap_pages_in_pgd(pgd_t *pgd, unsigned long address,return retval; } +#ifndef HIGHMEM +static struct page *__alloc_page_order (int node,gfp_t gfp_mask,int order) +{ +if (node ==NUMA_NO_NODE ) +return alloc_pages (gfp_mask, order); + +return alloc_pages_node (node, gfp_mask, order); +} + +static struct grouped_page_cache *__get_gpc_from_sc (struct shrinker *shrinker) +{ + return container_of(shrinker, struct grouped_page_cache, shrinker); +} + +static unsigned long grouped_shrink_count (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct grouped_page_cache *gpc =__get_gpc_from_sc (shrinker); +unsigned long page_cnt =list_lru_shrink_count (&gpc->lru, sc); + +return page_cnt ? page_cnt :SHRINK_EMPTY ; +} + +static enum lru_status grouped_isolate (struct list_head *item, +struct list_lru_one *list, +spinlock_t *lock,void *cb_arg) +{ +struct list_head *dispose = cb_arg; + +list_lru_isolate_move (list, item, dispose); + +return LRU_REMOVED ; +} + +static void __dispose_pages (struct grouped_page_cache *gpc,struct list_head *head) +{ +struct list_head *cur, *next; + +list_for_each_safe (cur, next, head) { + struct page *page = list_entry(head, struct page, lru); + +list_del (cur); + +__free_pages (page,0 ); + } +} + +static unsigned long grouped_shrink_scan (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct grouped_page_cache *gpc =__get_gpc_from_sc (shrinker); +unsigned long isolated; +LIST_HEAD (freeable); + +if (!(sc->gfp_mask & gpc->gfp)) +return SHRINK_STOP ; + + isolated =list_lru_shrink_walk (&gpc->lru, sc, grouped_isolate, + &freeable); +__dispose_pages (gpc, &freeable); + +/* Every item walked gets isolated */ + sc->nr_scanned += isolated; + +return isolated; +} + +static struct page *__remove_first_page (struct grouped_page_cache *gpc,int node) +{ +unsigned int start_nid, i; +struct list_head *head; + +if (node !=NUMA_NO_NODE ) { + head =list_lru_get_mru (&gpc->lru, node); +if (head) + return list_entry(head, struct page, lru); +return NULL ; + } + +/* If NUMA_NO_NODE, search the nodes in round robin for a page */ + start_nid = (unsigned int )atomic_fetch_inc (&gpc->nid_round_robin) % nr_node_ids; +for (i =0 ; i < nr_node_ids; i++) { +int cur_nid = (start_nid + i) % nr_node_ids; + + head =list_lru_get_mru (&gpc->lru, cur_nid); +if (head) + return list_entry(head, struct page, lru); + } + +return NULL ; +} + +/* Get and add some new pages to the cache to be used by VM_GROUP_PAGES */ +static struct page *__replenish_grouped_pages (struct grouped_page_cache *gpc,int node) +{ +const unsigned int hpage_cnt =HPAGE_SIZE >>PAGE_SHIFT ; +struct page *page; +int i; + + page =__alloc_page_order (node, gpc->gfp,HUGETLB_PAGE_ORDER ); +if (!page) +return __alloc_page_order (node, gpc->gfp,0 ); + +split_page (page,HUGETLB_PAGE_ORDER ); + +for (i =1 ; i < hpage_cnt; i++) +free_grouped_page (gpc, &page[i]); + +return &page[0 ]; +} + +int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp) +{ +int err =0 ; + +memset (gpc,0 ,sizeof (struct grouped_page_cache )); + +if (list_lru_init (&gpc->lru)) + goto out; + + gpc->shrinker.count_objects = grouped_shrink_count; + gpc->shrinker.scan_objects = grouped_shrink_scan; + gpc->shrinker.seeks =DEFAULT_SEEKS ; + gpc->shrinker.flags =SHRINKER_NUMA_AWARE ; + + err =register_shrinker (&gpc->shrinker); +if (err) +list_lru_destroy (&gpc->lru); + +out: +return err; +} + +struct page *get_grouped_page (int node,struct grouped_page_cache *gpc) +{ +struct page *page; + + page =__remove_first_page (gpc, node); + +if (page) +return page; + +return __replenish_grouped_pages (gpc, node); +} + +void free_grouped_page (struct grouped_page_cache *gpc,struct page *page) +{ +INIT_LIST_HEAD (&page->lru); +list_lru_add_node (&gpc->lru, &page->lru,page_to_nid (page)); +} +#endif /* !HIGHMEM */ /* * The testcases use internal knowledge of the implementation that shouldn't * be exposed to the rest of the kernel. Include these directly here. Signature · 1 line
2.30.2
Signature · 2 lines
Sincerely yours, Mike.
Peter Zijlstra
maintainerRe: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 16 lines · Mike Rapoport
Quoted 10 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results in fracturing large pages. Direct map fracturing can be reduced by locating pages that will have their permissions set close together.
Create a simple page cache that allocates pages from huge page size blocks. Don't guarantee that a page will come from a huge page grouping, instead fallback to non-grouped pages to fulfill the allocation if needed. Also, register a shrinker such that the system can ask for the pages back if needed. Since this is only needed when there is a direct map, compile it out on highmem systems.
I only had time to skim through the patches, I like the idea of having a simple cache that allocates larger pages with a fallback to basic page size.
Quoted 5 lines
I just think it should be more generic and closer to the page allocator. I was thinking about adding a GFP flag that will tell that the allocated pages should be removed from the direct map. Then
alloc_pages()could use such cache whenever this GFP flag is specified with a fallback for lower order allocations.
That doesn't provide enough information I think. Removing from direct map isn't the only consideration, you also want to group them by the target protection bits such that we don't get to use 4k pages quite so much.
Mike Rapoport
reviewerRe: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 23 lines · Peter Zijlstra
Quoted 16 lines · Mike Rapoport
Quoted 10 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results in fracturing large pages. Direct map fracturing can be reduced by locating pages that will have their permissions set close together.
Create a simple page cache that allocates pages from huge page size blocks. Don't guarantee that a page will come from a huge page grouping, instead fallback to non-grouped pages to fulfill the allocation if needed. Also, register a shrinker such that the system can ask for the pages back if needed. Since this is only needed when there is a direct map, compile it out on highmem systems.
I only had time to skim through the patches, I like the idea of having a simple cache that allocates larger pages with a fallback to basic page size.
Quoted 5 lines
I just think it should be more generic and closer to the page allocator. I was thinking about adding a GFP flag that will tell that the allocated pages should be removed from the direct map. Then
alloc_pages()could use such cache whenever this GFP flag is specified with a fallback for lower order allocations.
Quoted 4 lines
That doesn't provide enough information I think. Removing from direct map isn't the only consideration, you also want to group them by the target protection bits such that we don't get to use 4k pages quite so much.
Unless I'm missing something we anyway hand out 4k pages from the cache and the neighbouring 4k may end up with different protections.
This is also similar to what happens in the set Rick posted a while ago to support grouped vmalloc allocations:
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 45 lines · Mike Rapoport
Quoted 36 lines · Peter Zijlstra
Quoted 25 lines · Mike Rapoport
Quoted 6 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results in fracturing
large pages. Direct map fracturing can be reduced by locating pages that
will have their permissions set close together.Quoted 11 lines
Create a simple page cache that allocates pages from huge page size
blocks. Don't guarantee that a page will come from a huge page grouping,
instead fallback to non-grouped pages to fulfill the allocation if
needed. Also, register a shrinker such that the system can ask for the
pages back if needed. Since this is only needed when there is a direct
map, compile it out on highmem systems.I only had time to skim through the patches, I like the idea of having a
simple cache that allocates larger pages with a fallback to basic page
size.Quoted 9 lines
I just think it should be more generic and closer to the page allocator.
I was thinking about adding a GFP flag that will tell that the allocated
pages should be removed from the direct map. Thenalloc_pages()could use
such cache whenever this GFP flag is specified with a fallback for lower
order allocations.Quoted 7 lines
That doesn't provide enough information I think. Removing from direct
map isn't the only consideration, you also want to group them by the
target protection bits such that we don't get to use 4k pages quite so
much.
Quoted 8 lines
Unless I'm missing something we anyway hand out 4k pages from the cache and
the neighbouring 4k may end up with different protections.This is also similar to what happens in the set Rick posted a while ago to
support grouped vmalloc allocations:
One issue is with the shrinker callbacks. If you are just trying to reset and free a single page because the system is low on memory, it could be problematic to have to break a large page, which would require another page.
I think for vmalloc, eventually it should just have the direct map alias unmapped. The reason it was not in the linked patch, is just to iteratively move in the direction of having permissioned vmallocs be unmapped.
Mike Rapoport
reviewerRe: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 55 lines · Rick P
Quoted 45 lines · Mike Rapoport
Quoted 36 lines · Peter Zijlstra
Quoted 25 lines · Mike Rapoport
Quoted 6 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results in fracturing
large pages. Direct map fracturing can be reduced by locating pages that
will have their permissions set close together.Quoted 11 lines
Create a simple page cache that allocates pages from huge page size
blocks. Don't guarantee that a page will come from a huge page grouping,
instead fallback to non-grouped pages to fulfill the allocation if
needed. Also, register a shrinker such that the system can ask for the
pages back if needed. Since this is only needed when there is a direct
map, compile it out on highmem systems.I only had time to skim through the patches, I like the idea of having a
simple cache that allocates larger pages with a fallback to basic page
size.Quoted 9 lines
I just think it should be more generic and closer to the page allocator.
I was thinking about adding a GFP flag that will tell that the allocated
pages should be removed from the direct map. Thenalloc_pages()could use
such cache whenever this GFP flag is specified with a fallback for lower
order allocations.Quoted 7 lines
That doesn't provide enough information I think. Removing from direct
map isn't the only consideration, you also want to group them by the
target protection bits such that we don't get to use 4k pages quite so
much.Quoted 8 lines
Unless I'm missing something we anyway hand out 4k pages from the cache and
the neighbouring 4k may end up with different protections.This is also similar to what happens in the set Rick posted a while ago to
support grouped vmalloc allocations:
Quoted 4 lines
One issue is with the shrinker callbacks. If you are just trying to reset and free a single page because the system is low on memory, it could be problematic to have to break a large page, which would require another page.
I don't follow you here. Maybe I've misread the patches but AFAIU the large page is broken at allocation time and 4k pages remain 4k pages afterwards.
In my understanding the problem with a simple shrinker is that even if we have the entire 2M free it is not being reinstated as 2M page in the direct mapping.
Signature · 2 lines
Sincerely yours, Mike.
Re: [PATCH RFC 3/9] x86/mm/cpa: Add grouped page allocations
Quoted 79 lines · Mike Rapoport
Quoted 71 lines · Rick P
Quoted 60 lines · Mike Rapoport
Quoted 49 lines · Peter Zijlstra
Quoted 35 lines · Mike Rapoport
Quoted 8 lines · Rick Edgecombe
For x86, setting memory permissions on the direct map results
in fracturing
large pages. Direct map fracturing can be reduced by locating
pages that
will have their permissions set close together.Quoted 16 lines
Create a simple page cache that allocates pages from huge page
size
blocks. Don't guarantee that a page will come from a huge page
grouping,
instead fallback to non-grouped pages to fulfill the allocation
if
needed. Also, register a shrinker such that the system can ask
for the
pages back if needed. Since this is only needed when there is a
direct
map, compile it out on highmem systems.I only had time to skim through the patches, I like the idea of
having a
simple cache that allocates larger pages with a fallback to basic
page
size.Quoted 12 lines
I just think it should be more generic and closer to the page allocator.
I was thinking about adding a GFP flag that will tell that the
allocated
pages should be removed from the direct map. Thenalloc_pages()
could use
such cache whenever this GFP flag is specified with a fallback
for lower
order allocations.Quoted 9 lines
That doesn't provide enough information I think. Removing from direct
map isn't the only consideration, you also want to group them by
the
target protection bits such that we don't get to use 4k pages quite
so
much.Quoted 9 lines
Unless I'm missing something we anyway hand out 4k pages from the cache and
the neighbouring 4k may end up with different protections.This is also similar to what happens in the set Rick posted a while
ago to
support grouped vmalloc allocations:Quoted 6 lines
One issue is with the shrinker callbacks. If you are just trying to reset and free a single page because the system is low on memory, it
could be problematic to have to break a large page, which would require
another page.
Quoted 4 lines
I don't follow you here. Maybe I've misread the patches but AFAIU the large
page is broken at allocation time and 4k pages remain 4k pages afterwards.
Yea that's right.
I thought Peter was saying if the page allocator grouped all of the same permission together it could often leave the direct map as large pages, and so the page allocator would have to know about permissions.
So I was just trying to say, to leave large pages on the direct map, the shrinker has to handle breaking a page while freeing a single page. So that would have to be addressed to get large pages with permissions in the first place.
It doesn't seem impossible to solve I guess, so maybe not an important point. It could maybe just hold a page in reserve.
Now that I think about it, since this PKS tables series holds all potentially needed direct map page tables in reserve, it shouldn't actually be a problem for this case. So this could leave the PKS tables pages as large on the direct map.
Quoted 5 lines
In my understanding the problem with a simple shrinker is that even if we
have the entire 2M free it is not being reinstated as 2M page in the direct
mapping.
Yea, that is a downside to this simple shrinker.
RFC PATCH 4/9 mm: Explicitly zero page table lock ptr
In reply to Rick EdgecombeIn ptlock_init() there is a VM_BUG_ON_PAGE() check on the page table lock pointer. Explicitly zero the lock in ptlock_free() so a page table lock can be re-initialized without triggering the BUG_ON().
It appears this doesn't normally trigger because the private field shares the same space in struct page as ptl and page tables always return to the buddy allocator before being re-initialized as new page tables. When the page returns to the buddy allocator, private gets used to store the page order, so it inadvertently clears ptl as well. In future patches, pages will get re-initialized as page tables without returning to the buddy allocator so this is needed.
--- mm/memory.c | 1 + 1 file changed, 1 insertion(+)
mm/memory.c +1 -0
@@ -5225,5 +5225,6 @@ bool ptlock_alloc(struct page *page)void ptlock_free (struct page *page) {kmem_cache_free (page_ptl_cachep, page->ptl); + page->ptl =0 ; }#endif
Signature · 1 line
2.30.2
RFC PATCH 5/9 x86, mm: Use cache of page tables
In reply to Rick EdgecombeChange the page table allocation functions defined in pgalloc.h to use a cache of physically grouped pages. This will let the page tables to be set with PKS permissions later.
For userspace page tables, they are gathered up using mmu gather, and freed along with other types of pages in swap.c. Reuse the PageTable page flag to communicate that swap needs to return this page to the cache of page tables, and not free it to the page allocator. Set this flag in the free_tlb() family of functions called by mmu gather.
Do not set PKS permissions on the page tables, because the page table setting functions cannot handle it yet. This will be done in later patches.
--- arch/x86/include/asm/pgalloc.h | 4 ++ arch/x86/mm/pgtable.c | 75 ++++++++++++++++++++++++++++++++++ include/asm-generic/pgalloc.h | 42 +++++++++++++++---- include/linux/mm.h | 7 ++++ mm/swap.c | 7 ++++ mm/swap_state.c | 6 +++ 6 files changed, 132 insertions(+), 9 deletions(-)
arch/x86/include/asm/pgalloc.h +4 -0
@@ -7,6 +7,10 @@#include <linux/pagemap.h> #define __HAVE_ARCH_PTE_ALLOC_ONE +#ifdef CONFIG_PKS_PG_TABLES +#define __HAVE_ARCH_FREE_TABLE +#define __HAVE_ARCH_ALLOC_TABLE +#endif #define __HAVE_ARCH_PGD_FREE#include <asm-generic/pgalloc.h>
arch/x86/mm/pgtable.c +75 -0
135 lines not on this page · raw message
include/asm-generic/pgalloc.h +33 -9
104 lines not on this page · raw message
include/linux/mm.h +7 -0
@@ -2185,6 +2185,13 @@ static inline bool ptlock_init(struct page *page) { return true; }static inline void ptlock_free (struct page *page) {}#endif /* USE_SPLIT_PTE_PTLOCKS */ +#ifndef CONFIG_PKS_PG_TABLES +static inline void free_table (struct page *table_page) +{ +__free_pages (table_page,0 ); +} +#endif /* CONFIG_PKS_PG_TABLES */ +static inline void pgtable_init (void ) {ptlock_cache_init ();
mm/swap.c +7 -0
@@ -36,6 +36,7 @@#include <linux/hugetlb.h> #include <linux/page_idle.h> #include <linux/local_lock.h> +#include <asm/pgalloc.h> #include "internal.h"
@@ -888,6 +889,12 @@ void release_pages(struct page **pages, int nr)continue ; } +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +continue ; + } +if (!put_page_testzero (page))continue ;
mm/swap_state.c +6 -0
@@ -22,6 +22,7 @@#include <linux/swap_slots.h> #include <linux/huge_mm.h> #include <linux/shmem_fs.h> +#include <asm/pgalloc.h> #include "internal.h" /*
@@ -310,6 +311,11 @@ static inline void free_swap_cache(struct page *page)void free_page_and_swap_cache (struct page *page) {free_swap_cache (page); +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +return ; + }if (!is_huge_zero_page (page))put_page (page); }
Signature · 1 line
2.30.2
Peter Zijlstra
maintainerRe: [PATCH RFC 5/9] x86, mm: Use cache of page tables
Quoted 27 lines · Rick Edgecombe
arch/x86/mm/pgtable.c +6
-0@@ -54,6 +98,8 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) {pgtable_pte_page_dtor (pte);paravirt_release_pte (page_to_pfn (pte)); +/* Set Page Table so swap knows how to free it */ +__SetPageTable (pte);paravirt_tlb_remove_table (tlb, pte); }@@ -70,12 +116,16 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =1 ;#endif pgtable_pmd_page_dtor (page); +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pmd));paravirt_tlb_remove_table (tlb, page); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) { +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pud));paravirt_release_pud (__pa (pud) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (pud)); }
arch/x86/mm/pgtable.c void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) · line 140
Quoted 8 lines
arch/x86/mm/pgtable.c +2
-0@@ -83,6 +133,8 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) { +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (p4d));paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }
This, to me, seems like a really weird place to __SetPageTable(), why can't we do that on allocation?
Quoted 27 lines
mm/swap.c +6
-0@@ -888,6 +889,12 @@ void release_pages(struct page **pages, int nr)continue ; } +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +continue ; + } +if (!put_page_testzero (page))continue ;mm/swap_state.c +1
-0@@ -22,6 +22,7 @@#include <linux/swap_slots.h> #include <linux/huge_mm.h> #include <linux/shmem_fs.h> +#include <asm/pgalloc.h> #include "internal.h" /* @@ -310,6 +311,11 @@ static inline void free_swap_cache(struct page *page)void free_page_and_swap_cache (struct page *page)
mm/swap_state.c static inline void free_swap_cache(struct page *page) · line 321
Quoted 10 lines
mm/swap_state.c +5
-0{free_swap_cache (page); +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +return ; + }if (!is_huge_zero_page (page))put_page (page); }
And then free_table() can __ClearPageTable() and all is nice and symmetric and all this weirdness goes away, no?
Mike Rapoport
reviewerRe: [PATCH RFC 5/9] x86, mm: Use cache of page tables
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Rick Edgecombe
arch/x86/mm/pgtable.c +6
-0@@ -54,6 +98,8 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) {pgtable_pte_page_dtor (pte);paravirt_release_pte (page_to_pfn (pte)); +/* Set Page Table so swap knows how to free it */ +__SetPageTable (pte);paravirt_tlb_remove_table (tlb, pte); }@@ -70,12 +116,16 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =1 ;#endif pgtable_pmd_page_dtor (page); +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pmd));paravirt_tlb_remove_table (tlb, page); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) { +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pud));paravirt_release_pud (__pa (pud) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (pud)); }@@ -83,6 +133,8 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) {
Quoted 8 lines
arch/x86/mm/pgtable.c void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) · line 140
Quoted 5 lines
arch/x86/mm/pgtable.c +2
-0+/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (p4d));paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }This, to me, seems like a really weird place to
__SetPageTable(), why can't we do that on allocation?
We call __ClearPageTable() at pgtable_pxy_page_dtor(), so at least for pte and pmd we need to somehow tell release_pages() what kind of page it was.
Quoted 40 lines
Quoted 27 lines
mm/swap.c +6
-0@@ -888,6 +889,12 @@ void release_pages(struct page **pages, int nr)continue ; } +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +continue ; + } +if (!put_page_testzero (page))continue ;mm/swap_state.c +1
-0@@ -22,6 +22,7 @@#include <linux/swap_slots.h> #include <linux/huge_mm.h> #include <linux/shmem_fs.h> +#include <asm/pgalloc.h> #include "internal.h" /* @@ -310,6 +311,11 @@ static inline void free_swap_cache(struct page *page)void free_page_and_swap_cache (struct page *page)mm/swap_state.c static inline void free_swap_cache(struct page *page) · line 321
Quoted 10 lines
mm/swap_state.c +5
-0{free_swap_cache (page); +if (PageTable (page)) { +__ClearPageTable (page); +free_table (page); +return ; + }if (!is_huge_zero_page (page))put_page (page); }And then
free_table()can__ClearPageTable()and all is nice and symmetric and all this weirdness goes away, no?
Signature · 2 lines
Sincerely yours, Mike.
Peter Zijlstra
maintainerRe: [PATCH RFC 5/9] x86, mm: Use cache of page tables
Quoted 41 lines · Mike Rapoport
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Rick Edgecombe
arch/x86/mm/pgtable.c +6
-0@@ -54,6 +98,8 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) {pgtable_pte_page_dtor (pte);paravirt_release_pte (page_to_pfn (pte)); +/* Set Page Table so swap knows how to free it */ +__SetPageTable (pte);paravirt_tlb_remove_table (tlb, pte); }@@ -70,12 +116,16 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =1 ;#endif pgtable_pmd_page_dtor (page); +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pmd));paravirt_tlb_remove_table (tlb, page); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) { +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pud));paravirt_release_pud (__pa (pud) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (pud)); }@@ -83,6 +133,8 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) {Quoted 8 lines
arch/x86/mm/pgtable.c void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) · line 140
Quoted 5 lines
arch/x86/mm/pgtable.c +2
-0+/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (p4d));paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }This, to me, seems like a really weird place to
__SetPageTable(), why can't we do that on allocation?
Quoted 2 lines
We call
__ClearPageTable()atpgtable_pxy_page_dtor(), so at least for pte and pmd we need to somehow tellrelease_pages()what kind of page it was.
Hurph, right, but then the added comment is misleading; s/Set/Reset/g. Still I'm thinking that if we do these allocators, moving the set/clear to the allocator would be the most natural place, perhaps we can remove them from the {c,d}tor.
Re: [PATCH RFC 5/9] x86, mm: Use cache of page tables
Quoted 51 lines · Peter Zijlstra
Quoted 45 lines · Mike Rapoport
Quoted 35 lines · Peter Zijlstra
Quoted 34 lines · Rick Edgecombe
arch/x86/mm/pgtable.c +2
-0@@ -54,6 +98,8 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) { pgtable_pte_page_dtor(pte); paravirt_release_pte(page_to_pfn(pte)); + /* Set Page Table so swap knows how to free it */ + __SetPageTable(pte); paravirt_tlb_remove_table(tlb, pte); }arch/x86/mm/pgtable.c +2
-0@@ -70,12 +116,16 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all = 1; #endif pgtable_pmd_page_dtor(page); + /* Set Page Table so swap nows how to free it */ + __SetPageTable(virt_to_page(pmd)); paravirt_tlb_remove_table(tlb, page); }
#ifCONFIG_PGTABLE_LEVELS> 3
void___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud){
+ /* Set Page Table so swap nows how to free it */ + __SetPageTable(virt_to_page(pud));paravirt_release_pud(__pa(pud) >> PAGE_SHIFT);paravirt_tlb_remove_table(tlb, virt_to_page(pud)); }arch/x86/mm/pgtable.c +1
-0@@ -83,6 +133,8 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + /* Set Page Table so swap nows how to free it */Quoted 8 lines
Quoted 4 lines
arch/x86/mm/pgtable.c +1
-0+ __SetPageTable(virt_to_page(p4d)); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d));}
This, to me, seems like a really weird place to
__SetPageTable(), why
can't we do that on allocation?Quoted 4 lines
We call
__ClearPageTable()atpgtable_pxy_page_dtor(), so at least for pte
and pmd we need to somehow tellrelease_pages()what kind of page it was.
Quoted 7 lines
Hurph, right, but then the added comment is misleading; s/Set/Reset/g.
Still I'm thinking that if we do these allocators, moving the set/clear
to the allocator would be the most natural place, perhaps we can remove
them from the {c,d}tor.
Hmm, yes. I guess there could be just x86 specific versions of the ctor/dtor that don't set the flag. Seems like it should work and be less confusing. Thanks.
Re: [PATCH RFC 5/9] x86, mm: Use cache of page tables
In reply to Mike RapoportQuoted 41 lines · Mike Rapoport
Quoted 31 lines · Peter Zijlstra
Quoted 30 lines · Rick Edgecombe
arch/x86/mm/pgtable.c +6
-0@@ -54,6 +98,8 @@ void ___pte_free_tlb(struct mmu_gather *tlb, struct page *pte) {pgtable_pte_page_dtor (pte);paravirt_release_pte (page_to_pfn (pte)); +/* Set Page Table so swap knows how to free it */ +__SetPageTable (pte);paravirt_tlb_remove_table (tlb, pte); }@@ -70,12 +116,16 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) tlb->need_flush_all =1 ;#endif pgtable_pmd_page_dtor (page); +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pmd));paravirt_tlb_remove_table (tlb, page); }#if CONFIG_PGTABLE_LEVELS >3 void ___pud_free_tlb (struct mmu_gather *tlb,pud_t *pud) { +/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (pud));paravirt_release_pud (__pa (pud) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (pud)); }@@ -83,6 +133,8 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud)#if CONFIG_PGTABLE_LEVELS >4 void ___p4d_free_tlb (struct mmu_gather *tlb,p4d_t *p4d) {Quoted 8 lines
arch/x86/mm/pgtable.c void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) · line 140
Quoted 5 lines
arch/x86/mm/pgtable.c +2
-0+/* Set Page Table so swap nows how to free it */ +__SetPageTable (virt_to_page (p4d));paravirt_release_p4d (__pa (p4d) >>PAGE_SHIFT );paravirt_tlb_remove_table (tlb,virt_to_page (p4d)); }This, to me, seems like a really weird place to
__SetPageTable(), why can't we do that on allocation?
Quoted 2 lines
We call
__ClearPageTable()atpgtable_pxy_page_dtor(), so at least for pte and pmd we need to somehow tellrelease_pages()what kind of page it was.
One of the things I've been thinking about doing is removing the pgtable dtors and instead calling the pgtable dtor in __put_page() if PageTable(). Might work nicely with this ...
Re: [PATCH RFC 5/9] x86, mm: Use cache of page tables
In reply to Rick EdgecombeQuoted 1 line · Rick Edgecombe
[...]
Quoted 13 lines
arch/x86/mm/pgtable.c +13
-0+#ifdef CONFIG_PKS_PG_TABLES +struct page *alloc_table (gfp_t gfp) +{ +struct page *table; + +if (!pks_page_en) +return alloc_page (gfp); + + table =get_grouped_page (numa_node_id (), &gpc_pks); +if (!table) +return NULL ; + +if (gfp & __GFP_ZERO)
arch/x86/mm/pgtable.c pgtable_t pte_alloc_one(struct mm_struct *mm) · line 62
Quoted 10 lines
arch/x86/mm/pgtable.c +10
-0+memset (page_address (table),0 ,PAGE_SIZE ); + +if (memcg_kmem_enabled () && + gfp & __GFP_ACCOUNT && + !__memcg_kmem_charge_page (table, gfp,0 )) { +free_table (table); + table =NULL ; + } + +VM_BUG_ON_PAGE (*(unsigned long *)&table->ptl, table);
table can be NULL due to charge failure.
Re: [PATCH RFC 5/9] x86, mm: Use cache of page tables
Quoted 21 lines · Shakeel Butt
Quoted 1 line · Rick Edgecombe
[...]
Quoted 17 lines
arch/x86/mm/pgtable.c +17
-0+#ifdef CONFIG_PKS_PG_TABLES +struct page *alloc_table (gfp_t gfp) +{ + struct page *table; + + if (!pks_page_en) + return alloc_page(gfp); + + table = get_grouped_page(numa_node_id(), &gpc_pks); + if (!table) + return NULL; + + if (gfp & __GFP_ZERO) + memset(page_address(table), 0, PAGE_SIZE); + + if (memcg_kmem_enabled() && + gfp & __GFP_ACCOUNT &&
Quoted 8 lines
Quoted 6 lines
+ !__memcg_kmem_charge_page(table, gfp, 0)) { +
free_table(table);
+ table = NULL;
+ }
+
+VM_BUG_ON_PAGE(*(unsigned long *)&table->ptl, table);table can be NULL due to charge failure.
Argh, yes. Thank you. I'll remove the VM_BUG_ON, it was left accidentally.
RFC PATCH 6/9 x86/mm/cpa: Add set_memory_pks()
In reply to Rick EdgecombeAdd function for setting PKS key on kernel memory.
--- arch/x86/include/asm/set_memory.h | 1 + arch/x86/mm/pat/set_memory.c | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/include/asm/set_memory.h +1 -0
@@ -52,6 +52,7 @@ int set_memory_decrypted(unsigned long addr, int numpages);int set_memory_np_noalias (unsigned long addr,int numpages);int set_memory_nonglobal (unsigned long addr,int numpages);int set_memory_global (unsigned long addr,int numpages); +int set_memory_pks (unsigned long addr,int numpages,int key);int set_pages_array_uc (struct page **pages,int addrinarray);int set_pages_array_wc (struct page **pages,int addrinarray);
arch/x86/mm/pat/set_memory.c +7 -0
@@ -1914,6 +1914,13 @@ int set_memory_wb(unsigned long addr, int numpages) }EXPORT_SYMBOL (set_memory_wb); +int set_memory_pks (unsigned long addr,int numpages,int key) +{ +return change_page_attr_set_clr (&addr, numpages,__pgprot (_PAGE_PKEY (key)), +__pgprot (_PAGE_PKEY (0xF & ~(unsigned int )key)), +0 ,0 ,NULL ); +} +int set_memory_x (unsigned long addr,int numpages) {if (!(__supported_pte_mask & _PAGE_NX))
Signature · 1 line
2.30.2
RFC PATCH 7/9 x86/mm/cpa: Add perm callbacks to grouped pages
In reply to Rick EdgecombeFuture patches will need to set permissions on pages in the cache, so add some callbacks that let gouped page cache callers provide a callback the component can call when replenishing the cache or free-ing pages via the shrinker.
--- arch/x86/include/asm/set_memory.h | 8 +++++++- arch/x86/mm/pat/set_memory.c | 26 +++++++++++++++++++++++--- arch/x86/mm/pgtable.c | 3 ++- 3 files changed, 32 insertions(+), 5 deletions(-)
arch/x86/include/asm/set_memory.h +7 -1
@@ -139,14 +139,20 @@ static inline int clear_mce_nospec(unsigned long pfn)*/ #endif +typedef int (*gpc_callback )(struct page *,unsigned int ); +struct grouped_page_cache {struct shrinker shrinker;struct list_lru lru;gfp_t gfp; +gpc_callback pre_add_to_cache; +gpc_callback pre_shrink_free;atomic_t nid_round_robin; }; -int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp); +int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp, +gpc_callback pre_add_to_cache, +gpc_callback pre_shrink_free);struct page *get_grouped_page (int node,struct grouped_page_cache *gpc);void free_grouped_page (struct grouped_page_cache *gpc,struct page *page);
arch/x86/mm/pat/set_memory.c +23 -3
@@ -2356,6 +2356,9 @@ static void __dispose_pages(struct grouped_page_cache *gpc, struct list_head *helist_del (cur); +if (gpc->pre_shrink_free) + gpc->pre_shrink_free (page,1 ); +__free_pages (page,0 ); } }
@@ -2413,18 +2416,33 @@ static struct page *__replenish_grouped_pages(struct grouped_page_cache *gpc, inint i; page =__alloc_page_order (node, gpc->gfp,HUGETLB_PAGE_ORDER ); -if (!page) -return __alloc_page_order (node, gpc->gfp,0 ); +if (!page) { + page =__alloc_page_order (node, gpc->gfp,0 ); +if (gpc->pre_add_to_cache) + gpc->pre_add_to_cache (page,1 ); +return page; + }split_page (page,HUGETLB_PAGE_ORDER ); +/* If fail to prepare to be added, try to clean up and free */ +if (gpc->pre_add_to_cache && gpc->pre_add_to_cache (page, hpage_cnt)) { +if (gpc->pre_shrink_free) + gpc->pre_shrink_free (page, hpage_cnt); +for (i =0 ; i < hpage_cnt; i++) +__free_pages (&page[i],0 ); +return NULL ; + } +for (i =1 ; i < hpage_cnt; i++)free_grouped_page (gpc, &page[i]);return &page[0 ]; } -int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp) +int init_grouped_page_cache (struct grouped_page_cache *gpc,gfp_t gfp, +gpc_callback pre_add_to_cache, +gpc_callback pre_shrink_free) {int err =0 ;
@@ -2442,6 +2460,8 @@ int init_grouped_page_cache(struct grouped_page_cache *gpc, gfp_t gfp)if (err)list_lru_destroy (&gpc->lru); + gpc->pre_add_to_cache = pre_add_to_cache; + gpc->pre_shrink_free = pre_shrink_free; out:return err; }
arch/x86/mm/pgtable.c +2 -1
@@ -926,7 +926,8 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)#ifdef CONFIG_PKS_PG_TABLES static int __init pks_page_init (void ) { - pks_page_en = !init_grouped_page_cache (&gpc_pks,GFP_KERNEL |PGTABLE_HIGHMEM ); + pks_page_en = !init_grouped_page_cache (&gpc_pks,GFP_KERNEL |PGTABLE_HIGHMEM , +NULL ,NULL ); out:return !pks_page_en;
Signature · 1 line
2.30.2
RFC PATCH 8/9 x86, mm: Protect page tables with PKS
In reply to Rick EdgecombeWrite protect page tables with PKS. Toggle writeability inside the pgtable.h defined page table modifiction functions.
Do not protect the direct map page tables as it is more complicated and will come in a later patch.
--- arch/x86/boot/compressed/ident_map_64.c | 5 ++ arch/x86/include/asm/pgalloc.h | 2 + arch/x86/include/asm/pgtable.h | 26 ++++++++- arch/x86/include/asm/pgtable_64.h | 33 ++++++++++-- arch/x86/include/asm/pkeys_common.h | 8 ++- arch/x86/mm/pgtable.c | 72 ++++++++++++++++++++++--- mm/Kconfig | 6 ++- 7 files changed, 140 insertions(+), 12 deletions(-)
arch/x86/boot/compressed/ident_map_64.c +5 -0
@@ -349,3 +349,8 @@ void do_boot_page_fault(struct pt_regs *regs, unsigned long error_code)*/ add_identity_map (address, end); } + +#ifdef CONFIG_PKS_PG_TABLES +void enable_pgtable_write (void ) {} +void disable_pgtable_write (void ) {} +#endif
arch/x86/include/asm/pgalloc.h +2 -0
@@ -6,6 +6,8 @@#include <linux/mm.h> /* for struct page */ #include <linux/pagemap.h> +#define STATIC_TABLE_KEY 1 +#define __HAVE_ARCH_PTE_ALLOC_ONE#ifdef CONFIG_PKS_PG_TABLES #define __HAVE_ARCH_FREE_TABLE
arch/x86/include/asm/pgtable.h +25 -1
@@ -117,6 +117,14 @@ extern pmdval_t early_pmd_flags;#define arch_end_context_switch (prev) do {} while(0)#endif /* CONFIG_PARAVIRT_XXL */ +#ifdef CONFIG_PKS_PG_TABLES +void enable_pgtable_write (void ); +void disable_pgtable_write (void ); +#else /* CONFIG_PKS_PG_TABLES */ +static void enable_pgtable_write (void ) { } +static void disable_pgtable_write (void ) { } +#endif /* CONFIG_PKS_PG_TABLES */ +/* * The following only work if pte_present() is true. * Undefined behaviour if not..
@@ -1102,7 +1110,9 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm,static inline void ptep_set_wrprotect (struct mm_struct *mm,unsigned long addr,pte_t *ptep) { +enable_pgtable_write ();clear_bit (_PAGE_BIT_RW, (unsigned long *)&ptep->pte); +disable_pgtable_write (); }#define flush_tlb_fix_spurious_fault (vma, address) do { } while (0)
@@ -1152,7 +1162,9 @@ static inline pud_t pudp_huge_get_and_clear(struct mm_struct *mm,static inline void pmdp_set_wrprotect (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp) { +enable_pgtable_write ();clear_bit (_PAGE_BIT_RW, (unsigned long *)pmdp); +disable_pgtable_write (); }#define pud_write pud_write
@@ -1167,10 +1179,18 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t pmd) { if (IS_ENABLED(CONFIG_SMP)) { - return xchg(pmdp, pmd); + pmd_t ret; + +enable_pgtable_write (); + ret =xchg (pmdp, pmd); +disable_pgtable_write (); + +return ret; } else {pmd_t old = *pmdp; +enable_pgtable_write ();WRITE_ONCE (*pmdp, pmd); +disable_pgtable_write ();return old; } }
@@ -1253,13 +1273,17 @@ static inline p4d_t *user_to_kernel_p4dp(p4d_t *p4dp)*/ static inline void clone_pgd_range (pgd_t *dst,pgd_t *src,int count) { +enable_pgtable_write ();memcpy (dst, src, count *sizeof (pgd_t)); +disable_pgtable_write ();#ifdef CONFIG_PAGE_TABLE_ISOLATION if (!static_cpu_has (X86_FEATURE_PTI ))return ;/* Clone the user space pgd as well */ +enable_pgtable_write ();memcpy (kernel_to_user_pgdp (dst),kernel_to_user_pgdp (src), count *sizeof (pgd_t)); +disable_pgtable_write ();#endif }
arch/x86/include/asm/pgtable_64.h +30 -3
@@ -64,7 +64,9 @@ void set_pte_vaddr_pud(pud_t *pud_page, unsigned long vaddr, pte_t new_pte);static inline void native_set_pte (pte_t *ptep,pte_t pte) { +enable_pgtable_write ();WRITE_ONCE (*ptep, pte); +disable_pgtable_write (); } static inline void native_pte_clear(struct mm_struct *mm, unsigned long addr,
@@ -80,7 +82,9 @@ static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte)static inline void native_set_pmd (pmd_t *pmdp,pmd_t pmd) { +enable_pgtable_write ();WRITE_ONCE (*pmdp, pmd); +disable_pgtable_write (); }static inline void native_pmd_clear (pmd_t *pmd)
@@ -91,7 +95,12 @@ static inline void native_pmd_clear(pmd_t *pmd)static inline pte_t native_ptep_get_and_clear (pte_t *xp) {#ifdef CONFIG_SMP -return native_make_pte (xchg (&xp->pte,0 )); +pteval_t pte_val; + +enable_pgtable_write (); + pte_val =xchg (&xp->pte,0 ); +disable_pgtable_write (); +return native_make_pte (pte_val);#else /* native_local_ptep_get_and_clear, but duplicated because of cyclic dependency */
@@ -104,7 +113,12 @@ static inline pte_t native_ptep_get_and_clear(pte_t *xp)static inline pmd_t native_pmdp_get_and_clear (pmd_t *xp) {#ifdef CONFIG_SMP -return native_make_pmd (xchg (&xp->pmd,0 )); +pteval_t pte_val; + +enable_pgtable_write (); + pte_val =xchg (&xp->pmd,0 ); +disable_pgtable_write (); +return native_make_pmd (pte_val);#else /* native_local_pmdp_get_and_clear, but duplicated because of cyclic dependency */
@@ -116,7 +130,9 @@ static inline pmd_t native_pmdp_get_and_clear(pmd_t *xp)static inline void native_set_pud (pud_t *pudp,pud_t pud) { +enable_pgtable_write ();WRITE_ONCE (*pudp, pud); +disable_pgtable_write (); }static inline void native_pud_clear (pud_t *pud)
@@ -127,7 +143,12 @@ static inline void native_pud_clear(pud_t *pud)static inline pud_t native_pudp_get_and_clear (pud_t *xp) {#ifdef CONFIG_SMP -return native_make_pud (xchg (&xp->pud,0 )); +pteval_t pte_val; + +enable_pgtable_write (); + pte_val =xchg (&xp->pud,0 ); +disable_pgtable_write (); +return native_make_pud (pte_val);#else /* native_local_pudp_get_and_clear, * but duplicated because of cyclic dependency
@@ -144,13 +165,17 @@ static inline void native_set_p4d(p4d_t *p4dp, p4d_t p4d)pgd_t pgd;if (pgtable_l5_enabled () || !IS_ENABLED (CONFIG_PAGE_TABLE_ISOLATION )) { +enable_pgtable_write ();WRITE_ONCE (*p4dp, p4d); +disable_pgtable_write ();return ; } pgd =native_make_pgd (native_p4d_val (p4d)); pgd =pti_set_user_pgtbl ((pgd_t *)p4dp, pgd); +enable_pgtable_write ();WRITE_ONCE (*p4dp,native_make_p4d (native_pgd_val (pgd))); +disable_pgtable_write (); }static inline void native_p4d_clear (p4d_t *p4d)
@@ -160,7 +185,9 @@ static inline void native_p4d_clear(p4d_t *p4d)static inline void native_set_pgd (pgd_t *pgdp,pgd_t pgd) { +enable_pgtable_write ();WRITE_ONCE (*pgdp,pti_set_user_pgtbl (pgdp, pgd)); +disable_pgtable_write (); }static inline void native_pgd_clear (pgd_t *pgd)
arch/x86/include/asm/pkeys_common.h +7 -1
@@ -25,7 +25,13 @@* * NOTE: This needs to be a macro to be used as part of the INIT_THREAD macro. */ -#define INIT_PKRS_VALUE (PKR_AD_KEY(1) | PKR_AD_KEY(2) | PKR_AD_KEY(3) | \ + +/* +* HACK: There is no global pkeys support yet. We want the pg table key to be +* read only, not disabled. Assume the page table key will be key 1 and set it +* WD in the default mask. +*/ +#define INIT_PKRS_VALUE (PKR_WD_KEY(1) | PKR_AD_KEY(2) | PKR_AD_KEY(3) | \ PKR_AD_KEY(4) | PKR_AD_KEY(5) | PKR_AD_KEY(6) | \ PKR_AD_KEY(7) | PKR_AD_KEY(8) | PKR_AD_KEY(9) | \ PKR_AD_KEY(10) | PKR_AD_KEY(11) | PKR_AD_KEY(12) | \
arch/x86/mm/pgtable.c +66 -6
143 lines not on this page · raw message
mm/Kconfig +5 -1
@@ -812,7 +812,11 @@ config ARCH_HAS_SUPERVISOR_PKEYS bool config ARCH_ENABLE_SUPERVISOR_PKEYS def_bool y - depends on PKS_TEST + depends on PKS_TEST || PKS_PG_TABLES + +config PKS_PG_TABLES + def_bool y + depends on !PAGE_TABLE_ISOLATION && !HIGHMEM && !X86_PAE && PGTABLE_LEVELS = 4 config PERCPU_STATS bool "Collect percpu memory statistics"
Signature · 1 line
2.30.2
RFC PATCH 9/9 x86, cpa: PKS protect direct map page tables
In reply to Rick EdgecombeProtecting direct map page tables is a bit more difficult because a page table may be needed for a page split as part new setting the PKS permission the new page table. So in the case of an empty cache of page tables the page table allocator could get into a situation where it cannot create any more page tables.
Several solutions were looked at:
1. Break the direct map with pages allocated from the large page being converted to PKS. This would result in a window where the table could be written to right before it was linked into the page tables. It also depends on high order pages being available, and so would regress from the un-protecteed behavior in that respect.
2. Hold some page tables in reserve to be able to break the large page for a new 2MB page, but if there are no 2MB page's available we may need to add a single page to the cache, in which case we would use up the reserve of page tables needed to break a new page, but not get enough page tables back to replenish the resereve.
3. Always map the direct map at 4k when protecting page tables so that pages don't need to be broken to map them with a PKS key. This would have undesirable performance.
4. Lastly, the strategy employed in this patch, have a separate cache of page tables just used for the direct map. Early in boot, squirrel away enough page tables to map the direct map at 4k. This comes with the same memory overhead of mapping the direct map at 4k, but gets the other benefits of mapping the direct map as large pages.
Some direct map page tables currently still escape protection, so there are a few todos. It is a rough sketch of the idea.
--- arch/x86/include/asm/set_memory.h | 2 + arch/x86/mm/init.c | 40 +++++++++ arch/x86/mm/pat/set_memory.c | 134 +++++++++++++++++++++++++++++- 3 files changed, 172 insertions(+), 4 deletions(-)
arch/x86/include/asm/set_memory.h +2 -0
@@ -90,6 +90,8 @@ bool kernel_page_present(struct page *page);extern int kernel_set_to_readonly; +void add_pks_table (unsigned long addr); +#ifdef CONFIG_X86_64 /* * Prevent speculative access to the page by either unmapping
arch/x86/mm/init.c +40 -0
@@ -26,6 +26,7 @@#include <asm/pti.h> #include <asm/text-patching.h> #include <asm/memtype.h> +#include <asm/pgalloc.h> /* * We need to define the tracepoints somewhere, and tlb.c
@@ -119,6 +120,8 @@ __ref void *alloc_low_pages(unsigned int num)if (after_bootmem) {unsigned int order; +WARN_ON (IS_ENABLED (CONFIG_PKS_PG_TABLES )); +/* TODO: When does this happen, how to deal with the order? */ order =get_order ((unsigned long )num <<PAGE_SHIFT );return (void *)__get_free_pages (GFP_ATOMIC | __GFP_ZERO, order); }
@@ -153,6 +156,11 @@ __ref void *alloc_low_pages(unsigned int num)clear_page (adr); } +printk ("Allocing un-protected page table: %lx\n" , (unsigned long )__va (pfn <<PAGE_SHIFT )); +/* +* TODO: Save the va of this table to PKS protect post boot, but we need a small allocation +* for the list... +*/ return __va (pfn <<PAGE_SHIFT ); }
@@ -532,6 +540,36 @@ unsigned long __ref init_memory_mapping(unsigned long start,return ret >>PAGE_SHIFT ; } +/* TODO: Check this math */ +static u64 calc_tables_needed (unsigned int size) +{ +unsigned int puds = size >>PUD_SHIFT ; +unsigned int pmds = size >>PMD_SHIFT ; +unsigned int needed_to_map_tables =0 ;//?? + +return puds + pmds + needed_to_map_tables; +} + +static void __init reserve_page_tables (u64 start,u64 end) +{ +u64 reserve_size =calc_tables_needed (end - start); +u64 reserved =0 ; +u64 cur; +int i; + +while (reserved < reserve_size) { + cur =memblock_find_in_range (start, end,HPAGE_SIZE ,HPAGE_SIZE ); +if (!cur) { +WARN (1 ,"Could not reserve HPAGE size page tables" ); +return ; + } +memblock_reserve (cur,HPAGE_SIZE ); +for (i =0 ; i <HPAGE_SIZE ; i +=PAGE_SIZE ) +add_pks_table ((long unsigned int )__va (cur + i)); + reserved +=HPAGE_SIZE ; + } +} +/* * We need to iterate through the E820 memory map and create direct mappings * for only E820_TYPE_RAM and E820_KERN_RESERVED regions. We cannot simply
@@ -568,6 +606,8 @@ static unsigned long __init init_range_memory_mapping(init_memory_mapping (start, end,PAGE_KERNEL ); mapped_ram_size += end - start; can_use_brk_pgt = true; +if (IS_ENABLED (CONFIG_PKS_PG_TABLES )) +reserve_page_tables (start, end); }return mapped_ram_size;
arch/x86/mm/pat/set_memory.c +130 -4
170 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 7 lines · Rick Edgecombe
This is based on V6 [1] of the core PKS infrastructure patches. PKS infrastructure follow-on’s are planned to enable keys to be set to the same permissions globally. Since this usage needs a key to be set globally read-only by default, a small temporary solution is hacked up in patch 8. Long term, PKS protected page tables would use a better and more generic solution to achieve this.
Before you send this out I've been thinking about this more and I think I would prefer you not call this 'globally' setting the key. Because you don't really want to be able to update the key globally like I originally suggested for kmap(). What is required is to set a different default for the key which gets used by all threads by 'default'.
What is really missing is how to get the default changed after it may have been used by some threads... thus the 'global' nature... Perhaps I am picking nits here but I think it may go over better with Thomas and the maintainers. Or maybe not... :-)
Would it be too much trouble to call this a 'default' change? Because that is really what you implement?
Ira
Quoted 5 lines · Rick Edgecombe
This is a POC for write protecting page tables with PKS (Protection Keys for Supervisor) [1]. The basic idea is to make the page tables read only, except temporarily on a per-cpu basis when they need to be modified. I’m looking for opinions on whether people like the general direction of this in terms of value and implementation.
Yay!
Quoted 10 lines
Why would people want this?
===========================
Page tables are the basis for many types of protections and as such, are a juicy target for attackers. Mapping them read-only will make them harder to use in attacks.This protects against an attacker that has acquired the ability to write to the page tables. It's not foolproof because an attacker who can execute arbitrary code can either disable PKS directly, or simply call the same functions that the kernel uses for legitimate page table writes.
I think it absolutely has value. The exploit techniques I'm aware of that target the page table are usually attempting to upgrade an arbitrary write into execution (e.g. write to kernel text after setting kernel text writable in the page table) or similar "data only" attacks (make sensitive page writable).
It looks like PKS-protected page tables would be much like the RO-protected text pages in the sense that there is already code in the kernel to do things to make it writable, change text, and set it read-only again (alternatives, ftrace, etc). That said, making the PKS manipulation code be inline to page-writing code would make it less available for ROP/JOP, if an attack DID want to go that route.
Quoted 7 lines
Why use PKS for this?
=====================
PKS is an upcoming CPU feature that allows supervisor virtual memory permissions to be changed without flushing the TLB, like PKU does for user memory. Protecting page tables would normally be really expensive because you would have to do it with paging itself. PKS helps by providing a way to toggle the writability of the page tables with just a per-cpu MSR.
The per-cpu-ness is really important for both performance and for avoiding temporal attacks where an arbitrary write in one CPU is timed against a page table write in another CPU.
Quoted 10 lines
Performance impacts
===================
Setting direct map permissions on whatever random page gets allocated for a page table would result in a lot of kernel range shootdowns and direct map large page shattering. So the way the PKS page table memory is created is similar to this module page clustering series[2], where a cache of pages is replenished from 2MB pages such that the direct map permissions and associated breakage is localized on the direct map. In the PKS page tables case, a PKS key is pre-applied to the direct map for pages in the cache.
Quoted 4 lines
There would be some costs of memory overhead in order to protect the direct map page tables. There would also be some extra kernel range shootdowns to replenish the cache on occasion, from setting the PKS key on the direct map of the new pages. I don’t have any actual performance data yet.
What CPU models are expected to have PKS?
Quoted 8 lines
This is based on V6 [1] of the core PKS infrastructure patches. PKS infrastructure follow-on’s are planned to enable keys to be set to the same permissions globally. Since this usage needs a key to be set globally read-only by default, a small temporary solution is hacked up in patch 8. Long term, PKS protected page tables would use a better and more generic solution to achieve this.
Ah, neat!
Quoted 1 line
Ooh. What does this do for performance? It sounds like less TLB pressure, IIUC?
Signature · 1 line
Kees Cook
Peter Zijlstra
maintainerQuoted 4 lines · Kees Cook
It looks like PKS-protected page tables would be much like the RO-protected text pages in the sense that there is already code in the kernel to do things to make it writable, change text, and set it read-only again (alternatives, ftrace, etc).
We don't actually modify text by changing the mapping at all. We modify through a writable (but not executable) temporary alias on the page (on x86).
Once a mapping is RX it will *never* be writable again (until we tear it all down).
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Kees Cook
It looks like PKS-protected page tables would be much like the RO-protected text pages in the sense that there is already code in the kernel to do things to make it writable, change text, and set it read-only again (alternatives, ftrace, etc).
Quoted 6 lines
We don't actually modify text by changing the mapping at all. We modify through a writable (but not executable) temporary alias on the page (on x86).
Once a mapping is RX it will *never* be writable again (until we tear it all down).
Yes, quite true. I was trying to answer the concern about "is it okay that there is a routine in the kernel that can write to page tables (via temporary disabling of PKS)?" by saying "yes, this is fine -- we already have similar routines in the kernel that bypass memory protections, and that's okay because the defense is primarily about blocking flaws that allow attacker-controlled writes to be used to leverage greater control over kernel state, of which the page tables are pretty central. :)
Signature · 1 line
Kees Cook
Quoted 12 lines · Kees Cook
Quoted 11 lines
infrastructure follow-on’s are planned to enable keys to be set to the same
permissions globally. Since this usage needs a key to be set globally read-only by default, a small temporary solution is hacked up in patch 8. Long
term, PKS protected page tables would use a better and more generic solution
to achieve this.
Quoted 7 lines
Ah, neat!
Quoted 2 lines
Ooh. What does this do for performance? It sounds like less TLB pressure, IIUC?
Yea, less TLB pressure, faster page table walks in theory. There was some testing that showed having all 4k pages was bad for performance:
https://lore.kernel.org/linux-mm/[email protected]/
I'm not sure exactly how much breakage is needed before problems start to show up, but there was also someone posting that large amounts of tracing was noticeable for their workload.
Quoted 12 lines · Kees Cook
Quoted 10 lines · Rick Edgecombe
Performance impacts
===================
Setting direct map permissions on whatever random page gets allocated for a page table would result in a lot of kernel range shootdowns and direct map large page shattering. So the way the PKS page table memory is created is similar to this module page clustering series[2], where a cache of pages is replenished from 2MB pages such that the direct map permissions and associated breakage is localized on the direct map. In the PKS page tables case, a PKS key is pre-applied to the direct map for pages in the cache.
Quoted 6 lines
Quoted 4 lines
There would be some costs of memory overhead in order to protect the direct map page tables. There would also be some extra kernel range shootdowns to replenish the cache on occasion, from setting the PKS key on the direct map of the new pages. I don’t have any actual performance data yet.
What CPU models are expected to have PKS?
Supervisor Memory Protection Keys (PKS) is a feature which is found on Intel’s Sapphire Rapids (and later) “Scalable Processor” Server CPUs. It will also be available in future non-server Intel parts.
Also qemu has some support as well.
https://www.qemu.org/2021/04/30/qemu-6-0-0/
Ira
Quoted 12 lines · Rick Edgecombe
This is a POC for write protecting page tables with PKS (Protection Keys for Supervisor) [1]. The basic idea is to make the page tables read only, except temporarily on a per-cpu basis when they need to be modified. I’m looking for opinions on whether people like the general direction of this in terms of value and implementation.
Why would people want this?
===========================
Page tables are the basis for many types of protections and as such, are a juicy target for attackers. Mapping them read-only will make them harder to use in attacks.
Quoted 4 lines
This protects against an attacker that has acquired the ability to write to the page tables. It's not foolproof because an attacker who can execute arbitrary code can either disable PKS directly, or simply call the same functions that the kernel uses for legitimate page table writes.
Yeah, it's a good idea. I've once used a similar approach locally during debugging a problem that appeared to be stray writes hitting page tables, and without PKS I indeed made the whole pages read-only when not touched by the designated code.
Quoted 7 lines
Why use PKS for this?
=====================
PKS is an upcoming CPU feature that allows supervisor virtual memory permissions to be changed without flushing the TLB, like PKU does for user memory. Protecting page tables would normally be really expensive because you would have to do it with paging itself. PKS helps by providing a way to toggle the writability of the page tables with just a per-cpu MSR.
I can see in patch 8/9 that you are flipping the MSR around individual operations on page table entries. In my patch I hooked making the page table writable to obtaining the page table lock (IIRC I had only the PTE level fully handled though). Wonder if that would be better tradeoff even for your MSR approach?
Vlastimil
Quoted 63 lines
Performance impacts
===================
Setting direct map permissions on whatever random page gets allocated for a page table would result in a lot of kernel range shootdowns and direct map large page shattering. So the way the PKS page table memory is created is similar to this module page clustering series[2], where a cache of pages is replenished from 2MB pages such that the direct map permissions and associated breakage is localized on the direct map. In the PKS page tables case, a PKS key is pre-applied to the direct map for pages in the cache.There would be some costs of memory overhead in order to protect the direct map page tables. There would also be some extra kernel range shootdowns to replenish the cache on occasion, from setting the PKS key on the direct map of the new pages. I don’t have any actual performance data yet.
This is based on V6 [1] of the core PKS infrastructure patches. PKS infrastructure follow-on’s are planned to enable keys to be set to the same permissions globally. Since this usage needs a key to be set globally read-only by default, a small temporary solution is hacked up in patch 8. Long term, PKS protected page tables would use a better and more generic solution to achieve this.
[1]
https://lore.kernel.org/lkml/[email protected]/
[2]
https://lore.kernel.org/lkml/[email protected]
/Thanks,
Rick
Rick Edgecombe (9):
list: Support getting most recent element inlist_lru
list: Support list head not in object forlist_lrux86/mm/cpa: Add grouped page allocations
mm: Explicitly zero page table lock ptr
x86, mm: Use cache of page tables x86/mm/cpa: Addset_memory_pks()
x86/mm/cpa: Add perm callbacks to grouped pages x86, mm: Protect page tables with PKS
x86, cpa: PKS protect direct map page tablesarch/x86/boot/compressed/ident_map_64.c | 5 + arch/x86/include/asm/pgalloc.h | 6 + arch/x86/include/asm/pgtable.h | 26 +- arch/x86/include/asm/pgtable_64.h | 33 ++- arch/x86/include/asm/pkeys_common.h | 8 +- arch/x86/include/asm/set_memory.h | 23 ++ arch/x86/mm/init.c | 40 +++ arch/x86/mm/pat/set_memory.c | 312 +++++++++++++++++++++++- arch/x86/mm/pgtable.c | 144 ++++++++++- include/asm-generic/pgalloc.h | 42 +++- include/linux/list_lru.h | 26 ++ include/linux/mm.h | 7 + mm/Kconfig | 6 +- mm/list_lru.c | 38 ++- mm/memory.c | 1 + mm/swap.c | 7 + mm/swap_state.c | 6 + 17 files changed, 705 insertions(+), 25 deletions(-)
Peter Zijlstra
maintainerQuoted 10 lines · Vlastimil Babka
Quoted 7 lines · Rick Edgecombe
Why use PKS for this?
=====================
PKS is an upcoming CPU feature that allows supervisor virtual memory permissions to be changed without flushing the TLB, like PKU does for user memory. Protecting page tables would normally be really expensive because you would have to do it with paging itself. PKS helps by providing a way to toggle the writability of the page tables with just a per-cpu MSR.
Quoted 4 lines
I can see in patch 8/9 that you are flipping the MSR around individual operations on page table entries. In my patch I hooked making the page table writable to obtaining the page table lock (IIRC I had only the PTE level fully handled though). Wonder if that would be better tradeoff even for your MSR approach?
There's also the HIGHPTE code we could abuse to kmap an alias while we're at it.
Quoted 15 lines · Peter Zijlstra
Quoted 14 lines · Vlastimil Babka
Quoted 11 lines · Rick Edgecombe
Why use PKS for this?
=====================
PKS is an upcoming CPU feature that allows supervisor virtual memory
permissions to be changed without flushing the TLB, like PKU does for user
memory. Protecting page tables would normally be really expensive because you
would have to do it with paging itself. PKS helps by providing a way to toggle
the writability of the page tables with just a per-cpu MSR.
Quoted 9 lines
Quoted 8 lines
I can see in patch 8/9 that you are flipping the MSR around individual
operations on page table entries. In my patch I hooked making the page table
writable to obtaining the page table lock (IIRC I had only the PTE level fully
handled though). Wonder if that would be better tradeoff even for your MSR approach?
Hmm, I see, that could reduce the sprinkling of the enable/disable calls. It seems some(most?) of the kernel address space page table modification don't take the page table locks though so those callers would have to call something else to be able to write.
Quoted 2 lines
There's also the HIGHPTE code we could abuse to kmap an alias while we're at it.
For a non-PKS debug feature?
It might fit pretty easily into CONFIG_DEBUG_PAGEALLOC on top of this series. enable/disable_pgtable_write() could take a pointer that is ignored in PKS mode, but triggers a cpa call in a CONFIG_DEBUG_PAGETABLE_WRITE mode.