arm64: mm: support set_memory_encrypted/decrypted for vmalloc addresses
RFC PATCH 1/1 arm64: mm: support set_memory_encrypted/decrypted for vmalloc addresses
Currently __set_memory_enc_dec() only handles linear map (lm) addresses and returns -EINVAL for anything else. This means callers using vmalloc'd buffers cannot mark memory as shared/protected with the RMM via set_memory_decrypted()/set_memory_encrypted().
Extend the implementation to handle vmalloc (non-linear-map) addresses by introducing __set_va_addr_enc_dec(). For vmalloc addresses, the page table entries are not contiguous in the physical address space, so the function walks the vm_area's pages array and issues per-page RSI calls to transition each page between shared and protected states.
The original linear-map path is factored out into __set_lm_addr_enc_dec(), and __set_memory_enc_dec() now dispatches to the appropriate helper based on whether the address is a linear map address.
--- arch/arm64/mm/pageattr.c | 74 +++++++++++++++++++++++++++++++++++----- 1 file changed, 65 insertions(+), 9 deletions(-)
arch/arm64/mm/pageattr.c +65 -9
@@ -275,20 +275,12 @@ int set_direct_map_default_noflush(struct page *page) PAGE_SIZE, set_mask, clear_mask); } -static int __set_memory_enc_dec (unsigned long addr, -int numpages, -bool encrypt) +static int __set_lm_addr_enc_dec (unsigned long addr,int numpages,bool encrypt) {unsigned long set_prot =0 , clear_prot =0 ;phys_addr_t start, end;int ret; -if (!is_realm_world ()) -return 0 ; - -if (!__is_lm_address (addr)) -return -EINVAL ; - start =__virt_to_phys (addr); end = start + numpages *PAGE_SIZE ;
@@ -321,6 +313,70 @@ static int __set_memory_enc_dec(unsigned long addr, __pgprot(PTE_PRESENT_INVALID)); } +static int __set_va_addr_enc_dec (unsigned long addr,int numpages,bool encrypt) +{ +unsigned long set_prot =0 , clear_prot =0 , start_idx; +struct vm_struct *area; +int i, ret; + +if (encrypt) + clear_prot =PROT_NS_SHARED ; +else + set_prot =PROT_NS_SHARED ; + + area =find_vm_area ((void *)addr); +if (!area) +return -EINVAL ; + + start_idx = ((unsigned long )kasan_reset_tag ((void *)addr) - + (unsigned long )kasan_reset_tag (area->addr)) >> +PAGE_SHIFT ; + +if (start_idx + numpages > area->nr_pages) +return -EINVAL ; + +/* +* Break the mapping before we make any changes to avoid stale TLB +* entries or Synchronous External Aborts caused by RIPAS_EMPTY +*/ + ret =change_memory_common (addr, numpages, +__pgprot (set_prot |PTE_PRESENT_INVALID ), +__pgprot (clear_prot |PTE_PRESENT_VALID_KERNEL )); + +if (ret) +return ret; + +for (i =0 ; i < numpages; i++) { +struct page *page = area->pages[start_idx + i]; +phys_addr_t phys =page_to_phys (page); + +if (encrypt) { + ret =rsi_set_memory_range_protected (phys, + phys +PAGE_SIZE ); + }else { + ret =rsi_set_memory_range_shared (phys, + phys +PAGE_SIZE ); + } +if (ret) +return ret; + } + +return change_memory_common (addr, numpages, +__pgprot (PTE_PRESENT_VALID_KERNEL ), +__pgprot (PTE_PRESENT_INVALID )); +} + +static int __set_memory_enc_dec (unsigned long addr,int numpages,bool encrypt) +{ +if (!is_realm_world ()) +return 0 ; + +if (!__is_lm_address (addr)) +return __set_va_addr_enc_dec (addr, numpages, encrypt); + +return __set_lm_addr_enc_dec (addr, numpages, encrypt); +} +static int realm_set_memory_encrypted (unsigned long addr,int numpages) {int ret =__set_memory_enc_dec (addr, numpages, true);
base-commit: 25f6040ca43a78048466b949994b8d637fe2fe07
Signature · 1 line
2.45.4
Catalin Marinas
maintainerQuoted 11 lines · Kameron Carr
Currently
__set_memory_enc_dec()only handles linear map (lm) addresses and returns-EINVALfor anything else. This means callers using vmalloc'd buffers cannot mark memory as shared/protected with the RMM via set_memory_decrypted()/set_memory_encrypted().Extend the implementation to handle vmalloc (non-linear-map) addresses by introducing
__set_va_addr_enc_dec(). For vmalloc addresses, the page table entries are not contiguous in the physical address space, so the function walks the vm_area's pages array and issues per-page RSI calls to transition each page between shared and protected states.
Quoted 3 lines
The original linear-map path is factored out into
__set_lm_addr_enc_dec(), and__set_memory_enc_dec()now dispatches to the appropriate helper based on whether the address is a linear map address.
Could you give more details about the user of set_memory_decrypted() on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something like GFP_DECRYPTED would be simpler to implement (even posted a hack but without vmalloc() support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just use pgprot_decrypted() for vmap(). No need to rewrite the page table after mapping the pages.
Signature · 1 line
Catalin
Quoted 8 lines · Catalin Marinas
Could you give more details about the user of
set_memory_decrypted()on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something likeGFP_DECRYPTEDwould be simpler to implement (even posted a hack but withoutvmalloc()support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just usepgprot_decrypted()forvmap(). No need to rewrite the page table after mapping the pages.
Thank you for the review. I understand that my approach is not ideal in terms of speed / cost. For my use case it was not an issue since the memory is typically only initialized once during device initialization, but I understand there could be a larger performance impact in other uses.
The use case I am interested in is Hyper-V netvsc (drivers/net/hyperv/netvsc.c). The network driver allocates large send and receive buffers (typically 16MB+) with vzalloc(), then registers them as Guest Physical Address Descriptor Lists (GPADLs) via vmbus_establish_gpadl(). Inside __vmbus_establish_gpadl() (drivers/hv/channel.c), set_memory_decrypted() is called on the buffer so the hypervisor can access the shared memory.
In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like GFP_DECRYPTED is an option.
I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well. He likely had netvsc's use of vmalloc in mind when he made the point "GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.
What is the current status of your proposed GFP_DECRYPTED implementation? Is this something you are actively working on?
I'd be happy to work on an RFC following the GFP_DECRYPTED approach if you think that's the right direction.
Regards,
Kameron Carr
Catalin Marinas
maintainerQuoted 9 lines · Kameron Carr
Quoted 8 lines · Catalin Marinas
Could you give more details about the user of
set_memory_decrypted()on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something likeGFP_DECRYPTEDwould be simpler to implement (even posted a hack but withoutvmalloc()support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just usepgprot_decrypted()forvmap(). No need to rewrite the page table after mapping the pages.
[...]
Quoted 6 lines
In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like
GFP_DECRYPTEDis an option.I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well.
Yes, that's the thread. It started originally as a GICv3 need (eventually we went for genpool).
Quoted 8 lines
He likely had netvsc's use of vmalloc in mind when he made the point "
GFP_DECRYPTEDshould work for the three memory allocation interfaces and their variants:alloc_pages(),kmalloc(), andvmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.What is the current status of your proposed
GFP_DECRYPTEDimplementation? Is this something you are actively working on?
Not really. But I've been looking at it again and I think it adds more problems than it solves. A GFP flag would be passed down to kmem_cache_alloc() and confuse the slab management if some pages are encrypted, others not for the same kmem_cache (SLAB_NO_MERGE wouldn't help). I wonder whether something like SLAB_DECRYPTED would work better for this if we really need it (not aware of any user though).
Anyway, let's ignore slab for now and look at vmalloc(). I can see hv_ringbuffer_init() using an explicit vmap(pgprot_decrypted()). While you could do this, it might be better to just add a VM_DECRYPTED flag and a few wrappers like vmalloc_decrypted(). It would call set_memory_decrypted() for the allocated pages and use pgprot_decrypted() for vmap. On vfree(), it will have to set the pages back to encrypted. It should be fairly mechanical to do (or a 5 min job for an LLM ;)).
Signature · 1 line
Catalin
Hi Catalin,
Quoted 37 lines · Catalin Marinas
Quoted 9 lines · Kameron Carr
Quoted 8 lines · Catalin Marinas
Could you give more details about the user of
set_memory_decrypted()on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something likeGFP_DECRYPTEDwould be simpler to implement (even posted a hack but withoutvmalloc()support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just usepgprot_decrypted()forvmap(). No need to rewrite the page table after mapping the pages.[...]
Quoted 6 lines
In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like
GFP_DECRYPTEDis an option.I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well.Yes, that's the thread. It started originally as a GICv3 need (eventually we went for genpool).
Quoted 8 lines
He likely had netvsc's use of vmalloc in mind when he made the point "
GFP_DECRYPTEDshould work for the three memory allocation interfaces and their variants:alloc_pages(),kmalloc(), andvmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.What is the current status of your proposed
GFP_DECRYPTEDimplementation? Is this something you are actively working on?Not really. But I've been looking at it again and I think it adds more problems than it solves. A GFP flag would be passed down to
kmem_cache_alloc()and confuse the slab management if some pages are encrypted, others not for the samekmem_cache(SLAB_NO_MERGEwouldn't help). I wonder whether something likeSLAB_DECRYPTEDwould work better for this if we really need it (not aware of any user though).
Quoted 8 lines
Anyway, let's ignore slab for now and look at
vmalloc(). I can seehv_ringbuffer_init()using an explicit vmap(pgprot_decrypted()). While you could do this, it might be better to just add aVM_DECRYPTEDflag and a few wrappers likevmalloc_decrypted(). It would callset_memory_decrypted()for the allocated pages and usepgprot_decrypted()for vmap. Onvfree(), it will have to set the pages back to encrypted. It should be fairly mechanical to do (or a 5 min job for an LLM ;)).
A GFP + SLAB flag is semantically better than only implementing decrypted allocations only in vmalloc, but it's surely way more complex and intrusive and we are running low on gfp flags on 32 bits :)
But I think we can push the _decrypted to kvmalloc() that would do either kmalloc() + set_memory_decrypted() or vmalloc_decrypted() and it seems that x86 and drivers could use that instead of alloc_pages() + set_memory_decrypted().
We also would want also to make x86::set_memory_decrypted() to only accept linear^w direct map addresses to ensure consistency between architectures.
Quoted 3 lines
Signature · 1 line
Catalin
Signature · 2 lines
Sincerely yours, Mike.