Skip to contentSkip to conversation

arm64: mm: support set_memory_encrypted/decrypted for vmalloc addresses

RFC PATCH In review Kameron Carr 3 participants 5 messages Apr 6 – Apr 30 arm64

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 arm64: mm: support set_memory_encrypted/decrypted for vmalloc addresses

Currently __set_memory_enc_dec() only handles linear map (lm) addresses and returns -EINVAL for anything else. This means callers using vmalloc'd buffers cannot mark memory as shared/protected with the RMM via set_memory_decrypted()/set_memory_encrypted().

Extend the implementation to handle vmalloc (non-linear-map) addresses by introducing __set_va_addr_enc_dec(). For vmalloc addresses, the page table entries are not contiguous in the physical address space, so the function walks the vm_area's pages array and issues per-page RSI calls to transition each page between shared and protected states.

The original linear-map path is factored out into __set_lm_addr_enc_dec(), and __set_memory_enc_dec() now dispatches to the appropriate helper based on whether the address is a linear map address.

---
 arch/arm64/mm/pageattr.c | 74 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 65 insertions(+), 9 deletions(-)
arch/arm64/mm/pageattr.c +65 -9
index ce035e1b4eaf..45058f61b957 100644
@@ -275,20 +275,12 @@ int set_direct_map_default_noflush(struct page *page)
 				 PAGE_SIZE, set_mask, clear_mask);
 }
 
-static int __set_memory_enc_dec(unsigned long addr,
-				int numpages,
-				bool encrypt)
+static int __set_lm_addr_enc_dec(unsigned long addr, int numpages, bool encrypt)
 {
 	unsigned long set_prot = 0, clear_prot = 0;
 	phys_addr_t start, end;
 	int ret;
 
-	if (!is_realm_world())
-		return 0;
-
-	if (!__is_lm_address(addr))
-		return -EINVAL;
-
 	start = __virt_to_phys(addr);
 	end = start + numpages * PAGE_SIZE;
 
@@ -321,6 +313,70 @@ static int __set_memory_enc_dec(unsigned long addr,
 				      __pgprot(PTE_PRESENT_INVALID));
 }
 
+static int __set_va_addr_enc_dec(unsigned long addr, int numpages, bool encrypt)
+{
+	unsigned long set_prot = 0, clear_prot = 0, start_idx;
+	struct vm_struct *area;
+	int i, ret;
+
+	if (encrypt)
+		clear_prot = PROT_NS_SHARED;
+	else
+		set_prot = PROT_NS_SHARED;
+
+	area = find_vm_area((void *)addr);
+	if (!area)
+		return -EINVAL;
+
+	start_idx = ((unsigned long)kasan_reset_tag((void *)addr) -
+		     (unsigned long)kasan_reset_tag(area->addr)) >>
+		    PAGE_SHIFT;
+
+	if (start_idx + numpages > area->nr_pages)
+		return -EINVAL;
+
+	/*
+	 * Break the mapping before we make any changes to avoid stale TLB
+	 * entries or Synchronous External Aborts caused by RIPAS_EMPTY
+	 */
+	ret = change_memory_common(addr, numpages,
+		__pgprot(set_prot | PTE_PRESENT_INVALID),
+		__pgprot(clear_prot | PTE_PRESENT_VALID_KERNEL));
+
+	if (ret)
+		return ret;
+
+	for (i = 0; i < numpages; i++) {
+		struct page *page = area->pages[start_idx + i];
+		phys_addr_t phys = page_to_phys(page);
+
+		if (encrypt) {
+			ret = rsi_set_memory_range_protected(phys,
+							     phys + PAGE_SIZE);
+		} else {
+			ret = rsi_set_memory_range_shared(phys,
+							  phys + PAGE_SIZE);
+		}
+		if (ret)
+			return ret;
+	}
+
+	return change_memory_common(addr, numpages,
+				    __pgprot(PTE_PRESENT_VALID_KERNEL),
+				    __pgprot(PTE_PRESENT_INVALID));
+}
+
+static int __set_memory_enc_dec(unsigned long addr, int numpages, bool encrypt)
+{
+	if (!is_realm_world())
+		return 0;
+
+	if (!__is_lm_address(addr))
+		return __set_va_addr_enc_dec(addr, numpages, encrypt);
+
+	return __set_lm_addr_enc_dec(addr, numpages, encrypt);
+}
+
 static int realm_set_memory_encrypted(unsigned long addr, int numpages)
 {
 	int ret = __set_memory_enc_dec(addr, numpages, true);

base-commit: 25f6040ca43a78048466b949994b8d637fe2fe07

Signature · 1 line
2.45.4
maintainer
#1 lore raw
Quoted 11 lines · Kameron Carr

Currently __set_memory_enc_dec() only handles linear map (lm) addresses and returns -EINVAL for anything else. This means callers using vmalloc'd buffers cannot mark memory as shared/protected with the RMM via set_memory_decrypted()/set_memory_encrypted().

Extend the implementation to handle vmalloc (non-linear-map) addresses by introducing __set_va_addr_enc_dec(). For vmalloc addresses, the page table entries are not contiguous in the physical address space, so the function walks the vm_area's pages array and issues per-page RSI calls to transition each page between shared and protected states.

Quoted 3 lines

The original linear-map path is factored out into __set_lm_addr_enc_dec(), and __set_memory_enc_dec() now dispatches to the appropriate helper based on whether the address is a linear map address.

Could you give more details about the user of set_memory_decrypted() on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something like GFP_DECRYPTED would be simpler to implement (even posted a hack but without vmalloc() support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just use pgprot_decrypted() for vmap(). No need to rewrite the page table after mapping the pages.

Signature · 1 line
Catalin
#2 lore raw
Quoted 8 lines · Catalin Marinas

Could you give more details about the user of set_memory_decrypted() on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something like GFP_DECRYPTED would be simpler to implement (even posted a hack but without vmalloc() support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just use pgprot_decrypted() for vmap(). No need to rewrite the page table after mapping the pages.

Thank you for the review. I understand that my approach is not ideal in terms of speed / cost. For my use case it was not an issue since the memory is typically only initialized once during device initialization, but I understand there could be a larger performance impact in other uses.

The use case I am interested in is Hyper-V netvsc (drivers/net/hyperv/netvsc.c). The network driver allocates large send and receive buffers (typically 16MB+) with vzalloc(), then registers them as Guest Physical Address Descriptor Lists (GPADLs) via vmbus_establish_gpadl(). Inside __vmbus_establish_gpadl() (drivers/hv/channel.c), set_memory_decrypted() is called on the buffer so the hypervisor can access the shared memory.

In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like GFP_DECRYPTED is an option.

I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well. He likely had netvsc's use of vmalloc in mind when he made the point "GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.

What is the current status of your proposed GFP_DECRYPTED implementation? Is this something you are actively working on?

I'd be happy to work on an RFC following the GFP_DECRYPTED approach if you think that's the right direction.

Regards,
Kameron Carr

maintainer
#3 lore raw
Quoted 9 lines · Kameron Carr
Quoted 8 lines · Catalin Marinas

Could you give more details about the user of set_memory_decrypted() on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something like GFP_DECRYPTED would be simpler to implement (even posted a hack but without vmalloc() support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just use pgprot_decrypted() for vmap(). No need to rewrite the page table after mapping the pages.

[...]

Quoted 6 lines

In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like GFP_DECRYPTED is an option.

I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well.

Yes, that's the thread. It started originally as a GICv3 need (eventually we went for genpool).

Quoted 8 lines

He likely had netvsc's use of vmalloc in mind when he made the point "GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.

What is the current status of your proposed GFP_DECRYPTED implementation? Is this something you are actively working on?

Not really. But I've been looking at it again and I think it adds more problems than it solves. A GFP flag would be passed down to kmem_cache_alloc() and confuse the slab management if some pages are encrypted, others not for the same kmem_cache (SLAB_NO_MERGE wouldn't help). I wonder whether something like SLAB_DECRYPTED would work better for this if we really need it (not aware of any user though).

Anyway, let's ignore slab for now and look at vmalloc(). I can see hv_ringbuffer_init() using an explicit vmap(pgprot_decrypted()). While you could do this, it might be better to just add a VM_DECRYPTED flag and a few wrappers like vmalloc_decrypted(). It would call set_memory_decrypted() for the allocated pages and use pgprot_decrypted() for vmap. On vfree(), it will have to set the pages back to encrypted. It should be fairly mechanical to do (or a 5 min job for an LLM ;)).

Signature · 1 line
Catalin
#4 lore raw

Hi Catalin,

Quoted 37 lines · Catalin Marinas
Quoted 9 lines · Kameron Carr
Quoted 8 lines · Catalin Marinas

Could you give more details about the user of set_memory_decrypted() on vmalloc()'ed addresses? I think this came up in the past and I wondered whether something like GFP_DECRYPTED would be simpler to implement (even posted a hack but without vmalloc() support). If it is known upfront that the memory will be decrypted, it's easier/cheaper to do this on the page allocation time to change the linear map and just use pgprot_decrypted() for vmap(). No need to rewrite the page table after mapping the pages.

[...]

Quoted 6 lines

In this use case, whether to decrypt the memory can always be known at time of allocation, so a solution like GFP_DECRYPTED is an option.

I think I found the hack you mentioned
(https://lore.kernel.org/linux-arm-kernel/[email protected]/). The feedback in Michael Kelley's reply covers the key considerations well.

Yes, that's the thread. It started originally as a GICv3 need (eventually we went for genpool).

Quoted 8 lines

He likely had netvsc's use of vmalloc in mind when he made the point "GFP_DECRYPTED should work for the three memory allocation interfaces and their variants: alloc_pages(), kmalloc(), and vmalloc()." His other points already cover the concerns I had in mind around handling errors from set_memory_decrypted()/encrypted(), etc.

What is the current status of your proposed GFP_DECRYPTED implementation? Is this something you are actively working on?

Not really. But I've been looking at it again and I think it adds more problems than it solves. A GFP flag would be passed down to kmem_cache_alloc() and confuse the slab management if some pages are encrypted, others not for the same kmem_cache (SLAB_NO_MERGE wouldn't help). I wonder whether something like SLAB_DECRYPTED would work better for this if we really need it (not aware of any user though).

Quoted 8 lines

Anyway, let's ignore slab for now and look at vmalloc(). I can see hv_ringbuffer_init() using an explicit vmap(pgprot_decrypted()). While you could do this, it might be better to just add a VM_DECRYPTED flag and a few wrappers like vmalloc_decrypted(). It would call set_memory_decrypted() for the allocated pages and use pgprot_decrypted() for vmap. On vfree(), it will have to set the pages back to encrypted. It should be fairly mechanical to do (or a 5 min job for an LLM ;)).

A GFP + SLAB flag is semantically better than only implementing decrypted allocations only in vmalloc, but it's surely way more complex and intrusive and we are running low on gfp flags on 32 bits :)

But I think we can push the _decrypted to kvmalloc() that would do either kmalloc() + set_memory_decrypted() or vmalloc_decrypted() and it seems that x86 and drivers could use that instead of alloc_pages() + set_memory_decrypted().

We also would want also to make x86::set_memory_decrypted() to only accept linear^w direct map addresses to ensure consistency between architectures.

Quoted 3 lines
Signature · 1 line
Catalin
Signature · 2 lines
Sincerely yours,
Mike.