arm64: swiotlb: dma: its: Ensure shared buffers are properly aligned

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

This patch introduces a new helper, arch_shared_mem_alignment(), which

can be used to enforce proper alignment of shared buffers.

The actual implementation of arch_shared_mem_alignment() is deferred to a follow-up patch.

Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>

---
 arch/arm64/include/asm/mem_encrypt.h |  6 ++++++
 arch/arm64/mm/init.c                 |  4 +++-
 arch/arm64/mm/mem_encrypt.c          |  6 ++++++
 drivers/irqchip/irq-gic-v3-its.c     |  8 ++++++--
 include/linux/mem_encrypt.h          |  7 +++++++
 include/linux/swiotlb.h              |  7 ++++---
 kernel/dma/direct.c                  |  7 +++++++
 kernel/dma/pool.c                    |  1 +
 kernel/dma/swiotlb.c                 | 28 +++++++++++++++++-----------

Quoted 7 lines · Aneesh Kumar K. V.

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

This patch introduces a new helper, arch_shared_mem_alignment(), which

# git grep "This patch" Documentation/process/

Quoted 6 lines

can be used to enforce proper alignment of shared buffers.

The actual implementation of arch_shared_mem_alignment() is deferred to a follow-up patch.

Signed-off-by Aneesh Kumar K.V (Arm) <[email protected]>

Quoted 10 lines

---
 arch/arm64/include/asm/mem_encrypt.h |  6 ++++++
 arch/arm64/mm/init.c                 |  4 +++-
 arch/arm64/mm/mem_encrypt.c          |  6 ++++++
 drivers/irqchip/irq-gic-v3-its.c     |  8 ++++++--
 include/linux/mem_encrypt.h          |  7 +++++++
 include/linux/swiotlb.h              |  7 ++++---
 kernel/dma/direct.c                  |  7 +++++++
 kernel/dma/pool.c                    |  1 +
 kernel/dma/swiotlb.c                 | 28 +++++++++++++++++-----------

This does too many things at once and breaks all swiotlb users except arm64. Seriously?

include/linux/swiotlb.h static inline bool is_swiotlb_force_bounce(struct device *dev) · line 185

Quoted 2 lines

include/linux/swiotlb.h +1 -1

-void swiotlb_init(bool addressing_limited, unsigned int flags);
+void swiotlb_init(bool addressing_limited, unsigned int alignment, unsigned int flags);

Why do you need this alignment argument in the first place?

In quite some other places you use arch_shared_mem_alignment(), which defaults to PAGE_SIZE if the architecture does not implement it's own variant. What's preventing you from using that in the init functions as well?

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

arch/arm64/mm/init.c +2 -0

index ea84a61ed508..389070e9ee65 100644
@@ -337,12 +337,14 @@ void __init bootmem_init(void)
 
 void __init arch_mm_preinit(void)
 {
+	unsigned int swiotlb_align = PAGE_SIZE;
 	unsigned int flags = SWIOTLB_VERBOSE;
 	bool swiotlb = max_pfn > PFN_DOWN(arm64_dma_phys_limit);
 
 	if (is_realm_world()) {
 		swiotlb = true;
 		flags |= SWIOTLB_FORCE;
+		swiotlb_align = arch_shared_mem_alignment();
 	}
 
 	if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) {

drivers/irqchip/irq-gic-v3-its.c +2 -1

index 467cb78435a9..e2142bbca13b 100644
@@ -213,16 +213,20 @@ static gfp_t gfp_flags_quirk;
 static struct page *its_alloc_pages_node(int node, gfp_t gfp,
 					 unsigned int order)
 {
+	long new_order;
 	struct page *page;
 	int ret = 0;
 
-	page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
+	/* align things to hypervisor page size */

Hi Aneesh,

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

Quoted 14 lines

drivers/irqchip/irq-gic-v3-its.c +5 -1

@@ -213,16 +213,20 @@ static gfp_t gfp_flags_quirk;
 static struct page *its_alloc_pages_node(int node, gfp_t gfp,
 					 unsigned int order)
 {
+	long new_order;
 	struct page *page;
 	int ret = 0;
 
-	page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
+	/* align things to hypervisor page size */
+	new_order = get_order(ALIGN((PAGE_SIZE << order), arch_shared_mem_alignment()));
+
+	page = alloc_pages_node(node, gfp | gfp_flags_quirk, new_order);
 

drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 229

Quoted 6 lines

drivers/irqchip/irq-gic-v3-its.c +1 -1

 	if (!page)
 		return NULL;
 
 	ret = set_memory_decrypted((unsigned long)page_address(page),
-				   1 << order);
+				   1 << new_order);

At some point this could move to the DMA API.

Quoted 14 lines · Catalin Marinas

Quoted 14 lines

drivers/irqchip/irq-gic-v3-its.c +5 -1

@@ -213,16 +213,20 @@ static gfp_t gfp_flags_quirk;
 static struct page *its_alloc_pages_node(int node, gfp_t gfp,
 					 unsigned int order)
 {
+	long new_order;
 	struct page *page;
 	int ret = 0;
 
-	page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
+	/* align things to hypervisor page size */
+	new_order = get_order(ALIGN((PAGE_SIZE << order), arch_shared_mem_alignment()));
+
+	page = alloc_pages_node(node, gfp | gfp_flags_quirk, new_order);
 

Quoted 8 lines

drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 229

Quoted 6 lines

drivers/irqchip/irq-gic-v3-its.c +1 -1

 	if (!page)
 		return NULL;
 
 	ret = set_memory_decrypted((unsigned long)page_address(page),
-				   1 << order);
+				   1 << new_order);

At some point this could move to the DMA API.

I don't think we should be open coding these patterns.

Esepcially given the above, it makes no sense to 'alloc page' and then 'decrypt page' on ARM CCA. decryption is not really a OS page level operation. I suggest coming with some series to clean these up into a more sensible API.

Everything wanting decrypted memory should be going through some more general API that has some opportunity to use pools.

Quoted 3 lines · Catalin Marinas

Hi Aneesh,

On Fri, Sep 05, 2025 at 11:24:41AM +0530, Aneesh Kumar K.V (Arm) wrote:

Quoted 8 lines

Quoted 5 lines

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do

Quoted 3 lines

anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can

Quoted 2 lines

anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

+1 I'm also confused by this description.

I thought the issue here was in the RMM. The GPT or S2 min granule could be > 4k and in this case an unaligned set_memory_decrypted() from the guest would have to fail inside the RMM as impossible to execute?

Though I'm a little unclear on when and why the S2 needs to be manipulated. Can't the S2 fully map both the protected and unprotected IPA space and rely on the GPT for protection?

I do remember having a discussion that set_memory_decrypted() has nothing to do with the VM's S1 granule size, and it is a mistake to have linked these together. The VM needs to understand what granularity the RMM will support set_memory_decrypted() for and follow that.

I don't recall there is also an issue on the hypervisor? I thought GPT faults on ARM were going to work well, ie we could cleanly segfault the VMM process if it touches any protected memory that may have been mapped into it, and speculation was safe?

Quoted 15 lines

Quoted 15 lines

drivers/irqchip/irq-gic-v3-its.c +5 -1

@@ -213,16 +213,20 @@ static gfp_t gfp_flags_quirk;
  static struct page *its_alloc_pages_node(int node, gfp_t gfp,
  					 unsigned int order)
  {
+	long new_order;
  	struct page *page;
  	int ret = 0;
 
-	page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
+	/* align things to hypervisor page size */
+	new_order = get_order(ALIGN((PAGE_SIZE << order), arch_shared_mem_alignment()));
+
+	page = alloc_pages_node(node, gfp | gfp_flags_quirk, new_order);
 

Quoted 8 lines

drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 229

Quoted 6 lines

drivers/irqchip/irq-gic-v3-its.c +1 -1

  	if (!page)
  		return NULL;
 
  	ret = set_memory_decrypted((unsigned long)page_address(page),
-				   1 << order);
+				   1 << new_order);

At some point this could move to the DMA API.

I don't think we should be open coding these patterns.

Esepcially given the above, it makes no sense to 'alloc page' and then 'decrypt page' on ARM CCA. decryption is not really a OS page level operation. I suggest coming with some series to clean these up into a more sensible API.

Everything wanting decrypted memory should be going through some more general API that has some opportunity to use pools.

DMA API may be one choice, but I know we will need more options in RDMA land :|

Jason

Hi Aneesh,

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

Quoted 20 lines

arch/arm64/mm/init.c +2 -0

index ea84a61ed508..389070e9ee65 100644
@@ -337,12 +337,14 @@ void __init bootmem_init(void)
 
 void __init arch_mm_preinit(void)
 {
+	unsigned int swiotlb_align = PAGE_SIZE;
 	unsigned int flags = SWIOTLB_VERBOSE;
 	bool swiotlb = max_pfn > PFN_DOWN(arm64_dma_phys_limit);
 
 	if (is_realm_world()) {
 		swiotlb = true;
 		flags |= SWIOTLB_FORCE;
+		swiotlb_align = arch_shared_mem_alignment();
 	}
 
 	if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) {

arch/arm64/mm/init.c void __init arch_mm_preinit(void) · line 362

Quoted 6 lines

arch/arm64/mm/init.c +1 -1

@@ -356,7 +358,7 @@ void __init arch_mm_preinit(void)
 		swiotlb = true;
 	}
 
-	swiotlb_init(swiotlb, flags);
+	swiotlb_init(swiotlb, swiotlb_align, flags);
 	swiotlb_update_mem_attributes();

I think there's too much change just to pass down an alignment. We have IO_TLB_MIN_SLABS, we could add an IO_TLB_MIN_ALIGN that's PAGE_SIZE by default but can be overridden to something dynamic for arm64.

Quoted 14 lines

drivers/irqchip/irq-gic-v3-its.c +2 -1

index 467cb78435a9..e2142bbca13b 100644
@@ -213,16 +213,20 @@ static gfp_t gfp_flags_quirk;
 static struct page *its_alloc_pages_node(int node, gfp_t gfp,
 					 unsigned int order)
 {
+	long new_order;
 	struct page *page;
 	int ret = 0;
 
-	page = alloc_pages_node(node, gfp | gfp_flags_quirk, order);
+	/* align things to hypervisor page size */

drivers/irqchip/irq-gic-v3-its.c static gfp_t gfp_flags_quirk; · line 229

Quoted 10 lines

drivers/irqchip/irq-gic-v3-its.c +4 -1

+	new_order = get_order(ALIGN((PAGE_SIZE << order), arch_shared_mem_alignment()));
+
+	page = alloc_pages_node(node, gfp | gfp_flags_quirk, new_order);
 
 	if (!page)
 		return NULL;
 
 	ret = set_memory_decrypted((unsigned long)page_address(page),
-				   1 << order);
+				   1 << new_order);

At some point this could move to the DMA API.

Quoted 8 lines

kernel/dma/direct.c +0 -0

index 24c359d9c879..5db5baad5efa 100644
@@ -255,6 +255,9 @@ void *dma_direct_alloc(struct device *dev, size_t size,
 	    dma_direct_use_pool(dev, gfp))
 		return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
 

kernel/dma/direct.c void *dma_direct_alloc(struct device *dev, size_t size, · line 262

Quoted 5 lines

kernel/dma/direct.c +3 -0

+	if (force_dma_unencrypted(dev))
+		/*  align to hypervisor size */
+		size = ALIGN(size, arch_shared_mem_alignment());
 	/* we always manually zero the memory once we are done */
 	page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true);

You align the size but does __dma_direct_alloc_pages() guarantee a natural alignment? Digging through cma_alloc_aligned(), it guarantees the minimum of size and CONFIG_CMA_ALIGNMENT. The latter can be configured to a page order of 2.

kernel/dma/direct.c void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags, · line 420

Quoted 13 lines

kernel/dma/direct.c +1 -1

@@ -382,7 +384,7 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
 
 	nslabs = default_nslabs;
 	nareas = limit_nareas(default_nareas, nslabs);
-	while ((tlb = swiotlb_memblock_alloc(nslabs, flags, remap)) == NULL) {
+	while ((tlb = swiotlb_memblock_alloc(nslabs, alignment, flags, remap)) == NULL) {
 		if (nslabs <= IO_TLB_MIN_SLABS)
 			return;
 		nslabs = ALIGN(nslabs >> 1, IO_TLB_SEGSIZE);

@@ -417,9 +419,9 @@ void __init swiotlb_init_remap(bool addressing_limit, unsigned int flags,
 		swiotlb_print_info();
 }

We also have the dynamic swiotlb allocations via swiotlb_dyn_alloc() -> swiotlb_alloc_pool() -> swiotlb_alloc_tlb(). I don't see any alignment guarantees.

TBH, I'm not too keen on this patch. It feels like a problem to be solved by the host - don't advertise smaller page sizes to guest or cope with sub-page memory sharing. Currently the streaming DMA is handled via bouncing but we may, at some point, just allow set_memory_decrypted() on the DMA-mapped page. The above requirements will not allow this.

I cannot come up with a better solution though. I hope the hardware can cope with sub-page shared/non-shared ranges.

Quoted 7 lines · Catalin Marinas

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

Quoted 7 lines

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

With features like guest_memfd, the goal is to explicitly prevent the host from mapping private memory, rather than relying on the host to avoid accessing those regions.

As per Arm ARM:
RVJLXG: Accesses are checked against the GPC configuration for the physical granule being accessed, regardless of the stage 1 and stage 2 translation configuration.

For example, if GPCCR_EL3.PGS is configured to a smaller granule size than the configured stage 1 and stage 2 translation granule size, accesses are checked at the GPCCR_EL3.PGS granule size.

Quoted 16 lines · Aneesh Kumar K.V

Quoted 14 lines · Catalin Marinas

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

Quoted 3 lines

With features like guest_memfd, the goal is to explicitly prevent the host from mapping private memory, rather than relying on the host to avoid accessing those regions.

Yes, if all the memory is private. At some point the guest will start sharing memory with the host. In theory, the host could map more than it was given access to as long as it doesn't touch the area around the shared range. Not ideal and it may not match the current guest_memfd API

but I'd like to understand all the options we have.

Quoted 4 lines

As per Arm ARM:
RVJLXG: Accesses are checked against the GPC configuration for the physical granule being accessed, regardless of the stage 1 and stage 2 translation configuration.

OK, so this rule doesn't say anything about the granule size at stage 1 or stage 2. The check is purely done based on the PGS field configuration. The need for the host granule size to match PGS is just a software construct.

Quoted 3 lines

For example, if GPCCR_EL3.PGS is configured to a smaller granule size than the configured stage 1 and stage 2 translation granule size, accesses are checked at the GPCCR_EL3.PGS granule size.

I assume GPCCR_EL3.PGS is pre-configured on the system as 4K and part of the RMM spec.

Quoted 21 lines · Catalin Marinas

Quoted 19 lines · Aneesh Kumar K.V

Quoted 7 lines · Catalin Marinas

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

Quoted 7 lines

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

With features like guest_memfd, the goal is to explicitly prevent the host from mapping private memory, rather than relying on the host to avoid accessing those regions.

Quoted 4 lines

Yes, if all the memory is private. At some point the guest will start sharing memory with the host. In theory, the host could map more than it was given access to as long as it doesn't touch the area around the shared range. Not ideal and it may not match the current guest_memfd API

The kernel may be taught not to touch the area, but it is tricky when the shared page gets mapped into the usespace and what it does with it.

True. The GPC Page Size is going to be 4K. At present the RMM S2 page size is fixed to 4K.

Quoted 22 lines · Suzuki K Poulose

Quoted 21 lines · Catalin Marinas

Quoted 19 lines · Aneesh Kumar K.V

Quoted 7 lines · Catalin Marinas

Quoted 5 lines · Aneesh Kumar K.V (Arm)

When running with private memory guests, the guest kernel must allocate memory with specific constraints when sharing it with the hypervisor.

These shared memory buffers are also accessed by the host kernel, which means they must be aligned to the host kernel's page size.

Quoted 7 lines

So this is the case where the guest page size is smaller than the host one. Just trying to understand what would go wrong if we don't do anything here. Let's say the guest uses 4K pages and the host a 64K pages. Within a 64K range, only a 4K is shared/decrypted. If the host does not explicitly access the other 60K around the shared 4K, can anything still go wrong? Is the hardware ok with speculative loads from non-shared ranges?

With features like guest_memfd, the goal is to explicitly prevent the host from mapping private memory, rather than relying on the host to avoid accessing those regions.

Quoted 7 lines

Quoted 4 lines

Yes, if all the memory is private. At some point the guest will start sharing memory with the host. In theory, the host could map more than it was given access to as long as it doesn't touch the area around the shared range. Not ideal and it may not match the current guest_memfd API

The kernel may be taught not to touch the area, but it is tricky when the shared page gets mapped into the usespace and what it does with it.

But what happes?

The entire reason we have this nasty hyper-restrictive memfd private memory is beacuse Intel takes a machine check if anything does it wrong, and that is fatal and can't be handled.

Is ARM like that? I thought ARM had good faults on GPT violation that could be handled in the same way as a normal page fault?

If ARM has proper faulting then you don't have an issue mapping 64K into a userspace and just segfaulting the VMM if it does something wrong.

If not, then sure you need all this unmapping stuff like Intel does :\

Quoted 2 lines

True. The GPC Page Size is going to be 4K. At present the RMM S2 page size is fixed to 4K.

A 4k S2 is a pointless thing to do if the VMM is only going to approve 64k shared/private transitions :(

Quoted 3 lines · Jason Gunthorpe

If ARM has proper faulting then you don't have an issue mapping 64K into a userspace and just segfaulting the VMM if it does something wrong.

...the VMM can cause problems. If the VMM touches the memory itself then things are simple - we can detect that the fault was from user space and trigger a SIGBUS to kill of the VMM.

But the VMM can also attempt to pass the address into the kernel and cause the kernel to do a get_user_pages() call (and this is something we want to support for shared memory). The problem is if the kernel then touches the parts of the page which are protected we get a fault with no (easy) way to relate back to the VMM.

guest_memfd provided a nice way around this - a dedicated allocator which doesn't allow mmap(). This meant we don't need to worry about user space handing protected memory into the kernel. It's now getting extended to support mmap() but only when shared, and there was a lot of discussion about how to ensure that there are no mmap regions when converting memory back to private.

Quoted 8 lines · Steven Price

Quoted 3 lines · Jason Gunthorpe

If ARM has proper faulting then you don't have an issue mapping 64K into a userspace and just segfaulting the VMM if it does something wrong.

...the VMM can cause problems. If the VMM touches the memory itself then things are simple - we can detect that the fault was from user space and trigger a SIGBUS to kill of the VMM.

Similarly for uaccess.

Quoted 5 lines

But the VMM can also attempt to pass the address into the kernel and cause the kernel to do a get_user_pages() call (and this is something we want to support for shared memory). The problem is if the kernel then touches the parts of the page which are protected we get a fault with no (easy) way to relate back to the VMM.

I assume the host has a mechanism to check that the memory has been marked as shared by the guest and the guest cannot claim it back as private while the host is accessing it (I should dig out the CCA spec).

Quoted 6 lines

guest_memfd provided a nice way around this - a dedicated allocator which doesn't allow mmap(). This meant we don't need to worry about user space handing protected memory into the kernel. It's now getting extended to support mmap() but only when shared, and there was a lot of discussion about how to ensure that there are no mmap regions when converting memory back to private.

That's indeed problematic and we don't have a simple way to check that a user VMM address won't fault when accessed via the linear map. The vma checks we get with mmap are (host) page size based.

Can we instead only allow mismatched (or smaller) granule sizes in the guest if the VMM doesn't use the mmap() interface? It's not like trapping TCR_EL1 but simply rejecting such unaligned memory slots since the host will need to check that the memory has indeed been shared. KVM can advertise higher granules only, though the guest can ignore them.

Quoted 33 lines · Catalin Marinas

Quoted 8 lines · Steven Price

Quoted 3 lines · Jason Gunthorpe

If ARM has proper faulting then you don't have an issue mapping 64K into a userspace and just segfaulting the VMM if it does something wrong.

...the VMM can cause problems. If the VMM touches the memory itself then things are simple - we can detect that the fault was from user space and trigger a SIGBUS to kill of the VMM.

Similarly for uaccess.

Quoted 5 lines

But the VMM can also attempt to pass the address into the kernel and cause the kernel to do a get_user_pages() call (and this is something we want to support for shared memory). The problem is if the kernel then touches the parts of the page which are protected we get a fault with no (easy) way to relate back to the VMM.

I assume the host has a mechanism to check that the memory has been marked as shared by the guest and the guest cannot claim it back as private while the host is accessing it (I should dig out the CCA spec).

Quoted 6 lines

guest_memfd provided a nice way around this - a dedicated allocator which doesn't allow mmap(). This meant we don't need to worry about user space handing protected memory into the kernel. It's now getting extended to support mmap() but only when shared, and there was a lot of discussion about how to ensure that there are no mmap regions when converting memory back to private.

That's indeed problematic and we don't have a simple way to check that a user VMM address won't fault when accessed via the linear map. The vma checks we get with mmap are (host) page size based.

Quoted 6 lines

Can we instead only allow mismatched (or smaller) granule sizes in the guest if the VMM doesn't use the mmap() interface? It's not like trapping TCR_EL1 but simply rejecting such unaligned memory slots since the host will need to check that the memory has indeed been shared. KVM can advertise higher granules only, though the guest can ignore them.

Yes, mismatched granules sizes could be supported if we disallowed mmap(). This is assuming the RMM supports the required size - which is currently true, but the intention is to optimise the S2 in the future by matching the host page size.

But I'm not sure how useful that would be. The VMMs of today don't expect to have to perform read()/write() calls to access the guest's memory, so any user space emulation would need to also be updated to deal with this restriction.

But that seems like a lot of effort to support something that doesn't seem to have a use case. Whereas there's an obvious use case for the guest and VMM sharing one (or often more) pages of (mapped) memory. The part that CCA makes this tricky is that we need to pick the VMM's page size rather than the guest's.

guest_memfd provided a nice way around this - a dedicated allocator which doesn't allow mmap(). This meant we don't need to worry about user space handing protected memory into the kernel. It's now getting extended to support mmap() but only when shared, and there was a lot of discussion about how to ensure that there are no mmap regions when converting memory back to private.