Skip to contentSkip to conversation

RDMA: detect and handle CoCo DMA bounce buffering

PATCH Superseded Jiri Pirko 4 participants 16 messages May 6 – May 17 InfiniBand

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jiri Pirko

author
#0 lore raw

v2 cover RDMA: detect and handle CoCo DMA bounce buffering

From: Jiri Pirko <[email protected]>

In Confidential Computing (CoCo) guests, the DMA mapping layer redirects all device DMA through swiotlb bounce buffers to keep guest memory encrypted. This is transparent for regular devices because the CPU copies data between the bounce buffer and the real buffer on every DMA map/unmap cycle.

RDMA breaks this model. Once a memory region is registered, the device accesses the underlying pages directly for an extended period without CPU involvement. The swiotlb layer never gets a chance to synchronize, so the device operates on bounce buffer memory while the application works with its own pages - the two never see each other's updates.

This series adds detection and handling of this condition. A new IB_UVERBS_DEVICE_CC_DMA_BOUNCE flag is exposed in device_cap_flags_ex so userspace libraries can detect the situation and switch to dmabuf-based memory registration using "system_cc_shared" heap where available. Plain __ib_umem_get_va() is made to fail early with -EOPNOTSUPP to have more specific error code instead of existing -EIO caused by DMA_ATTR_REQUIRE_COHERENT.

---
See individual patches for changelog.

v1: https://lore.kernel.org/all/[email protected]/

based on top of:
https://lore.kernel.org/all/[email protected]/

Jiri Pirko (2):
RDMA/uverbs: expose CoCo DMA bounce requirement to userspace RDMA/umem: block plain userspace memory registration under CoCo bounce

 drivers/infiniband/core/device.c     | 6 ++++++
 drivers/infiniband/core/umem.c       | 3 +++
 drivers/infiniband/core/uverbs_cmd.c | 2 ++
 include/rdma/ib_verbs.h              | 3 +++
 include/uapi/rdma/ib_user_verbs.h    | 2 ++
 5 files changed, 16 insertions(+)
Signature · 1 line
2.53.0
author
#1 lore raw

PATCH v2 1/2 RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

From: Jiri Pirko <[email protected]>

In CoCo guests, device DMA to regular userspace memory does not work because the DMA mapping layer redirects all mappings through swiotlb bounce buffers. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Expose this condition to userspace as IB_UVERBS_DEVICE_CC_DMA_BOUNCE in device_cap_flags_exi.

---
 drivers/infiniband/core/device.c     | 6 ++++++
 drivers/infiniband/core/uverbs_cmd.c | 2 ++
 include/rdma/ib_verbs.h              | 3 +++
 include/uapi/rdma/ib_user_verbs.h    | 2 ++
 4 files changed, 13 insertions(+)
drivers/infiniband/core/device.c +6 -0
index b89efaaa81ec..ad3da92c9318 100644
@@ -42,6 +42,8 @@
 #include <linux/security.h>
 #include <linux/notifier.h>
 #include <linux/hashtable.h>
+#include <linux/cc_platform.h>
+#include <linux/swiotlb.h>
 #include <rdma/rdma_netlink.h>
 #include <rdma/ib_addr.h>
 #include <rdma/ib_cache.h>
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))
+		device->cc_dma_bounce = 1;
 
 	ret = setup_device(device);
 	if (ret)
drivers/infiniband/core/uverbs_cmd.c +2 -0
index 240f8a0cfd86..2a70774c639a 100644
@@ -3655,6 +3655,8 @@ static int ib_uverbs_ex_query_device(struct uverbs_attr_bundle *attrs)
 	resp.timestamp_mask = attr.timestamp_mask;
 	resp.hca_core_clock = attr.hca_core_clock;
 	resp.device_cap_flags_ex = attr.device_cap_flags;
+	if (ib_dev->cc_dma_bounce)
+		resp.device_cap_flags_ex |= IB_UVERBS_DEVICE_CC_DMA_BOUNCE;
 	resp.rss_caps.supported_qpts = attr.rss_caps.supported_qpts;
 	resp.rss_caps.max_rwq_indirection_tables =
 		attr.rss_caps.max_rwq_indirection_tables;
include/rdma/ib_verbs.h +3 -0
index 167fb924f0cf..d06071b87d96 100644
@@ -275,6 +275,7 @@ enum ib_device_cap_flags {
 	IB_DEVICE_FLUSH_GLOBAL = IB_UVERBS_DEVICE_FLUSH_GLOBAL,
 	IB_DEVICE_FLUSH_PERSISTENT = IB_UVERBS_DEVICE_FLUSH_PERSISTENT,
 	IB_DEVICE_ATOMIC_WRITE = IB_UVERBS_DEVICE_ATOMIC_WRITE,
+	IB_DEVICE_CC_DMA_BOUNCE = IB_UVERBS_DEVICE_CC_DMA_BOUNCE,
 };
 
 enum ib_kernel_cap_flags {
@@ -2950,6 +2951,8 @@ struct ib_device {
 	u16                          kverbs_provider:1;
 	/* CQ adaptive moderation (RDMA DIM) */
 	u16                          use_cq_dim:1;
+	/* CoCo guest with DMA bounce buffering required */
+	u16                          cc_dma_bounce:1;
 	u8                           node_type;
 	u32			     phys_port_cnt;
 	struct ib_device_attr        attrs;
include/uapi/rdma/ib_user_verbs.h +2 -0
index 3b7bd99813e9..d2aeadb6d2f9 100644
@@ -1368,6 +1368,8 @@ enum ib_uverbs_device_cap_flags {
 	IB_UVERBS_DEVICE_FLUSH_PERSISTENT = 1ULL << 39,
 	/* Atomic write attributes */
 	IB_UVERBS_DEVICE_ATOMIC_WRITE = 1ULL << 40,
+	/* CoCo guest with DMA bounce buffering required */
+	IB_UVERBS_DEVICE_CC_DMA_BOUNCE = 1ULL << 41,
 };
 
 enum ib_uverbs_raw_packet_caps {
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 33 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

In CoCo guests, device DMA to regular userspace memory does not work because the DMA mapping layer redirects all mappings through swiotlb bounce buffers. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Expose this condition to userspace as IB_UVERBS_DEVICE_CC_DMA_BOUNCE in device_cap_flags_exi.

---
 drivers/infiniband/core/device.c     | 6 ++++++
 drivers/infiniband/core/uverbs_cmd.c | 2 ++
 include/rdma/ib_verbs.h              | 3 +++
 include/uapi/rdma/ib_user_verbs.h    | 2 ++
 4 files changed, 13 insertions(+)
drivers/infiniband/core/device.c +2 -0
index b89efaaa81ec..ad3da92c9318 100644
@@ -42,6 +42,8 @@
 #include <linux/security.h>
 #include <linux/notifier.h>
 #include <linux/hashtable.h>
+#include <linux/cc_platform.h>
+#include <linux/swiotlb.h>
 #include <rdma/rdma_netlink.h>
 #include <rdma/ib_addr.h>
 #include <rdma/ib_cache.h>

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 6 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

Thanks

author
#3 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 44 lines · Tue, May 12, 2026 at 03:03:29PM CEST, [email protected] wrote:
Quoted 33 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

In CoCo guests, device DMA to regular userspace memory does not work because the DMA mapping layer redirects all mappings through swiotlb bounce buffers. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Expose this condition to userspace as IB_UVERBS_DEVICE_CC_DMA_BOUNCE in device_cap_flags_exi.

---
 drivers/infiniband/core/device.c     | 6 ++++++
 drivers/infiniband/core/uverbs_cmd.c | 2 ++
 include/rdma/ib_verbs.h              | 3 +++
 include/uapi/rdma/ib_user_verbs.h    | 2 ++
 4 files changed, 13 insertions(+)
drivers/infiniband/core/device.c +2 -0
index b89efaaa81ec..ad3da92c9318 100644
@@ -42,6 +42,8 @@
 #include <linux/security.h>
 #include <linux/notifier.h>
 #include <linux/hashtable.h>
+#include <linux/cc_platform.h>
+#include <linux/swiotlb.h>
 #include <rdma/rdma_netlink.h>
 #include <rdma/ib_addr.h>
 #include <rdma/ib_cache.h>

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 6 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

Quoted 4 lines

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

What do you suggest as alternative? We need to somehow tell the user what is the situation.

Quoted 2 lines

Thanks

#4 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 11 lines · Jiri Pirko
Quoted 11 lines

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 7 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

Quoted 7 lines
Quoted 4 lines

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

What do you suggest as alternative? We need to somehow tell the user what is the situation.

For now CC_ATTR_GUEST_MEM_ENCRYPT is likely sufficient.

Later we should be able to detect if the device is in T=1 mode directly.

Jason

author
#5 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 17 lines · Tue, May 12, 2026 at 04:05:10PM CEST, [email protected] wrote:
Quoted 16 lines · Jiri Pirko
Quoted 15 lines

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 7 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

Quoted 7 lines
Quoted 2 lines

What do you suggest as alternative? We need to somehow tell the user what is the situation.

For now CC_ATTR_GUEST_MEM_ENCRYPT is likely sufficient.

Later we should be able to detect if the device is in T=1 mode directly.

Okay, so we assume for now that every device is T=0 (which I believe is the reality). Once T=1 device appears, it changes this "if statement". Do I understand that correctly?

#6 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 21 lines · Jiri Pirko
Quoted 20 lines · Tue, May 12, 2026 at 04:05:10PM CEST, [email protected] wrote:
Quoted 18 lines · Jiri Pirko
Quoted 11 lines

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 7 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

Quoted 4 lines

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

What do you suggest as alternative? We need to somehow tell the user what is the situation.

Quoted 8 lines
Quoted 4 lines

For now CC_ATTR_GUEST_MEM_ENCRYPT is likely sufficient.

Later we should be able to detect if the device is in T=1 mode directly.

Okay, so we assume for now that every device is T=0 (which I believe is the reality). Once T=1 device appears, it changes this "if statement". Do I understand that correctly?

Yes, that is what I was thinking

Jason

author
#7 lore raw

Re: [PATCH rdma-next v2 1/2] RDMA/uverbs: expose CoCo DMA bounce requirement to userspace

Quoted 24 lines · Tue, May 12, 2026 at 04:34:02PM CEST, [email protected] wrote:
Quoted 23 lines · Jiri Pirko
Quoted 22 lines · Tue, May 12, 2026 at 04:05:10PM CEST, [email protected] wrote:
Quoted 11 lines · Jiri Pirko
Quoted 11 lines

drivers/infiniband/core/device.c int ib_register_device(struct ib_device *device, const char *name, · line 1426

Quoted 7 lines
drivers/infiniband/core/device.c +3 -0
@@ -1419,6 +1421,10 @@ int ib_register_device(struct ib_device *device, const char *name,
 	 */
 	WARN_ON(dma_device && !dma_device->dma_parms);
 	device->dma_device = dma_device;
+	if (dma_device &&
+	    cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+	    is_swiotlb_force_bounce(dma_device))

It is the wrong place. When I worked on my DMA series, I tried something similar (a call into SWIOTLB) to notify users that RDMA would not work.

Quoted 7 lines
Quoted 4 lines

The general feedback was that this is a layering violation, and that any knowledge of SWIOTLB (and its API) should not leak out of the DMA API.

You shouldn't call to is_swiotlb_force_bounce() here.

What do you suggest as alternative? We need to somehow tell the user what is the situation.

For now CC_ATTR_GUEST_MEM_ENCRYPT is likely sufficient.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Later we should be able to detect if the device is in T=1 mode directly.

Okay, so we assume for now that every device is T=0 (which I believe is the reality). Once T=1 device appears, it changes this "if statement". Do I understand that correctly?

Yes, that is what I was thinking

Okay, I'll leave some comment for future generations. Thanks!

author
#8 lore raw

PATCH v2 2/2 RDMA/umem: block plain userspace memory registration under CoCo bounce

In reply to Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

---
v1->v2:
- updated patch description with mention of DMA_ATTR_REQUIRE_COHERENT

---
 drivers/infiniband/core/umem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/core/umem.c +3 -0
index 611d693eb9a2..b1877b83b021 100644
@@ -167,6 +167,9 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	int pinned, ret;
 	unsigned int gup_flags = FOLL_LONGTERM;
 
+	if (device->cc_dma_bounce)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	/*
 	 * If the combination of the addr and size requested for this memory
 	 * region causes an integer overflow, return error.
Signature · 1 line
2.53.0
#9 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 10 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Quoted 4 lines

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

Thanks

Quoted 26 lines

---
v1->v2:
- updated patch description with mention of DMA_ATTR_REQUIRE_COHERENT

---
 drivers/infiniband/core/umem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/core/umem.c +3 -0
index 611d693eb9a2..b1877b83b021 100644
@@ -167,6 +167,9 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	int pinned, ret;
 	unsigned int gup_flags = FOLL_LONGTERM;
 
+	if (device->cc_dma_bounce)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	/*
 	 * If the combination of the addr and size requested for this memory
 	 * region causes an integer overflow, return error.
Signature · 1 line
2.53.0
author
#10 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 11 lines · Tue, May 12, 2026 at 03:05:15PM CEST, [email protected] wrote:
Quoted 10 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Quoted 6 lines
Quoted 4 lines

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

I'm not sure I follow. What's the issue you see?

Quoted 29 lines

Thanks

Quoted 26 lines

---
v1->v2:
- updated patch description with mention of DMA_ATTR_REQUIRE_COHERENT

---
 drivers/infiniband/core/umem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/core/umem.c +3 -0
index 611d693eb9a2..b1877b83b021 100644
@@ -167,6 +167,9 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	int pinned, ret;
 	unsigned int gup_flags = FOLL_LONGTERM;
 
+	if (device->cc_dma_bounce)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	/*
 	 * If the combination of the addr and size requested for this memory
 	 * region causes an integer overflow, return error.
Signature · 1 line
2.53.0
#11 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 12 lines · Jiri Pirko
Quoted 11 lines · Tue, May 12, 2026 at 03:05:15PM CEST, [email protected] wrote:
Quoted 10 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

I'm not sure I follow. What's the issue you see?

SWIOTLB is the layer below DMA API, RDMA is the layer above DMA API. You shouldn't call to SWIOTLB functions in RDMA code.

Thanks

Quoted 33 lines
Quoted 33 lines

Thanks

Quoted 26 lines

---
v1->v2:
- updated patch description with mention of DMA_ATTR_REQUIRE_COHERENT

---
 drivers/infiniband/core/umem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/infiniband/core/umem.c +3 -0
index 611d693eb9a2..b1877b83b021 100644
@@ -167,6 +167,9 @@ static struct ib_umem *__ib_umem_get_va(struct ib_device *device,
 	int pinned, ret;
 	unsigned int gup_flags = FOLL_LONGTERM;
 
+	if (device->cc_dma_bounce)
+		return ERR_PTR(-EOPNOTSUPP);
+
 	/*
 	 * If the combination of the addr and size requested for this memory
 	 * region causes an integer overflow, return error.
Signature · 1 line
2.53.0
author
#12 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 18 lines · Thu, May 14, 2026 at 06:25:06PM +0200, [email protected] wrote:
Quoted 17 lines · Jiri Pirko
Quoted 16 lines · Tue, May 12, 2026 at 03:05:15PM CEST, [email protected] wrote:
Quoted 14 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

I'm not sure I follow. What's the issue you see?

SWIOTLB is the layer below DMA API, RDMA is the layer above DMA API. You shouldn't call to SWIOTLB functions in RDMA code.

This patch doesn't do that. The patch description only describes the current situation and how the patch changes the behaviour.

[..]

#13 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 21 lines · Jiri Pirko
Quoted 20 lines · Thu, May 14, 2026 at 06:25:06PM +0200, [email protected] wrote:
Quoted 19 lines · Jiri Pirko
Quoted 17 lines · Tue, May 12, 2026 at 03:05:15PM CEST, [email protected] wrote:
Quoted 10 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Quoted 4 lines

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I'm not sure I follow. What's the issue you see?

SWIOTLB is the layer below DMA API, RDMA is the layer above DMA API. You shouldn't call to SWIOTLB functions in RDMA code.

This patch doesn't do that. The patch description only describes the current situation and how the patch changes the behaviour.

From previous patch:
+ if (dma_device &&
+ cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+ is_swiotlb_force_bounce(dma_device))
+ device->cc_dma_bounce = 1;

And this patch reads cc_dma_bounce.

Thanks

Quoted 2 lines

[..]

author
#14 lore raw

Re: [PATCH rdma-next v2 2/2] RDMA/umem: block plain userspace memory registration under CoCo bounce

Quoted 36 lines · Sun, May 17, 2026 at 01:50:06PM +0200, [email protected] wrote:
Quoted 21 lines · Jiri Pirko
Quoted 20 lines · Thu, May 14, 2026 at 06:25:06PM +0200, [email protected] wrote:
Quoted 19 lines · Jiri Pirko
Quoted 17 lines · Tue, May 12, 2026 at 03:05:15PM CEST, [email protected] wrote:
Quoted 10 lines · Jiri Pirko

From: Jiri Pirko <[email protected]>

When a device requires DMA bounce buffering inside a Confidential Computing guest, __ib_umem_get_va() cannot work. The DMA mapping layer redirects all mappings through swiotlb bounce buffers, so the device receives DMA addresses pointing to bounce buffer memory rather than the user's pages. Since RDMA devices access registered memory directly without CPU involvement, there is no opportunity for swiotlb to synchronize between the bounce buffer and the original pages.

Quoted 4 lines

The registration would already fail later on, since the umem mapping is requested with DMA_ATTR_REQUIRE_COHERENT and gets rejected under is_swiotlb_force_bounce() with -EIO. Fail early with -EOPNOTSUPP instead, so the user gets a specific error code to react to.

DMA_ATTR_REQUIRE_COHERENT was our answer to "layering violation claim".

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I'm not sure I follow. What's the issue you see?

SWIOTLB is the layer below DMA API, RDMA is the layer above DMA API. You shouldn't call to SWIOTLB functions in RDMA code.

This patch doesn't do that. The patch description only describes the current situation and how the patch changes the behaviour.

From previous patch:

(quoted diff) +4 -0
+       if (dma_device &&
+           cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) &&
+           is_swiotlb_force_bounce(dma_device))
+               device->cc_dma_bounce = 1;
Quoted 1 line

And this patch reads cc_dma_bounce.

Sure, I got that remark from the reply to the previous patch :) Already removed. Will send next v soon.

Thanks!