Skip to contentSkip to conversation

RDMA/rxe: reject non-8-byte ATOMIC_WRITE payloads

PATCH Applied Michael Bommarito 3 participants 6 messages Apr 18 – Apr 28 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/rxe: reject non-8-byte ATOMIC_WRITE payloads

atomic_write_reply() at drivers/infiniband/sw/rxe/rxe_resp.c unconditionally dereferences 8 bytes at payload_addr(pkt):

    value = *(u64 *)payload_addr(pkt);

check_rkey() previously accepted an ATOMIC_WRITE request with pktlen == resid == 0 because the length validation only compared pktlen against resid. A remote initiator that sets the RETH length to 0 therefore reaches atomic_write_reply() with a zero-byte logical payload, and the responder reads sizeof(u64) bytes from past the logical end of the packet into skb->head tailroom, then writes those 8 bytes into the attacker's MR via rxe_mr_do_atomic_write(). That is a remote disclosure of 4 bytes of kernel tailroom per probe (the other 4 bytes are the packet's own trailing ICRC).

IBA oA19-28 defines ATOMIC_WRITE as exactly 8 bytes. Anything else is protocol-invalid. Hoist a strict length check into check_rkey() so the responder never reaches the unchecked dereference, and keep the existing WRITE-family length logic for the normal RDMA WRITE path.

Reproduced on mainline with an unmodified rxe driver: a sustained zero-length ATOMIC_WRITE probe repeatedly leaks adjacent skb head-buffer bytes into the attacker's MR, including recognisable kernel strings and partial kernel-direct-map pointer words. With this patch applied the responder rejects the PDU and the MR stays all-zero.

Fixes 034e285f8b99 ("RDMA/rxe: Make responder support atomic write on RC service")
Assisted-by Claude:claude-opus-4-7

---
Previously reported to security@ (2026-04-18); reposting publicly at the maintainer's request.

Per-probe evidence from a 100K-packet run on the clean unpatched tree at 9ca18fc915c6 (single attacker QP against a hairpin target QP over a veth pair; each probe one crafted zero-length ATOMIC_WRITE PDU):

transmitted packets: 100,000
observed MR writes: 48,575
non-zero leaked tails: 33,297 (68.55% of observed writes)
mostly-printable tails: 3,796 (7.81%)
fully-printable tails: 2,241 (4.61%)
unique non-zero tails: 22,220

Each probe is a fresh skb head-buffer allocation, so the 4 attacker-visible bytes after the ICRC are an independent sample of slab-adjacent memory. Content distribution across the 48,575 observed writes: 31.45% zero, 4.61% fully printable, 3.20% mostly printable, 12.06% header/sentinel- looking (08004500, 08004508, ffffffff, ...), 48.68% other binary. 80.9% of unique non-zero tails were singletons, so the leak is not dominated by one repeated value.

Representative printable fragments observed on the attacker side:

    74 6f 70 2e   "top."
    66 72 65 65   "free"
    45 78 65 63   "Exec"
    2f 73 79 73   "/sys"
    72 6f 6f 74   "root"
    45 56 50 41   "EVPA"
    43 4f 44 45   "CODE"

Partial pointer-like recoveries (4-byte words ending in the kernel-direct-map prefix 0xffff....):

    3,361 observations ending in ffff
    1,364 unique ....ffff tails
    most common:
      81 88 ff ff   LE 0xffff8881   1.68% of observed writes
      80 88 ff ff   LE 0xffff8880   0.22%

The run did not recover full 64-bit kernel virtual addresses (only 4 bytes per probe are attacker-observable), but the partial pointer material is consistent with a KASLR-weakening primitive under sustained probing. With the fix applied, the same harness leaves the attacker MR all-zero.

---
 drivers/infiniband/sw/rxe/rxe_resp.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_resp.c +13 -1
index 711f73e0bbb1..09ba21d0f3c4 100644
@@ -526,7 +526,19 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
 	}
 
 skip_check_range:
-	if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
+	if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		/* IBA oA19-28: ATOMIC_WRITE payload is exactly 8 bytes.
+		 * Reject any other length before the responder reads
+		 * sizeof(u64) bytes from payload_addr(pkt); a shorter
+		 * payload would read past the logical end of the packet
+		 * into skb->head tailroom.
+		 */
+		if (resid != sizeof(u64) || pktlen != sizeof(u64) ||
+		    bth_pad(pkt)) {
+			state = RESPST_ERR_LENGTH;
+			goto err;
+		}
+	} else if (pkt->mask & RXE_WRITE_MASK) {
 		if (resid > mtu) {
 			if (pktlen != mtu || bth_pad(pkt)) {
 				state = RESPST_ERR_LENGTH;
Signature · 1 line
2.53.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 78 lines · Michael Bommarito

atomic_write_reply() at drivers/infiniband/sw/rxe/rxe_resp.c unconditionally dereferences 8 bytes at payload_addr(pkt):

     value = *(u64 *)payload_addr(pkt);

check_rkey() previously accepted an ATOMIC_WRITE request with pktlen == resid == 0 because the length validation only compared pktlen against resid. A remote initiator that sets the RETH length to 0 therefore reaches atomic_write_reply() with a zero-byte logical payload, and the responder reads sizeof(u64) bytes from past the logical end of the packet into skb->head tailroom, then writes those 8 bytes into the attacker's MR via rxe_mr_do_atomic_write(). That is a remote disclosure of 4 bytes of kernel tailroom per probe (the other 4 bytes are the packet's own trailing ICRC).

IBA oA19-28 defines ATOMIC_WRITE as exactly 8 bytes. Anything else is protocol-invalid. Hoist a strict length check into check_rkey() so the responder never reaches the unchecked dereference, and keep the existing WRITE-family length logic for the normal RDMA WRITE path.

Reproduced on mainline with an unmodified rxe driver: a sustained zero-length ATOMIC_WRITE probe repeatedly leaks adjacent skb head-buffer bytes into the attacker's MR, including recognisable kernel strings and partial kernel-direct-map pointer words. With this patch applied the responder rejects the PDU and the MR stays all-zero.

Fixes 034e285f8b99 ("RDMA/rxe: Make responder support atomic write on RC service")
Assisted-by Claude:claude-opus-4-7

---
Previously reported to security@ (2026-04-18); reposting publicly at the maintainer's request.

Per-probe evidence from a 100K-packet run on the clean unpatched tree at 9ca18fc915c6 (single attacker QP against a hairpin target QP over a veth pair; each probe one crafted zero-length ATOMIC_WRITE PDU):

transmitted packets: 100,000
observed MR writes: 48,575
non-zero leaked tails: 33,297 (68.55% of observed writes)
mostly-printable tails: 3,796 (7.81%)
fully-printable tails: 2,241 (4.61%)
unique non-zero tails: 22,220

Each probe is a fresh skb head-buffer allocation, so the 4 attacker-visible bytes after the ICRC are an independent sample of slab-adjacent memory. Content distribution across the 48,575 observed writes: 31.45% zero, 4.61% fully printable, 3.20% mostly printable, 12.06% header/sentinel- looking (08004500, 08004508, ffffffff, ...), 48.68% other binary. 80.9% of unique non-zero tails were singletons, so the leak is not dominated by one repeated value.

Representative printable fragments observed on the attacker side:

     74 6f 70 2e   "top."
     66 72 65 65   "free"
     45 78 65 63   "Exec"
     2f 73 79 73   "/sys"
     72 6f 6f 74   "root"
     45 56 50 41   "EVPA"
     43 4f 44 45   "CODE"

Partial pointer-like recoveries (4-byte words ending in the kernel-direct-map prefix 0xffff....):

     3,361 observations ending in ffff
     1,364 unique ....ffff tails
     most common:
       81 88 ff ff   LE 0xffff8881   1.68% of observed writes
       80 88 ff ff   LE 0xffff8880   0.22%
Quoted 5 lines

The run did not recover full 64-bit kernel virtual addresses (only 4 bytes per probe are attacker-observable), but the partial pointer material is consistent with a KASLR-weakening primitive under sustained probing. With the fix applied, the same harness leaves the attacker MR all-zero.

Thanks a lot. It would be great to have a corresponding negative test in tools/testing/selftests/rdma that sends malformed ATOMIC_WRITE requests (e.g., zero-length) and verifies that they are rejected and do not modify the target MR.

It is up to you. To this commit, I am fine with it.

Zhu Yanjun

Quoted 29 lines
---
  drivers/infiniband/sw/rxe/rxe_resp.c | 14 +++++++++++++-
  1 file changed, 13 insertions(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_resp.c +13 -1
index 711f73e0bbb1..09ba21d0f3c4 100644
@@ -526,7 +526,19 @@ static enum resp_states check_rkey(struct rxe_qp *qp,
  	}
 
  skip_check_range:
-	if (pkt->mask & (RXE_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) {
+	if (pkt->mask & RXE_ATOMIC_WRITE_MASK) {
+		/* IBA oA19-28: ATOMIC_WRITE payload is exactly 8 bytes.
+		 * Reject any other length before the responder reads
+		 * sizeof(u64) bytes from payload_addr(pkt); a shorter
+		 * payload would read past the logical end of the packet
+		 * into skb->head tailroom.
+		 */
+		if (resid != sizeof(u64) || pktlen != sizeof(u64) ||
+		    bth_pad(pkt)) {
+			state = RESPST_ERR_LENGTH;
+			goto err;
+		}
+	} else if (pkt->mask & RXE_WRITE_MASK) {
  		if (resid > mtu) {
  			if (pktlen != mtu || bth_pad(pkt)) {
  				state = RESPST_ERR_LENGTH;
#2 lore raw
Quoted 4 lines · Zhu Yanjun

Thanks a lot. It would be great to have a corresponding negative test in tools/testing/selftests/rdma that sends malformed ATOMIC_WRITE requests (e.g., zero-length) and verifies that they are rejected and do not modify the target MR.

Good idea. Do you want a v2 with this fix + a separate test case in rdma or should I submit it separately?

Thanks,
Mike Bommarito

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 3 lines · Zhu Yanjun

The fix should stand on its own, and the test case can be added as a follow-up patch under tools/testing/selftests/rdma (or rdma-core if more appropriate). This keeps each patch focused on a single logical change and makes review easier.

It seems like mainlining rdma tests might not be a good idea. I found the last time here:
Kamal Heib, 23 Oct 2019 — [PATCH for-next] selftests: rdma: Add rdma tests

I put the tests here and will PR it into linux-rdma/rdma-core tomorrow:
https://github.com/mjbommar/rdma-core/blob/4104d991a764de4aad9f645a2f0ec723f6076209/tests/test_rxe_atomic_write_oob.py

Thanks,
Mike Bommarito

#5 lore raw
Quoted 10 lines · Michael Bommarito
Quoted 3 lines · Zhu Yanjun

The fix should stand on its own, and the test case can be added as a follow-up patch under tools/testing/selftests/rdma (or rdma-core if more appropriate). This keeps each patch focused on a single logical change and makes review easier.

It seems like mainlining rdma tests might not be a good idea. I found the last time here:
Kamal Heib, 23 Oct 2019 — [PATCH for-next] selftests: rdma: Add rdma tests

I put the tests here and will PR it into linux-rdma/rdma-core tomorrow:
https://github.com/mjbommar/rdma-core/blob/4104d991a764de4aad9f645a2f0ec723f6076209/tests/test_rxe_atomic_write_oob.py

Thanks a lot. I am fine with the above link.

Zhu Yanjun

Quoted 3 lines

Thanks,
Mike Bommarito

Signature · 2 lines
Best Regards,
Yanjun.Zhu
maintainer Applied
#6 lore raw
In reply to Michael Bommarito
Quoted 29 lines · Michael Bommarito

atomic_write_reply() at drivers/infiniband/sw/rxe/rxe_resp.c unconditionally dereferences 8 bytes at payload_addr(pkt):

    value = *(u64 *)payload_addr(pkt);

check_rkey() previously accepted an ATOMIC_WRITE request with pktlen == resid == 0 because the length validation only compared pktlen against resid. A remote initiator that sets the RETH length to 0 therefore reaches atomic_write_reply() with a zero-byte logical payload, and the responder reads sizeof(u64) bytes from past the logical end of the packet into skb->head tailroom, then writes those 8 bytes into the attacker's MR via rxe_mr_do_atomic_write(). That is a remote disclosure of 4 bytes of kernel tailroom per probe (the other 4 bytes are the packet's own trailing ICRC).

IBA oA19-28 defines ATOMIC_WRITE as exactly 8 bytes. Anything else is protocol-invalid. Hoist a strict length check into check_rkey() so the responder never reaches the unchecked dereference, and keep the existing WRITE-family length logic for the normal RDMA WRITE path.

Reproduced on mainline with an unmodified rxe driver: a sustained zero-length ATOMIC_WRITE probe repeatedly leaks adjacent skb head-buffer bytes into the attacker's MR, including recognisable kernel strings and partial kernel-direct-map pointer words. With this patch applied the responder rejects the PDU and the MR stays all-zero.

Quoted 6 lines
Fixes 034e285f8b99 ("RDMA/rxe: Make responder support atomic write on RC service")
Assisted-by Claude:claude-opus-4-7

---

Applied to for-rc thanks

Jason