Skip to contentSkip to conversation

Question about extra 40 bytes needed for UD receive buffer

DISCUSSION Honggang LI 1 participant 2 messages Apr 15, 2024 InfiniBand

lore
Share & tools
lore
#0 lore raw

hi,

According to volume 1 of IBAT specification of release 1.4,

1) 11.4.1.2 POST RECEIVE REQUEST
Note that for UD QPs, the first 40 bytes of the buffer(s) referred to by the Scatter/Gather list will contain the GRH of the incoming message. If no GRH is present, the contents of first 40 bytes of the buffer(s) will be undefined. The presence of the GRH will be indicated by a bit in the Work Completion.

2) 11.4.2.1 POLL FOR COMPLETION
GRH Present indicator, for UD RQs only. If this indicator is set, the first 40 bytes of the buffer(s) referred to by the Scatter/Gather list will contain the GRH of the incoming message. If it is not set, the contents of first 40 bytes of the buffer(s) will be undefined. Contents of the payload of the message will begin after the first 40 bytes

3) A17.4.5.2 SCATTERING OF THE L3 HEADER IN UD
The first 40 bytes of user posted UD Receive Buffers are reserved for the L3 header of the incoming packet (as per the InfiniBand Spec Section 11.4.1.2). In RoCEv2, this area is filled up with the IP header. IPv6 header uses the entire 40 bytes. IPv4 headers use the 20 bytes in the second half of the reserved 40 bytes area (i.e. offset 20 from the beginning of the receive buffer). In this case, the content of the first 20 bytes is undefined.

After function ibv_poll_cq return 1, the dump of receive buffer of ibv_ud_pingpong shows there is 40 bytes GRH in the head of receive buffer. The first 20 bytes are zeros and the second 20 bytes is a valid IP header. At this point, everything works as required by the specification rules 1), 2) and 3).

However, I'm confused by the dump of mckey receive buffer. The flag IBV_WC_GRH is set in ibv_wc.wc_flags, but there is no GRH in the receive buffer. Received data starts from the *first* byte of receive buffer. As multicast over UD QP only, can someone please explian why there is no GRH in receive buffer and the data starts from the first bytes of receive buffer with IBV_WC_GRH was set?

Thanks

#1 lore raw
Quoted 10 lines · Honggang LI

Subject: Question about extra 40 bytes needed for UD receive buffer
From: Honggang LI <[email protected]>
Date: Mon, 15 Apr 2024 10:55:12 +0800

However, I'm confused by the dump of mckey receive buffer. The flag IBV_WC_GRH is set in ibv_wc.wc_flags, but there is no GRH in the receive buffer. Received data starts from the *first* byte of receive buffer. As multicast over UD QP only, can someone please explian why there is no GRH in receive buffer and the data starts from the first bytes of receive buffer with IBV_WC_GRH was set?

Please ignore this. I made a mistake with the offset of the grh.

There is a valid IP header in the receive buffer of 'mckey'.