Skip to contentSkip to conversation

libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

PATCH In review Or Gerlitz 3 participants 5 messages Nov 30, 2010 – Dec 1, 2010 8 messages not in the archive InfiniBand

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 4CF3E5FA.6010308-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Or Gerlitz

author
#5 lore raw

PATCH 1/1 libibverbs/doc: clarify selective signaling (Re: using IBV_SEND_SIGNALED)

Quoted 5 lines · Steve Wise

On 11/29/2010 11:34 AM, Robert D. Russell wrote:
See section 8.2.1 of the iWARP Verbs draft at:
http://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.2.1.
The last big paragraph in 8.2.1 spells it all out.
There should be a similar requirement in the IBTA IB Verbs specification...

Yes, I think the below patch to the man page can help people understand how to work with the thing

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

You must signal one WR for every SQ-depth worth of WRs posted. IE:
If your SQ depth is 16, you must signal at least one out of every 16 This ensures proper flow control for HW resources.

clarify selective signaling usage

---

 man/ibv_post_send.3 |    6 ++++++
 1 file changed, 6 insertions(+)
man/ibv_post_send.3 +6 -0
Index: libibverbs/man/ibv_post_send.3
===================================================================
@@ -112,6 +112,12 @@ request is fully executed and a work com
 from the corresponding completion queue (CQ). However, if the
 IBV_SEND_INLINE flag was set, the buffer can be reused immediately
 after the call returns.
+.PP
+Applications which use selective signaling should make sure to post
+a signaled WR (i.e one for which the IBV_SEND_SIGNALED flag is set) from
+time to time - depending on the requirements of the underlying rdma device.
+Typically the number of consecutive unsignaled WRs should not exceed the
+size of the send queue specified when the QP was created.
 .SH "SEE ALSO"
 .BR ibv_create_qp (3),
 .BR ibv_create_ah (3),

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CF4E65D.4070205-hKgKHo2Ms0FWk0Htik3J/[email protected]

#7 lore raw
Quoted 9 lines · Or Gerlitz
Quoted 5 lines · Steve Wise

On 11/29/2010 11:34 AM, Robert D. Russell wrote:
See section 8.2.1 of the iWARP Verbs draft at:
http://tools.ietf.org/html/draft-hilland-rddp-verbs-00#section-8.2.1.
The last big paragraph in 8.2.1 spells it all out.
There should be a similar requirement in the IBTA IB Verbs specification...

Yes, I think the below patch to the man page can help people understand how to work with the thing

I think the subtly here is that remote completion of a local WR does not indiciate local completion. This is the only operational case I can think of that might break the rules.

For instance a protocol that had remote acks for all SENDs might be tempted to not ever use signaled WR's on the local side - since receiving the remote ACK SEND must indicate the local SEND WR is complete. This is the incorrect assumption.

So, I'd suggest this language:

Applications which use selective signaling can only assume that unsignaled WRs are complete once a completion for a later signaled WR is received. In practice this means that a signaled WR must be used periodically, and that the send queue should never be filled with unsignaled WRs.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20101130185744.GK16788-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#9 lore raw
Quoted 1 line · Jason Gunthorpe

I think the subtly here is that remote completion of a local WR does not indiciate local completion. This is the only operational case I can think of that might break the rules.

The point you're bringing is sure valid and actually I can change the patch to address both issues, the one raised on the original thread (see "using IBV_SEND_SIGNALED in ewg from this week), where Robert says "However, as soon as I call ibv_post_send() more than max_send_wr times, it fails" so 1st we need to clarify that without posting a signaled WR the HCA will fail, and 2nd that the application can't consume back the resources associated with a send WR until a "later" WR completes.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw
In reply to a message not in the archive
Quoted 4 lines · Jason Gunthorpe

Applications which use selective signaling can only assume that unsignaled WRs are complete once a completion for a later signaled WR
is received. In practice this means that a signaled WR must be
used periodically, and that the send queue should never be filled with unsignaled WRs.

clarify selective signaling usage

---

Jason, how's this?

 man/ibv_post_send.3 |    9 +++++++++
 1 file changed, 9 insertions(+)
man/ibv_post_send.3 +9 -0
Index: libibverbs/man/ibv_post_send.3
===================================================================
@@ -112,6 +112,15 @@ request is fully executed and a work com
 from the corresponding completion queue (CQ). However, if the
 IBV_SEND_INLINE flag was set, the buffer can be reused immediately
 after the call returns.
+.PP
+Applications which use selective signaling can only assume that
+unsignaled WRs are complete once a completion for a later signaled WR
+is received. In practice this means that a signaled WR must be
+used periodically, and that the send queue should never be filled with
+unsignaled WRs. Such periodic signaling is also a requirement of most rdma
+devices. In the presence of completion with error, WCs are generated
+also for unsignaled WRs, so applications should make sure to provide a valid
+id (cookie) for all posted WRs.
 .SH "SEE ALSO"
 .BR ibv_create_qp (3),
 .BR ibv_create_ah (3),

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4CF60ED0.6000502-hKgKHo2Ms0FWk0Htik3J/[email protected]

#12 lore raw
Quoted 11 lines · Or Gerlitz
Quoted 4 lines · Jason Gunthorpe

Applications which use selective signaling can only assume that unsignaled WRs are complete once a completion for a later signaled WR
is received. In practice this means that a signaled WR must be
used periodically, and that the send queue should never be filled with unsignaled WRs.

clarify selective signaling usage

Jason, how's this?

Sure, but reviewing the original thread makes me think there is a greater mis-understanding at work about how resource ownership works in verbs than simply related to the unsignaled WRs.

Maybe this would be a better approach:

RESOURCE OWNERSHIP
ibv_post_* transfers ownership of the described buffer(s) to the device. While the device is processing the WR the buffer(s) should not be changed and a queue entry is
consumed to hold this WR while it is being processed.

Callers are required to track how many entries in the queue are owned by the device and how many are free for new WRs. Attempting to post more WRs that there are free queue entries results in E???.

Ownership of the buffer and queue entry is only passed back to the caller once a completion on the associated CQ is retrieved. A completion indicates all prior WR are completed, including unsignaled WRs.

Note that receiving a completion for the WR through a CQ is the only way to know that the queue entry and WR is passed back to the caller. Other implicit means, such as receiving a recv completion, observing data change in memory, or the like do not reliably indicate that the device is finished with the queue entry and WR.

When using unsignaled WRs it is necessary to periodically use a signaled WR and collect completions so that the available free space in the queue can be tracked.

Also, I like your sentence about errors for unsignaled WCs, that should be kept in the IBV_SEND_SIGNALED section..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html