About FRMR vec and SRQ post recv

Sorry.
sge[1]= {vec1_addr, vec1_length, dma_lkey};
should be
sge[1]= {vec2_addr, vec2_length, dma_lkey};

Quoted 4 lines

Hi, Steve.
I have the following questions.
1)FRMR
I want transfer the data in vec1 and vec2 in node A to the two vecs

with the same size in node B. Vecs' size is lower than a page.

Quoted 10 lines

A B

+--------+ +--------+
| vec1 | | vec1 |
+--------+ +--------+

+--------+ +--------+
| vec2 | | vec2 |
+--------+ +--------+

Could I use FRMR to register vec1 and vec2 as one Memory region?

Quoted 1 line

If so, then they have the same rkey, and I could send them to remote

peer for one time,right?

Quoted 3 lines

Or I have to register and send vec1 and vec2 seperately?

If I use Send/Recv not RDMA Read/Write to send them. ie. I will use

dma_mr's lkey/rkey, and let

Quoted 4 lines

sge[0]= {vec1_addr, vec1_length, dma_lkey};
sge[1]= {vec1_addr, vec1_length, dma_lkey};
ib_send_wr.sgl_list= sge;

Quoted 3 lines

and post send this wr. Is it allowed?
2) SRQ
SRQ's size is 1024, and I use ib_post_srq_recv 1024 times after

getting SRQ created.

Quoted 1 line

When cq_handler get a IB_WC_RECV event, do I need post another recv

wr?(since in the case without SRQ, it would be yes)

I try getting a summary. Give some comments on the following sentences,please.
For local side(A), where data to be transferred is placed. send or RDMA Read/Write operations.
the sge array in one send wr pointed by sgl_list should have the same [lr]key.
a) one send wr could send two arbitrary regions when using dma mr's [lr]ley or local_dma_lkey
b) one send wr could 'not' send two arbitrary regions registered by two FRWRs seperately.

For remote side(B), where data transferred will be placed.
2) RDMA Read/Write operations could put data to
a) a single physical continous region registerd
b) two or more regions with contstraints that they could be registerd by one FRWR.
c) could not put data to two regions registerd by two FRWR sperately since in side A, ib_send_wr.wr.rdma should specify one group {rkey,addr} for remote destination region.

and send/recv operatons can also put data to
c) two or more arbitrary regions using dma mr. since send operation in side A do not need remote {addr,rkey} and recv operation in side B could post two arbitrary regions using dma mr's lkey or local_dma_lkey in a singel recv_wr.

I want to transfer data from two arbitrary regions of A to two arbitrary regions of B for a single RDMA Read/Write operation originally.And it seems impossible now.

If I use send/recv operations to send the two arbitrary regions form A to B.How about the data order?
I mean
for side A
sge[0] = {A_vec1_addr,A_vec1_len, A_lkey}

sge[1] = {A_vec2_addr,A_vec2_len, A_lkey} ib_send_wr.sg_list = sge;
for side B
sge[0] = {B_vec1_addr,B_vec1_len, B_lkey}
sge[1] = {B_vec2_addr,B_vec2_len, B_lkey} ib_recv_wr.sg_list = sge;
Does lower level layers ensure that data in vec1 in A would be placed to vec1 in B and vec2(A) to vec2(B)?

According to krping and my experience, to use send/recv operations, I have to convert host byte order to net byte order when preparing send_buf, and convert back to host byte order when recv_buf's data is ready.Yet, RDMA Read/Write does not has this overhead,right?