Skip to contentSkip to conversation

WRITE/SEND to multiple destinations

DISCUSSION floaterions 2 participants 4 messages Nov 3, 2014 – Nov 4, 2014 4 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

Hello everyone,

As a newbie to the world of InfiniBand programming,
I would appreciate your helps on my somewhat general and basic question.

Assume that I have a server, and N clients (N > 1).
I would like my server to set all clients' memory buffers to some value V. How would I go about this problem?
So far, I've managed to establish 1-1 connections using READ/WRITE/SEND/RECEIVE. So I guess the core of my question is:
how 1-to-N connections are established in InfiniBand, and
if, in this regard, there is any difference between
RDMA verbs (READ/WRITE) vs. SEND/RECEIVE?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
In reply to a message not in the archive

You can do SEND/RECV over Unreliable Datagram multicast which could be beneficial in your situation (i.e., better than 1-1 connections). You can't use multicast for READs or WRITEs.

--Anuj

Quoted 14 lines · floaterions

Hello everyone,

As a newbie to the world of InfiniBand programming,
I would appreciate your helps on my somewhat general and basic question.

Assume that I have a server, and N clients (N > 1).
I would like my server to set all clients' memory buffers to some value V. How would I go about this problem?
So far, I've managed to establish 1-1 connections using READ/WRITE/SEND/RECEIVE. So I guess the core of my question is:
how 1-to-N connections are established in InfiniBand, and
if, in this regard, there is any difference between
RDMA verbs (READ/WRITE) vs. SEND/RECEIVE?

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
In reply to a message not in the archive

For the RC transport, the answer is yes. You definitely need one queue pair per client at the server, although it could be possible for the QPs at the server to share an event channel.

But UD transport is also pretty "reliable" as packet loss in any InfiniBand transport is extremely rare. A rather high ballpark is one in a billion.

--Anuj

Quoted 34 lines · Just Floaterions

Thank you for your help. So the question is, for reliable connections, do I need separate event channel/queue-pairs for each connection?

Quoted 28 lines · Anuj Kalia

You can do SEND/RECV over Unreliable Datagram multicast which could be beneficial in your situation (i.e., better than 1-1 connections). You can't use multicast for READs or WRITEs.

--Anuj

Quoted 20 lines · floaterions

Hello everyone,

As a newbie to the world of InfiniBand programming,
I would appreciate your helps on my somewhat general and basic question.

Assume that I have a server, and N clients (N > 1).
I would like my server to set all clients' memory buffers to some value V.
How would I go about this problem?
So far, I've managed to establish 1-1 connections using READ/WRITE/SEND/RECEIVE.
So I guess the core of my question is:
how 1-to-N connections are established in InfiniBand, and if, in this regard, there is any difference between
RDMA verbs (READ/WRITE) vs. SEND/RECEIVE?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw
In reply to a message not in the archive

Stupid me. I totally forgot about the XRC transport:
http://mvapich.cse.ohio-state.edu/static/media/publications/abstract/koop-cluster08.pdf.

--Anuj

Quoted 4 lines · Anuj Kalia

For the RC transport, the answer is yes. You definitely need one queue pair per client at the server, although it could be possible for the QPs at the server to share an event channel.

Quoted 41 lines

But UD transport is also pretty "reliable" as packet loss in any InfiniBand transport is extremely rare. A rather high ballpark is one in a billion.

--Anuj

Quoted 34 lines · Just Floaterions

Thank you for your help. So the question is, for reliable connections, do I need separate event channel/queue-pairs for each connection?

Quoted 28 lines · Anuj Kalia

You can do SEND/RECV over Unreliable Datagram multicast which could be beneficial in your situation (i.e., better than 1-1 connections). You can't use multicast for READs or WRITEs.

--Anuj

Quoted 20 lines · floaterions

Hello everyone,

As a newbie to the world of InfiniBand programming,
I would appreciate your helps on my somewhat general and basic question.

Assume that I have a server, and N clients (N > 1).
I would like my server to set all clients' memory buffers to some value V.
How would I go about this problem?
So far, I've managed to establish 1-1 connections using READ/WRITE/SEND/RECEIVE.
So I guess the core of my question is:
how 1-to-N connections are established in InfiniBand, and if, in this regard, there is any difference between
RDMA verbs (READ/WRITE) vs. SEND/RECEIVE?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html