APM support for IPoIB

Adding APM support for IPoIB seems like something worthwhile and we're thinking about it..but are wondering why has hasn't been done already..any comments/suggestions?

Quoted 3 lines · Bang Nguyen

Adding APM support for IPoIB seems like something worthwhile and we're thinking about it..but are wondering why has hasn't been done already..any comments/suggestions?

I've been quite supportive of APM in the past, IMHO I think it is something unique to IB that should be exploited as often as possible.

Improving the RDMA-CM to setup APM at all would be a great first step, but some SM work is also ultimately needed to get sufficiently independent/redundant paths.

I would like to see the full APM support eventually implemented - this would be fail over across adaptor ports, where each port is on a seperate subnet - so you get full redundancy of cabling, switching and management software.

Jason

Our version of OFED has full support of APM in RDMA-CM and it's being used by SDP and RDS..

Perhaps working on getting that merged upstream would be a better first step.

Quoted 1 line

Perhaps working on getting that merged upstream would be a better first step.

We have it on our plans to submit RDMA-CM APM to upstream.
Since we are in holidays season here it will take us some time to come back with plan.

In any case as Bang wrote this is not related to IPoIB-CM, that requires a different implementation, and any one is welcome to contribute here :-)

Tziporet

Just a crazy thought but IPoIB could use RDMA-CM for all of its connection management stuff..this way it can leverage the APM support in RDMA-CM..

comments?...

Quoted 5 lines

Just a crazy thought but IPoIB could use RDMA-CM for all of its connection management stuff..this way it can leverage the APM support in RDMA-CM..

comments?...

ipoib predates the rdma cm, which is why it's written to the ib_cm directly. Communication setup with the rdma cm follows these 3 steps.

1. address resolution - The rdma cm indirectly ends up using ipoib for this. With ipoib, this step isn't necessary. I think having AF_IB support would help this by avoiding this step, plus it restricts the rdma cm to using IB devices.
2. route resolution (aka path record query) - Ipoib maintains its own path record cache. It can set the rdma cm paths manually, or there would need to be some sort of rework.
3. connection setup - formats the ib cm private data and exchanges messages. The rdma cm would need to know the ipoib private data format.

Sooo... ipoib wouldn't use address resolution. It maintains its own path record cache, which is a significant piece of supporting APM. Ipoib probably doesn't need to maintain its own cache, but there's no decent alternative for kernel clients currently. And connection setup using the rdma cm is only slightly easier than using the ib cm. (I have no idea what APM support in the rdma cm looks like.)

- Sean

Here is the presentation on RDMA-CM APM

https://www.openfabrics.org/ofa-documents/presentations/doc_download/516-rdmacm-amp.html

Quoted 14 lines · Sean

Quoted 5 lines

Just a crazy thought but IPoIB could use RDMA-CM for all of its connection management stuff..this way it can leverage the APM support in RDMA-CM..

comments?...

ipoib predates the rdma cm, which is why it's written to the ib_cm directly. Communication setup with the rdma cm follows these 3 steps.

1. address resolution - The rdma cm indirectly ends up using ipoib for this. With ipoib, this step isn't necessary. I think having AF_IB support would help this by avoiding this step, plus it restricts the rdma cm to using IB devices.
2. route resolution (aka path record query) - Ipoib maintains its own path record cache. It can set the rdma cm paths manually, or there would need to be some sort of rework.
3. connection setup - formats the ib cm private data and exchanges messages. The rdma cm would need to know the ipoib private data format.

Sooo... ipoib wouldn't use address resolution. It maintains its own path record cache, which is a significant piece of supporting APM. Ipoib probably doesn't need to maintain its own cache, but there's no decent alternative for kernel clients currently. And connection setup using the rdma cm is only slightly easier than using the ib cm. (I have no idea what APM support in the rdma cm looks like.)

- Sean

-----Original Message-----
From: [email protected] [mailto:linux-rdma- [email protected]] On Behalf Of Jason Gunthorpe
Sent: Monday, October 01, 2012 4:53 PM
To: Bang Nguyen
Cc: [email protected]; Richard Frank
Subject: Re: APM support for IPoIB

Quoted 3 lines · Bang Nguyen

Adding APM support for IPoIB seems like something worthwhile and we're thinking about it..but are wondering why has hasn't been done already..any comments/suggestions?

I've been quite supportive of APM in the past, IMHO I think it is something unique to IB that should be exploited as often as possible.

Improving the RDMA-CM to setup APM at all would be a great first step, but some SM work is also ultimately needed to get sufficiently independent/redundant paths.

I would like to see the full APM support eventually implemented - this would be fail over across adaptor ports, where each port is on a seperate subnet - so you get full redundancy of cabling, switching and management software.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html

Jason - Like your idea for full APM support especially across subnets, but wouldn't that require some sort capability that lives above the subnet level and is able to query both SMs/SAs? Sounds vaguely router-ish?

Jason - Like your idea for full APM support especially across subnets, but wouldn't that require some sort capability that lives above the subnet level and is able to query both SMs/SAs? Sounds vaguely router-ish?