failure to get gid with rdma_bind_addr with >= 3.10 kernels

Hi Sean,

So somewhere between 3.10 and 3.11 a little bug was introduced in ucma/cma that when you bind to IPv4 address the resulted device address returned from the kernel is all zeros.

You can use the attached program to reproduce, just run it and provide an IPv4 address of IPoIB NIC

--> with 3.10

$ ./rb 192.168.20.47
event_channel: 0x6093c0
cm_id: 0x602cb0
sgid: fe 80 00 00 00 00 00 00 00 02 c9 03 00 e9 c0 81

--> with 3.12

$ ./rb 192.168.20.17
event_channel: 0x6093c0
cm_id: 0x6034c0
sgid: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

There many commits for the 3.11 AF_IB merge, hope you can dig it out quickly..

Or.

linux-2.6]# git log --oneline v3.10.. drivers/infiniband/core/cma.c drivers/infiniband/core/ucma.c 7c049d0 Merge tag 'rdma-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
27703bb Merge tag 'PTR_RET-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
24d44a3 RDMA/cma: Add IPv6 support for iWARP
5eb695c RDMA/cma: Only call cma_save_ib_info() for CM REQs
e511d1a RDMA/cma: Fix accessing invalid private data for UD
8fb488d RDMA/cma: Fix gcc warning
8c6ffba PTR_RET is now PTR_ERR_OR_ZERO(): Replace most.
ce117ff RDMA/cma: Export AF_IB statistics
5bc2b7b RDMA/ucma: Allow user space to specify AF_IB when joining multicast
209cf2a RDMA/ucma: Allow user space to pass AF_IB into resolve
eebe4c3 RDMA/ucma: Allow user space to bind to AF_IB
05ad945 RDMA/ucma: Name changes to indicate only IP addresses supported
edaa7a5 RDMA/ucma: Add ability to query GID addresses
cf53936 RDMA/cma: Export cma_get_service_id()
ac53b26 RDMA/ucma: Support querying when IB paths are not reversible
ee7aed4 RDMA/ucma: Support querying for AF_IB addresses
94d0c93 RDMA/cma: Only listen on IB devices when using AF_IB
5c43813 RDMA/cma: Set qkey for AF_IB
e8160e1 RDMA/cma: Expose private data when using AF_IB
fbaa1a6 RDMA/cma: Merge cma_get/save_net_info
01602f1 RDMA/cma: Remove unused SDP related code
496ce3c RDMA/cma: Add support for AF_IB to cma_get_service_id()
f68194c RDMA/cma: Add support for AF_IB to rdma_resolve_route()
f17df3b RDMA/cma: Add support for AF_IB to rdma_resolve_addr()
4ae7152 RDMA/cma: Verify that source and dest sa_family are the same
b0569e4 RDMA/cma: Restrict AF_IB loopback to binding to IB devices only
f475383 RDMA/cma: Add helper functions to return id address information
6a3e362 RDMA/cma: Do not modify sa_family when setting loopback address
680f920 RDMA/cma: Allow user to specify AF_IB when binding
58afdcb RDMA/cma: Update port reservation to support AF_IB
ef56086 IB/addr: Add AF_IB support to ip_addr_size
2e2d190 RDMA/cma: Include AF_IB in loopback and any address checks
c8dea2f9 RDMA/cma: Allow enabling reuseaddr in any state
351638e net: pass info struct via netdevice notifier

#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <errno.h>
#include <string.h>
#include <rdma/rdma_cma.h>

int main( int argc, char **argv ) {
struct rdma_cm_id * cm_id = NULL;
struct rdma_event_channel * event_channel = NULL; int i = 0;
struct sockaddr_in sa;

  if( argc < 2 ) {
    fprintf( stderr, "Specify interface address\n" );
    exit(1);
  }
 
  if( !inet_aton( argv[1], &sa.sin_addr.s_addr ) ) {
    fprintf( stderr, "Address invalid: %s\n", argv[1] );
    exit(1);
  }

  sa.sin_port   = 0;
  sa.sin_family = AF_INET;

  event_channel = rdma_create_event_channel();
  if( !event_channel ) {
    fprintf( stderr, "rdma_create_event_channel failed" );
    exit(1);
  }
  printf( "event_channel: %p\n", event_channel );

  if( rdma_create_id( event_channel, &cm_id, 0, RDMA_PS_IPOIB ) ) {
    fprintf( stderr, "rdma_create_id failed (%s)", strerror(errno) );
    exit(1);
  }
  printf( "cm_id: %p\n", cm_id );

  if( rdma_bind_addr( cm_id, (void*)&sa ) ) {
    fprintf( stderr, "rdma_bind_addr failed (%s)", strerror(errno) );
    exit(1);
  }

  uint8_t * x  = (uint8_t*)&cm_id->route.addr.addr.ibaddr.sgid;
  size_t    sz = sizeof( cm_id->route.addr.addr.ibaddr.sgid );

  printf( "sgid: " );

  for( i = 0; i < sz; ++i ) {
    printf( "%2.2x ", x[i] );
  }

printf( "\n" );
return 0;
}

between 3.10 and 3.12
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I'll take a look.

What is the impact to the application?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Application will run with MOFED 2.0/OFED 1.5.3 but wont start with Linux 3.12.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

..

As part of the AF_IB changes, additional queries were introduced that separated out retrieving the different kernel data -- assigned address, GID mappings, and IB PR data. (New queries were necessary, since sockaddr_ib is larger than sockaddr_in6, and we want to eventually handle non-reversible paths.) The librdmacm 1.0.17 changed which query was called after rdma_bind_addr was invoked to only retrieve the assigned address, versus retrieving everything. I think this is why you see an SGID of all 0's after calling rdma_bind_addr.

librdmacm 1.0.16 does not enable AF_IB for user space, and should return the SGID. Assuming that this interpretation of the problem is correct, a fix for 1.0.17 would be to add a call to ucma_query_gid() from rdma_bind_addr2() - note '2' - if the specified address is not 'any address'.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Is there another call that allows the retrieval of the SGID? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes - something like this patch should help, but I don't think this is the correct behavior when the IP address is 'any' addresss.

Retrieve SGID after calling rdma_bind_addr

From: Sean Hefty <[email protected]>

A change was made to rdma_bind_addr when AF_IB is enabled to only retrieve the resulting bound address. Previously, rdma_bind_addr would retrieve the corresponding SGID as well. This breaks some apps which were checking the SGID after binding to an IP address. Revert to the previous behavior of also retrieving the SGID after calling rdma_bind_addr.

---
 src/cma.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

index 4f41879..0cf4203 100644
@@ -753,7 +753,10 @@ static int rdma_bind_addr2(struct rdma_cm_id *id, struct sockaddr *addr,
 	if (ret != sizeof cmd)
 		return (ret >= 0) ? ERR(ENODATA) : -1;
 
-	return ucma_query_addr(id);
+	ret = ucma_query_addr(id);
+	if (!ret)
+		ret = ucma_query_gid(id);
+	return ret;
 }
 
 int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Well it works fine. Application starts now.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I will merge it, since it fixes the issue.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

cool
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Sean,

Can you please prepare a fresh (gift to 2014) release of librdmacm, such that distributions && people using kernels >= 3.11 will be able to have this functionality restored once they upgrade librdmacm.

thanks --

Or
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi,

Why is this issue addressed by a patch in librdmacm ?

After reading quickly the thread, I still believe that's sound like a kernel regression which broke existing userspace applications.

So a patch must be applied on the kernel to fix that regression.

Introducing AF_IB must not have changed the behavor for existing applications. Existing applications must not need a new librdmacm when they don't use newer kernel extensions.

Regards.

PS: Happy new (gregorian calendar's) year 2014 and best wishes ;)

Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

PiBBZnRlciByZWFkaW5nIHF1aWNrbHkgdGhlIHRocmVhZCwgSSBzdGlsbCBiZWxpZXZlIHRoYXQn cyBzb3VuZCBsaWtlIGENCj4ga2VybmVsIHJlZ3Jlc3Npb24gd2hpY2ggYnJva2UgZXhpc3Rpbmcg dXNlcnNwYWNlIGFwcGxpY2F0aW9ucy4NCj4gDQo+IFNvIGEgcGF0Y2ggbXVzdCBiZSBhcHBsaWVk IG9uIHRoZSBrZXJuZWwgdG8gZml4IHRoYXQgcmVncmVzc2lvbi4NCg0KV2hlbiBBRl9JQiB3YXMg YWRkZWQgdG8gdGhlIGtlcm5lbCwgY29ycmVzcG9uZGluZyBjaGFuZ2VzIHdlcmUgYWRkZWQgdG8g dGhlIGxpYnJkbWFjbS4gIFRob3NlIGNoYW5nZXMgdG8gdGhlIGxpYnJkbWFjbSBhcmUgdGhlIGNh dXNlIG9mIHRoZSBpc3N1ZS4NCg0KVGhlIGtlcm5lbCByZG1hIGNtIHByZXZpb3VzbHkgc3VwcG9y dGVkIGEgc2luZ2xlICdxdWVyeScgY2FsbC4gIFdpdGggQUZfSUIsIGl0IHN1cHBvcnRzIHRoZSBv cmlnaW5hbCBjYWxsLCBwbHVzIDMgYWRkaXRpb25hbCBjYWxscy4gIFRoZSBwcm9ibGVtIGlzIHRo ZSBsaWJyZG1hY20gdXNpbmcgdGhlIG5ld2VyIHF1ZXJ5IGNhbGxzLg0KIA0KPiBJbnRyb2R1Y2lu ZyBBRl9JQiBtdXN0IG5vdCBoYXZlIGNoYW5nZWQgdGhlIGJlaGF2b3IgZm9yIGV4aXN0aW5nDQo+ IGFwcGxpY2F0aW9ucy4gRXhpc3RpbmcgYXBwbGljYXRpb25zIG11c3Qgbm90IG5lZWQgYSBuZXcg bGlicmRtYWNtIHdoZW4NCj4gdGhleSBkb24ndCB1c2UgbmV3ZXIga2VybmVsIGV4dGVuc2lvbnMu DQoNCkFuIG9sZGVyIHZlcnNpb24gb2YgdGhlIGxpYnJkbWFjbSB0aGF0IGRvZXMgTk9UIHN1cHBv cnQgQUZfSUIgc2hvdWxkIHdvcmsgZmluZS4NCg0KLSBTZWFuDQo=
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I was missing this part, the original message was misleading for me.

So it's the combination of librdmacm >= 1.0.17 and kernel >= 3.10+ which is affected by the problem reported.

Thanks for clarifying this for me.

Regards.

Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

There's a 1.0.17.1 release available with this fix.

I will release 1.0.18 within a couple of weeks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

good to know... was it announced on the list?

cool
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html