Is there a way to transfer a rdma connection between userspace processes?

DISCUSSION Stefan (metze) Metzmacher 4 participants 6 messages Oct 15, 2012 – Oct 17, 2012 4 messages not in the archive InfiniBand

lore

Share & tools

lore

Stefan (metze) Metzmacher

author

2012-10-15 08:27 UTC #0 lore raw

Hi,

I'm currently researching how to implement SMBDirect [MS-SMBD] together with the multi channel feature of SMB 3.0 in Samba.

As Samba currently uses one process per tcp connection
and maintains a lot of in memory state within the process (e.g. for the SMB_VFS modules) it would require a lot of work to change Samba to coordinate two (or more) processes for one logical multi channel connection.

My current plan tries to pass the socket fd of new connections (which join an existing multi channel session) via fd-passing to the existing process.

Now I'm wondering if this would also be possible with a rdma connection (struct rdma_cm_i ).

From reading the code of rdma_create_event_channel()/rdma_create_id() and rdma_migrate_id(), it seems that the connection state is in partly in userspace (structures) and partly in the kernel space (hidden behind the channel fd)

ibv_create_comp_channel() and ibv_create_cq() seem to have a similar design.

As the ibverbs interface typically has a userspace driver I'm wondering if it's always true that there's also some kernel state maintained via the rdma/ibv_comp event channels?

As far as I can see there's currently no way to transfer the rdma/ibv state to
another process (for me it's enough to transfer it, using it from both processes
is not strictly needed).

Is anybody working on this already?

metze

A message that is not in the archive (sent off-list or not yet received): [email protected]

Yann Droneaud

2012-10-15 09:07 UTC #2 lore raw

Le lundi 15 octobre 2012 à 10:27 +0200, Stefan (metze) Metzmacher a écrit :

Quoted 11 lines

Hi,
I'm currently researching how to implement SMBDirect [MS-SMBD] together with the multi channel feature of SMB 3.0 in Samba.
As Samba currently uses one process per tcp connection
and maintains a lot of in memory state within the process (e.g. for the SMB_VFS modules) it would require a lot of work to change Samba to coordinate two (or more) processes for one logical multi channel connection.

Quoted 7 lines

My current plan tries to pass the socket fd of new connections (which join an existing multi channel session) via fd-passing to the existing process.
Now I'm wondering if this would also be possible with a rdma connection (struct rdma_cm_i ).

RDMA / verbs ressources are tied to a process (especially Memory Registration), but it's ending up in the HCA, which is probably unaware of processes.

Additionally, an RDMA_CM connection is not identified by a FD, so this kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not going to work.

Forking might already be a challenge for a RDMA/verbs application, so I don't think that sharing/moving an RDMA_CM connection across different processes is supported.

But other people on this list (especially Roland Dreier and Sean Hefty) could find a solution.

Regards.

Signature · 8 lines

Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1350292042.2750.11.camel-vNW8ozRvgWupuGC+iAP0z+TW4wlIGRCZ@public.gmane.org

Steve Wise

2012-10-15 15:31 UTC #4 lore raw

Quoted 28 lines · Yann Droneaud

Le lundi 15 octobre 2012 à 10:27 +0200, Stefan (metze) Metzmacher a écrit :
Quoted 11 lines
Hi,
I'm currently researching how to implement SMBDirect [MS-SMBD] together with the multi channel feature of SMB 3.0 in Samba.
As Samba currently uses one process per tcp connection
and maintains a lot of in memory state within the process (e.g. for the SMB_VFS modules) it would require a lot of work to change Samba to coordinate two (or more) processes for one logical multi channel connection.
Quoted 7 lines
My current plan tries to pass the socket fd of new connections (which join an existing multi channel session) via fd-passing to the existing process.
Now I'm wondering if this would also be possible with a rdma connection (struct rdma_cm_i ).
RDMA / verbs ressources are tied to a process (especially Memory Registration), but it's ending up in the HCA, which is probably unaware of processes.
Additionally, an RDMA_CM connection is not identified by a FD, so this kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not going to work.

Quoted 9 lines

Forking might already be a challenge for a RDMA/verbs application, so I don't think that sharing/moving an RDMA_CM connection across different processes is supported.
But other people on this list (especially Roland Dreier and Sean Hefty) could find a solution.
Regards.

fork() support like you need is not there in Linux RDMA verbs. Another alternative is to fork() before you setup the RDMA connection. IE if a regular TCP socket is first used to negotiate RDMA mode, then maybe you could fork() after negotiation but before setting up the RDMA connection and other resources?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 507C2C6E.7010507-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Stefan (metze) Metzmacher

author

2012-10-16 14:25 UTC #6 lore raw

@Yann and Steve: thanks for the feedback!

Quoted 31 lines

Quoted 31 lines
Quoted 9 lines
I'm currently researching how to implement SMBDirect [MS-SMBD] together with the multi channel feature of SMB 3.0 in Samba.
As Samba currently uses one process per tcp connection
and maintains a lot of in memory state within the process (e.g. for the SMB_VFS modules) it would require a lot of work to change Samba to coordinate two (or more) processes for one logical multi channel connection.
Quoted 7 lines
My current plan tries to pass the socket fd of new connections (which join an existing multi channel session) via fd-passing to the existing process.
Now I'm wondering if this would also be possible with a rdma connection (struct rdma_cm_i ).
RDMA / verbs ressources are tied to a process (especially Memory Registration), but it's ending up in the HCA, which is probably unaware of processes.
Additionally, an RDMA_CM connection is not identified by a FD, so this kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not going to work.
Forking might already be a challenge for a RDMA/verbs application, so I don't think that sharing/moving an RDMA_CM connection across different processes is supported.
But other people on this list (especially Roland Dreier and Sean Hefty) could find a solution.

Quoted 8 lines

Quoted 2 lines
Regards.
fork() support like you need is not there in Linux RDMA verbs. Another alternative is to fork() before you setup the RDMA connection. IE if a regular TCP socket is first used to negotiate RDMA mode, then maybe you could fork() after negotiation but before setting up the RDMA connection and other resources?

For client connections that would work, but it can't work for a server that uses fork() after listen().

Would it be possible to extend the librdmacm and libibverbs to:
- handle fork() after getting RDMA_CM_EVENT_CONNECT_REQUEST which means there have to be some functions to destroy contexts like struct rdma_cm_id, but without affecting the real connection. Similar to close() on an FD that is used shared between two processes.
- export/import an interprocess token attached to the hierarchy of objects similar to gss_export_sec_context/gss_import_sec_context.
Or a function which exports the contexts into a FD (of a pipe), which could also transfer open FDs to the other end.
Maybe this could pass the information through a rdma/ibverb event channel?

Are there any other possible solutions?

metze

A message that is not in the archive (sent off-list or not yet received): [email protected]

Stefan (metze) Metzmacher

author

2012-10-16 16:46 UTC #8 lore raw

Am 16.10.2012 16:25, schrieb Stefan (metze) Metzmacher:

Quoted 45 lines

@Yann and Steve: thanks for the feedback!
Quoted 31 lines
Quoted 31 lines
Quoted 9 lines
I'm currently researching how to implement SMBDirect [MS-SMBD] together with the multi channel feature of SMB 3.0 in Samba.
As Samba currently uses one process per tcp connection
and maintains a lot of in memory state within the process (e.g. for the SMB_VFS modules) it would require a lot of work to change Samba to coordinate two (or more) processes for one logical multi channel connection.
Quoted 7 lines
My current plan tries to pass the socket fd of new connections (which join an existing multi channel session) via fd-passing to the existing process.
Now I'm wondering if this would also be possible with a rdma connection (struct rdma_cm_i ).
RDMA / verbs ressources are tied to a process (especially Memory Registration), but it's ending up in the HCA, which is probably unaware of processes.
Additionally, an RDMA_CM connection is not identified by a FD, so this kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not going to work.
Forking might already be a challenge for a RDMA/verbs application, so I don't think that sharing/moving an RDMA_CM connection across different processes is supported.
But other people on this list (especially Roland Dreier and Sean Hefty) could find a solution.
Quoted 8 lines
Quoted 2 lines
Regards.
fork() support like you need is not there in Linux RDMA verbs. Another alternative is to fork() before you setup the RDMA connection. IE if a regular TCP socket is first used to negotiate RDMA mode, then maybe you could fork() after negotiation but before setting up the RDMA connection and other resources?
For client connections that would work, but it can't work for a server that uses fork() after listen().

Quoted 10 lines

Would it be possible to extend the librdmacm and libibverbs to:
- handle fork() after getting RDMA_CM_EVENT_CONNECT_REQUEST which means there have to be some functions to destroy contexts like struct rdma_cm_id, but without affecting the real connection. Similar to close() on an FD that is used shared between two processes.
- export/import an interprocess token attached to the hierarchy of objects similar to gss_export_sec_context/gss_import_sec_context.
Or a function which exports the contexts into a FD (of a pipe), which could also transfer open FDs to the other end.
Maybe this could pass the information through a rdma/ibverb event channel?

It would be ok if there's a limitation, which means that the transfer to a different process is only possible if there're:
- no posted work requests
- no queue pairs
- completion queues
active on the connection.

metze

Hefty, Sean

2012-10-17 19:01 UTC #9 lore raw

In reply to a message not in the archive

PiBGb3IgY2xpZW50IGNvbm5lY3Rpb25zIHRoYXQgd291bGQgd29yaywgYnV0IGl0IGNhbid0IHdv cmsgZm9yIGEgc2VydmVyDQo+IHRoYXQgdXNlcyBmb3JrKCkgYWZ0ZXIgbGlzdGVuKCkuDQo+IA0K PiBXb3VsZCBpdCBiZSBwb3NzaWJsZSB0byBleHRlbmQgdGhlIGxpYnJkbWFjbSBhbmQgbGliaWJ2 ZXJicyB0bzoNCj4gLSBoYW5kbGUgZm9yaygpIGFmdGVyIGdldHRpbmcgUkRNQV9DTV9FVkVOVF9D T05ORUNUX1JFUVVFU1QNCj4gICB3aGljaCBtZWFucyB0aGVyZSBoYXZlIHRvIGJlIHNvbWUgZnVu Y3Rpb25zIHRvIGRlc3Ryb3kgY29udGV4dHMgbGlrZQ0KPiAgIHN0cnVjdCByZG1hX2NtX2lkLCBi dXQgd2l0aG91dCBhZmZlY3RpbmcgdGhlIHJlYWwgY29ubmVjdGlvbi4NCj4gICBTaW1pbGFyIHRv IGNsb3NlKCkgb24gYW4gRkQgdGhhdCBpcyB1c2VkIHNoYXJlZCBiZXR3ZWVuIHR3byBwcm9jZXNz ZXMuDQo+IC0gZXhwb3J0L2ltcG9ydCBhbiBpbnRlcnByb2Nlc3MgdG9rZW4gYXR0YWNoZWQgdG8g dGhlIGhpZXJhcmNoeSBvZiBvYmplY3RzDQo+ICAgc2ltaWxhciB0byBnc3NfZXhwb3J0X3NlY19j b250ZXh0L2dzc19pbXBvcnRfc2VjX2NvbnRleHQuDQo+ICAgT3IgYSBmdW5jdGlvbiB3aGljaCBl eHBvcnRzIHRoZSBjb250ZXh0cyBpbnRvIGEgRkQgKG9mIGEgcGlwZSksDQo+ICAgd2hpY2ggY291 bGQgYWxzbyB0cmFuc2ZlciBvcGVuIEZEcyB0byB0aGUgb3RoZXIgZW5kLg0KPiAgIE1heWJlIHRo aXMgY291bGQgcGFzcyB0aGUgaW5mb3JtYXRpb24gdGhyb3VnaCBhIHJkbWEvaWJ2ZXJiIGV2ZW50 IGNoYW5uZWw/DQo+IA0KPiBBcmUgdGhlcmUgYW55IG90aGVyIHBvc3NpYmxlIHNvbHV0aW9ucz8N Cg0KSXQgd291bGQgYmUgZ3JlYXQgaWYgYW4gZW50aXJlIFJETUEgY29ubmVjdGlvbiBjb3VsZCBi ZSByZWZlcnJlZCB0byBieSBhbiBmZC4gIFRvZGF5IHRoYXQncyBub3QgdGhlIGNhc2UuICBEb2lu ZyB0aGlzIHJlcXVpcmVzIGFuIGFic3RyYWN0aW9uIGFib3ZlIHZlcmJzLiAgUnNvY2tldHMgcHJv dmlkZXMgYSBzaW1pbGFyIGFic3RyYWN0aW9uLCBidXQgaXQgZG9lc24ndCBhY3R1YWxseSByZXR1 cm4gYW4gZmQuDQoNClJzb2NrZXRzIGRvZXMgc3VwcG9ydCBmb3JrKCkgdG8gc29tZSBkZWdyZWUu ICBJdCBkb2VzIHRoaXMgYnkgZXN0YWJsaXNoaW5nIGEgVENQIGNvbm5lY3Rpb24uICBUaGUgUkRN QSBjb25uZWN0aW9uIGlzIG5vdCBhY3R1YWxseSBzZXR1cCB1bnRpbCB0aGUgZmlyc3QgZGF0YSB0 cmFuc2ZlciBvY2N1cnMuICBUaGlzIGhhbmRsZXMgdGhlIGNhc2Ugd2hlcmUgYSBzZXJ2ZXIgY2Fs bHMgZm9yaygpIGFmdGVyIGxpc3RlbigpLiAgSXQgd29ya3Mgd2l0aCB0aGUgYXBwcyB0aGF0IEkn dmUgdGVzdGVkLCBidXQgc3RpbGwgaW1wb3NlcyBzb21lIHJlc3RyaWN0aW9ucyBvbiBob3cgZm9y aygpIGlzIHVzZWQuDQoNCkkgaGl0IGludG8gb3RoZXIgaXNzdWVzIGJlc2lkZXMgZm9yaygpIHRy eWluZyB0byBzdXBwb3J0IGV4aXN0aW5nIHNvY2tldCBhcHBsaWNhdGlvbnMuICBXaXRob3V0IGhh dmluZyBhbiBmZCwgb3RoZXIgY2FsbHMgcmVxdWlyZSB3b3JraW5nIGFyb3VuZC4gIFRoaXMgaW5j bHVkZWQgZHVwMigpLCBzZW5kZmlsZSgpLCBhbmQgZnN0YXQoKSwgYnV0IEknbSBzdXJlIHRoZXJl IGFyZSBtYW55IG90aGVycy4NCg0KSXQgd291bGQgYmUgbmljZSBpZiB0aGVyZSB3ZXJlIGEgd2F5 IHRvIGFzc29jaWF0ZSBhbiBSRE1BIGNvbm5lY3Rpb24gd2l0aCBhbiBmZCBhbmQgdW5kZXIgaWRl YWwgY29uZGl0aW9ucyBhbGxvdyBkYXRhIHRyYW5zZmVycyB0byBvY2N1ciB3aXRoaW4gdGhlIHVz ZXJzcGFjZSBjb250ZXh0LiAgQnV0IGFsbG93IHRoZSBjb21tdW5pY2F0aW9uIHRvIG1pZ3JhdGUg dG8gdGhlIGtlcm5lbCBpZiBjZXJ0YWluIGNhbGxzIGFyZSBpbnZva2VkLiAgVGhpcyB3b3VsZCBw cm92aWRlIHZlcnkgZ29vZCBwZXJmb3JtYW5jZSB1bmRlciBtb3N0IGNvbmRpdGlvbnMsIHlldCBz dXBwb3J0IGxlZ2FjeSBhcHBzLg0KDQotIFNlYW4NCg==
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html