nfsrdma broken on 2.6.34-rc1?

DISCUSSION Steve Wise 5 participants 16 messages Mar 29, 2010 – Apr 1, 2010 10 messages not in the archive

Subsystems (2)

NFS server InfiniBand

lore

Share & tools

lore

Steve Wise

author

2010-03-29 17:42 UTC #0 lore raw

Hey Sean,

I'm trying NFSRDMA on net-next and the server side fails when registering the rdma transport. I think its due to the INET6 support added to the rdma-cm. I'm still debugging though.

In fs/nfsd/nfsctl.c:__write_ports_addxprt(), it tries to create a new svc transport for PF_INET, and PF_INET6 using the same port and the wildcard address. If the INET6 fails with anything other than -EAFNOSUPPORT, then the entire transport registration fails (ie no RDMA/INET support is added).

When I do echo "rdma 20049" > /proc/fs/nfsd/portlist I see the PF_INET transport get created successfully, but the INET6 transport create fails with -EADDRNOTAVAIL.

Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X and PF_INET6, IN6ADDR_ANY_INIT, port=X ? Apparently the native stack allows this (which makes sense seeing as how they are different protocol families).

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BB0E670.3010507-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Steve Wise

author

2010-03-29 18:03 UTC #2 lore raw

Actually, this looks like a recent nfs change that now creates an INET and INET6 transport when you add one via the /proc/fs/nfsd/portlist file. Looking at 2.6.30, which works, the nfsctl.c code is very different and only creates an INET transport...

Quoted 4 lines · Steve Wise

Hey Sean,
I'm trying NFSRDMA on net-next and the server side fails when registering the rdma transport. I think its due to the INET6 support added to the rdma-cm. I'm still debugging though.

Quoted 13 lines

In fs/nfsd/nfsctl.c:__write_ports_addxprt(), it tries to create a new svc transport for PF_INET, and PF_INET6 using the same port and the wildcard address. If the INET6 fails with anything other than -EAFNOSUPPORT, then the entire transport registration fails (ie no
RDMA/INET support is added).
When I do echo "rdma 20049" > /proc/fs/nfsd/portlist I see the PF_INET transport get created successfully, but the INET6 transport create fails with -EADDRNOTAVAIL.
Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X and PF_INET6, IN6ADDR_ANY_INIT, port=X ? Apparently the native stack allows this (which makes sense seeing as how they are different protocol families).
Steve.
Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Sean Hefty

2010-03-29 18:43 UTC #3 lore raw

In reply to a message not in the archive

Quoted 2 lines

Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X and PF_INET6, IN6ADDR_ANY_INIT, port=X ?

No, since this shows up in the rdma_cm as using the same port space.

The rdma_cm might be able to support this if the port space were separated based on the address family, depending on how PS IB ends up.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): B18454F3200D4A43B57F175388845AAC-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org

Steve Wise

author

2010-03-29 18:51 UTC #5 lore raw

Quoted 3 lines · Sean Hefty

The rdma_cm might be able to support this if the port space were separated based on the address family, depending on how PS IB ends up.

I think separate port spaces is the correct solution.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Roland Dreier

2010-03-29 19:01 UTC #6 lore raw

In reply to a message not in the archive

Quoted 4 lines

Quoted 2 lines
The rdma_cm might be able to support this if the port space were separated based on the address family, depending on how PS IB ends up.
I think separate port spaces is the correct solution.

This gets a bit tricky -- for normal IP stuff, there's the "bindv6only" sysctl (and the IPV6_V6ONLY socket option). Without that, you can't bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6 socket will accept IPv4 connections via an v4->v6 mapped address. (You can look at inet_csk_bind_conflict() to see the full complexity of the checking done when binding an IPv4 socket)

I wonder what the right way from the RDMA CM to stay close to Linux sockets semantics without adding too much horror is. (Adding Jason to the CC list since he usually has an opinion about things like this :)

- R.

Signature · 6 lines

Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaiq8fvtos.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

Jason Gunthorpe

2010-03-29 19:37 UTC #8 lore raw

Quoted 11 lines · Roland Dreier

Quoted 4 lines
Quoted 2 lines
The rdma_cm might be able to support this if the port space were separated based on the address family, depending on how PS IB ends up.
I think separate port spaces is the correct solution.
This gets a bit tricky -- for normal IP stuff, there's the "bindv6only" sysctl (and the IPV6_V6ONLY socket option). Without that, you can't bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6 socket will accept IPv4 connections via an v4->v6 mapped address. (You can look at inet_csk_bind_conflict() to see the full complexity of the checking done when binding an IPv4 socket)

Yeah, exactly, it is very complex and there is a real need for things pretending to be IP to capture all this subtlety. The details can't just be skipped over, people will notice :(

Though, I'm also not entirely certain that NFS-RDMA is right to bind to both AFs, generally speaking on Linux for a multi-protocol app you only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?

Quoted 3 lines

I wonder what the right way from the RDMA CM to stay close to Linux sockets semantics without adding too much horror is. (Adding Jason to the CC list since he usually has an opinion about things like this :)

Clearly the best way is to figure out some way to work with the existing routines in the kernel. This stuff is complex and duplicating all of it in rdma_cm would be annoying..

To match the semantics each CM ID would still register to one SID but an incoming connection request on a v4 PF SID could be matched to a v6 SID, etc.

I don't think new port spaces in the API are desirable.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20100329193715.GA28070-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

Steve Wise

author

2010-03-29 19:51 UTC #10 lore raw

Quoted 15 lines · Jason Gunthorpe

Quoted 12 lines · Roland Dreier
Quoted 4 lines
Quoted 2 lines
The rdma_cm might be able to support this if the port space were separated based on the address family, depending on how PS IB ends up.
I think separate port spaces is the correct solution.
This gets a bit tricky -- for normal IP stuff, there's the "bindv6only" sysctl (and the IPV6_V6ONLY socket option). Without that, you can't bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6 socket will accept IPv4 connections via an v4->v6 mapped address. (You can look at inet_csk_bind_conflict() to see the full complexity of the checking done when binding an IPv4 socket)

Quoted 9 lines

Yeah, exactly, it is very complex and there is a real need for things pretending to be IP to capture all this subtlety. The details can't just be skipped over, people will notice :(
Though, I'm also not entirely certain that NFS-RDMA is right to bind to both AFs, generally speaking on Linux for a multi-protocol app you only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?

This issue is really not in the NFS-RDMA code. the nfsd code is doing the binding. See commit:

37498292aa97658a5d0a9bb84699ce8c1016bb74

Author: Chuck Lever <[email protected]>
Date:   Tue Jan 26 14:04:22 2010 -0500

NFSD: Create PF_INET6 listener in write_ports

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BB104D2.9080309-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Jason Gunthorpe

2010-03-29 19:55 UTC #12 lore raw

Quoted 8 lines · Steve Wise

Quoted 7 lines
Yeah, exactly, it is very complex and there is a real need for things pretending to be IP to capture all this subtlety. The details can't just be skipped over, people will notice :(
Though, I'm also not entirely certain that NFS-RDMA is right to bind to both AFs, generally speaking on Linux for a multi-protocol app you only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?

Quoted 8 lines

This issue is really not in the NFS-RDMA code. the nfsd code is doing the binding. See commit:
37498292aa97658a5d0a9bb84699ce8c1016bb74
Author: Chuck Lever <[email protected]>
Date:   Tue Jan 26 14:04:22 2010 -0500
NFSD: Create PF_INET6 listener in write_ports

Sure.. but it relies on the behavior of svcsock.c which does this:

        if (family == PF_INET6)
                kernel_setsockopt(sock, SOL_IPV6, IPV6_V6ONLY,
                                        (char *)&val, sizeof(val));

And the NFS-RDMA has no equivalent. Having the common code explicitly rely on IPV6_V6ONLY is quite troublesome when you can't implement it :)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Roland Dreier

2010-03-29 20:26 UTC #13 lore raw

In reply to a message not in the archive

Quoted 2 lines

This issue is really not in the NFS-RDMA code. the nfsd code is doing the binding. See commit:

I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY flag on PF_INET6 RPC listener sockets") and followups. NFS expects to have one IPv6-only socket and one IPv4-only socket.

It seems RDMA CM should create a similar V6ONLY option for binding (and probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to handle this.

Signature · 6 lines

Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaaatqx4bo.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

Tom Tucker

2010-03-29 20:32 UTC #15 lore raw

Quoted 8 lines · Roland Dreier

Quoted 2 lines
This issue is really not in the NFS-RDMA code. the nfsd code is doing the binding. See commit:
I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY flag on PF_INET6 RPC listener sockets") and followups. NFS expects to have one IPv6-only socket and one IPv4-only socket.

Yes, this new behavior is common to both sockets and rdma transports.

Quoted 4 lines

It seems RDMA CM should create a similar V6ONLY option for binding (and probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to handle this.

This makes sense to me.

Tom

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Steve Wise

author

2010-04-01 17:46 UTC #16 lore raw

In reply to a message not in the archive

Quoted 11 lines · Roland Dreier

Quoted 2 lines
This issue is really not in the NFS-RDMA code. the nfsd code is doing the binding. See commit:
I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY flag on PF_INET6 RPC listener sockets") and followups. NFS expects to have one IPv6-only socket and one IPv4-only socket.
It seems RDMA CM should create a similar V6ONLY option for binding (and probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to handle this.

Sean, will you add this to the rdma_cm?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BB4DBF4.3040506-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Sean Hefty

2010-04-01 18:03 UTC #18 lore raw

Quoted 1 line

Sean, will you add this to the rdma_cm?

Not immediately because I lack the time to do it.

It would be really nice to share the kernel's port space code and remove the port code in the rdma_cm.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4A8AA06C165A47A69C73CDA744EBDD48-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org

Tom Tucker

2010-04-01 18:11 UTC #20 lore raw

Quoted 9 lines · Sean Hefty

Quoted 2 lines
Sean, will you add this to the rdma_cm?
Not immediately because I lack the time to do it.
It would be really nice to share the kernel's port space code and remove the port code in the rdma_cm.

LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?

Quoted 5 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BB4E1C5.3030905-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Roland Dreier

2010-04-01 18:17 UTC #22 lore raw

Quoted 1 line

LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?

umm....

seriously is there anyway to un-screw this for 2.6.34?

hack around it in the NFS-RDMA server so it just skips the second bind to the same port?

- R.

Signature · 6 lines

Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adatyrvm40m.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

Steve Wise

author

2010-04-01 18:32 UTC #24 lore raw

Quoted 10 lines · Roland Dreier

Quoted 1 line
LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?
umm....
seriously is there anyway to un-screw this for 2.6.34?
hack around it in the NFS-RDMA server so it just skips the second bind to the same port?

If the nfsd code would ignore the error on the INET6 xprt create, then it works:

fs/nfsd/nfsctl.c +2 -0

index 0f0e77f..d6abab9 100644
@@ -1008,8 +1008,10 @@ static ssize_t __write_ports_addxprt(char *buf)
 
       err = svc_create_xprt(nfsd_serv, transport,
                               PF_INET6, port, SVC_SOCK_ANONYMOUS);
+#if 0
       if (err < 0 && err != -EAFNOSUPPORT)
               goto out_close;
+#endif
       return 0;

out_close:

       xprt = svc_find_xprt(nfsd_serv, transport, PF_INET, port);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Tom Tucker

2010-04-01 18:53 UTC #25 lore raw

In reply to a message not in the archive

Quoted 10 lines · Roland Dreier

Quoted 1 line
LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?
umm....
seriously is there anyway to un-screw this for 2.6.34?
hack around it in the NFS-RDMA server so it just skips the second bind to the same port?

Yes, I think all we have to do is have the nfsctl.c code ignore any transport error on the IPV6 call as opposed to _only_ NOTSUP

Quoted 2 lines

- R.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html