Skip to contentSkip to conversation

using rsockets via librspreload: poll() support?

DISCUSSION Andreas Bluemle 2 participants 4 messages Aug 8, 2013 InfiniBand

lore
Share & tools
lore

Andreas Bluemle

author
#0 lore raw

Hi Sean,

I am currently testing rsockets in connection with ceph. I am using LD_PRELOAD and the librspreload.so to force the application (ceph) to use rsockets instead of regular tcp/ip sockets.

All this works pretty well - until the point where an established connection is shut down: this seems to not work and never finishes (unless the application is killed...).

The way ceph uses sockets is in a nonblocking mode. When reading from a socket, it polls the socket first with an event mask of POLLIN and POLLRDHUP.

On the return from the poll() I see that
- POLLIN and POLLHUP are set in the returned events (POLLRDHUP is *not* set)
- errno is 95 (EOPNOTSUPP)

(The POLLHUP makes me believe that in this case the other end has shutdown the socket already.)

The EOPNOTSUPP confuses ceph quite a bit and prevents it from shutting down it's side of the socket connection properly.

Question: is it possible that the POLLRDHUP causes the EOPNOTSUPP to be set by librspreload::poll() or rpoll()?

Best Regards

Andreas Bluemle

Signature · 10 lines
Andreas Bluemle                     mailto:[email protected]
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#1 lore raw
Quoted 5 lines

I am currently testing rsockets in connection with ceph. I am using LD_PRELOAD and the librspreload.so to force the application (ceph) to use rsockets instead of regular tcp/ip sockets.

Quoted 7 lines

All this works pretty well - until the point where an established connection is shut down: this seems to not work and never finishes (unless the application is killed...).

The way ceph uses sockets is in a nonblocking mode. When reading from a socket, it polls the socket first with an event mask of POLLIN and POLLRDHUP.

rsockets does not handle POLLRDHUP. Support for that would need to be added.

Quoted 8 lines

On the return from the poll() I see that
- POLLIN and POLLHUP are set in the returned events (POLLRDHUP is *not* set)
- errno is 95 (EOPNOTSUPP)

(The POLLHUP makes me believe that in this case the other end has shutdown the socket already.)

Quoted 7 lines

The EOPNOTSUPP confuses ceph quite a bit and prevents it from shutting down it's side of the socket connection properly.

Question: is it possible that the POLLRDHUP causes the EOPNOTSUPP to be set by librspreload::poll() or rpoll()?

That's likely.

A call to poll() from the app eventually winds its way down to the real poll() routine. The 'fd' representing the rsocket ends up being converted to an fd that's basically an open file used to communicate with the RDMA stack in the kernel. Returning EOPNOTSUPP for POLLRDHUP on an open file looks reasonable.

If this is the case, POLLRDHUP needs to be removed from the real poll() call and handled within the rsocket rpoll() implementation.

Btw, if it's possible, copying the [email protected] mail list on this would be helpful in showing both the use of rsocket and where additional work is needed in a more public way.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#2 lore raw
In reply to Andreas Bluemle

Hi Sean,

I begin to believe that this may be a more general
problem: it seems to me that errno is not always initialized to 0 when the librspreload wrapper for a socket system call or the corresponding r*() routine from rsocket.c is called.

For the poll() I have cleared the errno explicitly before polling the socket - and it is still cleared on return from poll(). Hence: where I used to encounter an EOPNOTSUPP, I now see errno 0 (i.e. "Success").

Best Regards

Andreas Bluemle

Quoted 38 lines · On Thu, 8 Aug 2013 17:46:29 +0200 Andreas Bluemle…

Hi Sean,

I am currently testing rsockets in connection with ceph. I am using LD_PRELOAD and the librspreload.so to force the application (ceph) to use rsockets instead of regular tcp/ip sockets.

All this works pretty well - until the point where an established connection is shut down: this seems to not work and never finishes (unless the application is killed...).

The way ceph uses sockets is in a nonblocking mode. When reading from a socket, it polls the socket first with an event mask of POLLIN and POLLRDHUP.

On the return from the poll() I see that
- POLLIN and POLLHUP are set in the returned events (POLLRDHUP is *not* set)
- errno is 95 (EOPNOTSUPP)

(The POLLHUP makes me believe that in this case the other end has shutdown the socket already.)

The EOPNOTSUPP confuses ceph quite a bit and prevents it from shutting down it's side of the socket connection properly.

Question: is it possible that the POLLRDHUP causes the EOPNOTSUPP to be set by librspreload::poll() or rpoll()?

Best Regards

Andreas Bluemle

Signature · 10 lines
Andreas Bluemle                     mailto:[email protected]
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw

I don't believe that errno is supposed to be cleared by calls. It is only set when an error occurs. This way, errno does not get lost. Is poll() returning -1?

Quoted 61 lines

I begin to believe that this may be a more general
problem: it seems to me that errno is not always initialized to 0 when the librspreload wrapper for a socket system call or the corresponding r*() routine from rsocket.c is called.

For the poll() I have cleared the errno explicitly before polling the socket - and it is still cleared on return from poll(). Hence: where I used to encounter an EOPNOTSUPP, I now see errno 0 (i.e. "Success").

Best Regards

Andreas Bluemle

Quoted 27 lines · On Thu, 8 Aug 2013 17:46:29 +0200 Andreas Bluemle…

Hi Sean,

I am currently testing rsockets in connection with ceph. I am using LD_PRELOAD and the librspreload.so to force the application (ceph) to use rsockets instead of regular tcp/ip sockets.

All this works pretty well - until the point where an established connection is shut down: this seems to not work and never finishes (unless the application is killed...).

The way ceph uses sockets is in a nonblocking mode. When reading from a socket, it polls the socket first with an event mask of POLLIN and POLLRDHUP.

On the return from the poll() I see that
- POLLIN and POLLHUP are set in the returned events (POLLRDHUP is *not* set)
- errno is 95 (EOPNOTSUPP)

(The POLLHUP makes me believe that in this case the other end has shutdown the socket already.)

The EOPNOTSUPP confuses ceph quite a bit and prevents it from shutting down it's side of the socket connection properly.

Quoted 11 lines

Question: is it possible that the POLLRDHUP causes the EOPNOTSUPP to be set by librspreload::poll() or rpoll()?

Best Regards

Andreas Bluemle

Quoted 7 lines

--

Andreas Bluemle                     mailto:[email protected]
ITXperts GmbH                       http://www.itxperts.de
Balanstrasse 73, Geb. 08            Phone: (+49) 89 89044917
D-81541 Muenchen (Germany)          Fax:   (+49) 89 89044910

Company details: http://www.itxperts.de/imprint.htm

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html