Adjustable timeout
Subsystems (2)
Hello,
I'm using network storage for voicemail recording, which works pretty well on responsive servers. However in case of server crash on malfunction even open syscall takes ages to return error, which is just unfortunate.
Situation was described in thread "Timeout waaay too long"
http://lists.samba.org/archive/linux-cifs-client/2006-February/001203.html
and now, afer more than four years, it is not any better.
My very problem could be probably solved in userspace with "guard" thread killing stuck open or write syscall and moving to next storage available, but I found such a solution unly.
There is interesting notion in post "[PATCH] cifs: hard mount option behaviour" http://lists.samba.org/archive/linux-cifs-client/2010-June/006291.html
about what is considered a responsive server today.
For now I modified timeouts in SendReceive(2), which improved situation for me, but the real qustion is, how should widely acceptable solution look like.
Thanks for your suggestions,
ladis
A message that is not in the archive (sent off-list or not yet received): 20100713104810.GA3001-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Quoted 18 lines · On Tue, 13 Jul 2010 12:48:10 +0200 Ladislav Michl…
Hello,
I'm using network storage for voicemail recording, which works pretty well on responsive servers. However in case of server crash on malfunction even open syscall takes ages to return error, which is just unfortunate.
Situation was described in thread "Timeout waaay too long"
http://lists.samba.org/archive/linux-cifs-client/2006-February/001203.html
and now, afer more than four years, it is not any better.My very problem could be probably solved in userspace with "guard" thread killing stuck open or write syscall and moving to next storage available, but I found such a solution unly.
There is interesting notion in post "[PATCH] cifs: hard mount option behaviour" http://lists.samba.org/archive/linux-cifs-client/2010-June/006291.html
about what is considered a responsive server today.
Quoted 5 lines
For now I modified timeouts in SendReceive(2), which improved situation for me, but the real qustion is, how should widely acceptable solution look like.
Thanks for your suggestions,
ladis
Agreed that the situation still sucks and it's high time we start this discussion from first principles.
cifs always uses reliable transport (TCP primarily, but there has been some discussion of using SCTP in the past). With a reliable transport the kernel knows when the server has received the packet (it's ACK'ed).
Here's how I think it ought to work (at least, it's a starting point for discussion):
When the client sends a call to the server, the thread waits indefinitely for a reply. That wait is generally interruptible via signal, however.
If the socket changes state (possibly indicating that the server crashed before a reply could be sent to the call), then any calls in flight should be reissued once the socket is set back up.
Now, what to do in the situation where the client sends a call, the server crashes and no further calls are sent? The client might block indefinitely in that case since there is no more network traffic on the socket.
The sockets should do some sort of keepalive. A normal TCP keepalive might be ok, or we could consider doing SMB "pings" for this. That should make sure that we notice state changes in a timely fashion and should also help prevent disconnection on low-traffic sockets.
Thoughts?
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20100713082151.3d231b8b-4QP7MXygkU+dMjc06nkz3ljfA9RmPOcC@public.gmane.org
Quoted 34 lines · On Tue, 13 Jul 2010 08:21:51 -0400 Jeff Layton…
Quoted 18 lines · On Tue, 13 Jul 2010 12:48:10 +0200 Ladislav Michl…
Hello,
I'm using network storage for voicemail recording, which works pretty well on responsive servers. However in case of server crash on malfunction even open syscall takes ages to return error, which is just unfortunate.
Situation was described in thread "Timeout waaay too long"
http://lists.samba.org/archive/linux-cifs-client/2006-February/001203.html
and now, afer more than four years, it is not any better.My very problem could be probably solved in userspace with "guard" thread killing stuck open or write syscall and moving to next storage available, but I found such a solution unly.
There is interesting notion in post "[PATCH] cifs: hard mount option behaviour" http://lists.samba.org/archive/linux-cifs-client/2010-June/006291.html
about what is considered a responsive server today.Quoted 5 lines
For now I modified timeouts in SendReceive(2), which improved situation for me, but the real qustion is, how should widely acceptable solution look like.
Thanks for your suggestions,
ladisAgreed that the situation still sucks and it's high time we start this discussion from first principles.
cifs always uses reliable transport (TCP primarily, but there has been some discussion of using SCTP in the past). With a reliable transport the kernel knows when the server has received the packet (it's ACK'ed).
Quoted 6 lines
Here's how I think it ought to work (at least, it's a starting point for discussion):
When the client sends a call to the server, the thread waits indefinitely for a reply. That wait is generally interruptible via signal, however.
I should clarify too that wait only comes into play when the packet is ACK'ed. If it's not received by the server, then we'll need to determine what to do.
I think we need to be very careful about returning errors by default due to network partition. Apps generally don't expect it, so cifs ought to aggressively retry rather than returning errors to syscalls just because the server isn't responding.
Quoted 16 lines
If the socket changes state (possibly indicating that the server crashed before a reply could be sent to the call), then any calls in flight should be reissued once the socket is set back up.
Now, what to do in the situation where the client sends a call, the server crashes and no further calls are sent? The client might block indefinitely in that case since there is no more network traffic on the socket.
The sockets should do some sort of keepalive. A normal TCP keepalive might be ok, or we could consider doing SMB "pings" for this. That should make sure that we notice state changes in a timely fashion and should also help prevent disconnection on low-traffic sockets.
Thoughts?
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20100713082916.03767113-4QP7MXygkU+dMjc06nkz3ljfA9RmPOcC@public.gmane.org
Quoted 16 lines · Jeff Layton
Quoted 7 lines · On Tue, 13 Jul 2010 08:21:51 -0400 Jeff Layton…
Agreed that the situation still sucks and it's high time we start this discussion from first principles.
cifs always uses reliable transport (TCP primarily, but there has been some discussion of using SCTP in the past). With a reliable transport the kernel knows when the server has received the packet (it's ACK'ed).
Quoted 6 lines
Here's how I think it ought to work (at least, it's a starting point for discussion):
When the client sends a call to the server, the thread waits indefinitely for a reply. That wait is generally interruptible via signal, however.
Quoted 8 lines
I should clarify too that wait only comes into play when the packet is ACK'ed. If it's not received by the server, then we'll need to determine what to do.
I think we need to be very careful about returning errors by default due to network partition. Apps generally don't expect it, so cifs ought to aggressively retry rather than returning errors to syscalls just because the server isn't responding.
Here we probably want to distinguish 'soft' and 'hard' mount option. While 'hard' should do the best to get data written (and man page states syscall block until server comes back), which is perfectly okay, 'soft' is allowed to return error on server unresponsiveness. So, if above describes 'hard' mount case, I do agree. 'Soft' mount option currently poses a limit how long to wait and I'd like to see this timeout adjustable (something like timeo parameter is okay).
Or does your proposal generally mean to stop distinguishing between 'soft' and 'hard' mounts and make 'hard' default?
Quoted 9 lines
Quoted 9 lines
If the socket changes state (possibly indicating that the server crashed before a reply could be sent to the call), then any calls in flight should be reissued once the socket is set back up.
Now, what to do in the situation where the client sends a call, the server crashes and no further calls are sent? The client might block indefinitely in that case since there is no more network traffic on the socket.
Quoted 4 lines
Quoted 4 lines
The sockets should do some sort of keepalive. A normal TCP keepalive might be ok, or we could consider doing SMB "pings" for this. That should make sure that we notice state changes in a timely fashion and should also help prevent disconnection on low-traffic sockets.
I'd go for SMB echo, just because it proves server side is still alive more reliably. It would be also nice to have some way to let userspace know that server didn't reply to echo request for last n seconds.
A message that is not in the archive (sent off-list or not yet received): 20100713221750.GA2471-bi+AKbBUZKY6gyzm1THtWbp2dZbC/[email protected]
Quoted 34 lines · On Wed, 14 Jul 2010 00:17:50 +0200 Ladislav Michl…
Quoted 16 lines · Jeff Layton
Quoted 7 lines · On Tue, 13 Jul 2010 08:21:51 -0400 Jeff Layton…
Agreed that the situation still sucks and it's high time we start this discussion from first principles.
cifs always uses reliable transport (TCP primarily, but there has been some discussion of using SCTP in the past). With a reliable transport the kernel knows when the server has received the packet (it's ACK'ed).
Quoted 6 lines
Here's how I think it ought to work (at least, it's a starting point for discussion):
When the client sends a call to the server, the thread waits indefinitely for a reply. That wait is generally interruptible via signal, however.
Quoted 8 lines
I should clarify too that wait only comes into play when the packet is ACK'ed. If it's not received by the server, then we'll need to determine what to do.
I think we need to be very careful about returning errors by default due to network partition. Apps generally don't expect it, so cifs ought to aggressively retry rather than returning errors to syscalls just because the server isn't responding.
Here we probably want to distinguish 'soft' and 'hard' mount option. While 'hard' should do the best to get data written (and man page states syscall block until server comes back), which is perfectly okay, 'soft' is allowed to return error on server unresponsiveness. So, if above describes 'hard' mount case, I do agree. 'Soft' mount option currently poses a limit how long to wait and I'd like to see this timeout adjustable (something like timeo parameter is okay).
Quoted 3 lines
Or does your proposal generally mean to stop distinguishing between 'soft' and 'hard' mounts and make 'hard' default?
Well, it means that "hard" should be the default. I think that makes sense since that's better for data and application integrity in the face of network partitions. I'm not opposed to keeping some sort of "soft" semantics, but I don't think it ought to be the default.
Many applications don't deal well with getting EIO back on a file operation. The vast majority will crash in the face of it. I think "soft" semantics really make the most sense when you have an interactive application and in that situation, the user can generally signal to break out of a program that's stuck.
We have to take note however that SMB is not RPC. Much of the state of a SMB session is tied up with the socket. When the socket is reconnected, open files and (more importantly) file locks are lost and must be reclaimed.
There is also no concept of a grace period like with NFS/NLM so when this is done, other clients can race in and steal your file locks. Linux has no SIGLOST or anything so the application isn't aware when this occurs.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
If the socket changes state (possibly indicating that the server crashed before a reply could be sent to the call), then any calls in flight should be reissued once the socket is set back up.
Now, what to do in the situation where the client sends a call, the server crashes and no further calls are sent? The client might block indefinitely in that case since there is no more network traffic on the socket.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
The sockets should do some sort of keepalive. A normal TCP keepalive might be ok, or we could consider doing SMB "pings" for this. That should make sure that we notice state changes in a timely fashion and should also help prevent disconnection on low-traffic sockets.
I'd go for SMB echo, just because it proves server side is still alive more reliably. It would be also nice to have some way to let userspace know that server didn't reply to echo request for last n seconds.
Personally, I think TCP keepalive makes much more sense. It's easier to implement, and at least some older samba servers didn't return SMB echoes when they were stuck on blocking I/O. Again, we need to take great care not to give up and reconnect the socket unnecessarily as so much of the client's state is bound to it.
I don't think we should return error if the socket still seems to be working but the application bound to it is just not responding. A printk in this situation to alert the admin that the server is stuck is ok though.
Also, the syscall interface doesn't really provide a way to tell the application info about server responsiveness, but we could display it with a /proc or /sys file or something, or maybe display it as part of the "not responding" printk.
Signature · 1 line
Jeff Layton <[email protected]>
A message that is not in the archive (sent off-list or not yet received): 20100714081222.35b998ce-4QP7MXygkU+dMjc06nkz3ljfA9RmPOcC@public.gmane.org
Quoted 4 lines · Jeff Layton
Well, it means that "hard" should be the default. I think that makes sense since that's better for data and application integrity in the face of network partitions. I'm not opposed to keeping some sort of "soft" semantics, but I don't think it ought to be the default.
Ok then. NFS defaults to "hard" as well, but CIFS defaults to "soft". With regard to session state lose on reconnect, changing default makes sense.
Quoted 20 lines
Many applications don't deal well with getting EIO back on a file operation. The vast majority will crash in the face of it. I think "soft" semantics really make the most sense when you have an interactive application and in that situation, the user can generally signal to break out of a program that's stuck.
We have to take note however that SMB is not RPC. Much of the state of a SMB session is tied up with the socket. When the socket is reconnected, open files and (more importantly) file locks are lost and must be reclaimed.
There is also no concept of a grace period like with NFS/NLM so when this is done, other clients can race in and steal your file locks. Linux has no SIGLOST or anything so the application isn't aware when this occurs.
Quoted 3 lines
I'd go for SMB echo, just because it proves server side is still alive more reliably. It would be also nice to have some way to let userspace know that server didn't reply to echo request for last n seconds.
Quoted 5 lines
Personally, I think TCP keepalive makes much more sense. It's easier to implement, and at least some older samba servers didn't return SMB echoes when they were stuck on blocking I/O. Again, we need to take great care not to give up and reconnect the socket unnecessarily as so much of the client's state is bound to it.
Okay, you conviced me.
Quoted 9 lines
I don't think we should return error if the socket still seems to be working but the application bound to it is just not responding. A printk in this situation to alert the admin that the server is stuck is ok though.
Also, the syscall interface doesn't really provide a way to tell the application info about server responsiveness, but we could display it with a /proc or /sys file or something, or maybe display it as part of the "not responding" printk.
I'm not against _also_ displaying this information as a part of printk, but having it somewhere in /sys or /proc makes obtaining it more scripting friendly.