Skip to contentSkip to conversation

Desired RPC client behaviour on socket errors?

DISCUSSION Jamie Bainbridge 2 participants 4 messages May 1, 2015 – May 13, 2015 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.

Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.

After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.

However, this commit also causes those returns to call rpc_delay(task,3*HZ) and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.

In practice this means if a client tries to mount and there is a permanent network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.

It seems 3ed5e2a solves a problem for transient network errors but creates a problem for permanent network errors.

I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.

Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?

Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables clients in both security zones to use the same NFS mount configuration.

Cheers,
Jamie

#2 lore raw
Quoted 16 lines · On Fri, 1 May 2015 01:22:35 -0400 (EDT) Jamie Bainbridge <[email protected]>…

Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.

Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.

After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.

However, this commit also causes those returns to call rpc_delay(task,3*HZ) and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.

In practice this means if a client tries to mount and there is a permanent network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.

It seems 3ed5e2a solves a problem for transient network errors but creates a problem for permanent network errors.

I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.

Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?

Quoted 8 lines

Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables clients in both security zones to use the same NFS mount configuration.

Cheers,
Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'd say no...

One thing to consider is that it's quite common for servers (Linux or otherwise) to return those sorts of errors as they are coming up after a reboot. Network interfaces are often brought online before the nfs server is ready to accept connections.

If we were to change that then you'd likely see RPCs failing in those situations, which is almost certainly not what you want.

AFAICT, mount requests should end up trying to do a rpc_ping first, which should have RPC_TASK_SOFTCONN set. Is that not working for some reason?

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw

----- Original Message -----

Quoted 71 lines

From: "Jeff Layton" <[email protected]>
To: "Jamie Bainbridge" <[email protected]>
Cc: [email protected], [email protected]
Sent: Friday, 1 May, 2015 11:39:49 PM
Subject: Re: Desired RPC client behaviour on socket errors?

Quoted 40 lines · On Fri, 1 May 2015 01:22:35 -0400 (EDT) Jamie Bainbridge <[email protected]>…

Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.

Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.

After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.

However, this commit also causes those returns to call rpc_delay(task,3*HZ) and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.

In practice this means if a client tries to mount and there is a permanent network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.

It seems 3ed5e2a solves a problem for transient network errors but creates a problem for permanent network errors.

I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.

Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?

Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables clients in both security zones to use the same NFS mount configuration.

Quoted 6 lines

Cheers,
Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'd say no...

One thing to consider is that it's quite common for servers (Linux or otherwise) to return those sorts of errors as they are coming up after a reboot. Network interfaces are often brought online before the nfs server is ready to accept connections.

If we were to change that then you'd likely see RPCs failing in those situations, which is almost certainly not what you want.

AFAICT, mount requests should end up trying to do a rpc_ping first, which should have RPC_TASK_SOFTCONN set. Is that not working for some reason?

Quoted 7 lines

--
Jeff Layton <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Thanks for the confirmation of connect behaviour.

Debug shows a NULL procedure being generated, so it seems rpc_ping is called, but the RPC task ends up with a status of -EAGAIN and not -ECONNRESET, so it doesn't break out of the switch:case to call rpc_exit().

It seems socket status is not being passed back to become task status. I'll look into why that is.

Cheers,
Jamie

#4 lore raw

----- Original Message -----

Quoted 91 lines

From: "Jamie Bainbridge" <[email protected]>
To: "Jeff Layton" <[email protected]>
Cc: [email protected], [email protected]
Sent: Monday, 4 May, 2015 4:23:06 PM
Subject: Re: Desired RPC client behaviour on socket errors?

----- Original Message -----

Quoted 76 lines

From: "Jeff Layton" <[email protected]>
To: "Jamie Bainbridge" <[email protected]>
Cc: [email protected], [email protected]
Sent: Friday, 1 May, 2015 11:39:49 PM
Subject: Re: Desired RPC client behaviour on socket errors?

Quoted 45 lines · On Fri, 1 May 2015 01:22:35 -0400 (EDT) Jamie Bainbridge <[email protected]>…

Commit 3ed5e2a introduced a change to the RPC client's handling of socket return on connect.

Prior to this commit, any error return was considered instantly fatal and rpc_exit(task,-EIO) was called.

After this commit, socket returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH are passed back to the caller. This is a good idea and works well.

However, this commit also causes those returns to call rpc_delay(task,3*HZ)
and the RPC connect to retry until the RPC times out. The timeout can be modified with soft/timeo/retrans but defaults to 3 minutes.

In practice this means if a client tries to mount and there is a permanent
network error outside the client, a TCP Reset or an ICMP error might get returned, bu the mount will hang and the client will keep trying to connect many times until the RPC times out. Previously a mount would fail almost straight away.

It seems 3ed5e2a solves a problem for transient network errors but creates
a problem for permanent network errors.

I agree it's probably desirable for a client application (RPC in this instance) to keep trying to connect until a timeout, and it's good the timeout is configurable, but it's bad that the timeout must be tied to all
RPC operations. Someone wanting a quick mount timeout must also suffer a quick NFS operation timeout, not to mention the data corruption risk that goes along with soft.

Should the RPC client call rpc_exit() on an xprt connect which returns ECONNREFUSED ECONNRESET ECONNABORTED ENETUNREACH EHOSTUNREACH because those returns imply a "more permanent" network issue?

Disclosure: We came across this because a customer is (ab)using NFSv4 Migrations in a strange way. One server in fs_locations is firewalled behind a TCP Reset and one is not. Depending on which security zone a client is in, it can connect to one server but not the other. This enables
clients in both security zones to use the same NFS mount configuration.

Quoted 6 lines

Cheers,
Jamie
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I'd say no...

One thing to consider is that it's quite common for servers (Linux or otherwise) to return those sorts of errors as they are coming up after a reboot. Network interfaces are often brought online before the nfs server is ready to accept connections.

If we were to change that then you'd likely see RPCs failing in those situations, which is almost certainly not what you want.

AFAICT, mount requests should end up trying to do a rpc_ping first, which should have RPC_TASK_SOFTCONN set. Is that not working for some reason?

Quoted 7 lines

--
Jeff Layton <[email protected]>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 8 lines

Thanks for the confirmation of connect behaviour.

Debug shows a NULL procedure being generated, so it seems rpc_ping is called, but the RPC task ends up with a status of -EAGAIN and not -ECONNRESET, so it doesn't break out of the switch:case to call rpc_exit().

It seems socket status is not being passed back to become task status. I'll look into why that is.

Turns out this is working fine in 4.0.1 but not in EL6.

The EL6 kernel is currently missing xs_error_report() which sets sock->sk_err to task->status. Without that function, the default state of -EAGAIN is set in xprt_disconnect_done().

I'll handle this further via Red Hat Bugzilla.

Thanks for the help Jeff!

Cheers,
Jamie