Skip to contentSkip to conversation

dlm: bound TCP failure detection with TCP_USER_TIMEOUT

PATCH In review Mark Syms 2 participants 2 messages Aug 24 – Oct 1 Distributed lock manager (DLM)

lore
Share & tools
lore

Mark Syms

author
#0 lore raw

PATCH 1/1 dlm: bound TCP failure detection with TCP_USER_TIMEOUT

DLM sets no timeout of any kind on its TCP sockets. When a connection blackholes - the peer is still up, but the flow is silently broken - DLM only finds out once TCP itself gives up after tcp_retries2 retransmissions, which is around 15 minutes with the default value of 15.

Until then the reliable transmission layer cannot make progress:
dlm_midcomms_unack_msg_resend() is only ever called from the socket error/close path, so the unacknowledged message stays queued and every node waiting on a lock held by the stuck node waits with it. This was observed as a cluster-wide GFS2 stall of just under 16 minutes, with all five nodes blocked in gfs2_glock_wait() on a single inode glock whose EX holder could not be demoted because one DLM message to one peer was stranded on a dead socket. Once TCP finally reported the error the socket was reconnected, the message retransmitted, and the cluster recovered in about two seconds.

The only lever available today is the system wide net.ipv4.tcp_retries2 sysctl, which also governs iSCSI, NFS and every other TCP user on the host.

Set TCP_USER_TIMEOUT on DLM sockets instead, so that DLM bounds its own failure detection independently of the global TCP tuning, and expose it as a new cluster configfs attribute tcp_user_timeout_secs alongside mark. Zero keeps the previous behaviour of relying on tcp_retries2.

On expiry the socket is aborted with ETIMEDOUT, which drives the existing lowcomms_error_report() -> dlm_midcomms_unack_msg_resend() path, i.e. exactly the recovery that already unwound the stall, just promptly. Because the option is applied before connect() it also caps SYN retransmission, which is harmless here as dlm_connect() retries indefinitely.

The value is read when a socket is created, so a change made through configfs takes effect for subsequent connections.

Accepted sockets now get the protocol socket options set explicitly rather than inheriting them from the listening socket, so that inbound connections also pick up the current configuration.

SCTP is not affected; TCP_USER_TIMEOUT has no SCTP equivalent.

---
 fs/dlm/config.c   | 16 ++++++++++++++++
 fs/dlm/config.h   |  1 +
 fs/dlm/lowcomms.c | 20 ++++++++++++++++++++
 3 files changed, 37 insertions(+)
fs/dlm/config.c +16 -0
index 53cd33293042..dd4d741c8797 100644
@@ -98,6 +98,7 @@ enum {
 	CLUSTER_ATTR_NEW_RSB_COUNT,
 	CLUSTER_ATTR_RECOVER_CALLBACKS,
 	CLUSTER_ATTR_CLUSTER_NAME,
+	CLUSTER_ATTR_TCP_USER_TIMEOUT_SECS,
 };
 
 static ssize_t cluster_cluster_name_show(struct config_item *item, char *buf)
@@ -220,6 +221,17 @@ static int dlm_check_zero(unsigned int x)
 	return 0;
 }
 
+static int dlm_check_tcp_user_timeout_secs(unsigned int x)
+{
+	/* zero disables the timeout, anything else must survive the
+	 * conversion to the milliseconds tcp_sock_set_user_timeout() takes
+	 */
+	if (x > INT_MAX / MSEC_PER_SEC)
+		return -EINVAL;
+
+	return 0;
+}
+
 static int dlm_check_buffer_size(unsigned int x)
 {
 	if (x < DLM_MAX_SOCKET_BUFSIZE)
@@ -237,6 +249,7 @@ CLUSTER_ATTR(log_debug, NULL);
 CLUSTER_ATTR(log_info, NULL);
 CLUSTER_ATTR(protocol, dlm_check_protocol_and_dlm_running);
 CLUSTER_ATTR(mark, NULL);
+CLUSTER_ATTR(tcp_user_timeout_secs, dlm_check_tcp_user_timeout_secs);
 CLUSTER_ATTR(new_rsb_count, NULL);
 CLUSTER_ATTR(recover_callbacks, NULL);
 
@@ -254,6 +267,7 @@ static struct configfs_attribute *cluster_attrs[] = {
 	[CLUSTER_ATTR_NEW_RSB_COUNT] = &cluster_attr_new_rsb_count,
 	[CLUSTER_ATTR_RECOVER_CALLBACKS] = &cluster_attr_recover_callbacks,
 	[CLUSTER_ATTR_CLUSTER_NAME] = &cluster_attr_cluster_name,
+	[CLUSTER_ATTR_TCP_USER_TIMEOUT_SECS] = &cluster_attr_tcp_user_timeout_secs,
 	NULL,
 };
 
@@ -1031,6 +1045,7 @@ int dlm_our_addr(struct sockaddr_storage *addr, int num)
 #define DEFAULT_LOG_INFO           1
 #define DEFAULT_PROTOCOL           DLM_PROTO_TCP
 #define DEFAULT_MARK               0
+#define DEFAULT_TCP_USER_TIMEOUT_SECS 0
 #define DEFAULT_NEW_RSB_COUNT    128
 #define DEFAULT_RECOVER_CALLBACKS  0
 #define DEFAULT_CLUSTER_NAME      ""
@@ -1046,6 +1061,7 @@ struct dlm_config_info dlm_config = {
 	.ci_log_info = DEFAULT_LOG_INFO,
 	.ci_protocol = DEFAULT_PROTOCOL,
 	.ci_mark = DEFAULT_MARK,
+	.ci_tcp_user_timeout_secs = DEFAULT_TCP_USER_TIMEOUT_SECS,
 	.ci_new_rsb_count = DEFAULT_NEW_RSB_COUNT,
 	.ci_recover_callbacks = DEFAULT_RECOVER_CALLBACKS,
 	.ci_cluster_name = DEFAULT_CLUSTER_NAME
fs/dlm/config.h +1 -0
index 4ebd45f75276..3843d3bf0d0e 100644
@@ -41,6 +41,7 @@ struct dlm_config_info {
 	unsigned int ci_log_info;
 	unsigned int ci_protocol;
 	unsigned int ci_mark;
+	unsigned int ci_tcp_user_timeout_secs;
 	unsigned int ci_new_rsb_count;
 	unsigned int ci_recover_callbacks;
 	char ci_cluster_name[DLM_LOCKSPACE_LEN];
fs/dlm/lowcomms.c +20 -0
index 2aff1c7c17de..571a4d4094ec 100644
@@ -1049,6 +1049,12 @@ static int accept_from_sock(void)
 
 	sock_set_mark(newsock->sk, mark);
 
+	/* set the socket options explicitly instead of relying on them being
+	 * inherited from the listening socket, so that an accepted socket
+	 * picks up the current configuration
+	 */
+	dlm_proto_ops->sockopts(newsock);
+
 	down_write(&newcon->sock_lock);
 	if (newcon->sock) {
 		struct connection *othercon = newcon->othercon;
@@ -1839,8 +1845,22 @@ static int dlm_tcp_listen_validate(void)
 
 static void dlm_tcp_sockopts(struct socket *sock)
 {
+	unsigned int timeout_secs;
+
 	/* Turn off Nagle's algorithm */
 	tcp_sock_set_nodelay(sock->sk);
+
+	/* Bound the time a message may stay unacknowledged on a silently
+	 * broken flow. Without this DLM only finds out that a connection is
+	 * dead once TCP gives up after tcp_retries2, which is roughly 15
+	 * minutes with the default value of 15, and every lockspace user
+	 * waiting on a lock held by that node stalls for as long. On expiry
+	 * the socket is aborted, the error report path resends the
+	 * unacknowledged messages and lowcomms reconnects.
+	 */
+	timeout_secs = READ_ONCE(dlm_config.ci_tcp_user_timeout_secs);
+	if (timeout_secs)
+		tcp_sock_set_user_timeout(sock->sk, timeout_secs * MSEC_PER_SEC);
 }
 
 static void dlm_tcp_listen_sockopts(struct socket *sock)
Signature · 1 line
2.55.0
maintainer
#1 lore raw

Hi,

sorry for the late reply,

Quoted 7 lines · Mark Syms

DLM sets no timeout of any kind on its TCP sockets. When a connection blackholes - the peer is still up, but the flow is silently broken - DLM only finds out once TCP itself gives up after tcp_retries2 retransmissions, which is around 15 minutes with the default value of 15.

Quoted 11 lines

Until then the reliable transmission layer cannot make progress:
dlm_midcomms_unack_msg_resend() is only ever called from the socket error/close path, so the unacknowledged message stays queued and every node waiting on a lock held by the stuck node waits with it. This was observed as a cluster-wide GFS2 stall of just under 16 minutes, with all five nodes blocked in gfs2_glock_wait() on a single inode glock whose EX holder could not be demoted because one DLM message to one peer was stranded on a dead socket. Once TCP finally reported the error the socket was reconnected, the message retransmitted, and the cluster recovered in about two seconds.

I never experienced this behaviour. What you describe suggests the master node would send a GRANT message to nodes in contention if it can be granted.
As soon as this message cannot be delivered for whatever reason and sendmsg() fails it should close the socket and retransmit.

However, if this parameter makes your life easier and resolves an issue, I believe upstreaming this tweak should do no harm and will improve the situations you describe above.

Quoted 111 lines

The only lever available today is the system wide net.ipv4.tcp_retries2 sysctl, which also governs iSCSI, NFS and every other TCP user on the host.

Set TCP_USER_TIMEOUT on DLM sockets instead, so that DLM bounds its own failure detection independently of the global TCP tuning, and expose it as a new cluster configfs attribute tcp_user_timeout_secs alongside mark. Zero keeps the previous behaviour of relying on tcp_retries2.

On expiry the socket is aborted with ETIMEDOUT, which drives the existing lowcomms_error_report() -> dlm_midcomms_unack_msg_resend() path, i.e. exactly the recovery that already unwound the stall, just promptly. Because the option is applied before connect() it also caps SYN retransmission, which is harmless here as dlm_connect() retries indefinitely.

The value is read when a socket is created, so a change made through configfs takes effect for subsequent connections.

Accepted sockets now get the protocol socket options set explicitly rather than inheriting them from the listening socket, so that inbound connections also pick up the current configuration.

SCTP is not affected; TCP_USER_TIMEOUT has no SCTP equivalent.

---
 fs/dlm/config.c   | 16 ++++++++++++++++
 fs/dlm/config.h   |  1 +
 fs/dlm/lowcomms.c | 20 ++++++++++++++++++++
 3 files changed, 37 insertions(+)
fs/dlm/config.c +16 -0
index 53cd33293042..dd4d741c8797 100644
@@ -98,6 +98,7 @@ enum {
        CLUSTER_ATTR_NEW_RSB_COUNT,
        CLUSTER_ATTR_RECOVER_CALLBACKS,
        CLUSTER_ATTR_CLUSTER_NAME,
+       CLUSTER_ATTR_TCP_USER_TIMEOUT_SECS,
 };
 
 static ssize_t cluster_cluster_name_show(struct config_item *item, char *buf)
@@ -220,6 +221,17 @@ static int dlm_check_zero(unsigned int x)
        return 0;
 }
 
+static int dlm_check_tcp_user_timeout_secs(unsigned int x)
+{
+       /* zero disables the timeout, anything else must survive the
+        * conversion to the milliseconds tcp_sock_set_user_timeout() takes
+        */
+       if (x > INT_MAX / MSEC_PER_SEC)
+               return -EINVAL;
+
+       return 0;
+}
+
 static int dlm_check_buffer_size(unsigned int x)
 {
        if (x < DLM_MAX_SOCKET_BUFSIZE)
@@ -237,6 +249,7 @@ CLUSTER_ATTR(log_debug, NULL);
 CLUSTER_ATTR(log_info, NULL);
 CLUSTER_ATTR(protocol, dlm_check_protocol_and_dlm_running);
 CLUSTER_ATTR(mark, NULL);
+CLUSTER_ATTR(tcp_user_timeout_secs, dlm_check_tcp_user_timeout_secs);
 CLUSTER_ATTR(new_rsb_count, NULL);
 CLUSTER_ATTR(recover_callbacks, NULL);
 
@@ -254,6 +267,7 @@ static struct configfs_attribute *cluster_attrs[] = {
        [CLUSTER_ATTR_NEW_RSB_COUNT] = &cluster_attr_new_rsb_count,
        [CLUSTER_ATTR_RECOVER_CALLBACKS] = &cluster_attr_recover_callbacks,
        [CLUSTER_ATTR_CLUSTER_NAME] = &cluster_attr_cluster_name,
+       [CLUSTER_ATTR_TCP_USER_TIMEOUT_SECS] = &cluster_attr_tcp_user_timeout_secs,
        NULL,
 };
 
@@ -1031,6 +1045,7 @@ int dlm_our_addr(struct sockaddr_storage *addr, int num)
 #define DEFAULT_LOG_INFO           1
 #define DEFAULT_PROTOCOL           DLM_PROTO_TCP
 #define DEFAULT_MARK               0
+#define DEFAULT_TCP_USER_TIMEOUT_SECS 0
 #define DEFAULT_NEW_RSB_COUNT    128
 #define DEFAULT_RECOVER_CALLBACKS  0
 #define DEFAULT_CLUSTER_NAME      ""
@@ -1046,6 +1061,7 @@ struct dlm_config_info dlm_config = {
        .ci_log_info = DEFAULT_LOG_INFO,
        .ci_protocol = DEFAULT_PROTOCOL,
        .ci_mark = DEFAULT_MARK,
+       .ci_tcp_user_timeout_secs = DEFAULT_TCP_USER_TIMEOUT_SECS,
        .ci_new_rsb_count = DEFAULT_NEW_RSB_COUNT,
        .ci_recover_callbacks = DEFAULT_RECOVER_CALLBACKS,
        .ci_cluster_name = DEFAULT_CLUSTER_NAME
fs/dlm/config.h +1 -0
index 4ebd45f75276..3843d3bf0d0e 100644
@@ -41,6 +41,7 @@ struct dlm_config_info {
        unsigned int ci_log_info;
        unsigned int ci_protocol;
        unsigned int ci_mark;
+       unsigned int ci_tcp_user_timeout_secs;
        unsigned int ci_new_rsb_count;
        unsigned int ci_recover_callbacks;
        char ci_cluster_name[DLM_LOCKSPACE_LEN];
fs/dlm/lowcomms.c +0 -0
index 2aff1c7c17de..571a4d4094ec 100644

fs/dlm/lowcomms.c static int accept_from_sock(void) · line 1057

Quoted 9 lines
fs/dlm/lowcomms.c +6 -0
@@ -1049,6 +1049,12 @@ static int accept_from_sock(void)
 
        sock_set_mark(newsock->sk, mark);
 
+       /* set the socket options explicitly instead of relying on them being
+        * inherited from the listening socket, so that an accepted socket
+        * picks up the current configuration
+        */
+       dlm_proto_ops->sockopts(newsock);
+

Yes you can do that, but someone else will remove it later as the socket implementation should already handle this; if it doesn't, the problem lies there.

Quoted 13 lines
fs/dlm/lowcomms.c +4 -0
        down_write(&newcon->sock_lock);
        if (newcon->sock) {
                struct connection *othercon = newcon->othercon;
@@ -1839,8 +1845,22 @@ static int dlm_tcp_listen_validate(void)
 
 static void dlm_tcp_sockopts(struct socket *sock)
 {
+       unsigned int timeout_secs;
+
        /* Turn off Nagle's algorithm */
        tcp_sock_set_nodelay(sock->sk);
+
+       /* Bound the time a message may stay unacknowledged on a silently

fs/dlm/lowcomms.c static int dlm_tcp_listen_validate(void) · line 1863

Quoted 10 lines
fs/dlm/lowcomms.c +10 -0
+        * broken flow. Without this DLM only finds out that a connection is
+        * dead once TCP gives up after tcp_retries2, which is roughly 15
+        * minutes with the default value of 15, and every lockspace user
+        * waiting on a lock held by that node stalls for as long. On expiry
+        * the socket is aborted, the error report path resends the
+        * unacknowledged messages and lowcomms reconnects.
+        */
+       timeout_secs = READ_ONCE(dlm_config.ci_tcp_user_timeout_secs);
+       if (timeout_secs)
+               tcp_sock_set_user_timeout(sock->sk, timeout_secs * MSEC_PER_SEC);

why making this value not the same dimension as
tcp_sock_set_user_timeout() handles it, msecs?

- Alex