Skip to contentSkip to conversation

fs: dlm: receive and mark fixes

PATCH New Alexander Aring 1 participant 6 messages Sep 24, 2020 Distributed lock manager (DLM)

lore
Share & tools
lore

Alexander Aring

author maintainer
#0 lore raw

v2 cover fs: dlm: receive and mark fixes

Hi,

This patch-series contains some improvements for the receive handling and fixes for really setting per nodeid mark settings. The cluster mark value becomes now the meaning of default mark value, a per nodeid mark value will overwrite this mark value only for this node traffic.

- Alex

changes since v2:

- drop patch of rework transmit side:

After I got some review regarding to performance issues with sendmsg()/sendpage() I will drop this patch. And yes [0] all branches of tcp sendmsg() will copy except the zerocopy branch [1]. However for now zerocopy seems to be very only developed for user space thing.

I changed to a "per dlm message" behaviour to have it easier to implement a possible "retransmit" handling to provide that dlm can survive a spam of tcp reset messages (tcpkill). However I will schedule a different solution when I implement that feature later. This solution will hopefully combine to send large page buffers and having the possibility to retransmit specific dlm messages of it. Unlikely that a retransmit ends in a large page buffer but they _should_ be happen very rarely.

- drop patch of release connection with call_rcu:

This patch makes problems regarding to cleaning up the writequeue, according to [2] the call_rcu() callback can be invoked by softirq context. We using spin_lock() primitives there without _bh(). I came to conclusion it's fine as it for now without the transmit handling changes.

- add patch to fix/change skb mark handling:

Yea, I got finally aware why some mark values wasn't working as expected. The cluster mark value will now be used always. If a per nodeid mark is specified it will be used instead of the default one. I will send another patch for dlm_controld man page to update the meaning.

- fix in receive rework patch a con vs othercon regarding to rx_buflen

[0] https://elixir.bootlin.com/linux/v5.9-rc6/source/net/ipv4/tcp.c#L1315
[1] https://elixir.bootlin.com/linux/v5.9-rc6/source/net/ipv4/tcp.c#L1359
[2] https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt

Alexander Aring (5):
fs: dlm: remove lock dependency warning
fs: dlm: fix mark per nodeid setting
fs: dlm: handle range check as callback
fs: dlm: disallow buffer size below default
fs: dlm: rework receive handling

 fs/dlm/config.c   |  63 +++++++++------
 fs/dlm/config.h   |   4 +-
 fs/dlm/lowcomms.c | 197 ++++++++++++++++++++++------------------------
 fs/dlm/midcomms.c | 136 +++++++++++++-------------------
 fs/dlm/midcomms.h |   3 +-
 5 files changed, 188 insertions(+), 215 deletions(-)
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH v2 1/5 fs: dlm: remove lock dependency warning

During my experiments to make dlm robust against tcpkill application I was able to run sometimes in a circular lock dependency warning between clusters_root.subsys.su_mutex and con->sock_mutex. We don't need to held the sock_mutex when getting the mark value which held the clusters_root.subsys.su_mutex. This patch moves the specific handling just before the sock_mutex will be held.

---
 fs/dlm/lowcomms.c | 18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
fs/dlm/lowcomms.c +8 -10
index 1bf1808bfa6b0..24f5e55313d83 100644
@@ -971,6 +971,10 @@ static void sctp_connect_to_sock(struct connection *con)
 		return;
 	}
 
+	result = dlm_comm_mark(con->nodeid, &mark);
+	if (result < 0)
+		return;
+
 	mutex_lock(&con->sock_mutex);
 
 	/* Some odd races can cause double-connects, ignore them */
@@ -995,11 +999,6 @@ static void sctp_connect_to_sock(struct connection *con)
 	if (result < 0)
 		goto socket_err;
 
-	/* set skb mark */
-	result = dlm_comm_mark(con->nodeid, &mark);
-	if (result < 0)
-		goto bind_err;
-
 	sock_set_mark(sock->sk, mark);
 
 	con->rx_action = receive_from_sock;
@@ -1072,6 +1071,10 @@ static void tcp_connect_to_sock(struct connection *con)
 		return;
 	}
 
+	result = dlm_comm_mark(con->nodeid, &mark);
+	if (result < 0)
+		return;
+
 	mutex_lock(&con->sock_mutex);
 	if (con->retries++ > MAX_CONNECT_RETRIES)
 		goto out;
@@ -1086,11 +1089,6 @@ static void tcp_connect_to_sock(struct connection *con)
 	if (result < 0)
 		goto out_err;
 
-	/* set skb mark */
-	result = dlm_comm_mark(con->nodeid, &mark);
-	if (result < 0)
-		goto out_err;
-
 	sock_set_mark(sock->sk, mark);
 
 	memset(&saddr, 0, sizeof(saddr));
Signature · 1 line
2.26.2
author maintainer
#2 lore raw

PATCH v2 2/5 fs: dlm: fix mark per nodeid setting

In reply to Alexander Aring

This patch fixes to set per nodeid mark configuration for accepted sockets as well. Before this patch only the listen socket mark value was used for all accepted connections. This patch will ensure that the cluster mark attribute value will be always used for all sockets, if a per nodeid mark value is specified dlm will use this value for the specific node.

---
 fs/dlm/config.c   | 16 ++++++++++------
 fs/dlm/config.h   |  2 +-
 fs/dlm/lowcomms.c | 12 ++++++------
 3 files changed, 17 insertions(+), 13 deletions(-)
fs/dlm/config.c +10 -6
index f33a7e4ae917b..ca4a9795afbe2 100644
@@ -860,18 +860,22 @@ int dlm_comm_seq(int nodeid, uint32_t *seq)
 	return 0;
 }
 
-int dlm_comm_mark(int nodeid, unsigned int *mark)
+void dlm_comm_mark(int nodeid, unsigned int *mark)
 {
 	struct dlm_comm *cm;
 
 	cm = get_comm(nodeid);
-	if (!cm)
-		return -ENOENT;
+	if (!cm) {
+		*mark = dlm_config.ci_mark;
+		return;
+	}
 
-	*mark = cm->mark;
-	put_comm(cm);
+	if (cm->mark)
+		*mark = cm->mark;
+	else
+		*mark = dlm_config.ci_mark;
 
-	return 0;
+	put_comm(cm);
 }
 
 int dlm_our_nodeid(void)
fs/dlm/config.h +1 -1
index f62996cad5616..3b284ae9aeebd 100644
@@ -46,7 +46,7 @@ void dlm_config_exit(void);
 int dlm_config_nodes(char *lsname, struct dlm_config_node **nodes_out,
 		     int *count_out);
 int dlm_comm_seq(int nodeid, uint32_t *seq);
-int dlm_comm_mark(int nodeid, unsigned int *mark);
+void dlm_comm_mark(int nodeid, unsigned int *mark);
 int dlm_our_nodeid(void);
 int dlm_our_addr(struct sockaddr_storage *addr, int num);
 
fs/dlm/lowcomms.c +6 -6
index 24f5e55313d83..96f84541867c4 100644
@@ -790,6 +790,7 @@ static int accept_from_sock(struct connection *con)
 	int nodeid;
 	struct connection *newcon;
 	struct connection *addcon;
+	unsigned int mark;
 
 	if (!dlm_allow_conn) {
 		return -1;
@@ -826,6 +827,9 @@ static int accept_from_sock(struct connection *con)
 		return -1;
 	}
 
+	dlm_comm_mark(nodeid, &mark);
+	sock_set_mark(newsock->sk, mark);
+
 	log_print("got connection from %d", nodeid);
 
 	/*  Check to see if we already have a connection to this node. This
@@ -971,9 +975,7 @@ static void sctp_connect_to_sock(struct connection *con)
 		return;
 	}
 
-	result = dlm_comm_mark(con->nodeid, &mark);
-	if (result < 0)
-		return;
+	dlm_comm_mark(con->nodeid, &mark);
 
 	mutex_lock(&con->sock_mutex);
 
@@ -1071,9 +1073,7 @@ static void tcp_connect_to_sock(struct connection *con)
 		return;
 	}
 
-	result = dlm_comm_mark(con->nodeid, &mark);
-	if (result < 0)
-		return;
+	dlm_comm_mark(con->nodeid, &mark);
 
 	mutex_lock(&con->sock_mutex);
 	if (con->retries++ > MAX_CONNECT_RETRIES)
Signature · 1 line
2.26.2
author maintainer
#3 lore raw

PATCH v2 3/5 fs: dlm: handle range check as callback

In reply to Alexander Aring

This patch adds a callback to CLUSTER_ATTR macro to allow individual callbacks for attributes which might have a more complex attribute range checking just than non zero.

---
 fs/dlm/config.c | 41 +++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)
fs/dlm/config.c +23 -18
index ca4a9795afbe2..e03b409a4df01 100644
@@ -125,7 +125,7 @@ static ssize_t cluster_cluster_name_store(struct config_item *item,
 CONFIGFS_ATTR(cluster_, cluster_name);
 
 static ssize_t cluster_set(struct dlm_cluster *cl, unsigned int *cl_field,
-			   int *info_field, int check_zero,
+			   int *info_field, bool (*check_cb)(unsigned int x),
 			   const char *buf, size_t len)
 {
 	unsigned int x;
@@ -137,7 +137,7 @@ static ssize_t cluster_set(struct dlm_cluster *cl, unsigned int *cl_field,
 	if (rc)
 		return rc;
 
-	if (check_zero && !x)
+	if (check_cb && check_cb(x))
 		return -EINVAL;
 
 	*cl_field = x;
@@ -146,13 +146,13 @@ static ssize_t cluster_set(struct dlm_cluster *cl, unsigned int *cl_field,
 	return len;
 }
 
-#define CLUSTER_ATTR(name, check_zero)                                        \
+#define CLUSTER_ATTR(name, check_cb)                                          \
 static ssize_t cluster_##name##_store(struct config_item *item, \
 		const char *buf, size_t len) \
 {                                                                             \
 	struct dlm_cluster *cl = config_item_to_cluster(item);		      \
 	return cluster_set(cl, &cl->cl_##name, &dlm_config.ci_##name,         \
-			   check_zero, buf, len);                             \
+			   check_cb, buf, len);                               \
 }                                                                             \
 static ssize_t cluster_##name##_show(struct config_item *item, char *buf)     \
 {                                                                             \
@@ -161,20 +161,25 @@ static ssize_t cluster_##name##_show(struct config_item *item, char *buf)     \
 }                                                                             \
 CONFIGFS_ATTR(cluster_, name);
 
-CLUSTER_ATTR(tcp_port, 1);
-CLUSTER_ATTR(buffer_size, 1);
-CLUSTER_ATTR(rsbtbl_size, 1);
-CLUSTER_ATTR(recover_timer, 1);
-CLUSTER_ATTR(toss_secs, 1);
-CLUSTER_ATTR(scan_secs, 1);
-CLUSTER_ATTR(log_debug, 0);
-CLUSTER_ATTR(log_info, 0);
-CLUSTER_ATTR(protocol, 0);
-CLUSTER_ATTR(mark, 0);
-CLUSTER_ATTR(timewarn_cs, 1);
-CLUSTER_ATTR(waitwarn_us, 0);
-CLUSTER_ATTR(new_rsb_count, 0);
-CLUSTER_ATTR(recover_callbacks, 0);
+static bool dlm_check_zero(unsigned int x)
+{
+	return !x;
+}
+
+CLUSTER_ATTR(tcp_port, dlm_check_zero);
+CLUSTER_ATTR(buffer_size, dlm_check_zero);
+CLUSTER_ATTR(rsbtbl_size, dlm_check_zero);
+CLUSTER_ATTR(recover_timer, dlm_check_zero);
+CLUSTER_ATTR(toss_secs, dlm_check_zero);
+CLUSTER_ATTR(scan_secs, dlm_check_zero);
+CLUSTER_ATTR(log_debug, NULL);
+CLUSTER_ATTR(log_info, NULL);
+CLUSTER_ATTR(protocol, NULL);
+CLUSTER_ATTR(mark, NULL);
+CLUSTER_ATTR(timewarn_cs, dlm_check_zero);
+CLUSTER_ATTR(waitwarn_us, NULL);
+CLUSTER_ATTR(new_rsb_count, NULL);
+CLUSTER_ATTR(recover_callbacks, NULL);
 
 static struct configfs_attribute *cluster_attrs[] = {
 	[CLUSTER_ATTR_TCP_PORT] = &cluster_attr_tcp_port,
Signature · 1 line
2.26.2
author maintainer
#4 lore raw

PATCH v2 4/5 fs: dlm: disallow buffer size below default

In reply to Alexander Aring

I observed that the upper layer will not send messages above this value. As conclusion the application receive buffer should not below that value, otherwise we are not capable to deliver the dlm message to the upper layer. This patch forbids to set the receive buffer below the maximum possible dlm message size.

---
 fs/dlm/config.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
fs/dlm/config.c +7 -2
index e03b409a4df01..a4bed304a8430 100644
@@ -166,8 +166,14 @@ static bool dlm_check_zero(unsigned int x)
 	return !x;
 }
 
+#define DEFAULT_BUFFER_SIZE     4096
+static bool dlm_check_buffer_size(unsigned int x)
+{
+	return (x < DEFAULT_BUFFER_SIZE);
+}
+
 CLUSTER_ATTR(tcp_port, dlm_check_zero);
-CLUSTER_ATTR(buffer_size, dlm_check_zero);
+CLUSTER_ATTR(buffer_size, dlm_check_buffer_size);
 CLUSTER_ATTR(rsbtbl_size, dlm_check_zero);
 CLUSTER_ATTR(recover_timer, dlm_check_zero);
 CLUSTER_ATTR(toss_secs, dlm_check_zero);
@@ -901,7 +907,6 @@ int dlm_our_addr(struct sockaddr_storage *addr, int num)
 
 /* Config file defaults */
 #define DEFAULT_TCP_PORT       21064
-#define DEFAULT_BUFFER_SIZE     4096
 #define DEFAULT_RSBTBL_SIZE     1024
 #define DEFAULT_RECOVER_TIMER      5
 #define DEFAULT_TOSS_SECS         10
Signature · 1 line
2.26.2
author maintainer
#5 lore raw

PATCH v2 5/5 fs: dlm: rework receive handling

In reply to Alexander Aring

This patch reworks the current receive handling of dlm. As I tried to change the send handling to fix reorder issues I took a look into the receive handling and simplified it, it works as the following:

Each connection has a preallocated receive buffer with a minimum length of 4096. On receive, the upper layer protocol will process all dlm message until there is not enough data anymore. If there exists "leftover" data at the end of the receive buffer because the dlm message wasn't fully received it will be copied to the begin of the preallocated receive buffer. Next receive more data will be appended to the previous "leftover" data and processing will begin again.

This will remove a lot of code of the current mechanism. Inside the processing functionality we will ensure with a memmove() that the dlm message should be memory aligned. To have a dlm message always started at the beginning of the buffer will reduce some amount of memmove() calls because src and dest pointers are the same.

The cluster attribute "buffer_size" becomes a new meaning, it's now the size of application layer receive buffer size. If this is changed during runtime the receive buffer will be reallocated. It's important that the receive buffer size has at minimum the size of the maximum possible dlm message size otherwise the received message cannot be placed inside the receive buffer size.

---
 fs/dlm/config.c   |   1 -
 fs/dlm/config.h   |   2 +
 fs/dlm/lowcomms.c | 179 ++++++++++++++++++++++------------------------
 fs/dlm/midcomms.c | 136 ++++++++++++++---------------------
 fs/dlm/midcomms.h |   3 +-
 5 files changed, 141 insertions(+), 180 deletions(-)
fs/dlm/config.c +0 -1
index a4bed304a8430..49c5f9407098e 100644
@@ -166,7 +166,6 @@ static bool dlm_check_zero(unsigned int x)
 	return !x;
 }
 
-#define DEFAULT_BUFFER_SIZE     4096
 static bool dlm_check_buffer_size(unsigned int x)
 {
 	return (x < DEFAULT_BUFFER_SIZE);
fs/dlm/config.h +2 -0
index 3b284ae9aeebd..c210250a25818 100644
@@ -12,6 +12,8 @@
 #ifndef __CONFIG_DOT_H__
 #define __CONFIG_DOT_H__
 
+#define DEFAULT_BUFFER_SIZE     4096
+
 struct dlm_config_node {
 	int nodeid;
 	int weight;
fs/dlm/lowcomms.c +85 -94

265 lines not on this page · raw message

fs/dlm/midcomms.c +53 -83

167 lines not on this page · raw message

fs/dlm/midcomms.h +1 -2
index 2e122e81c8d0d..61e90a9218493 100644
@@ -12,8 +12,7 @@
 #ifndef __MIDCOMMS_DOT_H__
 #define __MIDCOMMS_DOT_H__
 
-int dlm_process_incoming_buffer(int nodeid, const void *base, unsigned offset,
-				unsigned len, unsigned limit);
+int dlm_process_incoming_buffer(int nodeid, unsigned char *buf, int buflen);
 
 #endif				/* __MIDCOMMS_DOT_H__ */
 
Signature · 1 line
2.26.2