Skip to contentSkip to conversation

fs: dlm: locking and memory fixes

PATCH New Alexander Aring 1 participant 8 messages Aug 27, 2020 Distributed lock manager (DLM)

lore
Share & tools
lore

Alexander Aring

author maintainer
#0 lore raw

cover fs: dlm: locking and memory fixes

Hi,

this patch series contains some locking and memory issues which I found while working on a bigger fix to make dlm secured against tcpkill.

- Alex

Alexander Aring (7):
fs: dlm: synchronize dlm before shutdown
fs: dlm: make connection hash lockless
fs: dlm: fix dlm_local_addr memory leak
fs: dlm: fix configfs memory leak
fs: dlm: move free writequeue into con free
fs: dlm: handle possible othercon writequeues
fs: dlm: use free_con to free connection

 fs/dlm/Kconfig    |   1 +
 fs/dlm/config.c   |   3 ++
 fs/dlm/lowcomms.c | 122 +++++++++++++++++++++-------------------------
 3 files changed, 60 insertions(+), 66 deletions(-)
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH 1/7 fs: dlm: synchronize dlm before shutdown

This patch moves the dlm workqueue dlm synchronization before shutdown handling. The patch just flushes all pending work before starting to shutdown the connection. At least for the send_workqeue we should flush the workqueue to make sure there is no new connection handling going on as dlm_allow_conn switch is turned to false before.

---
 fs/dlm/lowcomms.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
fs/dlm/lowcomms.c +6 -4
index 5050fe05769b..ed098870ba0d 100644
@@ -1624,10 +1624,6 @@ static void work_flush(void)
 	struct hlist_node *n;
 	struct connection *con;
 
-	if (recv_workqueue)
-		flush_workqueue(recv_workqueue);
-	if (send_workqueue)
-		flush_workqueue(send_workqueue);
 	do {
 		ok = 1;
 		foreach_conn(stop_conn);
@@ -1659,6 +1655,12 @@ void dlm_lowcomms_stop(void)
 	mutex_lock(&connections_lock);
 	dlm_allow_conn = 0;
 	mutex_unlock(&connections_lock);
+
+	if (recv_workqueue)
+		flush_workqueue(recv_workqueue);
+	if (send_workqueue)
+		flush_workqueue(send_workqueue);
+
 	foreach_conn(shutdown_conn);
 	work_flush();
 	clean_writequeues();
Signature · 1 line
2.26.2
author maintainer
#2 lore raw

PATCH 2/7 fs: dlm: make connection hash lockless

In reply to Alexander Aring

There are some problems with the connections_lock. During my experiements I saw sometimes circular dependencies with sock_lock. The reason here might be code parts which runs nodeid2con() before or after sock_lock is acquired.

Another issue are missing locks in for_conn() iteration. Maybe this works fine because for_conn() is running in a context where connection_hash cannot be manipulated by others anymore.

However this patch changes the connection_hash to be protected by sleepable rcu. The hotpath function __find_con() is implemented lockless as it is only a reader of connection_hash and this hopefully fixes the circular locking dependencies. The iteration for_conn() will still call some sleepable functionality, that's why we use sleepable rcu in this case.

This patch removes the kmemcache functionality as I think I need to make some free() functionality via call_rcu(). However allocation time isn't here an issue. The dlm_allow_con will not be protected by a lock anymore as I think it's enough to just set and flush workqueues afterwards.

---
 fs/dlm/Kconfig    |  1 +
 fs/dlm/lowcomms.c | 86 ++++++++++++++++++++---------------------------
 2 files changed, 37 insertions(+), 50 deletions(-)
fs/dlm/Kconfig +1 -0
index f82a4952769d..ee92634196a8 100644
@@ -4,6 +4,7 @@ menuconfig DLM
 	depends on INET
 	depends on SYSFS && CONFIGFS_FS && (IPV6 || IPV6=n)
 	select IP_SCTP
+	select SRCU
 	help
 	A general purpose distributed lock manager for kernel or userspace
 	applications.
fs/dlm/lowcomms.c +36 -50

209 lines not on this page · raw message

Signature · 1 line
2.26.2
author maintainer
#3 lore raw

PATCH 3/7 fs: dlm: fix dlm_local_addr memory leak

In reply to Alexander Aring

This patch fixes the following memory detected by kmemleak and umount gfs2 filesystem which removed the last lockspace:

unreferenced object 0xffff9264f4f48f00 (size 128):
comm "mount", pid 425, jiffies 4294690253 (age 48.159s) hex dump (first 32 bytes):

    02 00 52 48 c0 a8 7a fb 00 00 00 00 00 00 00 00  ..RH..z.........
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  backtrace:
    [<0000000067a34940>] kmemdup+0x18/0x40
    [<00000000c935f9ab>] init_local+0x4c/0xa0
    [<00000000bbd286ef>] dlm_lowcomms_start+0x28/0x160
    [<00000000a86625cb>] dlm_new_lockspace+0x7e/0xb80
    [<000000008df6cd63>] gdlm_mount+0x1cc/0x5de
    [<00000000b67df8c7>] gfs2_lm_mount.constprop.0+0x1a3/0x1d3
    [<000000006642ac5e>] gfs2_fill_super+0x717/0xba9
    [<00000000d3ab7118>] get_tree_bdev+0x17f/0x280
    [<000000001975926e>] gfs2_get_tree+0x21/0x90
    [<00000000561ce1c4>] vfs_get_tree+0x28/0xc0
    [<000000007fecaf63>] path_mount+0x434/0xc00
    [<00000000636b9594>] __x64_sys_mount+0xe3/0x120
    [<00000000cc478a33>] do_syscall_64+0x33/0x40
    [<00000000ce9ccf01>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
---
 fs/dlm/lowcomms.c | 9 +++++++++
 1 file changed, 9 insertions(+)
fs/dlm/lowcomms.c +9 -0
index 9db7126de793..d0ece252a0d9 100644
@@ -1234,6 +1234,14 @@ static void init_local(void)
 	}
 }
 
+static void deinit_local(void)
+{
+	int i;
+
+	for (i = 0; i < dlm_local_count; i++)
+		kfree(dlm_local_addr[i]);
+}
+
 /* Initialise SCTP socket and bind to all interfaces */
 static int sctp_listen_for_all(void)
 {
@@ -1663,6 +1671,7 @@ void dlm_lowcomms_stop(void)
 	clean_writequeues();
 	foreach_conn(free_conn);
 	work_stop();
+	deinit_local();
 }
 
 int dlm_lowcomms_start(void)
Signature · 1 line
2.26.2
author maintainer
#4 lore raw

PATCH 4/7 fs: dlm: fix configfs memory leak

In reply to Alexander Aring

This patch fixes the following memory detected by kmemleak and umount gfs2 filesystem which removed the last lockspace:

unreferenced object 0xffff9264f482f600 (size 192):
comm "dlm_controld", pid 325, jiffies 4294690276 (age 48.136s) hex dump (first 32 bytes):

    00 00 00 00 00 00 00 00 6e 6f 64 65 73 00 00 00  ........nodes...
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  backtrace:
    [<00000000060481d7>] make_space+0x41/0x130
    [<000000008d905d46>] configfs_mkdir+0x1a2/0x5f0
    [<00000000729502cf>] vfs_mkdir+0x155/0x210
    [<000000000369bcf1>] do_mkdirat+0x6d/0x110
    [<00000000cc478a33>] do_syscall_64+0x33/0x40
    [<00000000ce9ccf01>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

The patch just remembers the "nodes" entry pointer in space as I think it's created as subdirectory when parent "spaces" is created. In function drop_space() we will lost the pointer reference to nds because configfs_remove_default_groups(). However as this subdirectory is always available when "spaces" exists it will just be freed when "spaces" will be freed.

---
 fs/dlm/config.c | 3 +++
 1 file changed, 3 insertions(+)
fs/dlm/config.c +3 -0
index 47f0b98b707f..f33a7e4ae917 100644
@@ -221,6 +221,7 @@ struct dlm_space {
 	struct list_head members;
 	struct mutex members_lock;
 	int members_count;
+	struct dlm_nodes *nds;
 };
 
 struct dlm_comms {
@@ -430,6 +431,7 @@ static struct config_group *make_space(struct config_group *g, const char *name)
 	INIT_LIST_HEAD(&sp->members);
 	mutex_init(&sp->members_lock);
 	sp->members_count = 0;
+	sp->nds = nds;
 	return &sp->group;
 
  fail:
@@ -451,6 +453,7 @@ static void drop_space(struct config_group *g, struct config_item *i)
 static void release_space(struct config_item *i)
 {
 	struct dlm_space *sp = config_item_to_space(i);
+	kfree(sp->nds);
 	kfree(sp);
 }
 
Signature · 1 line
2.26.2
author maintainer
#5 lore raw

PATCH 5/7 fs: dlm: move free writequeue into con free

In reply to Alexander Aring

This patch just move the free of struct connection member writequeue into the functionality when struct connection will be freed instead of doing two iterations.

---
 fs/dlm/lowcomms.c | 9 +--------
 1 file changed, 1 insertion(+), 8 deletions(-)
fs/dlm/lowcomms.c +1 -8
index d0ece252a0d9..04afc7178afb 100644
@@ -1550,13 +1550,6 @@ static void process_send_sockets(struct work_struct *work)
 		send_to_sock(con);
 }
 
-
-/* Discard all entries on the write queues */
-static void clean_writequeues(void)
-{
-	foreach_conn(clean_one_writequeue);
-}
-
 static void work_stop(void)
 {
 	if (recv_workqueue)
@@ -1620,6 +1613,7 @@ static void free_conn(struct connection *con)
 	spin_lock(&connections_lock);
 	hlist_del_rcu(&con->list);
 	spin_unlock(&connections_lock);
+	clean_one_writequeue(con);
 	kfree_rcu(con, rcu);
 }
 
@@ -1668,7 +1662,6 @@ void dlm_lowcomms_stop(void)
 
 	foreach_conn(shutdown_conn);
 	work_flush();
-	clean_writequeues();
 	foreach_conn(free_conn);
 	work_stop();
 	deinit_local();
Signature · 1 line
2.26.2
author maintainer
#6 lore raw

PATCH 6/7 fs: dlm: handle possible othercon writequeues

In reply to Alexander Aring

This patch adds free of possible other writequeue entries in othercon member of struct connection.

---
 fs/dlm/lowcomms.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/dlm/lowcomms.c +4 -2
index 04afc7178afb..794216eb728c 100644
@@ -1608,11 +1608,13 @@ static void shutdown_conn(struct connection *con)
 static void free_conn(struct connection *con)
 {
 	close_connection(con, true, true, true);
-	if (con->othercon)
-		kfree_rcu(con->othercon, rcu);
 	spin_lock(&connections_lock);
 	hlist_del_rcu(&con->list);
 	spin_unlock(&connections_lock);
+	if (con->othercon) {
+		clean_one_writequeue(con->othercon);
+		kfree_rcu(con->othercon, rcu);
+	}
 	clean_one_writequeue(con);
 	kfree_rcu(con, rcu);
 }
Signature · 1 line
2.26.2
author maintainer
#7 lore raw

PATCH 7/7 fs: dlm: use free_con to free connection

In reply to Alexander Aring

This patch use free_con() functionality to free the listen connection if listen fails. It also fixes an issue that a freed resource is still part of the connection_hash as hlist_del() is not called in this case. The only difference is that free_con() handles othercon as well, but this is never been set for the listen connection.

---
 fs/dlm/lowcomms.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
fs/dlm/lowcomms.c +2 -4
index 794216eb728c..1bf1808bfa6b 100644
@@ -1704,10 +1704,8 @@ int dlm_lowcomms_start(void)
 fail_unlisten:
 	dlm_allow_conn = 0;
 	con = nodeid2con(0,0);
-	if (con) {
-		close_connection(con, false, true, true);
-		kfree_rcu(con, rcu);
-	}
+	if (con)
+		free_conn(con);
 fail:
 	return error;
 }
Signature · 1 line
2.26.2