Skip to contentSkip to conversation

fs: dlm: misc and tracing

PATCH RESEND New Alexander Aring 1 participant 5 messages Aug 18, 2021

Subsystems (2)
lore
Share & tools
lore

Alexander Aring

author maintainer
#0 lore raw

cover fs: dlm: misc and tracing

Hi,

this series is a resend of my pending patches which I sent some days ago. For the recent dlm tracepoint patches I did some changes, see below.

changes in "fs: dlm: initial support for tracepoints":

- switch from e.g. uin32_t/u32 to __u32 in dlm tracing header as I figured out that this is common used. However it seems it really doesn't care, they are not uapi headers.
- switch from %d to %x printout for lkb_id as it is used for dlm debugfs. The lockspace id is still %d as it is used for sysfs.
- move trace_dlm_lock_start after error, we don't trace the error and we need to garuantee that lkb pointer is not dangled.
- add tracepoint start/end for dlm_unlock aswell, otherwise there are ast callbacks which are not recognizable, especially on DLM_EUNLOCK inside the status block.
- add a note that dlm_lock/unlock_start will only be traced if ls/lkb is valid. I think it makes no sense without any reference to do any lock/unlock tracing.

- Alex

Alexander Aring (4):
fs: dlm: implement delayed ack handling
fs: dlm: fix return -EINTR on recovery stopped
fs: dlm: initial support for tracepoints
fs: dlm: trace socket handling

 fs/dlm/ast.c               |   4 +
 fs/dlm/dir.c               |   4 +-
 fs/dlm/lock.c              |  10 ++
 fs/dlm/lowcomms.c          |   5 +
 fs/dlm/lowcomms.h          |   1 +
 fs/dlm/main.c              |   3 +
 fs/dlm/member.c            |   4 +-
 fs/dlm/midcomms.c          |  56 +++++++--
 fs/dlm/recoverd.c          |   4 +-
 include/trace/events/dlm.h | 252 +++++++++++++++++++++++++++++++++++++
 10 files changed, 332 insertions(+), 11 deletions(-)
 create mode 100644 include/trace/events/dlm.h
Signature · 1 line
2.27.0
author maintainer
#1 lore raw

PATCH RESEND 1/4 fs: dlm: implement delayed ack handling

This patch changes that we don't ack each message. Lowcomms will take care about to send an ack back after a bulk of messages was processed. Currently it's only when the whole receive buffer was processed, there might better positions to send an ack back but only the lowcomms implementation know when there are more data to receive. This patch has also disadvantages that we might retransmit more on errors, however this is a very rare case.

Tested with make_panic on gfs2 with three nodes by running:

trace-cmd record -p function -l 'dlm_send_ack' sleep 100

and

trace-cmd report | wc -l

Before patch:
- 20548
- 21376
- 21398

After patch:
- 18338
- 20679
- 19949

---
 fs/dlm/lowcomms.c |  1 +
 fs/dlm/lowcomms.h |  1 +
 fs/dlm/midcomms.c | 56 ++++++++++++++++++++++++++++++++++++++++-------
 3 files changed, 50 insertions(+), 8 deletions(-)
fs/dlm/lowcomms.c +1 -0
index cce1d50aa09f..8f715c620e1f 100644
@@ -947,6 +947,7 @@ static int receive_from_sock(struct connection *con)
 		}
 	}
 
+	dlm_midcomms_receive_done(con->nodeid);
 	mutex_unlock(&con->sock_mutex);
 	return 0;
 
fs/dlm/lowcomms.h +1 -0
index aaae7115c00d..4ccae07cf005 100644
@@ -46,6 +46,7 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg);
 int dlm_lowcomms_connect_node(int nodeid);
 int dlm_lowcomms_nodes_set_mark(int nodeid, unsigned int mark);
 int dlm_lowcomms_addr(int nodeid, struct sockaddr_storage *addr, int len);
+void dlm_midcomms_receive_done(int nodeid);
 
 #endif				/* __LOWCOMMS_DOT_H__ */
 
fs/dlm/midcomms.c +48 -8

89 lines not on this page · raw message

Signature · 1 line
2.27.0
author maintainer
#2 lore raw

PATCH RESEND 2/4 fs: dlm: fix return -EINTR on recovery stopped

In reply to Alexander Aring

This patch will return -EINTR instead of 1 if recovery is stopped. In case of ping_members() the return value will be checked if the error is -EINTR for signaling another recovery was triggered and the whole recovery process will come to a clean end to process the next one. Returning 1 will abort the recovery process and can leave the recovery in a broken state.

It was reported with the following kernel log message attached and a gfs2 mount stopped working:

"dlm: bobvirt1: dlm_recover_members error 1"

whereas 1 was returned because of a conversion of "dlm_recovery_stopped()" to an errno was missing which this patch will introduce. While on it all other possible missing errno conversions at other places were added as they are done as in other places.

It might be worth to check the error case at this recovery level, because some of the functionality also returns -ENOBUFS and check why recovery ends in a broken state. However this will fix the issue if another recovery was triggered at some points of recovery handling.

---
 fs/dlm/dir.c      | 4 +++-
 fs/dlm/member.c   | 4 +++-
 fs/dlm/recoverd.c | 4 +++-
 3 files changed, 9 insertions(+), 3 deletions(-)
fs/dlm/dir.c +3 -1
index 10c36ae1a8f9..45ebbe602bbf 100644
@@ -85,8 +85,10 @@ int dlm_recover_directory(struct dlm_ls *ls)
 		for (;;) {
 			int left;
 			error = dlm_recovery_stopped(ls);
-			if (error)
+			if (error) {
+				error = -EINTR;
 				goto out_free;
+			}
 
 			error = dlm_rcom_names(ls, memb->nodeid,
 					       last_name, last_len);
fs/dlm/member.c +3 -1
index d9e1e4170eb1..731d489aa323 100644
@@ -443,8 +443,10 @@ static int ping_members(struct dlm_ls *ls)
 
 	list_for_each_entry(memb, &ls->ls_nodes, list) {
 		error = dlm_recovery_stopped(ls);
-		if (error)
+		if (error) {
+			error = -EINTR;
 			break;
+		}
 		error = dlm_rcom_status(ls, memb->nodeid, 0);
 		if (error)
 			break;
fs/dlm/recoverd.c +3 -1
index 85e245392715..97d052cea5a9 100644
@@ -125,8 +125,10 @@ static int ls_recover(struct dlm_ls *ls, struct dlm_recover *rv)
 	dlm_recover_waiters_pre(ls);
 
 	error = dlm_recovery_stopped(ls);
-	if (error)
+	if (error) {
+		error = -EINTR;
 		goto fail;
+	}
 
 	if (neg || dlm_no_directory(ls)) {
 		/*
Signature · 1 line
2.27.0
author maintainer
#3 lore raw

PATCH RESEND 3/4 fs: dlm: initial support for tracepoints

In reply to Alexander Aring

This patch adds initial support for dlm tracepoints. It will introduce tracepoints to dlm main functionality dlm_lock()/dlm_unlock() and their complete ast() callback or blocking bast() callback.

The lock/unlock functionality has a start and end tracepoint, this is because there exists a race in case if would have a tracepoint at the end position only the complete/blocking callbacks could occur before. To work with eBPF tracing and using their lookup hash functionality there could be problems that an entry was not inserted yet. However use the start functionality for hash insert and check again in end functionality if there was an dlm internal error so there is no ast callback. In further it might also that locks with local masters will occur those callbacks immediately so we must have such functionality.

I did not make everything accessible yet, although it seems eBPF can be used to access a lot of internal datastructures if it's aware of the struct definitions of the running kernel instance. We still can change it, if you do eBPF experiments e.g. time measurements between lock and callback functionality you can simple use the local lkb_id field as hash value in combination with the lockspace id if you have multiple lockspaces. Otherwise you can simple use trace-cmd for some functionality, e.g. trace-cmd record -e dlm and trace-cmd report afterwards.

---
 fs/dlm/ast.c               |   4 +
 fs/dlm/lock.c              |  10 ++
 fs/dlm/main.c              |   3 +
 include/trace/events/dlm.h | 212 +++++++++++++++++++++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100644 include/trace/events/dlm.h
fs/dlm/ast.c +4 -0
index 283c7b94edda..0e7c6f6819ac 100644
@@ -9,6 +9,8 @@
 *******************************************************************************
 ******************************************************************************/
 
+#include <trace/events/dlm.h>
+
 #include "dlm_internal.h"
 #include "lock.h"
 #include "user.h"
@@ -254,10 +256,12 @@ void dlm_callback_work(struct work_struct *work)
 			continue;
 		} else if (callbacks[i].flags & DLM_CB_BAST) {
 			bastfn(lkb->lkb_astparam, callbacks[i].mode);
+			trace_dlm_bast(ls, lkb, callbacks[i].mode);
 		} else if (callbacks[i].flags & DLM_CB_CAST) {
 			lkb->lkb_lksb->sb_status = callbacks[i].sb_status;
 			lkb->lkb_lksb->sb_flags = callbacks[i].sb_flags;
 			castfn(lkb->lkb_astparam);
+			trace_dlm_ast(ls, lkb, lkb->lkb_lksb);
 		}
 	}
 
fs/dlm/lock.c +10 -0
index c502c065d007..c9ac75a38ff8 100644
@@ -53,6 +53,8 @@
                                    R: do_xxxx()
    L: receive_xxxx_reply()     <-  R: send_xxxx_reply()
 */
+#include <trace/events/dlm.h>
+
 #include <linux/types.h>
 #include <linux/rbtree.h>
 #include <linux/slab.h>
@@ -3437,6 +3439,8 @@ int dlm_lock(dlm_lockspace_t *lockspace,
 	if (error)
 		goto out;
 
+	trace_dlm_lock_start(ls, lkb, mode, flags);
+
 	error = set_lock_args(mode, lksb, flags, namelen, 0, ast,
 			      astarg, bast, &args);
 	if (error)
@@ -3454,6 +3458,8 @@ int dlm_lock(dlm_lockspace_t *lockspace,
 		__put_lkb(ls, lkb);
 	if (error == -EAGAIN || error == -EDEADLK)
 		error = 0;
+
+	trace_dlm_lock_end(ls, lkb, mode, flags, error);
  out:
 	dlm_unlock_recovery(ls);
 	dlm_put_lockspace(ls);
@@ -3481,6 +3487,8 @@ int dlm_unlock(dlm_lockspace_t *lockspace,
 	if (error)
 		goto out;
 
+	trace_dlm_unlock_start(ls, lkb, flags);
+
 	error = set_unlock_args(flags, astarg, &args);
 	if (error)
 		goto out_put;
@@ -3496,6 +3504,8 @@ int dlm_unlock(dlm_lockspace_t *lockspace,
 		error = 0;
  out_put:
 	dlm_put_lkb(lkb);
+
+	trace_dlm_unlock_end(ls, lkb, flags, error);
  out:
 	dlm_unlock_recovery(ls);
 	dlm_put_lockspace(ls);
fs/dlm/main.c +3 -0
index afc66a1346d3..1c5be4b70ac1 100644
@@ -19,6 +19,9 @@
 #include "config.h"
 #include "lowcomms.h"
 
+#define CREATE_TRACE_POINTS
+#include <trace/events/dlm.h>
+
 static int __init init_dlm(void)
 {
 	int error;
include/trace/events/dlm.h new file +212 -0

212 lines not on this page · raw message

Signature · 1 line
2.27.0
author maintainer
#4 lore raw

PATCH RESEND 4/4 fs: dlm: trace socket handling

In reply to Alexander Aring

This patch adds tracepoints for dlm socket receive and send functionality. We can use it to track how much data was send or received to or from a specific nodeid.

---
 fs/dlm/lowcomms.c          |  4 ++++
 include/trace/events/dlm.h | 40 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
fs/dlm/lowcomms.c +4 -0
index 8f715c620e1f..e5438f2e586a 100644
@@ -53,6 +53,8 @@
 #include <net/sctp/sctp.h>
 #include <net/ipv6.h>
 
+#include <trace/events/dlm.h>
+
 #include "dlm_internal.h"
 #include "lowcomms.h"
 #include "midcomms.h"
@@ -925,6 +927,7 @@ static int receive_from_sock(struct connection *con)
 		msg.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL;
 		ret = kernel_recvmsg(con->sock, &msg, &iov, 1, iov.iov_len,
 				     msg.msg_flags);
+		trace_dlm_recv(con->nodeid, ret);
 		if (ret == -EAGAIN)
 			break;
 		else if (ret <= 0)
@@ -1411,6 +1414,7 @@ static void send_to_sock(struct connection *con)
 
 		ret = kernel_sendpage(con->sock, e->page, offset, len,
 				      msg_flags);
+		trace_dlm_send(con->nodeid, ret);
 		if (ret == -EAGAIN || ret == 0) {
 			if (ret == -EAGAIN &&
 			    test_bit(SOCKWQ_ASYNC_NOSPACE, &con->sock->flags) &&
include/trace/events/dlm.h +40 -0
index ad21d1759dd4..bdc4890114fb 100644
@@ -206,6 +206,46 @@ TRACE_EVENT(dlm_unlock_end,
 
 );
 
+TRACE_EVENT(dlm_send,
+
+	TP_PROTO(int nodeid, int ret),
+
+	TP_ARGS(nodeid, ret),
+
+	TP_STRUCT__entry(
+		__field(int, nodeid)
+		__field(int, ret)
+	),
+
+	TP_fast_assign(
+		__entry->nodeid = nodeid;
+		__entry->ret = ret;
+	),
+
+	TP_printk("nodeid=%d ret=%d", __entry->nodeid, __entry->ret)
+
+);
+
+TRACE_EVENT(dlm_recv,
+
+	TP_PROTO(int nodeid, int ret),
+
+	TP_ARGS(nodeid, ret),
+
+	TP_STRUCT__entry(
+		__field(int, nodeid)
+		__field(int, ret)
+	),
+
+	TP_fast_assign(
+		__entry->nodeid = nodeid;
+		__entry->ret = ret;
+	),
+
+	TP_printk("nodeid=%d ret=%d", __entry->nodeid, __entry->ret)
+
+);
+
 #endif /* if !defined(_TRACE_DLM_H) || defined(TRACE_HEADER_MULTI_READ) */
 
 /* This part must be outside protection */
Signature · 1 line
2.27.0