fs: dlm: misc and tracing
Subsystems (2)
Alexander Aring
maintainercover fs: dlm: misc and tracing
Hi,
this series is a resend of my pending patches which I sent some days ago. For the recent dlm tracepoint patches I did some changes, see below.
changes in "fs: dlm: initial support for tracepoints":
- switch from e.g. uin32_t/u32 to __u32 in dlm tracing header as I figured out that this is common used. However it seems it really doesn't care, they are not uapi headers.
- switch from %d to %x printout for lkb_id as it is used for dlm debugfs. The lockspace id is still %d as it is used for sysfs.
- move trace_dlm_lock_start after error, we don't trace the error and we need to garuantee that lkb pointer is not dangled.
- add tracepoint start/end for dlm_unlock aswell, otherwise there are ast callbacks which are not recognizable, especially on DLM_EUNLOCK inside the status block.
- add a note that dlm_lock/unlock_start will only be traced if ls/lkb is valid. I think it makes no sense without any reference to do any lock/unlock tracing.
- Alex
Alexander Aring (4):
fs: dlm: implement delayed ack handling
fs: dlm: fix return -EINTR on recovery stopped
fs: dlm: initial support for tracepoints
fs: dlm: trace socket handling
fs/dlm/ast.c | 4 + fs/dlm/dir.c | 4 +- fs/dlm/lock.c | 10 ++ fs/dlm/lowcomms.c | 5 + fs/dlm/lowcomms.h | 1 + fs/dlm/main.c | 3 + fs/dlm/member.c | 4 +- fs/dlm/midcomms.c | 56 +++++++-- fs/dlm/recoverd.c | 4 +- include/trace/events/dlm.h | 252 +++++++++++++++++++++++++++++++++++++ 10 files changed, 332 insertions(+), 11 deletions(-) create mode 100644 include/trace/events/dlm.h
Signature · 1 line
2.27.0
Alexander Aring
maintainerPATCH RESEND 1/4 fs: dlm: implement delayed ack handling
This patch changes that we don't ack each message. Lowcomms will take care about to send an ack back after a bulk of messages was processed. Currently it's only when the whole receive buffer was processed, there might better positions to send an ack back but only the lowcomms implementation know when there are more data to receive. This patch has also disadvantages that we might retransmit more on errors, however this is a very rare case.
Tested with make_panic on gfs2 with three nodes by running:
trace-cmd record -p function -l 'dlm_send_ack' sleep 100
and
trace-cmd report | wc -l
Before patch:
- 20548
- 21376
- 21398
After patch:
- 18338
- 20679
- 19949
--- fs/dlm/lowcomms.c | 1 + fs/dlm/lowcomms.h | 1 + fs/dlm/midcomms.c | 56 ++++++++++++++++++++++++++++++++++++++++------- 3 files changed, 50 insertions(+), 8 deletions(-)
fs/dlm/lowcomms.c +1 -0
@@ -947,6 +947,7 @@ static int receive_from_sock(struct connection *con) } } +dlm_midcomms_receive_done (con->nodeid);mutex_unlock (&con->sock_mutex);return 0 ;
fs/dlm/lowcomms.h +1 -0
@@ -46,6 +46,7 @@ int dlm_lowcomms_resend_msg(struct dlm_msg *msg);int dlm_lowcomms_connect_node (int nodeid);int dlm_lowcomms_nodes_set_mark (int nodeid,unsigned int mark);int dlm_lowcomms_addr (int nodeid,struct sockaddr_storage *addr,int len); +void dlm_midcomms_receive_done (int nodeid);#endif /* __LOWCOMMS_DOT_H__ */
fs/dlm/midcomms.c +48 -8
89 lines not on this page · raw message
Signature · 1 line
2.27.0
Alexander Aring
maintainerPATCH RESEND 2/4 fs: dlm: fix return -EINTR on recovery stopped
In reply to Alexander AringThis patch will return -EINTR instead of 1 if recovery is stopped. In case of ping_members() the return value will be checked if the error is -EINTR for signaling another recovery was triggered and the whole recovery process will come to a clean end to process the next one. Returning 1 will abort the recovery process and can leave the recovery in a broken state.
It was reported with the following kernel log message attached and a gfs2 mount stopped working:
"dlm: bobvirt1: dlm_recover_members error 1"
whereas 1 was returned because of a conversion of "dlm_recovery_stopped()" to an errno was missing which this patch will introduce. While on it all other possible missing errno conversions at other places were added as they are done as in other places.
It might be worth to check the error case at this recovery level, because some of the functionality also returns -ENOBUFS and check why recovery ends in a broken state. However this will fix the issue if another recovery was triggered at some points of recovery handling.
--- fs/dlm/dir.c | 4 +++- fs/dlm/member.c | 4 +++- fs/dlm/recoverd.c | 4 +++- 3 files changed, 9 insertions(+), 3 deletions(-)
fs/dlm/dir.c +3 -1
@@ -85,8 +85,10 @@ int dlm_recover_directory(struct dlm_ls *ls)for (;;) {int left; error =dlm_recovery_stopped (ls); -if (error) +if (error) { + error = -EINTR ; goto out_free; + } error =dlm_rcom_names (ls, memb->nodeid, last_name, last_len);
fs/dlm/member.c +3 -1
@@ -443,8 +443,10 @@ static int ping_members(struct dlm_ls *ls)list_for_each_entry (memb, &ls->ls_nodes, list) { error =dlm_recovery_stopped (ls); -if (error) +if (error) { + error = -EINTR ;break ; + } error =dlm_rcom_status (ls, memb->nodeid,0 );if (error)break ;
fs/dlm/recoverd.c +3 -1
@@ -125,8 +125,10 @@ static int ls_recover(struct dlm_ls *ls, struct dlm_recover *rv)dlm_recover_waiters_pre (ls); error =dlm_recovery_stopped (ls); -if (error) +if (error) { + error = -EINTR ; goto fail; + }if (neg ||dlm_no_directory (ls)) {/*
Signature · 1 line
2.27.0
Alexander Aring
maintainerPATCH RESEND 3/4 fs: dlm: initial support for tracepoints
In reply to Alexander AringThis patch adds initial support for dlm tracepoints. It will introduce tracepoints to dlm main functionality dlm_lock()/dlm_unlock() and their complete ast() callback or blocking bast() callback.
The lock/unlock functionality has a start and end tracepoint, this is because there exists a race in case if would have a tracepoint at the end position only the complete/blocking callbacks could occur before. To work with eBPF tracing and using their lookup hash functionality there could be problems that an entry was not inserted yet. However use the start functionality for hash insert and check again in end functionality if there was an dlm internal error so there is no ast callback. In further it might also that locks with local masters will occur those callbacks immediately so we must have such functionality.
I did not make everything accessible yet, although it seems eBPF can be used to access a lot of internal datastructures if it's aware of the struct definitions of the running kernel instance. We still can change it, if you do eBPF experiments e.g. time measurements between lock and callback functionality you can simple use the local lkb_id field as hash value in combination with the lockspace id if you have multiple lockspaces. Otherwise you can simple use trace-cmd for some functionality, e.g. trace-cmd record -e dlm and trace-cmd report afterwards.
--- fs/dlm/ast.c | 4 + fs/dlm/lock.c | 10 ++ fs/dlm/main.c | 3 + include/trace/events/dlm.h | 212 +++++++++++++++++++++++++++++++++++++ 4 files changed, 229 insertions(+) create mode 100644 include/trace/events/dlm.h
fs/dlm/ast.c +4 -0
@@ -9,6 +9,8 @@******************************************************************************* ******************************************************************************/ +#include <trace/events/dlm.h> +#include "dlm_internal.h" #include "lock.h" #include "user.h"
@@ -254,10 +256,12 @@ void dlm_callback_work(struct work_struct *work)continue ; }else if (callbacks[i].flags &DLM_CB_BAST ) {bastfn (lkb->lkb_astparam, callbacks[i].mode); +trace_dlm_bast (ls, lkb, callbacks[i].mode); }else if (callbacks[i].flags &DLM_CB_CAST ) { lkb->lkb_lksb->sb_status = callbacks[i].sb_status; lkb->lkb_lksb->sb_flags = callbacks[i].sb_flags;castfn (lkb->lkb_astparam); +trace_dlm_ast (ls, lkb, lkb->lkb_lksb); } }
fs/dlm/lock.c +10 -0
@@ -53,6 +53,8 @@R: do_xxxx() L: receive_xxxx_reply() <- R: send_xxxx_reply() */ +#include <trace/events/dlm.h> +#include <linux/types.h> #include <linux/rbtree.h> #include <linux/slab.h>
@@ -3437,6 +3439,8 @@ int dlm_lock(dlm_lockspace_t *lockspace,if (error) goto out; +trace_dlm_lock_start (ls, lkb, mode, flags); + error =set_lock_args (mode, lksb, flags, namelen,0 , ast, astarg, bast, &args);if (error)
@@ -3454,6 +3458,8 @@ int dlm_lock(dlm_lockspace_t *lockspace,__put_lkb (ls, lkb);if (error == -EAGAIN || error == -EDEADLK ) error =0 ; + +trace_dlm_lock_end (ls, lkb, mode, flags, error); out:dlm_unlock_recovery (ls);dlm_put_lockspace (ls);
@@ -3481,6 +3487,8 @@ int dlm_unlock(dlm_lockspace_t *lockspace,if (error) goto out; +trace_dlm_unlock_start (ls, lkb, flags); + error =set_unlock_args (flags, astarg, &args);if (error) goto out_put;
@@ -3496,6 +3504,8 @@ int dlm_unlock(dlm_lockspace_t *lockspace, error =0 ; out_put:dlm_put_lkb (lkb); + +trace_dlm_unlock_end (ls, lkb, flags, error); out:dlm_unlock_recovery (ls);dlm_put_lockspace (ls);
fs/dlm/main.c +3 -0
@@ -19,6 +19,9 @@#include "config.h" #include "lowcomms.h" +#define CREATE_TRACE_POINTS +#include <trace/events/dlm.h> +static int __init init_dlm (void ) {int error;
include/trace/events/dlm.h new file +212 -0
212 lines not on this page · raw message
Signature · 1 line
2.27.0
Alexander Aring
maintainerPATCH RESEND 4/4 fs: dlm: trace socket handling
In reply to Alexander AringThis patch adds tracepoints for dlm socket receive and send functionality. We can use it to track how much data was send or received to or from a specific nodeid.
--- fs/dlm/lowcomms.c | 4 ++++ include/trace/events/dlm.h | 40 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+)
fs/dlm/lowcomms.c +4 -0
@@ -53,6 +53,8 @@#include <net/sctp/sctp.h> #include <net/ipv6.h> +#include <trace/events/dlm.h> +#include "dlm_internal.h" #include "lowcomms.h" #include "midcomms.h"
@@ -925,6 +927,7 @@ static int receive_from_sock(struct connection *con) msg.msg_flags =MSG_DONTWAIT |MSG_NOSIGNAL ; ret =kernel_recvmsg (con->sock, &msg, &iov,1 , iov.iov_len, msg.msg_flags); +trace_dlm_recv (con->nodeid, ret);if (ret == -EAGAIN )break ;else if (ret <=0 )
@@ -1411,6 +1414,7 @@ static void send_to_sock(struct connection *con) ret =kernel_sendpage (con->sock, e->page, offset, len, msg_flags); +trace_dlm_send (con->nodeid, ret);if (ret == -EAGAIN || ret ==0 ) { if (ret == -EAGAIN && test_bit(SOCKWQ_ASYNC_NOSPACE, &con->sock->flags) &&
include/trace/events/dlm.h +40 -0
@@ -206,6 +206,46 @@ TRACE_EVENT(dlm_unlock_end, ); +TRACE_EVENT (dlm_send, + + TP_PROTO(int nodeid, int ret), + +TP_ARGS (nodeid, ret), + +TP_STRUCT__entry ( + __field(int, nodeid) +__field (int, ret) + ), + +TP_fast_assign ( + __entry->nodeid = nodeid; + __entry->ret = ret; + ), + +TP_printk ("nodeid=%d ret=%d" , __entry->nodeid, __entry->ret) + +); + +TRACE_EVENT (dlm_recv, + + TP_PROTO(int nodeid, int ret), + +TP_ARGS (nodeid, ret), + +TP_STRUCT__entry ( + __field(int, nodeid) +__field (int, ret) + ), + +TP_fast_assign ( + __entry->nodeid = nodeid; + __entry->ret = ret; + ), + +TP_printk ("nodeid=%d ret=%d" , __entry->nodeid, __entry->ret) + +); +#endif /* if !defined(_TRACE_DLM_H) || defined(TRACE_HEADER_MULTI_READ) */ /* This part must be outside protection */
Signature · 1 line
2.27.0