From: Alexander Aring Date: Fri, 4 Sep 2020 10:29:46 -0400 Subject: [Cluster-devel] [PATCH dlm-tool 2/2] dlm_controld: set SO_RCVBUF for netlink socket In-Reply-To: <20200904142946.8684-1-aahringo@redhat.com> References: <20200904142946.8684-1-aahringo@redhat.com> Message-ID: <20200904142946.8684-2-aahringo@redhat.com> List-Id: To: cluster-devel.redhat.com MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Saw some: 1597148652 uevent recv error -1 errno 105 on a dlm_tool dump. The errno 105 is ENOBUFS on an recv of an AF_NETLINK socket. Further investigations showed that we dropping uevents in such case, see the added comment. The above error message was on a node which hung inside do_uevent() of dlm kernel code which means that the node is waiting for a sysfs write of "event_done". My guess is that dlm_controld dropped some "important" messages and never writes to "event_done" in this case. However we should prevent such ENOBUFS cases in netlink which this patch is trying to do in a simple way. --- dlm_controld/dlm_daemon.h | 2 ++ dlm_controld/main.c | 19 +++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/dlm_controld/dlm_daemon.h b/dlm_controld/dlm_daemon.h index 0b4ae5f2..95848201 100644 --- a/dlm_controld/dlm_daemon.h +++ b/dlm_controld/dlm_daemon.h @@ -83,6 +83,8 @@ #define DEFAULT_LOGFILE_PRIORITY LOG_INFO #define DEFAULT_LOGFILE LOG_FILE_PATH +#define DEFAULT_NETLINK_RCVBUF (2 * 1024 * 1024) + enum { no_arg = 0, req_arg_bool = 1, diff --git a/dlm_controld/main.c b/dlm_controld/main.c index 470a067c..a82fc9c2 100644 --- a/dlm_controld/main.c +++ b/dlm_controld/main.c @@ -765,6 +765,7 @@ static void process_uevent(int ci) static int setup_uevent(void) { struct sockaddr_nl snl; + int rcvbuf; int s, rv; s = socket(AF_NETLINK, SOCK_DGRAM, NETLINK_KOBJECT_UEVENT); @@ -773,6 +774,24 @@ static int setup_uevent(void) return s; } + /* man 7 netlink: + * + * However, reliable transmissions from kernel to user are impossible in + * any case. The kernel can't send a netlink message if the socket buffer + * is full: the message will be dropped and the kernel and the user-space + * process will no longer have the same view of kernel state. It is up to + * the application to detect when this happens (via the ENOBUFS error + * returned by recvmsg(2)) and resynchronize. + * + * To prevent ENOBUFS errors we just set the receive buffer to two + * megabyte as other applications do it. This will not ensure that we never + * receive ENOBUFS but it's more unlikely. May it's worth to handle ENOBUFS + * errors on a different way in future. + */ + rcvbuf = DEFAULT_NETLINK_RCVBUF; + setsockopt(s, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof(rcvbuf)); + setsockopt(s, SOL_SOCKET, SO_RCVBUFFORCE, &rcvbuf, sizeof(rcvbuf)); + memset(&snl, 0, sizeof(snl)); snl.nl_family = AF_NETLINK; snl.nl_pid = getpid(); -- 2.26.2