eal/linux: harden uevent recv error handling

The Linux uevent handler is harded for non-blocking receive behavior

Signed-off-by Randy Tice <[email protected]>

---

lib/eal/linux/eal_dev.c +3 -1

-	if (ret < 0 && errno == EAGAIN)
+	if (ret < 0 &&
+		(errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR)) {
+		/* non-blocking or interrupted */
 		return;

The EAL uevent only needs/wants messages from the kernel.
Better to let kernel side do the filtering to avoid any overrun issues in DPDK interrupt thread.

There is no exposed API definition here. The convention is kernel is 1 and libudev uses 2.

Since kernel is now filtering can remove step in uevent parsing.

Fixes 0d0f478d0483 ("eal/linux: add uevent parse and process")

Cc [email protected]

Reported-by Randy Tice <[email protected]>

Signed-off-by Stephen Hemminger <[email protected]>

---
 .mailmap                | 1 +
 lib/eal/linux/eal_dev.c | 8 ++++----
 2 files changed, 5 insertions(+), 4 deletions(-)

.mailmap +0 -0

index fcb3d1bb3f..9b7713b6df 100644
@@ -1380,6 +1380,7 @@ Ralf Hoffmann <ralf.hoffmann@allegro-

packets.com>
Rami Rosen <[email protected]> <[email protected]> Rami Rosen
<[email protected]> <[email protected]> Randy Schacher <[email protected]>
+Randy Tice <[email protected]>
Rani Sharoni <[email protected]>
Ranjit Menon <[email protected]>
Rasesh Mody <[email protected]> <[email protected]> diff --git
a/lib/eal/linux/eal_dev.c b/lib/eal/linux/eal_dev.c index ec408649d0..666967667f 100644

lib/eal/linux/eal_dev.c +0 -0

@@ -29,6 +29,9 @@ static bool hotplug_handle;  #define EAL_UEV_MSG_LEN

4096 #define EAL_UEV_MSG_ELEM_LEN 128

lib/eal/linux/eal_dev.c +4 -4

+/* Listen only to messages from kernel (not libudev) */ #define
+EAL_UEV_GROUP_KERNEL 1
+
 /*
  * spinlock for device hot-unplug failure handling. If it try to access bus or
  * device, such as handle sigbus on bus or handle memory failure for device

@@ -116,7 +119,7 @@ dev_uev_socket_fd_create(void)
 	memset(&addr, 0, sizeof(addr));
 	addr.nl_family = AF_NETLINK;
 	addr.nl_pid = 0;
-	addr.nl_groups = 0xffffffff;
+	addr.nl_groups = EAL_UEV_GROUP_KERNEL;
 
 	ret = bind(fd, (struct sockaddr *) &addr, sizeof(addr));
 	if (ret < 0) {

@@ -164,9 +167,6 @@ dev_uev_parse(const char *buf, struct rte_dev_event
 *event, int length)
 		 * check device uevent from kernel side, no need to check
 		 * uevent from udev.
 		 */
-		if (!strncmp(buf, "libudev", 7)) {
-			return -1;
-		}
 		if (!strncmp(buf, "ACTION=", 7)) {
 			buf += 7;

i += 7;
--
2.53.0

The kernel netlink socket into DPDK can get overrun if DPDK interrupt thread is not keeping up. To reduce the possibility increase the socket receive buffer to 4M.

Fixes 0d0f478d0483 ("eal/linux: add uevent parse and process")

Cc [email protected]

Reported-by Randy Tice <[email protected]>

Signed-off-by Stephen Hemminger <[email protected]>

---
 lib/eal/linux/eal_dev.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

lib/eal/linux/eal_dev.c index renamed from lib/eal/linux/eal_dev.c +0 -0

666967667f..fb5c8bf070 100644

lib/eal/linux/eal_dev.c +0 -0

@@ -32,6 +32,9 @@ static bool hotplug_handle;
 /* Listen only to messages from kernel (not libudev) */  #define

EAL_UEV_GROUP_KERNEL 1

lib/eal/linux/eal_dev.c +28 -1

+/* Large uevent buffer */
+#define EAL_UEV_MSG_RCVBUF (4 * 1024 * 1024)
+
 /*
  * spinlock for device hot-unplug failure handling. If it try to access bus or
  * device, such as handle sigbus on bus or handle memory failure for device

@@ -103,6 +106,29 @@ static int cmp_dev_name(const struct rte_device
 *dev,
 	return strcmp(dev->name, name);
 }
 
+/*
+ * To avoid losing uevents increase the netlink receive buffer size,
+ * and override the kernel clamp value if necessary.
+ */
+static int
+dev_uev_set_rcvbuf(int fd, int n)
+{
+	int ret, val;
+	socklen_t len = sizeof(val);
+
+	ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(n));
+	if (ret < 0)
+		return ret;
+
+	/* kernel may have clamped our request */
+	ret = getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &val, &len);
+	if (ret >= 0 && len == sizeof(val) && val == n * 2)
+		return 0; /* request worked */
+
+	/* try again to override kernel restriction */
+	return setsockopt(fd, SOL_SOCKET, SO_RCVBUFFORCE, &n, sizeof(n)); }
+
 static int
 dev_uev_socket_fd_create(void)
 {

@@ -127,6 +153,9 @@ dev_uev_socket_fd_create(void)
 		goto err;
 	}
 
+	if (dev_uev_set_rcvbuf(fd, EAL_UEV_MSG_RCVBUF) < 0)
+		EAL_LOG(NOTICE, "Failed to set rcvbuf.");
+
 	if (rte_intr_fd_set(intr_handle, fd))
 		goto err;
 
--

2.53.0

Make uevent handler handle case where kernel netlink receive buffer is overrun. If this happens the DPDK device state will potentially be out of sync, but better not to give up.

The code already handles the case of kernel sending a zero length message; it just ignores it.

Fixes 0d0f478d0483 ("eal/linux: add uevent parse and process")

Cc [email protected]

Reported-by Randy Tice <[email protected]>

Signed-off-by Stephen Hemminger <[email protected]>

---
 lib/eal/linux/eal_dev.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

lib/eal/linux/eal_dev.c index renamed from lib/eal/linux/eal_dev.c +0 -0

fb5c8bf070..b535122ff6 100644

lib/eal/linux/eal_dev.c +14 -5

@@ -270,11 +270,20 @@ dev_uev_handler(__rte_unused void *param)
 
 	ret = recv(rte_intr_fd_get(intr_handle), buf, EAL_UEV_MSG_LEN,
 		   MSG_DONTWAIT);
-	if (ret < 0 && errno == EAGAIN)
-		return;
-	else if (ret <= 0) {
-		/* connection is closed or broken, can not up again. */
-		EAL_LOG(ERR, "uevent socket connection is broken.");
+	if (ret < 0) {
+		/* transient error */
+		if (errno == EAGAIN || errno == EWOULDBLOCK || errno ==
 EINTR)
+			return;
+
+		/* kernel netlink messages lost */
+		if (errno == ENOBUFS) {
+			EAL_LOG(NOTICE, "kernel receive buffer overrun");
+			return;
+		}
+
+		EAL_LOG(ERR, "unexpected error on uevent recv: %s",
+			strerror(errno));
+
 		rte_eal_alarm_set(1, dev_delayed_unregister, NULL);
 		return;

}
--
2.53.0

This set of patches is an extension of the proposed patch to handle netlink uevent socket overrun reported by Randy Tice.
Mostly impacts users of Azure VF events (MLX5 and MANA).

DPDK uses netlink uevent to track hotplug events reported by the kernel. Since this is a one-way non-blocking channel there is no fool proof way to prevent missing events if DPDK interrupt thread does not keep up.

What this series does is:
- use netlink group to only receive events from kernel
- increase socket buffer to have more headroom
- report the problem more clearly in log
- document requirement for control threads

Stephen Hemminger (4):
eal/linux: filter uevent Netlink in kernel
eal/linux: increase uevent socket buffer
eal/linux: report uevent socket overrun
doc: add warning about control threads

 .mailmap                                      |  1 +
 .../prog_guide/env_abstraction_layer.rst      | 14 +++++
 lib/eal/linux/eal_dev.c                       | 56 ++++++++++++++++---
 3 files changed, 62 insertions(+), 9 deletions(-)

--
2.53.0

This set of patches is an extension of the proposed patch to handle netlink uevent socket overrun reported by Randy Tice. Mostly impacts users of Azure VF events (MLX5 and MANA).

DPDK uses netlink uevent to track hotplug events reported by the kernel. Since this is a one-way non-blocking channel there is no fool proof way to prevent missing events if DPDK interrupt thread does not keep up.

What this series does is:
- use netlink group to only receive events from kernel
- increase socket buffer to have more headroom
- report the problem more clearly in log
- document requirement for control threads

Stephen Hemminger (4):
eal/linux: filter uevent Netlink in kernel
eal/linux: increase uevent socket buffer
eal/linux: report uevent socket overrun
doc: add warning about control threads

 .mailmap                                      |  1 +
 .../prog_guide/env_abstraction_layer.rst      | 14 +++++
 lib/eal/linux/eal_dev.c                       | 56 ++++++++++++++++---
 3 files changed, 62 insertions(+), 9 deletions(-)