Skip to contentSkip to conversation

src: try SO_SNDBUF before SO_SNDBUFFORCE

PATCH In review Dave Pifke 2 participants 7 messages Apr 7, 2023 – Apr 18, 2023 netfilter

lore
Share & tools
lore

Dave Pifke

author
#0 lore raw

PATCH 1/1 src: try SO_SNDBUF before SO_SNDBUFFORCE

Prior to this patch, nft inside a systemd-nspawn container was failing to install my ruleset (which includes a large-ish map), with the error

netlink: Error: Could not process rule: Message too long

strace reveals:

setsockopt(3, SOL_SOCKET, SO_SNDBUFFORCE, [524288], 4) = -1 EPERM (Operation not permitted)

This is despite the nspawn process supposedly having CAP_NET_ADMIN, and despite /proc/sys/net/core/wmem_max (in the main host namespace) being set larger than the requested size:

net.core.wmem_max = 16777216

A web search reveals at least one other user having the same issue:

https://old.reddit.com/r/Proxmox/comments/scnoav/lxc_container_debian_11_nftables_geoblocking/

After this patch, nft succeeds.

---
 src/mnl.c | 7 +++++++
 1 file changed, 7 insertions(+)
src/mnl.c +7 -0
index 26f943db..ab6750c8 100644
@@ -260,6 +260,13 @@ static void mnl_set_sndbuffer(const struct mnl_socket *nl,
 		return;
 
 	/* Rise sender buffer length to avoid hitting -EMSGSIZE */
+	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
+		       &newbuffsiz, sizeof(socklen_t)) == 0)
+		return;
+
+	/* If the above fails (probably because it exceeds
+	 * /proc/sys/net/core/wmem_max), try again with SO_SNDBUFFORCE.
+	 * This requires CAP_NET_ADMIN. */
 	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUFFORCE,
 		       &newbuffsiz, sizeof(socklen_t)) < 0)
 		return;
Signature · 1 line
2.20.1
#1 lore raw
Quoted 13 lines · Dave Pifke

Prior to this patch, nft inside a systemd-nspawn container was failing to install my ruleset (which includes a large-ish map), with the error

netlink: Error: Could not process rule: Message too long

strace reveals:

setsockopt(3, SOL_SOCKET, SO_SNDBUFFORCE, [524288], 4) = -1 EPERM (Operation not permitted)

This is despite the nspawn process supposedly having CAP_NET_ADMIN, and despite /proc/sys/net/core/wmem_max (in the main host namespace) being set larger than the requested size:

Quoted 7 lines

net.core.wmem_max = 16777216

A web search reveals at least one other user having the same issue:

https://old.reddit.com/r/Proxmox/comments/scnoav/lxc_container_debian_11_nftables_geoblocking/

After this patch, nft succeeds.

Patch LGTM.

May I add your Signed-off-by: tag to this patch before applying it?

Thanks.

author
#2 lore raw
Quoted 1 line · Pablo Neira Ayuso

May I add your Signed-off-by: tag to this patch before applying it?

Yes. Thank you!

Signature · 1 line
Dave Pifke, [email protected]
#3 lore raw
In reply to Dave Pifke

Hi again,

Let me revisit this.

Quoted 7 lines · Dave Pifke

Prior to this patch, nft inside a systemd-nspawn container was failing to install my ruleset (which includes a large-ish map), with the error

netlink: Error: Could not process rule: Message too long

strace reveals:

Quoted 7 lines

setsockopt(3, SOL_SOCKET, SO_SNDBUFFORCE, [524288], 4) = -1 EPERM (Operation not permitted)

This is despite the nspawn process supposedly having CAP_NET_ADMIN, and despite /proc/sys/net/core/wmem_max (in the main host namespace) being set larger than the requested size:

net.core.wmem_max = 16777216

OK, so you indeed increased net.core.wmem_max on the host namespace.

Quoted 14 lines

A web search reveals at least one other user having the same issue:

https://old.reddit.com/r/Proxmox/comments/scnoav/lxc_container_debian_11_nftables_geoblocking/

After this patch, nft succeeds.

---
 src/mnl.c | 7 +++++++
 1 file changed, 7 insertions(+)
src/mnl.c +0 -0
index 26f943db..ab6750c8 100644

src/mnl.c static void mnl_set_sndbuffer(const struct mnl_socket *nl, · line 265

Quoted 6 lines
src/mnl.c +3 -0
@@ -260,6 +260,13 @@ static void mnl_set_sndbuffer(const struct mnl_socket *nl,
 		return;
 
 	/* Rise sender buffer length to avoid hitting -EMSGSIZE */
+	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
+		       &newbuffsiz, sizeof(socklen_t)) == 0)
+		return;

setsockopt() with SO_SNDBUF never fails: it trims the newbuffsiz that is specified by net.core.wmem_max

This needs to call:

	setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
		   &newbuffsiz, sizeof(socklen_t));

without checking the return value. Otherwise, SO_SNDBUFFORCE is never going to be called after this patch. This needs a v2.

On top of this patch, you still needed to increase net.core.wmem_max in your host container for this to work.

Quoted 9 lines
src/mnl.c +3 -0
+	/* If the above fails (probably because it exceeds
+	 * /proc/sys/net/core/wmem_max), try again with SO_SNDBUFFORCE.
+	 * This requires CAP_NET_ADMIN. */
 	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUFFORCE,
 		       &newbuffsiz, sizeof(socklen_t)) < 0)
 		return;
Signature · 1 line
2.20.1
#4 lore raw

Hi,

Quoted 15 lines · Pablo Neira Ayuso

src/mnl.c static void mnl_set_sndbuffer(const struct mnl_socket *nl, · line 265

Quoted 11 lines
src/mnl.c +3 -0
index 26f943db..ab6750c8 100644
@@ -260,6 +260,13 @@ static void mnl_set_sndbuffer(const struct mnl_socket *nl,
 		return;
 
 	/* Rise sender buffer length to avoid hitting -EMSGSIZE */
+	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
+		       &newbuffsiz, sizeof(socklen_t)) == 0)
+		return;

setsockopt() with SO_SNDBUF never fails: it trims the newbuffsiz that is specified by net.core.wmem_max

Quoted 7 lines

This needs to call:

	setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
		   &newbuffsiz, sizeof(socklen_t));

without checking the return value. Otherwise, SO_SNDBUFFORCE is never going to be called after this patch. This needs a v2.

I think this patch should be fine.

src/mnl.c +7 -2
index 26f943dbb4c8..ee62c0c9c2a0 100644
@@ -261,8 +261,13 @@ static void mnl_set_sndbuffer(const struct mnl_socket *nl,
 
 	/* Rise sender buffer length to avoid hitting -EMSGSIZE */
 	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUFFORCE,
-		       &newbuffsiz, sizeof(socklen_t)) < 0)
-		return;
+		       &newbuffsiz, sizeof(socklen_t)) < 0) {
+		/* Fall back to SO_SNDBUF, this never fails, kernel trims down
+		 * the size to net.core.wmem_max.
+		 */
+		setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
+			   &newbuffsiz, sizeof(socklen_t));
+	}
 }
 
 static unsigned int nlsndbufsiz;
author
#5 lore raw
In reply to Pablo Neira Ayuso
Quoted 2 lines · Pablo Neira Ayuso

setsockopt() with SO_SNDBUF never fails: it trims the newbuffsiz that is specified by net.core.wmem_max

Oh, good catch! Your revised patch LGTM, and is closer to what was being done in the immediately proceeding function, mnl_set_rcvbuffer.

However, after thinking about it, I feel we should be checking the receiver value after setsockopt returns. If someone is running e.g. AppArmor, it seems better to me to attempt the non-privileged operation first, to avoid adding noise in the logs.

Also, I don't think there are any current situations where SO_SNDBUFFORCE might also trim down the value, but after re-reading the man page, I'm not sure the contract precludes that in the future.

Attached is a V3 patch for consideration, which also changes the code to attempt the non-privileged SO_RCVBUF before SO_RCVBUFFORCE. I defer to your judgment on which version is actually better; I tested both and they both work a) in a container where SO_SNDBUFFORCE fails, and b) outside a container with wmem_max set to a small-ish value where SO_SNDBUFFORCE is required.

src/mnl.c +22 -11
index 26f943db..dcc22b82 100644
@@ -259,10 +259,19 @@ static void mnl_set_sndbuffer(const struct mnl_socket *nl,
 	if (newbuffsiz <= sndnlbuffsiz)
 		return;
 
-	/* Rise sender buffer length to avoid hitting -EMSGSIZE */
-	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUFFORCE,
-		       &newbuffsiz, sizeof(socklen_t)) < 0)
-		return;
+	/* Raise sender buffer length to avoid hitting -EMSGSIZE.  The kernel may
+	 * reduce this to /proc/sys/net/core/wmem_max, see socket(7).
+	 */
+	sndnlbuffsiz = newbuffsiz;
+	if (setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUF,
+		   &sndnlbuffsiz, sizeof(socklen_t)) < 0 || sndnlbuffsiz < newbuffsiz) {
+		/* If SO_SNDBUF failed or the resulting size is still too small, try
+		 * again with SO_SNDBUFFORCE.  This requires CAP_NET_ADMIN.
+		 */
+		sndnlbuffsiz = newbuffsiz;
+		setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_SNDBUFFORCE,
+		   &sndnlbuffsiz, sizeof(socklen_t));
+	}
 }
 
 static unsigned int nlsndbufsiz;
@@ -280,14 +289,16 @@ static int mnl_set_rcvbuffer(const struct mnl_socket *nl, socklen_t bufsiz)
 	if (nlsndbufsiz >= bufsiz)
 		return 0;
 
-	ret = setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_RCVBUFFORCE,
-			 &bufsiz, sizeof(socklen_t));
-	if (ret < 0) {
-		/* If this doesn't work, try to reach the system wide maximum
-		 * (or whatever the user requested).
+	nlsndbufsiz = bufsiz;
+	ret = setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_RCVBUF,
+			 &nlsndbufsiz, sizeof(socklen_t));
+	if (ret < 0 || nlsndbufsiz < bufsiz) {
+		/* If this doesn't work, try again with SO_RCVBUFFORCE.  This requires
+		 * CAP_NET_ADMIN.
 		 */
-		ret = setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_RCVBUF,
-				 &bufsiz, sizeof(socklen_t));
+		nlsndbufsiz = bufsiz;
+		ret = setsockopt(mnl_socket_get_fd(nl), SOL_SOCKET, SO_RCVBUFFORCE,
+				 &nlsndbufsiz, sizeof(socklen_t));
 	}
 
 	return ret;
Signature · 2 lines
-- 
Dave Pifke, [email protected]
#6 lore raw
Quoted 17 lines · Dave Pifke
Quoted 2 lines · Pablo Neira Ayuso

setsockopt() with SO_SNDBUF never fails: it trims the newbuffsiz that is specified by net.core.wmem_max

Oh, good catch! Your revised patch LGTM, and is closer to what was being done in the immediately proceeding function, mnl_set_rcvbuffer.

However, after thinking about it, I feel we should be checking the receiver value after setsockopt returns. If someone is running e.g. AppArmor, it seems better to me to attempt the non-privileged operation first, to avoid adding noise in the logs.

Also, I don't think there are any current situations where SO_SNDBUFFORCE might also trim down the value, but after re-reading the man page, I'm not sure the contract precludes that in the future.

Quoted 6 lines

Attached is a V3 patch for consideration, which also changes the code to attempt the non-privileged SO_RCVBUF before SO_RCVBUFFORCE. I defer to your judgment on which version is actually better; I tested both and they both work a) in a container where SO_SNDBUFFORCE fails, and b) outside a container with wmem_max set to a small-ish value where SO_SNDBUFFORCE is required.

Thanks for your patch.

setsockopt() does not update the &sndnlbuffsiz that is passed as argument in Linux.

I have posted this patch:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/