Skip to contentSkip to conversation

tcp: Use per-vma locking for receive zerocopy

PATCH Applied Arjun Roy 4 participants 4 messages Jun 16, 2023 – Jul 7, 2023

Subsystems (5)
lore
Share & tools
lore

Arjun Roy

author
#0 lore raw

PATCH v2 1/1 tcp: Use per-vma locking for receive zerocopy

From: Arjun Roy <[email protected]>

Per-VMA locking allows us to lock a struct vm_area_struct without taking the process-wide mmap lock in read mode.

Consider a process workload where the mmap lock is taken constantly in write mode. In this scenario, all zerocopy receives are periodically blocked during that period of time - though in principle, the memory ranges being used by TCP are not touched by the operations that need the mmap write lock. This results in performance degradation.

Now consider another workload where the mmap lock is never taken in write mode, but there are many TCP connections using receive zerocopy that are concurrently receiving. These connections all take the mmap lock in read mode, but this does induce a lot of contention and atomic ops for this process-wide lock. This results in additional CPU overhead caused by contending on the cache line for this lock.

However, with per-vma locking, both of these problems can be avoided.

As a test, I ran an RPC-style request/response workload with 4KB payloads and receive zerocopy enabled, with 100 simultaneous TCP connections. I measured perf cycles within the find_tcp_vma/mmap_read_lock/mmap_read_unlock codepath, with and without per-vma locking enabled.

When using process-wide mmap semaphore read locking, about 1% of measured perf cycles were within this path. With per-VMA locking, this value dropped to about 0.45%.

---
v2 change:
Fixed linker error on builds without CONFIG_INET.

---
 MAINTAINERS            |  1 +
 include/linux/net_mm.h | 17 ++++++++++++++++
 include/net/tcp.h      |  1 +
 mm/memory.c            |  7 ++++---
 net/ipv4/tcp.c         | 45 ++++++++++++++++++++++++++++++++++--------
 5 files changed, 60 insertions(+), 11 deletions(-)
 create mode 100644 include/linux/net_mm.h
MAINTAINERS +1 -0
index c6fa6ed454f4..a7c495e3323b 100644
@@ -14727,6 +14727,7 @@ NETWORKING [TCP]
 M:	Eric Dumazet <[email protected]>
 L:	[email protected]
 S:	Maintained
+F:	include/linux/net_mm.h
 F:	include/linux/tcp.h
 F:	include/net/tcp.h
 F:	include/trace/events/tcp.h
include/linux/net_mm.h new file +17 -0
new file mode 100644
index 000000000000..b298998bd5a0
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifdef CONFIG_MMU
+
+#ifdef CONFIG_INET
+extern const struct vm_operations_struct tcp_vm_ops;
+static inline bool vma_is_tcp(const struct vm_area_struct *vma)
+{
+	return vma->vm_ops == &tcp_vm_ops;
+}
+#else
+static inline bool vma_is_tcp(const struct vm_area_struct *vma)
+{
+	return false;
+}
+#endif /* CONFIG_INET*/
+
+#endif /* CONFIG_MMU */
include/net/tcp.h +1 -0
index 5066e4586cf0..bfa5e27205ba 100644
@@ -45,6 +45,7 @@
 #include <linux/memcontrol.h>
 #include <linux/bpf-cgroup.h>
 #include <linux/siphash.h>
+#include <linux/net_mm.h>
 
 extern struct inet_hashinfo tcp_hashinfo;
 
mm/memory.c +4 -3
index f69fbc251198..3e46b4d881dc 100644
@@ -77,6 +77,7 @@
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
 #include <linux/sched/sysctl.h>
+#include <linux/net_mm.h>
 
 #include <trace/events/kmem.h>
 
@@ -5280,12 +5281,12 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	if (!vma)
 		goto inval;
 
-	/* Only anonymous vmas are supported for now */
-	if (!vma_is_anonymous(vma))
+	/* Only anonymous and tcp vmas are supported for now */
+	if (!vma_is_anonymous(vma) && !vma_is_tcp(vma))
 		goto inval;
 
 	/* find_mergeable_anon_vma uses adjacent vmas which are not locked */
-	if (!vma->anon_vma)
+	if (!vma->anon_vma && !vma_is_tcp(vma))
 		goto inval;
 
 	if (!vma_start_read(vma))
net/ipv4/tcp.c +37 -8

76 lines not on this page · raw message

Signature · 1 line
2.41.0.162.gfafddb0af9-goog
#1 lore raw
This patch was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Hello:

This patch was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:

Quoted 5 lines · On Fri, 16 Jun 2023 12:34:27 -0700 you wrote:

From: Arjun Roy <[email protected]>

Per-VMA locking allows us to lock a struct vm_area_struct without taking the process-wide mmap lock in read mode.

Quoted 7 lines

Consider a process workload where the mmap lock is taken constantly in write mode. In this scenario, all zerocopy receives are periodically blocked during that period of time - though in principle, the memory ranges being used by TCP are not touched by the operations that need the mmap write lock. This results in performance degradation.

[...]

Here is the summary with links:
- [net-next,v2] tcp: Use per-vma locking for receive zerocopy

    https://git.kernel.org/netdev/net-next/c/7a7f09463534

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
#2 lore raw
In reply to Arjun Roy
Quoted 1 line · Arjun Roy

However, with per-vma locking, both of these problems can be avoided.

I appreciate your enthusiasm for this. However, applying this patch completely wrecks my patch series to push per-vma locking down for file-backed mappings. It would be helpful if we can back this out for now and then apply it after that patch series.

Would it make life hard for this patch to go through the mm tree?

#3 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 1 line · Arjun Roy

However, with per-vma locking, both of these problems can be avoided.

I appreciate your enthusiasm for this. However, applying this patch completely wrecks my patch series to push per-vma locking down for file-backed mappings. It would be helpful if we can back this out for now and then apply it after that patch series.

Would it make life hard for this patch to go through the mm tree?

No worries, can you send a formal revert then ?

(With some details, because I do not see why you can not simply add the revert in front of your series ?)

Thanks.