Skip to contentSkip to conversation

tcp_mmap: optmizations

PATCH In review Eric Dumazet 4 participants 7 messages Aug 20, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover tcp_mmap: optmizations

This series updates tcp_mmap reference tool to use best pratices.

First patch is using madvise(MADV_DONTNEED) to decrease pressure on the socket lock.

Last patches try to use huge pages when available.

Eric Dumazet (3):
selftests: net: tcp_mmap: use madvise(MADV_DONTNEED)
selftests: net: tcp_mmap: Use huge pages in send path
selftests: net: tcp_mmap: Use huge pages in receive path

 tools/testing/selftests/net/tcp_mmap.c | 42 +++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 7 deletions(-)
Signature · 1 line
2.28.0.297.g1956fa8f8d-goog
#1 lore raw

PATCH 1/3 selftests: net: tcp_mmap: use madvise(MADV_DONTNEED)

When TCP_ZEROCOPY_RECEIVE operation has been added, I made the mistake of automatically un-mapping prior content before mapping new pages.

This has the unfortunate effect of adding potentially long MMU operations (like TLB flushes) while socket lock is held.

Using madvise(MADV_DONTNEED) right after pages has been used has two benefits :

1) This releases pages sooner, allowing pages to be recycled if they were part of a page pool in a NIC driver.

2) No more long unmap operations while preventing immediate processing of incoming packets.

The cost of the added system call is small enough.

Arjun will submit a kernel patch allowing to opt out from the unmap attempt in tcp_zerocopy_receive()

Cc Arjun Roy <[email protected]>
Cc Soheil Hassas Yeganeh <[email protected]>
---
 tools/testing/selftests/net/tcp_mmap.c | 4 ++++
 1 file changed, 4 insertions(+)
tools/testing/selftests/net/tcp_mmap.c +4 -0
index a61b7b3da5496285876b0e16b18a3060850b0803..59ec0b59f7b76ff75685bd96901d8237e0665b2b 100644
@@ -179,6 +179,10 @@ void *child_thread(void *arg)
 				total_mmap += zc.length;
 				if (xflg)
 					hash_zone(addr, zc.length);
+				/* It is more efficient to unmap the pages right now,
+				 * instead of doing this in next TCP_ZEROCOPY_RECEIVE.
+				 */
+				madvise(addr, zc.length, MADV_DONTNEED);
 				total += zc.length;
 			}
 			if (zc.recv_skip_hint) {
Signature · 1 line
2.28.0.297.g1956fa8f8d-goog
#2 lore raw

PATCH 2/3 selftests: net: tcp_mmap: Use huge pages in send path

In reply to Eric Dumazet

There are significant gains using huge pages when available, as shown in [1].

This patch adds mmap_large_buffer() and uses it in client side (tx path of this reference tool)

Following patch will use the feature for server side.

[1] https://patchwork.ozlabs.org/project/netdev/patch/[email protected]/

Cc Arjun Roy <[email protected]>
Cc Soheil Hassas Yeganeh <[email protected]>
---
 tools/testing/selftests/net/tcp_mmap.c | 29 +++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)
tools/testing/selftests/net/tcp_mmap.c +26 -3
index 59ec0b59f7b76ff75685bd96901d8237e0665b2b..ca2618f3e7a12ab6863665f465dea2e8d469131b 100644
@@ -123,6 +123,28 @@ void hash_zone(void *zone, unsigned int length)
 #define ALIGN_UP(x, align_to)	(((x) + ((align_to)-1)) & ~((align_to)-1))
 #define ALIGN_PTR_UP(p, ptr_align_to)	((typeof(p))ALIGN_UP((unsigned long)(p), ptr_align_to))
 
+
+static void *mmap_large_buffer(size_t need, size_t *allocated)
+{
+	void *buffer;
+	size_t sz;
+
+	/* Attempt to use huge pages if possible. */
+	sz = ALIGN_UP(need, map_align);
+	buffer = mmap(NULL, sz, PROT_READ | PROT_WRITE,
+		      MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+
+	if (buffer == (void *)-1) {
+		sz = need;
+		buffer = mmap(NULL, sz, PROT_READ | PROT_WRITE,
+			      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+		if (buffer != (void *)-1)
+			fprintf(stderr, "MAP_HUGETLB attempt failed, look at /sys/kernel/mm/hugepages for optimal performance\n");
+	}
+	*allocated = sz;
+	return buffer;
+}
+
 void *child_thread(void *arg)
 {
 	unsigned long total_mmap = 0, total = 0;
@@ -351,6 +373,7 @@ int main(int argc, char *argv[])
 	uint64_t total = 0;
 	char *host = NULL;
 	int fd, c, on = 1;
+	size_t buffer_sz;
 	char *buffer;
 	int sflg = 0;
 	int mss = 0;
@@ -441,8 +464,8 @@ int main(int argc, char *argv[])
 		}
 		do_accept(fdlisten);
 	}
-	buffer = mmap(NULL, chunk_size, PROT_READ | PROT_WRITE,
-			      MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+
+	buffer = mmap_large_buffer(chunk_size, &buffer_sz);
 	if (buffer == (char *)-1) {
 		perror("mmap");
 		exit(1);
@@ -488,6 +511,6 @@ int main(int argc, char *argv[])
 		total += wr;
 	}
 	close(fd);
-	munmap(buffer, chunk_size);
+	munmap(buffer, buffer_sz);
 	return 0;
 }
Signature · 1 line
2.28.0.297.g1956fa8f8d-goog
#3 lore raw

PATCH 3/3 selftests: net: tcp_mmap: Use huge pages in receive path

In reply to Eric Dumazet

One down side of using TCP rx zerocopy is one extra TLB miss per page after the mapping operation.

While if the application is using hugepages, the non zerocopy recvmsg() will not have to pay these TLB costs.

This patch allows server side to use huge pages for the non zero copy case, to allow fair comparisons when both solutions use optimal conditions.

Cc Arjun Roy <[email protected]>
Cc Soheil Hassas Yeganeh <[email protected]>
---
 tools/testing/selftests/net/tcp_mmap.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
tools/testing/selftests/net/tcp_mmap.c +5 -4
index ca2618f3e7a12ab6863665f465dea2e8d469131b..00f837c9bc6c4549c19dfc27aa2d08c454ea169e 100644
@@ -157,6 +157,7 @@ void *child_thread(void *arg)
 	void *addr = NULL;
 	double throughput;
 	struct rusage ru;
+	size_t buffer_sz;
 	int lu, fd;
 
 	fd = (int)(unsigned long)arg;
@@ -164,9 +165,9 @@ void *child_thread(void *arg)
 	gettimeofday(&t0, NULL);
 
 	fcntl(fd, F_SETFL, O_NDELAY);
-	buffer = malloc(chunk_size);
-	if (!buffer) {
-		perror("malloc");
+	buffer = mmap_large_buffer(chunk_size, &buffer_sz);
+	if (buffer == (void *)-1) {
+		perror("mmap");
 		goto error;
 	}
 	if (zflg) {
@@ -256,7 +257,7 @@ void *child_thread(void *arg)
 				ru.ru_nvcsw);
 	}
 error:
-	free(buffer);
+	munmap(buffer, buffer_sz);
 	close(fd);
 	if (zflg)
 		munmap(raddr, chunk_size + map_align);
Signature · 1 line
2.28.0.297.g1956fa8f8d-goog
Acked-by on the v1 series
#4 lore raw
In reply to Eric Dumazet
Quoted 6 lines · Eric Dumazet

This series updates tcp_mmap reference tool to use best pratices.

First patch is using madvise(MADV_DONTNEED) to decrease pressure on the socket lock.

Quoted 6 lines

Last patches try to use huge pages when available.

Eric Dumazet (3):
selftests: net: tcp_mmap: use madvise(MADV_DONTNEED)
selftests: net: tcp_mmap: Use huge pages in send path
selftests: net: tcp_mmap: Use huge pages in receive path

Thank you for the patches!

Quoted 6 lines
 tools/testing/selftests/net/tcp_mmap.c | 42 +++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 7 deletions(-)

--
2.28.0.297.g1956fa8f8d-goog

Acked-by
#5 lore raw
Quoted 10 lines · Soheil Hassas Yeganeh
Quoted 8 lines · Eric Dumazet

This series updates tcp_mmap reference tool to use best pratices.

First patch is using madvise(MADV_DONTNEED) to decrease pressure on the socket lock.

Last patches try to use huge pages when available.

Quoted 9 lines
Quoted 4 lines

Eric Dumazet (3):
selftests: net: tcp_mmap: use madvise(MADV_DONTNEED)
selftests: net: tcp_mmap: Use huge pages in send path
selftests: net: tcp_mmap: Use huge pages in receive path

Thank you for the patches!

-Arjun

Quoted 6 lines
Quoted 6 lines
 tools/testing/selftests/net/tcp_mmap.c | 42 +++++++++++++++++++++-----
 1 file changed, 35 insertions(+), 7 deletions(-)

--
2.28.0.297.g1956fa8f8d-goog

maintainer
#6 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 20 Aug 2020 10:11:15 -0700

Quoted 6 lines

This series updates tcp_mmap reference tool to use best pratices.

First patch is using madvise(MADV_DONTNEED) to decrease pressure on the socket lock.

Last patches try to use huge pages when available.

Series applied, thanks Eric.