Sockperf: Initial multi-threaded throughput client

RFC PATCH In review Doug Ledford 2 participants 6 messages Dec 18, 2017 – Dec 20, 2017 1 message not in the archive InfiniBand

lore

Share & tools

lore

Doug Ledford

author

2017-12-18 15:26 UTC #0 lore raw

cover Sockperf: Initial multi-threaded throughput client

During testing, it has become painfully clear that a single threaded UDP test client can not exercise a 100Gig link due to issues related to single core maximum throughput. This patchset implements a multi-threaded throughput test in sockperf. This is just an initial implementation, there is still more work to be done. In particular:

1) Although the speed improved with this change, it did not improve drastically. As soon as the client send bottleneck was removed, it became clear there is another bottleneck on the server. When sending to a server from one client, all data is received on a single queue pair, and due to how interrupts are spread in the RDMA stack (namely that each queue pair goes to a single interrupt and we rely on multiple queue pairs being in use to balance interrupts across different cores), we take all interrupts from a specific host on a single core and the receiving side then becomes the bottleneck with single core IPoIB receive processing being the limiting factor. On a slower machine, I clocked 30GBit/s throughput. On a faster machine as the server, I was able to get up to 70GBit/s throughput.

2) I thought I might try an experiment to get around the queue pair is on one CPU issue. We use P_Keys in our internal lab setup, and so on the specific link in question, I actually have a total of three different IP interfaces on different P_Keys. I tried to open tests on multiple of these interfaces to see how that would impact performance (so a multithreaded server listening on ports on three different P_Key interfaces all on the same physical link, which should use three different queue pairs, and a multithreaded client sending to those three different P_Key interfaces from three different P_Key interfaces of its own). It tanked it. Like less than gigabit ethernet speeds. This warrants some investigation moving forward I think.

3) I thought I might try sending from two clients to the server at once and summing their throughput. That was fun. With UDP the clients are able to send enough data that flow control on the link kicks in, at which point each client starts dropping packets on the floor (they're UDP after all), and so the net result is that one client claimed 200GBit/s and the other about 175GBit/s. Meanwhile, the server thought we were just kidding and didn't actually run a test at all.

4) I reran the test using TCP instead of UDP. That's a non-starter. Whether due to my changes, or just because it is the way it is, the TCP tests all failed. For larger message sizes, they failed instantly. For smaller message sizes the test might run for a few seconds, but would eventually fail too. Always the failure was that the server would get a message it deemed too large and would forcibly close all of the TCP connections, at which point the client just bails.

I should point out that I don't program C++. Issues with me not doing these patches in a C++ typical manner are related to that.

Doug Ledford (4):
Rename a few variables
Move num-threads and cpu-affinity to common opts Move server thread handler to SockPerf.cpp Initial implementation of threaded throughput client

 src/Client.cpp   | 140 +++++++++++++++---------
 src/Client.h     |   3 +-
 src/Defs.h       |  10 +-
 src/Server.cpp   | 137 +----------------------
 src/SockPerf.cpp | 324 ++++++++++++++++++++++++++++++++++++++++++-------------
 5 files changed, 357 insertions(+), 257 deletions(-)

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): cover.1513609601.git.dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org

Doug Ledford

author

2017-12-18 15:26 UTC #2 lore raw

PATCH 1/4 Rename a few variables

In preparation for making sockperf use multiple threads both in server and client mode, rename a few variables related to the multi-thread operation that had hints that they only applied to server mode to more generic names that don't provide what will soon be a false hint.

Signed-off-by Doug Ledford <[email protected]>

---
 src/Defs.h       |  2 +-
 src/Server.cpp   |  4 ++--
 src/SockPerf.cpp | 58 ++++++++++++++++++++++++++++----------------------------
 3 files changed, 32 insertions(+), 32 deletions(-)

src/Defs.h +1 -1

index 51098e8ba098..0ee0f1e229c7 100644
@@ -590,7 +590,7 @@ struct user_params_t {
 	unsigned int  burst_size;
 	bool packetrate_stats_print_details;
 //	bool stream_mode; - use b_stream instead
-	int mthread_server;
+	int mthread;
 	struct timeval* select_timeout;
 	int sock_buff_size;
 	int threads_num;

src/Server.cpp +2 -2

index efc0222197bf..adbc29f07b65 100644
@@ -146,7 +146,7 @@ void ServerBase::cleanupAfterLoop() {
 	// cleanup
 	log_dbg("thread %lu released allocations",(unsigned long)os_getthread().tid);
 
-	if (!g_pApp->m_const_params.mthread_server) {
+	if (!g_pApp->m_const_params.mthread) {
 		log_msg("%s() exit", __func__);
 	}
 }

@@ -461,7 +461,7 @@ void server_sig_handler(int signum) {
 	   (g_pApp->m_const_params.packetrate_stats_print_ratio < g_receiveCount))
 		printf("\n");
 
-	if (g_pApp->m_const_params.mthread_server) {
+	if (g_pApp->m_const_params.mthread) {
 		if (os_getthread().tid == thread_pid_array[0].tid) {  //main thread
 			if (g_debug_level >= LOG_LVL_DEBUG) {
 				log_dbg("Main thread %lu got signal %d - exiting",(unsigned long)os_getthread().tid,signum);

src/SockPerf.cpp +29 ~~-29~~

165 lines not on this page · raw message

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Doug Ledford

author

2017-12-18 15:26 UTC #3 lore raw

PATCH 2/4 Move num-threads and cpu-affinity to common opts

In reply to a message not in the archive

The option settings are split into common, server, client (which is really client common), and client mode specific settings. Move the two listed options to the common settings. This does not make the options *work* for clients yet, just makes them accepted for clients. With the exception of changing the help comment for num-threads, this made no other change except to move the code location.

Signed-off-by Doug Ledford <[email protected]>

---
 src/SockPerf.cpp | 90 ++++++++++++++++++++++++++++----------------------------
 1 file changed, 45 insertions(+), 45 deletions(-)

src/SockPerf.cpp +45 ~~-45~~

index a7bc55e6bbc8..7f8d6969e645 100644
@@ -169,6 +169,14 @@ static const AOPT_DESC  common_opt_desc[] =
 		'f', AOPT_ARG, aopt_set_literal( 'f' ), aopt_set_string( "file" ),
 		"Read multiple ip+port combinations from file <file> (will use IO muxer '-F' such as epoll, poll or select)"
 	},
+	{
+		OPT_THREADS_NUM, AOPT_ARG, aopt_set_literal( 0 ), aopt_set_string( "threads-num" ),
+		"Run <N> threads to process sockets (requires '-f' option)."
+	},
+	{
+		OPT_THREADS_AFFINITY, AOPT_ARG,	aopt_set_literal( 0 ),	aopt_set_string( "cpu-affinity" ),
+		"Set threads affinity to the given core ids in list format (see: cat /proc/cpuinfo)."
+	},
 	{
 		'F', AOPT_ARG, aopt_set_literal( 'F' ), aopt_set_string( "iomux-type" ),
 #ifdef WIN32

@@ -1421,14 +1429,6 @@ static int proc_mode_server( int id, int argc, const char **argv )
 			"Run in Bridge mode."
 		}, 
 */		
-		{
-			 OPT_THREADS_NUM, AOPT_ARG, aopt_set_literal( 0 ), aopt_set_string( "threads-num" ),
-			 "Run <N> threads on server side (requires '-f' option)."
-		 },
-		 {
-			 OPT_THREADS_AFFINITY, AOPT_ARG,	aopt_set_literal( 0 ),	aopt_set_string( "cpu-affinity" ),
-			 "Set threads affinity to the given core ids in list format (see: cat /proc/cpuinfo)."
-		 },
 #ifndef WIN32
 		 {
 			 OPT_VMARXFILTERCB, AOPT_NOARG,	aopt_set_literal( 0 ),	aopt_set_string( "vmarxfiltercb" ),

@@ -1493,43 +1493,6 @@ static int proc_mode_server( int id, int argc, const char **argv )
 			s_user_params.mode = MODE_BRIDGE;
 			p_addr->sin_port = htons(5001); /*iperf's default port*/
 		}
-
-		if ( !rc && aopt_check(server_obj, OPT_THREADS_NUM) ) {
-			if (aopt_check(common_obj, 'f')) {
-				const char* optarg = aopt_value(server_obj, OPT_THREADS_NUM);
-				if (optarg) {
-					s_user_params.mthread = 1;
-					errno = 0;
-					int threads_num = strtol(optarg, NULL, 0);
-					if (errno != 0  || threads_num <= 0) {
-						log_msg("'-%d' Invalid threads number: %s", OPT_THREADS_NUM, optarg);
-						rc = SOCKPERF_ERR_BAD_ARGUMENT;
-					}
-					else {
-						s_user_params.threads_num = threads_num;
-					}
-				}
-				else {
-					log_msg("'-%d' Invalid value", OPT_THREADS_NUM);
-					rc = SOCKPERF_ERR_BAD_ARGUMENT;
-				}
-			}
-			else {
-				log_msg("--threads-num must be used with feed file (option '-f')");
-				rc = SOCKPERF_ERR_BAD_ARGUMENT;
-			}
-		}
-
-		if ( !rc && aopt_check(server_obj, OPT_THREADS_AFFINITY) ) {
-			const char* optarg = aopt_value(server_obj, OPT_THREADS_AFFINITY);
-			if (optarg) {
-				strcpy(s_user_params.threads_affinity, optarg);
-			}
-			else {
-				log_msg("'-%d' Invalid threads affinity", OPT_THREADS_AFFINITY);
-				rc = SOCKPERF_ERR_BAD_ARGUMENT;
-			}
-		}
 #ifndef WIN32
 		if ( !rc && aopt_check(server_obj, OPT_VMARXFILTERCB) ) {
 			s_user_params.is_vmarxfiltercb = true;

@@ -1691,6 +1654,43 @@ static int parse_common_opt( const AOPT_OBJECT *common_obj )
 			}
 		}
 
+		if ( !rc && aopt_check(common_obj, OPT_THREADS_NUM) ) {
+			if (aopt_check(common_obj, 'f')) {
+				const char* optarg = aopt_value(common_obj, OPT_THREADS_NUM);
+				if (optarg) {
+					s_user_params.mthread = 1;
+					errno = 0;
+					int threads_num = strtol(optarg, NULL, 0);
+					if (errno != 0  || threads_num <= 0) {
+						log_msg("'-%d' Invalid threads number: %s", OPT_THREADS_NUM, optarg);
+						rc = SOCKPERF_ERR_BAD_ARGUMENT;
+					}
+					else {
+						s_user_params.threads_num = threads_num;
+					}
+				}
+				else {
+					log_msg("'-%d' Invalid value", OPT_THREADS_NUM);
+					rc = SOCKPERF_ERR_BAD_ARGUMENT;
+				}
+			}
+			else {
+				log_msg("--threads-num must be used with feed file (option '-f')");
+				rc = SOCKPERF_ERR_BAD_ARGUMENT;
+			}
+		}
+
+		if ( !rc && aopt_check(common_obj, OPT_THREADS_AFFINITY) ) {
+			const char* optarg = aopt_value(common_obj, OPT_THREADS_AFFINITY);
+			if (optarg) {
+				strcpy(s_user_params.threads_affinity, optarg);
+			}
+			else {
+				log_msg("'-%d' Invalid threads affinity", OPT_THREADS_AFFINITY);
+				rc = SOCKPERF_ERR_BAD_ARGUMENT;
+			}
+		}
+
 		if ( !rc && aopt_check(common_obj, 'F') ) {
 			if (aopt_check(common_obj, 'f')) {
 				const char* optarg = aopt_value(common_obj, 'F');

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Doug Ledford

author

2017-12-18 15:26 UTC #4 lore raw

PATCH 3/4 Move server thread handler to SockPerf.cpp

In reply to a message not in the archive

In order to use the thread starting code as common, move it to SockPerf.cpp and make it no longer specific to the server. Next we'll add a Client handler and call that from the common code.

Signed-off-by Doug Ledford <[email protected]>

---
 src/Server.cpp   | 133 ++----------------------------------------------------
 src/SockPerf.cpp | 135 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 136 insertions(+), 132 deletions(-)

src/Server.cpp +3 ~~-130~~

index adbc29f07b65..f6d79a245ff4 100644
@@ -32,8 +32,8 @@
 
 // static members initialization
 /*static*/ seq_num_map SwitchOnCalcGaps::ms_seq_num_map;
-static CRITICAL_SECTION	thread_exit_lock;
-static os_thread_t *thread_pid_array = NULL;
+extern CRITICAL_SECTION	thread_exit_lock;
+extern os_thread_t *thread_pid_array;
 
 //==============================================================================

@@ -407,7 +407,7 @@ void server_handler(handler_info *p_info)
 
 
 //------------------------------------------------------------------------------
-void *server_handler_for_multi_threaded(void *arg)
+void *server_handler_multi_thread(void *arg)
 {
 	handler_info *p_info = (handler_info *)arg;

@@ -431,23 +431,6 @@ void *server_handler_for_multi_threaded(void *arg)
 }
 
 
-//------------------------------------------------------------------------------
-void find_min_max_fds(int start_look_from, int len, int* p_fd_min, int* p_fd_max) {
-	int num_of_detected_fds;
-	int i;
-
-	for(num_of_detected_fds = 0, i = start_look_from; num_of_detected_fds < len;i++) {
-		if (g_fds_array[i]) {
-			if (!num_of_detected_fds) {
-				*p_fd_min = i;
-			}
-			num_of_detected_fds++;
-		}
-	}
-	*p_fd_max = i - 1;
-}
-
-
 //------------------------------------------------------------------------------
 void server_sig_handler(int signum) {
 	if (g_b_exit) {

@@ -497,116 +480,6 @@ void server_sig_handler(int signum) {
 }
 
 
-//------------------------------------------------------------------------------
-void server_select_per_thread(int _fd_num) {
-	int rc = SOCKPERF_ERR_NONE;
-	int i;
-	os_thread_t thread;
-	int fd_num;
-	int num_of_remainded_fds;
-	int last_fds = 0;
-	handler_info *handler_info_array = NULL;
-
-	handler_info_array = (handler_info*)MALLOC(sizeof(handler_info) * g_pApp->m_const_params.threads_num);
-	memset(handler_info_array, 0, sizeof(handler_info) * g_pApp->m_const_params.threads_num);
-	if (!handler_info_array) {
-		log_err("Failed to allocate memory for handler_info_arr");
-		rc = SOCKPERF_ERR_NO_MEMORY;
-	}
-
-	if (rc == SOCKPERF_ERR_NONE) {
-		thread_pid_array = (os_thread_t*)MALLOC(sizeof(os_thread_t)*(g_pApp->m_const_params.threads_num + 1));
-		if(!thread_pid_array) {
-			log_err("Failed to allocate memory for pid array");
-			rc = SOCKPERF_ERR_NO_MEMORY;
-		}
-		else {
-			memset(thread_pid_array, 0, sizeof(os_thread_t)*(g_pApp->m_const_params.threads_num + 1));
-			log_msg("Running %d threads to manage %d sockets", g_pApp->m_const_params.threads_num, _fd_num);
-		}
-	}
-
-	if (rc == SOCKPERF_ERR_NONE) {
-		INIT_CRITICAL(&thread_exit_lock);
-
-		thread_pid_array[0].tid = os_getthread().tid;
-
-		/* Divide fds_arr between threads */
-		num_of_remainded_fds = _fd_num % g_pApp->m_const_params.threads_num;
-		fd_num = _fd_num / g_pApp->m_const_params.threads_num;
-
-		for (i = 0; i < g_pApp->m_const_params.threads_num; i++) {
-			handler_info *cur_handler_info = (handler_info_array + i);
-
-			/* Set ID of handler (thread) */
-			cur_handler_info->id = i;
-
-			/* Set number of processed sockets */
-			cur_handler_info->fd_num = fd_num;
-			if (num_of_remainded_fds) {
-				cur_handler_info->fd_num++;
-				num_of_remainded_fds--;
-			}
-
-			/* Set min/max possible socket to be processed */
-			find_min_max_fds(last_fds, cur_handler_info->fd_num, &(cur_handler_info->fd_min), &(cur_handler_info->fd_max));
-
-			/* Launch handler */
-			errno = 0;
-			int ret = os_thread_exec(&thread, server_handler_for_multi_threaded, (void *)cur_handler_info);
-
-			/*
-			 * There is undocumented behaviour for early versions of libc (for example libc 2.5, 2.6, 2.7)
-			 * as pthread_create() call returns error code 12 ENOMEM and return value 0
-			 * Note: libc-2.9 demonstrates expected behaivour
-			 */
-			if ( (ret != 0) || (errno == ENOMEM) ) {
-				log_err("create thread has failed");
-				rc = SOCKPERF_ERR_FATAL;
-				break;
-			}
-			thread_pid_array[i + 1].tid = thread.tid;
-			last_fds = cur_handler_info->fd_max + 1;
-		}
-
-		/* Wait for ^C */
-		while ((rc == SOCKPERF_ERR_NONE) && !g_b_exit) {
-			sleep(1);
-		}
-
-		/* Stop all launched threads (the first index is reserved for main thread) */
-		for (i = 1; i <= g_pApp->m_const_params.threads_num; i++) {
-			os_thread_t cur_thread_pid;
-			cur_thread_pid.tid = 0;
-
-			ENTER_CRITICAL(&thread_exit_lock);
-			cur_thread_pid.tid = thread_pid_array[i].tid;
-			if (cur_thread_pid.tid) {
-				os_thread_kill(&cur_thread_pid);
-			}
-			LEAVE_CRITICAL(&thread_exit_lock);
-			if (cur_thread_pid.tid) {
-				os_thread_join(&cur_thread_pid);
-			}
-		}
-
-		DELETE_CRITICAL(&thread_exit_lock);
-	}
-
-	/* Free thread info allocated data */
-	if (handler_info_array) {
-		FREE(handler_info_array);
-	}
-
-	/* Free thread TID array */
-	if (thread_pid_array) {
-		FREE(thread_pid_array);
-	}
-
-	log_msg("%s() exit", __func__);
-}
-
-
 // Temp location because of compilation issue (inline-unit-growth=200) with the way this method was inlined
 void SwitchOnCalcGaps::execute(struct sockaddr_in *clt_addr, uint64_t seq_num, bool is_warmup)
 {

src/SockPerf.cpp +133 -2

158 lines not on this page · raw message

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Doug Ledford

author

2017-12-18 15:26 UTC #5 lore raw

PATCH 4/4 Initial implementation of threaded throughput client

In reply to a message not in the archive

We add fields to the handler_info struct and use that to store the results of individual threads since the thread destructor would otherwise delete our performance data before we could sum it up for a total. Otherwise, we reuse most of the implementation of the server thread code.

Signed-off-by Doug Ledford <[email protected]>

---
 src/Client.cpp   | 140 ++++++++++++++++++++++++++++++++++++-------------------
 src/Client.h     |   3 +-
 src/Defs.h       |   8 ++++
 src/SockPerf.cpp |  53 +++++++++++++++++++--
 4 files changed, 150 insertions(+), 54 deletions(-)

src/Client.cpp +92 ~~-48~~

index f8c1fdd11256..0edf9c36c3fc 100644
@@ -33,7 +33,9 @@
 #include "PacketTimes.h"
 #include "Switches.h"
 
-TicksTime s_startTime, s_endTime;
+extern CRITICAL_SECTION	thread_exit_lock;
+extern os_thread_t *thread_pid_array;
+TicksTime g_c_startTime, g_c_endTime;
 
 //==============================================================================
 //==============================================================================

@@ -119,7 +121,7 @@ void client_statistics(int serverNo, Message *pMsgRequest)
 
 	/* Print total statistic that is independent on server count */
 	if (SERVER_NO == 0) {
-		TicksDuration totalRunTime = s_endTime - s_startTime;
+		TicksDuration totalRunTime = g_c_endTime - g_c_startTime;
 		if (g_skipCount) {
 			log_msg_file2(f, "[Total Run] RunTime=%.3lf sec; Warm up time=%" PRIu32 " msec; SentMessages=%" PRIu64 "; ReceivedMessages=%" PRIu64 "; SkippedMessages=%" PRIu64 "",
 				totalRunTime.toDecimalUsec()/1000000, g_pApp->m_const_params.warmup_msec, sendCount, receiveCount, g_skipCount);

@@ -261,24 +263,34 @@ void client_statistics(int serverNo, Message *pMsgRequest)
 }
 
 //------------------------------------------------------------------------------
-void stream_statistics(Message *pMsgRequest)
+void stream_statistics(struct handler_info *p_info)
 {
-	TicksDuration totalRunTime = s_endTime - s_startTime;
+	TicksDuration totalRunTime = p_info->c_endTime - p_info->c_startTime;
+	uint64_t sendCount = p_info->sendCount;
+	char prefix[20];
+
+	if (g_pApp->m_const_params.mthread) {
+		if (p_info->id)
+			snprintf(prefix, sizeof prefix, "[TID: %d] ", p_info->id);
+		else
+			snprintf(prefix, sizeof prefix, "[TID: ALL] ");
+	}
+	else {
+		prefix[0] = '\0';
+	}
 
 	if (totalRunTime <= TicksDuration::TICKS0) return;
 	if (!g_pApp->m_const_params.b_stream) return;
 
-	const uint64_t sendCount = pMsgRequest->getSequenceCounter();
-
 	// Send only mode!
 	if (g_skipCount) {
-		log_msg("Total of %" PRIu64 " messages sent in %.3lf sec (%" PRIu64 " messages skipped)\n",
-				sendCount, totalRunTime.toDecimalUsec()/1000000, g_skipCount);
+		log_msg("%sTotal of %" PRIu64 " messages sent in %.3lf sec (%" PRIu64 " messages skipped)\n",
+				prefix, sendCount, totalRunTime.toDecimalUsec()/1000000, g_skipCount);
 	}
 	else 
 	{
-		log_msg("Total of %" PRIu64 " messages sent in %.3lf sec\n",
-				sendCount, totalRunTime.toDecimalUsec()/1000000);
+		log_msg("%sTotal of %" PRIu64 " messages sent in %.3lf sec\n",
+				prefix, sendCount, totalRunTime.toDecimalUsec()/1000000);
 	}
 	if (g_pApp->m_const_params.mps != MPS_MAX) {
 		if (g_pApp->m_const_params.msg_size_range)

@@ -308,17 +320,17 @@ void stream_statistics(Message *pMsgRequest)
 	int total_line_ip_data = g_pApp->m_const_params.msg_size;
 	double MBps = ((double)msgps * total_line_ip_data)/1024/1024; /* No including IP + UDP Headers per fragment */
 	if (ip_frags_per_msg == 1)
-		log_msg("Summary: Message Rate is %d [msg/sec]", msgps);
+		log_msg("%sSummary: Message Rate is %d [msg/sec]", prefix, msgps);
 	else
-		log_msg("Summary: Message Rate is %d [msg/sec], Packet Rate is about %d [pkt/sec] (%d ip frags / msg)", msgps, pktps, ip_frags_per_msg);
+		log_msg("%sSummary: Message Rate is %d [msg/sec], Packet Rate is about %d [pkt/sec] (%d ip frags / msg)", prefix, msgps, pktps, ip_frags_per_msg);
 	if (g_pApp->m_const_params.giga_size){
-		log_msg("Summary: BandWidth is %.3f GBps (%.3f Gbps)", MBps/1000, MBps*8/1000);
+		log_msg("%sSummary: BandWidth is %.3f GBps (%.3f Gbps)", prefix, MBps/1000, MBps*8/1000);
 	}
 	else if (g_pApp->m_const_params.increase_output_precision){
-			log_msg("Summary: BandWidth is %.9f GBps (%.9f Gbps)", MBps, MBps*8);
+			log_msg("%sSummary: BandWidth is %.9f GBps (%.9f Gbps)", prefix, MBps, MBps*8);
 	}
 	else{
-		log_msg("Summary: BandWidth is %.3f MBps (%.3f Mbps)", MBps, MBps*8);
+		log_msg("%sSummary: BandWidth is %.3f MBps (%.3f Mbps)", prefix, MBps, MBps*8);
 	}
 }

@@ -329,7 +341,7 @@ void client_sig_handler(int signum)
 		log_msg("Test end (interrupted by signal %d)", signum);
 		return;
 	}
-	s_endTime.setNowNonInline();
+	g_c_endTime.setNowNonInline();
 	g_b_exit = true;
 
 	// Just in case not Activity updates where logged add a '\n'

@@ -370,13 +382,15 @@ ClientBase::~ClientBase()
 	delete m_pMsgRequest;
 }
 
+
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo, class SwitchCycleDuration, class SwitchMsgSize , class PongModeCare >
-Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize , PongModeCare>::Client(int _fd_min, int _fd_max, int _fd_num):
+Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize , PongModeCare>::Client(struct handler_info *_p_info):
 	ClientBase(),
-	m_ioHandler(_fd_min, _fd_max, _fd_num),
+	m_ioHandler(_p_info->fd_min, _p_info->fd_max, _p_info->fd_num),
 	m_pongModeCare(m_pMsgRequest)
 {
+	p_info = _p_info;
 	os_thread_init (&m_receiverTid);
 }

@@ -408,6 +422,8 @@ template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo, cla
 void Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize , PongModeCare>
 ::cleanupAfterLoop()
 {
+	p_info->c_endTime.setNowNonInline();
+
 	usleep(100*1000);//0.1 sec - wait for rx packets for last sends (in normal flow)
 	if (m_receiverTid.tid) {
 		os_thread_kill(&m_receiverTid);

@@ -426,7 +442,9 @@ void Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration
 	}
 	else if (g_pApp->m_const_params.b_stream)
 	{
-		stream_statistics(m_pMsgRequest);
+		p_info->sendCount = m_pMsgRequest->getSequenceCounter();
+
+		stream_statistics(p_info);
 	}
 	else
 	{

@@ -581,9 +599,12 @@ int Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration,
 					}
 
 					if (rc == SOCKPERF_ERR_NONE) {
-						s_startTime.setNowNonInline();
-						g_lastTicks = s_startTime;
-						g_cycleStartTime = s_startTime - g_pApp->m_const_params.cycleDuration;
+						p_info->c_startTime.setNowNonInline();
+						if (g_c_startTime == TicksTime::TICKS0) {
+							g_c_startTime = p_info->c_startTime;
+							g_lastTicks = p_info->c_startTime;
+							g_cycleStartTime = p_info->c_startTime - g_pApp->m_const_params.cycleDuration;
+						}
 					}
 				}
 			}

@@ -635,7 +656,7 @@ void Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration
 ::doPlayback()
 {
 	usleep(100*1000);//wait for receiver thread to start (since we don't use warmup) //TODO: configure!
-	s_startTime.setNowNonInline();//reduce code size by calling non inline func from slow path
+	p_info->c_startTime.setNowNonInline();//reduce code size by calling non inline func from slow path
 	const PlaybackVector &pv = * g_pApp->m_const_params.pPlaybackVector;
 
 	size_t i = 0;

@@ -655,7 +676,7 @@ void Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration
 		m_switchActivityInfo.execute(m_pMsgRequest->getSequenceCounter());
 	}
 	g_cycle_wait_loop_counter++; //for silenting waring at the end
-	s_endTime.setNowNonInline();//reduce code size by calling non inline func from slow path
+	p_info->c_endTime.setNowNonInline();//reduce code size by calling non inline func from slow path
 	usleep(20*1000);//wait for reply of last packet //TODO: configure!
 	g_b_exit = true;
 }

@@ -684,61 +705,61 @@ void Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration
 
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo, class SwitchCycleDuration, class SwitchMsgSize, class PongModeCare>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
-	Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeCare> c(_fd_min, _fd_max, _fd_num);
+void client_handler(struct handler_info *_p_info) {
+	Client<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeCare> c(_p_info);
 	c.doHandler();
 }
 
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo, class SwitchCycleDuration, class SwitchMsgSize>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
+void client_handler(struct handler_info *_p_info) {
 	if (g_pApp->m_const_params.b_stream)
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeNever> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeNever> (_p_info);
 	else if (g_pApp->m_const_params.reply_every == 1)
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeAlways> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeAlways> (_p_info);
 	else
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeNormal> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchMsgSize, PongModeNormal> (_p_info);
 }
 
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo, class SwitchCycleDuration>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
+void client_handler(struct handler_info *_p_info) {
 	if (g_pApp->m_const_params.msg_size_range > 0)
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchOnMsgSize> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchOnMsgSize> (_p_info);
 	else
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchOff> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchCycleDuration, SwitchOff> (_p_info);
 }
 
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity, class SwitchActivityInfo>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
+void client_handler(struct handler_info *_p_info) {
 	if (g_pApp->m_const_params.cycleDuration > TicksDuration::TICKS0) {
 		if (g_pApp->m_const_params.dummy_mps) {
-			client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOnDummySend> (_fd_min, _fd_max, _fd_num);
+			client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOnDummySend> (_p_info);
 		} else {
-			client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOnCycleDuration> (_fd_min, _fd_max, _fd_num);
+			client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOnCycleDuration> (_p_info);
 		}
 	}
 	else
-		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOff> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchActivityInfo, SwitchOff> (_p_info);
 }
 
 //------------------------------------------------------------------------------
 template <class IoType, class SwitchDataIntegrity>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
+void client_handler(struct handler_info *_p_info) {
 	if (g_pApp->m_const_params.packetrate_stats_print_ratio > 0)
-		client_handler<IoType, SwitchDataIntegrity, SwitchOnActivityInfo> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchOnActivityInfo> (_p_info);
 	else
-		client_handler<IoType, SwitchDataIntegrity, SwitchOff> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchDataIntegrity, SwitchOff> (_p_info);
 }
 
 //------------------------------------------------------------------------------
 template <class IoType>
-void client_handler(int _fd_min, int _fd_max, int _fd_num) {
+void client_handler(struct handler_info *_p_info) {
 	if (g_pApp->m_const_params.data_integrity)
-		client_handler<IoType, SwitchOnDataIntegrity> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchOnDataIntegrity> (_p_info);
 	else
-		client_handler<IoType, SwitchOff> (_fd_min, _fd_max, _fd_num);
+		client_handler<IoType, SwitchOff> (_p_info);
 }
 
 //------------------------------------------------------------------------------

@@ -748,29 +769,29 @@ void client_handler(handler_info *p_info)
 		switch (g_pApp->m_const_params.fd_handler_type) {
 			case SELECT:
 			{
-				client_handler<IoSelect> (p_info->fd_min, p_info->fd_max, p_info->fd_num);
+				client_handler<IoSelect> (p_info);
 				break;
 			}
 			case RECVFROM:
 			{
-				client_handler<IoRecvfrom> (p_info->fd_min, p_info->fd_max, p_info->fd_num);
+				client_handler<IoRecvfrom> (p_info);
 				break;
 			}
 			case RECVFROMMUX:
 			{
-				client_handler<IoRecvfromMUX> (p_info->fd_min, p_info->fd_max, p_info->fd_num);
+				client_handler<IoRecvfromMUX> (p_info);
 				break;
 			}
 #ifndef WIN32
 			case POLL:
 			{
-				client_handler<IoPoll> (p_info->fd_min, p_info->fd_max, p_info->fd_num);
+				client_handler<IoPoll> (p_info);
 				break;
 			}
 #ifndef __FreeBSD__
 			case EPOLL:
 			{
-				client_handler<IoEpoll> (p_info->fd_min, p_info->fd_max, p_info->fd_num);
+				client_handler<IoEpoll> (p_info);
 				break;
 			}
 #endif

@@ -783,3 +804,26 @@ void client_handler(handler_info *p_info)
 		}
 	}
 }
+
+void *client_handler_multi_thread(void *arg)
+{
+	struct handler_info *p_info = (handler_info *)arg;
+
+	if (p_info) {
+		client_handler(p_info);
+
+		/* Mark this thread as complete (the first index is reserved for main thread) */
+		{
+			int i = p_info->id + 1;
+			if (p_info->id < g_pApp->m_const_params.threads_num) {
+				if (thread_pid_array && thread_pid_array[i].tid && (thread_pid_array[i].tid == os_getthread().tid)) {
+					ENTER_CRITICAL(&thread_exit_lock);
+					thread_pid_array[i].tid = 0;
+					LEAVE_CRITICAL(&thread_exit_lock);
+				}
+			}
+		}
+	}
+
+	return 0;
+}

src/Client.h +2 -1

index 965b3b847cad..a3898705987a 100644
@@ -53,6 +53,7 @@ private:
 	os_thread_t m_receiverTid;
 	IoType m_ioHandler;
 	addr_to_id   m_ServerList;
+	struct handler_info *p_info;
 
 	SwitchDataIntegrity m_switchDataIntegrity;
 	SwitchActivityInfo  m_switchActivityInfo;

@@ -61,7 +62,7 @@ private:
 	PongModeCare        m_pongModeCare; // has msg_sendto() method and can be one of: PongModeNormal, PongModeAlways, PongModeNever
 
 public:
-	Client(int _fd_min, int _fd_max, int _fd_num);
+	Client(struct handler_info *_p_info);
 	virtual ~Client();
 	void doHandler();
 	void client_receiver_thread();

src/Defs.h +8 -0

index 0ee0f1e229c7..97c54cd3b248 100644
@@ -467,6 +467,14 @@ typedef struct handler_info {
 	int fd_min;					/**< minimum descriptor (fd) */
 	int fd_max;					/**< maximum socket descriptor (fd) */
 	int fd_num;					/**< number of socket descriptors */
+	/* These are all of the stats relevant to a single thread's streaming
+	 * I/O performance.  When running a throughput test as client and
+	 * running in multiple threads, we sum these up to across the
+	 * threads to get a total
+	 */
+	uint64_t sendCount;
+	TicksTime c_startTime;
+	TicksTime c_endTime;
 }handler_info;
 
 typedef struct clt_session_info {

src/SockPerf.cpp +48 -5

97 lines not on this page · raw message

Signature · 6 lines

2.14.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Leon Romanovsky

2017-12-20 08:52 UTC #6 lore raw

In reply to a message not in the archive

Quoted 47 lines · Doug Ledford

During testing, it has become painfully clear that a single threaded UDP test client can not exercise a 100Gig link due to issues related to single core maximum throughput. This patchset implements a multi-threaded throughput test in sockperf. This is just an initial implementation, there is still more work to be done. In particular:
1) Although the speed improved with this change, it did not improve drastically. As soon as the client send bottleneck was removed, it became clear there is another bottleneck on the server. When sending to a server from one client, all data is received on a single queue pair, and due to how interrupts are spread in the RDMA stack (namely that each queue pair goes to a single interrupt and we rely on multiple queue pairs being in use to balance interrupts across different cores), we take all interrupts from a specific host on a single core and the receiving side then becomes the bottleneck with single core IPoIB receive processing being the limiting factor. On a slower machine, I clocked 30GBit/s throughput. On a faster machine as the server, I was able to get up to 70GBit/s throughput.
2) I thought I might try an experiment to get around the queue pair is on one CPU issue. We use P_Keys in our internal lab setup, and so on the specific link in question, I actually have a total of three different IP interfaces on different P_Keys. I tried to open tests on multiple of these interfaces to see how that would impact performance (so a multithreaded server listening on ports on three different P_Key interfaces all on the same physical link, which should use three different queue pairs, and a multithreaded client sending to those three different P_Key interfaces from three different P_Key interfaces of its own). It tanked it. Like less than gigabit ethernet speeds. This warrants some investigation moving forward I think.
3) I thought I might try sending from two clients to the server at once and summing their throughput. That was fun. With UDP the clients are able to send enough data that flow control on the link kicks in, at which point each client starts dropping packets on the floor (they're UDP after all), and so the net result is that one client claimed 200GBit/s and the other about 175GBit/s. Meanwhile, the server thought we were just kidding and didn't actually run a test at all.
4) I reran the test using TCP instead of UDP. That's a non-starter. Whether due to my changes, or just because it is the way it is, the TCP tests all failed. For larger message sizes, they failed instantly. For smaller message sizes the test might run for a few seconds, but would eventually fail too. Always the failure was that the server would get a message it deemed too large and would forcibly close all of the TCP connections, at which point the client just bails.

Quoted 3 lines

I should point out that I don't program C++. Issues with me not doing these patches in a C++ typical manner are related to that.

Doug,

I contacted the group which is responsible for the development of sockperf
https://github.com/Mellanox/sockperf

Their maintainer is on vacation till third week of January and unluckily there are no other people right now who can take a look onto your proposal.

In meanwhile, it is better to push the code to github, because their development flow is based on github and not on mailing list.

Thanks