Skip to contentSkip to conversation

vdpa/mlx5: set polling mode default delay to zero

PATCH In review Xueming Li 2 participants 13 messages Dec 2, 2020 – Jan 8, 2021

Subsystems (2)
lore
Share & tools
lore

Xueming Li

author
#0 lore raw

PATCH 1/4 vdpa/mlx5: set polling mode default delay to zero

To improve throughput and latency, this patch allows Rx polling timer delay to 0us.

---
 doc/guides/vdpadevs/mlx5.rst  | 3 +--
 drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++------
 2 files changed, 4 insertions(+), 8 deletions(-)
doc/guides/vdpadevs/mlx5.rst +1 -2
index 3a6d88362d..903fdb0e60 100644
@@ -125,8 +125,7 @@ Driver options
   - 0, A nonzero value to set timer step in micro-seconds. The timer thread
     dynamic delay change steps according to this value. Default value is 1us.
 
-  - 1, A nonzero value to set fixed timer delay in micro-seconds. Default value
-    is 100us.
+  - 1, A value to set fixed timer delay in micro-seconds. Default value is 0us.
 
 - ``no_traffic_time`` parameter [int]
 
drivers/vdpa/mlx5/mlx5_vdpa.c +3 -6
index b64f364eb7..5020a99fae 100644
@@ -651,12 +651,9 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 		return;
 	rte_kvargs_process(kvlist, NULL, mlx5_vdpa_args_check_handler, priv);
 	rte_kvargs_free(kvlist);
-	if (!priv->event_us) {
-		if (priv->event_mode == MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER)
-			priv->event_us = MLX5_VDPA_DEFAULT_TIMER_STEP_US;
-		else if (priv->event_mode == MLX5_VDPA_EVENT_MODE_FIXED_TIMER)
-			priv->event_us = MLX5_VDPA_DEFAULT_TIMER_DELAY_US;
-	}
+	if (!priv->event_us &&
+	    priv->event_mode == MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER)
+		priv->event_us = MLX5_VDPA_DEFAULT_TIMER_STEP_US;
 	DRV_LOG(DEBUG, "event mode is %d.", priv->event_mode);
 	DRV_LOG(DEBUG, "event_us is %u us.", priv->event_us);
 	DRV_LOG(DEBUG, "no traffic time is %u s.", priv->no_traffic_time_s);
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 2/4 vdpa/mlx5: default polling mode delay time to zero

To improve performance and latency, this patch set Rx polling mode default delay time to zero.

---
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.h +1 -1
index d039ada65b..08e04a86c4 100644
@@ -36,7 +36,7 @@
 #define VIRTIO_F_RING_PACKED 34
 #endif
 
-#define MLX5_VDPA_DEFAULT_TIMER_DELAY_US 100u
+#define MLX5_VDPA_DEFAULT_TIMER_DELAY_US 0u
 #define MLX5_VDPA_DEFAULT_TIMER_STEP_US 1u
 
 struct mlx5_vdpa_cq {
drivers/vdpa/mlx5/mlx5_vdpa_event.c +2 -1
index 3aeaeb893f..5366937e03 100644
@@ -265,7 +265,8 @@ mlx5_vdpa_timer_sleep(struct mlx5_vdpa_priv *priv, uint32_t max)
 			break;
 		}
 	}
-	usleep(priv->timer_delay_us);
+	if (priv->timer_delay_us)
+		usleep(priv->timer_delay_us);
 }
 
 static void *
Signature · 1 line
2.25.1
Reviewed-by on v1 2/4
#2 lore raw

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: default polling mode delay time to zero

Quoted 1 line · Xueming Li

To improve performance and latency, this patch set Rx polling mode

s/set/sets/

I'll fix while applying.

Quoted 28 lines

default delay time to zero.

---
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/vdpa/mlx5/mlx5_vdpa.h +1 -1
index d039ada65b..08e04a86c4 100644
@@ -36,7 +36,7 @@
 #define VIRTIO_F_RING_PACKED 34
 #endif
 
-#define MLX5_VDPA_DEFAULT_TIMER_DELAY_US 100u
+#define MLX5_VDPA_DEFAULT_TIMER_DELAY_US 0u
 #define MLX5_VDPA_DEFAULT_TIMER_STEP_US 1u
 
 struct mlx5_vdpa_cq {
drivers/vdpa/mlx5/mlx5_vdpa_event.c +0 -0
index 3aeaeb893f..5366937e03 100644

drivers/vdpa/mlx5/mlx5_vdpa_event.c mlx5_vdpa_timer_sleep(struct mlx5_vdpa_priv *priv, uint32_t max) · line 272

Quoted 9 lines
drivers/vdpa/mlx5/mlx5_vdpa_event.c +2 -1
@@ -265,7 +265,8 @@ mlx5_vdpa_timer_sleep(struct mlx5_vdpa_priv *priv, uint32_t max)
 			break;
 		}
 	}
-	usleep(priv->timer_delay_us);
+	if (priv->timer_delay_us)
+		usleep(priv->timer_delay_us);
 }
 
 static void *

Thanks,
Maxime

#3 lore raw

Re: [dpdk-dev] [PATCH 2/4] vdpa/mlx5: default polling mode delay time to zero

In reply to Xueming Li
Quoted 9 lines · Xueming Li

To improve performance and latency, this patch set Rx polling mode default delay time to zero.

---
 drivers/vdpa/mlx5/mlx5_vdpa.h       | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

Series applied to dpdk-next-virtio/main.

Thanks,
Maxime

author
#4 lore raw

PATCH 3/4 vdpa/mlx5: add cpu core parameter to bind polling thread

In reply to Xueming Li

This patch adds new device argument to specify cpu core affinity to event polling thread for better latency and throughput. The thread could be also located by name "vDPA-mlx5-<id>".

---
 doc/guides/vdpadevs/mlx5.rst        |  5 +++++
 drivers/vdpa/mlx5/mlx5_vdpa.c       |  7 +++++++
 drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 23 ++++++++++++++++++++++-
 4 files changed, 35 insertions(+), 1 deletion(-)
doc/guides/vdpadevs/mlx5.rst +5 -0
index 903fdb0e60..20254257c9 100644
@@ -134,6 +134,11 @@ Driver options
   interrupts are configured to the device in order to notify traffic for the
   driver. Default value is 2s.
 
+- ``event_core`` parameter [int]
+
+  CPU core number to set polling thread affinity to, default to control plane
+  cpu.
+
 Error handling
 ^^^^^^^^^^^^^^
 
drivers/vdpa/mlx5/mlx5_vdpa.c +7 -0
index 5020a99fae..1f92c529c9 100644
@@ -612,6 +612,7 @@ mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
 {
 	struct mlx5_vdpa_priv *priv = opaque;
 	unsigned long tmp;
+	int n_cores = sysconf(_SC_NPROCESSORS_ONLN);
 
 	if (strcmp(key, "class") == 0)
 		return 0;
@@ -630,6 +631,11 @@ mlx5_vdpa_args_check_handler(const char *key, const char *val, void *opaque)
 		priv->event_us = (uint32_t)tmp;
 	} else if (strcmp(key, "no_traffic_time") == 0) {
 		priv->no_traffic_time_s = (uint32_t)tmp;
+	} else if (strcmp(key, "event_core") == 0) {
+		if (tmp >= (unsigned long)n_cores)
+			DRV_LOG(WARNING, "Invalid event_core %s.", val);
+		else
+			priv->event_core = tmp;
 	} else {
 		DRV_LOG(WARNING, "Invalid key %s.", key);
 	}
@@ -643,6 +649,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 
 	priv->event_mode = MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER;
 	priv->event_us = 0;
+	priv->event_core = -1;
 	priv->no_traffic_time_s = MLX5_VDPA_DEFAULT_NO_TRAFFIC_TIME_S;
 	if (devargs == NULL)
 		return;
drivers/vdpa/mlx5/mlx5_vdpa.h +1 -0
index 08e04a86c4..b4dd3834aa 100644
@@ -131,6 +131,7 @@ struct mlx5_vdpa_priv {
 	pthread_cond_t timer_cond;
 	volatile uint8_t timer_on;
 	int event_mode;
+	int event_core; /* Event thread cpu affinity core. */
 	uint32_t event_us;
 	uint32_t timer_delay_us;
 	uint32_t no_traffic_time_s;
drivers/vdpa/mlx5/mlx5_vdpa_event.c +22 -1
index 5366937e03..f731c80004 100644
@@ -532,6 +532,9 @@ int
 mlx5_vdpa_cqe_event_setup(struct mlx5_vdpa_priv *priv)
 {
 	int ret;
+	rte_cpuset_t cpuset;
+	pthread_attr_t attr;
+	char name[16];
 
 	if (!priv->eventc)
 		/* All virtqs are in poll mode. */
@@ -540,12 +543,30 @@ mlx5_vdpa_cqe_event_setup(struct mlx5_vdpa_priv *priv)
 		pthread_mutex_init(&priv->timer_lock, NULL);
 		pthread_cond_init(&priv->timer_cond, NULL);
 		priv->timer_on = 0;
-		ret = pthread_create(&priv->timer_tid, NULL,
+		pthread_attr_init(&attr);
+		CPU_ZERO(&cpuset);
+		if (priv->event_core != -1)
+			CPU_SET(priv->event_core, &cpuset);
+		else
+			cpuset = rte_lcore_cpuset(rte_get_main_lcore());
+		ret = pthread_attr_setaffinity_np(&attr, sizeof(cpuset),
+						  &cpuset);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread affinity.");
+			return -1;
+		}
+		ret = pthread_create(&priv->timer_tid, &attr,
 				     mlx5_vdpa_poll_handle, (void *)priv);
 		if (ret) {
 			DRV_LOG(ERR, "Failed to create timer thread.");
 			return -1;
 		}
+		snprintf(name, sizeof(name), "vDPA-mlx5-%d", priv->vid);
+		ret = pthread_setname_np(priv->timer_tid, name);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set timer thread name.");
+			return -1;
+		}
 	}
 	priv->intr_handle.fd = priv->eventc->fd;
 	priv->intr_handle.type = RTE_INTR_HANDLE_EXT;
Signature · 1 line
2.25.1
Reviewed-by on v1 3/4
#5 lore raw

Re: [dpdk-dev] [PATCH 3/4] vdpa/mlx5: add cpu core parameter to bind polling thread

Quoted 4 lines · Xueming Li

This patch adds new device argument to specify cpu core affinity to event polling thread for better latency and throughput. The thread could be also located by name "vDPA-mlx5-<id>".

Quoted 9 lines
---
 doc/guides/vdpadevs/mlx5.rst        |  5 +++++
 drivers/vdpa/mlx5/mlx5_vdpa.c       |  7 +++++++
 drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 23 ++++++++++++++++++++++-
 4 files changed, 35 insertions(+), 1 deletion(-)

Thanks,
Maxime

#6 lore raw

Re: [dpdk-dev] [PATCH 3/4] vdpa/mlx5: add cpu core parameter to bind polling thread

In reply to Xueming Li
Quoted 4 lines · Xueming Li

This patch adds new device argument to specify cpu core affinity to event polling thread for better latency and throughput. The thread could be also located by name "vDPA-mlx5-<id>".

Quoted 8 lines
---
 doc/guides/vdpadevs/mlx5.rst        |  5 +++++
 drivers/vdpa/mlx5/mlx5_vdpa.c       |  7 +++++++
 drivers/vdpa/mlx5/mlx5_vdpa.h       |  1 +
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 23 ++++++++++++++++++++++-
 4 files changed, 35 insertions(+), 1 deletion(-)

Series applied to dpdk-next-virtio/main.

Thanks,
Maxime

author
#7 lore raw

PATCH 4/4 vdpa/mlx5: set default event mode to polling

In reply to Xueming Li

For better performance and latency, this patch sets default event handling mode to polling mode which uses dedicate thread per device to poll and process event.

---
 doc/guides/vdpadevs/mlx5.rst  | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +1 -1
index 20254257c9..730e171ba3 100644
@@ -116,7 +116,7 @@ Driver options
   - 2, Completion queue scheduling will be managed by interrupts. Each CQ burst
     arms the CQ in order to get an interrupt event in the next traffic burst.
 
-  - Default mode is 0.
+  - Default mode is 1.
 
 - ``event_us`` parameter [int]
 
drivers/vdpa/mlx5/mlx5_vdpa.c +1 -1
index 1f92c529c9..5d954d48ce 100644
@@ -647,7 +647,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 {
 	struct rte_kvargs *kvlist;
 
-	priv->event_mode = MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER;
+	priv->event_mode = MLX5_VDPA_EVENT_MODE_FIXED_TIMER;
 	priv->event_us = 0;
 	priv->event_core = -1;
 	priv->no_traffic_time_s = MLX5_VDPA_DEFAULT_NO_TRAFFIC_TIME_S;
Signature · 1 line
2.25.1
Reviewed-by on v1 4/4
#8 lore raw

Re: [dpdk-dev] [PATCH 4/4] vdpa/mlx5: set default event mode to polling

Quoted 30 lines · Xueming Li

For better performance and latency, this patch sets default event handling mode to polling mode which uses dedicate thread per device to poll and process event.

---
 doc/guides/vdpadevs/mlx5.rst  | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
doc/guides/vdpadevs/mlx5.rst +1 -1
index 20254257c9..730e171ba3 100644
@@ -116,7 +116,7 @@ Driver options
   - 2, Completion queue scheduling will be managed by interrupts. Each CQ burst
     arms the CQ in order to get an interrupt event in the next traffic burst.
 
-  - Default mode is 0.
+  - Default mode is 1.
 
 - ``event_us`` parameter [int]
 
drivers/vdpa/mlx5/mlx5_vdpa.c +0 -0
index 1f92c529c9..5d954d48ce 100644

drivers/vdpa/mlx5/mlx5_vdpa.c mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) · line 653

Quoted 8 lines
drivers/vdpa/mlx5/mlx5_vdpa.c +1 -1
@@ -647,7 +647,7 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 {
 	struct rte_kvargs *kvlist;
 
-	priv->event_mode = MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER;
+	priv->event_mode = MLX5_VDPA_EVENT_MODE_FIXED_TIMER;
 	priv->event_us = 0;
 	priv->event_core = -1;
 	priv->no_traffic_time_s = MLX5_VDPA_DEFAULT_NO_TRAFFIC_TIME_S;

Thanks,
Maxime

#9 lore raw

Re: [dpdk-dev] [PATCH 4/4] vdpa/mlx5: set default event mode to polling

In reply to Xueming Li
Quoted 10 lines · Xueming Li

For better performance and latency, this patch sets default event handling mode to polling mode which uses dedicate thread per device to poll and process event.

---
 doc/guides/vdpadevs/mlx5.rst  | 2 +-
 drivers/vdpa/mlx5/mlx5_vdpa.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

Series applied to dpdk-next-virtio/main.

Thanks,
Maxime

Reviewed-by on v1 1/4
#10 lore raw
In reply to Xueming Li
Quoted 35 lines · Xueming Li

To improve throughput and latency, this patch allows Rx polling timer delay to 0us.

---
 doc/guides/vdpadevs/mlx5.rst  | 3 +--
 drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++------
 2 files changed, 4 insertions(+), 8 deletions(-)
doc/guides/vdpadevs/mlx5.rst +1 -2
index 3a6d88362d..903fdb0e60 100644
@@ -125,8 +125,7 @@ Driver options
   - 0, A nonzero value to set timer step in micro-seconds. The timer thread
     dynamic delay change steps according to this value. Default value is 1us.
 
-  - 1, A nonzero value to set fixed timer delay in micro-seconds. Default value
-    is 100us.
+  - 1, A value to set fixed timer delay in micro-seconds. Default value is 0us.
 
 - ``no_traffic_time`` parameter [int]
 
drivers/vdpa/mlx5/mlx5_vdpa.c +0 -2
index b64f364eb7..5020a99fae 100644
@@ -651,12 +651,9 @@ mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv)
 		return;
 	rte_kvargs_process(kvlist, NULL, mlx5_vdpa_args_check_handler, priv);
 	rte_kvargs_free(kvlist);
-	if (!priv->event_us) {
-		if (priv->event_mode == MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER)

drivers/vdpa/mlx5/mlx5_vdpa.c mlx5_vdpa_config_get(struct rte_devargs *devargs, struct mlx5_vdpa_priv *priv) · line 659

Quoted 10 lines
drivers/vdpa/mlx5/mlx5_vdpa.c +3 -4
-			priv->event_us = MLX5_VDPA_DEFAULT_TIMER_STEP_US;
-		else if (priv->event_mode == MLX5_VDPA_EVENT_MODE_FIXED_TIMER)
-			priv->event_us = MLX5_VDPA_DEFAULT_TIMER_DELAY_US;
-	}
+	if (!priv->event_us &&
+	    priv->event_mode == MLX5_VDPA_EVENT_MODE_DYNAMIC_TIMER)
+		priv->event_us = MLX5_VDPA_DEFAULT_TIMER_STEP_US;
 	DRV_LOG(DEBUG, "event mode is %d.", priv->event_mode);
 	DRV_LOG(DEBUG, "event_us is %u us.", priv->event_us);
 	DRV_LOG(DEBUG, "no traffic time is %u s.", priv->no_traffic_time_s);

Thanks,
Maxime

#11 lore raw
In reply to Xueming Li
Quoted 10 lines · Xueming Li

To improve throughput and latency, this patch allows Rx polling timer delay to 0us.

---
 doc/guides/vdpadevs/mlx5.rst  | 3 +--
 drivers/vdpa/mlx5/mlx5_vdpa.c | 9 +++------
 2 files changed, 4 insertions(+), 8 deletions(-)

Series applied to dpdk-next-virtio/main.

Please when multiple patches, add a cover letter to avoid me having to reply to every patches when applying your series.

Thanks,
Maxime

#12 lore raw

Hi Maxime,

Quoted 22 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Friday, January 8, 2021 5:13 PM
To: Xueming(Steven) Li <[email protected]>; Matan Azrad <[email protected]>; Slava Ovsiienko <[email protected]>
Cc: [email protected]; Asaf Penso <[email protected]>
Subject: Re: [PATCH 1/4] vdpa/mlx5: set polling mode default delay to zero

Quoted 10 lines · Xueming Li

To improve throughput and latency, this patch allows Rx polling timer delay to 0us.

---
doc/guides/vdpadevs/mlx5.rst | 3 +-- drivers/vdpa/mlx5/mlx5_vdpa.c
| 9 +++------

 2 files changed, 4 insertions(+), 8 deletions(-)
Quoted 4 lines

Series applied to dpdk-next-virtio/main.

Please when multiple patches, add a cover letter to avoid me having to reply to every patches when applying your series.

Got it, thanks!

Quoted 3 lines

Thanks,
Maxime