Skip to contentSkip to conversation

vdpa/mlx5: fix polling threads scheduling

PATCH Applied Matan Azrad 4 participants 4 messages Feb 8, 2021 – Feb 10, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vdpa/mlx5: fix polling threads scheduling

When the event mode is with 0 fixed delay, the polling-thread will never give-up CPU.

So, when multi-polling-threads are active, the context-switch between them will be managed by the system which may affect latency according to the time-out decided by the system.

In order to fix multi-devices polling thread scheduling, this patch forces rescheduling for each CQ poll iteration.

Move the polling thread to SCHED_RR mode with maximum priority to complete the fairness.

Fixes 6956a48cabbb ("vdpa/mlx5: set polling mode default delay to zero")
---
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/vdpa/mlx5/mlx5_vdpa_event.c +16 -0
index 0f635ff..86adc86 100644
@@ -232,6 +232,9 @@
 	}
 	if (priv->timer_delay_us)
 		usleep(priv->timer_delay_us);
+	else
+		/* Give-up CPU to improve polling threads scheduling. */
+		pthread_yield();
 }
 
 static void *
@@ -500,6 +503,9 @@
 	rte_cpuset_t cpuset;
 	pthread_attr_t attr;
 	char name[16];
+	const struct sched_param sp = {
+		.sched_priority = sched_get_priority_max(SCHED_RR),
+	};
 
 	if (!priv->eventc)
 		/* All virtqs are in poll mode. */
@@ -520,6 +526,16 @@
 			DRV_LOG(ERR, "Failed to set thread affinity.");
 			return -1;
 		}
+		ret = pthread_attr_setschedpolicy(&attr, SCHED_RR);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread sched policy = RR.");
+			return -1;
+		}
+		ret = pthread_attr_setschedparam(&attr, &sp);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread priority.");
+			return -1;
+		}
 		ret = pthread_create(&priv->timer_tid, &attr,
 				     mlx5_vdpa_poll_handle, (void *)priv);
 		if (ret) {
Signature · 1 line
1.8.3.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 52 lines · Matan Azrad

When the event mode is with 0 fixed delay, the polling-thread will never give-up CPU.

So, when multi-polling-threads are active, the context-switch between them will be managed by the system which may affect latency according to the time-out decided by the system.

In order to fix multi-devices polling thread scheduling, this patch forces rescheduling for each CQ poll iteration.

Move the polling thread to SCHED_RR mode with maximum priority to complete the fairness.

Fixes 6956a48cabbb ("vdpa/mlx5: set polling mode default delay to zero")
---
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/vdpa/mlx5/mlx5_vdpa_event.c +9 -0
index 0f635ff..86adc86 100644
@@ -232,6 +232,9 @@
 	}
 	if (priv->timer_delay_us)
 		usleep(priv->timer_delay_us);
+	else
+		/* Give-up CPU to improve polling threads scheduling. */
+		pthread_yield();
 }
 
 static void *
@@ -500,6 +503,9 @@
 	rte_cpuset_t cpuset;
 	pthread_attr_t attr;
 	char name[16];
+	const struct sched_param sp = {
+		.sched_priority = sched_get_priority_max(SCHED_RR),
+	};
 
 	if (!priv->eventc)
 		/* All virtqs are in poll mode. */
@@ -520,6 +526,16 @@
 			DRV_LOG(ERR, "Failed to set thread affinity.");
 			return -1;
 		}
+		ret = pthread_attr_setschedpolicy(&attr, SCHED_RR);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread sched policy = RR.");

drivers/vdpa/mlx5/mlx5_vdpa_event.c line 541

Quoted 10 lines
drivers/vdpa/mlx5/mlx5_vdpa_event.c +7 -0
+			return -1;
+		}
+		ret = pthread_attr_setschedparam(&attr, &sp);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread priority.");
+			return -1;
+		}
 		ret = pthread_create(&priv->timer_tid, &attr,
 				     mlx5_vdpa_poll_handle, (void *)priv);
 		if (ret) {
Acked-by Xueming Li
#2 lore raw
Quoted 67 lines

-----Original Message-----
From: dev <[email protected]> On Behalf Of Maxime Coquelin
Sent: Monday, February 8, 2021 7:17 PM
To: Matan Azrad <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH] vdpa/mlx5: fix polling threads scheduling

Quoted 58 lines · Matan Azrad

When the event mode is with 0 fixed delay, the polling-thread will never give-up CPU.

So, when multi-polling-threads are active, the context-switch between them will be managed by the system which may affect latency according to the time-out decided by the system.

In order to fix multi-devices polling thread scheduling, this patch forces rescheduling for each CQ poll iteration.

Move the polling thread to SCHED_RR mode with maximum priority to complete the fairness.

Fixes 6956a48cabbb ("vdpa/mlx5: set polling mode default delay to

zero")

---
 drivers/vdpa/mlx5/mlx5_vdpa_event.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/vdpa/mlx5/mlx5_vdpa_event.c +14 -0
index 0f635ff..86adc86 100644
@@ -232,6 +232,9 @@
 	}
 	if (priv->timer_delay_us)
 		usleep(priv->timer_delay_us);
+	else
+		/* Give-up CPU to improve polling threads scheduling. */
+		pthread_yield();
 }
 
 static void *
@@ -500,6 +503,9 @@
 	rte_cpuset_t cpuset;
 	pthread_attr_t attr;
 	char name[16];
+	const struct sched_param sp = {
+		.sched_priority = sched_get_priority_max(SCHED_RR),
+	};
 
 	if (!priv->eventc)
 		/* All virtqs are in poll mode. */
@@ -520,6 +526,16 @@
 			DRV_LOG(ERR, "Failed to set thread affinity.");
 			return -1;
 		}
+		ret = pthread_attr_setschedpolicy(&attr, SCHED_RR);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread sched policy = RR.");
+			return -1;
+		}
+		ret = pthread_attr_setschedparam(&attr, &sp);
+		if (ret) {
+			DRV_LOG(ERR, "Failed to set thread priority.");
Quoted 8 lines

drivers/vdpa/mlx5/mlx5_vdpa_event.c line 541

Quoted 6 lines
drivers/vdpa/mlx5/mlx5_vdpa_event.c +2 -0
+			return -1;
+		}
 		ret = pthread_create(&priv->timer_tid, &attr,
 				     mlx5_vdpa_poll_handle, (void *)priv);
 		if (ret) {
#3 lore raw
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When the event mode is with 0 fixed delay, the polling-thread will never give-up CPU.

So, when multi-polling-threads are active, the context-switch between them will be managed by the system which may affect latency according to the time-out decided by the system.

In order to fix multi-devices polling thread scheduling, this patch forces rescheduling for each CQ poll iteration.

Move the polling thread to SCHED_RR mode with maximum priority to complete the fairness.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Fixes 6956a48cabbb ("vdpa/mlx5: set polling mode default delay to

zero")

converted to nvidia.com

Applied, thanks