Skip to contentSkip to conversation

sched: fix port time rounding error

PATCH Superseded [email protected] 7 participants 14 messages Apr 16, 2020 – Sep 7, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: fix port time rounding error

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +10 -2
index c0983ddda..c656dba2d 100644
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		goto end;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
 
+end:
 	/* Reset pipe loop detection */
 	for (i = 0; i < port->n_subports_per_port; i++)
 		port->subports[i]->pipe_loop = RTE_SCHED_PIPE_INVALID;
Signature · 1 line
2.17.1
#1 lore raw
Quoted 89 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, April 16, 2020 9:48 AM
To: Dumitrescu, Cristian <[email protected]>
Cc: [email protected]; Alan Dewar <[email protected]>
Subject: [PATCH] sched: fix port time rounding error

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +1 -0
index c0983ddda..c656dba2d 100644
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +1 -0
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct
 rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +8 -2
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		goto end;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff <<
 RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >>
 RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
 
+end:
 	/* Reset pipe loop detection */
lib/librte_sched/rte_sched.c +0 -1
 	for (i = 0; i < port->n_subports_per_port; i++)
 		port->subports[i]->pipe_loop = RTE_SCHED_PIPE_INVALID;
--
Quoted 1 line

2.17.1

Adding Jasvinder.

#2 lore raw
Quoted 76 lines

-----Original Message-----
From: Dumitrescu, Cristian <[email protected]>
Sent: Friday, April 17, 2020 10:19 PM
To: [email protected]
Cc: [email protected]; Alan Dewar <[email protected]>; Singh, Jasvinder <[email protected]>
Subject: RE: [PATCH] sched: fix port time rounding error

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 47 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, April 16, 2020 9:48 AM
To: Dumitrescu, Cristian <[email protected]>
Cc: [email protected]; Alan Dewar <[email protected]>
Subject: [PATCH] sched: fix port time rounding error

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte.
This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..c656dba2d 100644

lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +0 -0
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params

Quoted 16 lines
lib/librte_sched/rte_sched.c +1 -0
 *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +1 -1
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
Quoted 2 lines

lib/librte_sched/rte_sched.c static inline void · line 2683

Quoted 2 lines
lib/librte_sched/rte_sched.c +2 -0
+	if (cycles < port->time_cpu_cycles)
+		goto end;

Above check seems redundant as port->time_cpu_cycles will always be less than the current cycles due to roundoff in previous iteration.

Quoted 23 lines
Quoted 20 lines
lib/librte_sched/rte_sched.c +4 -2
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff <<
 RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >>
 RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
 
+end:
 	/* Reset pipe loop detection */
 	for (i = 0; i < port->n_subports_per_port; i++)
 		port->subports[i]->pipe_loop = RTE_SCHED_PIPE_INVALID;
--
Quoted 1 line

2.17.1

Adding Jasvinder.

#3 lore raw
Quoted 84 lines

-----Original Message-----
From: Singh, Jasvinder <[email protected]>
Sent: Monday, April 20, 2020 12:23 PM
To: Dumitrescu, Cristian <[email protected]>; [email protected]
Cc: [email protected]; Alan Dewar <[email protected]>
Subject: RE: [PATCH] sched: fix port time rounding error

Quoted 75 lines

-----Original Message-----
From: Dumitrescu, Cristian <[email protected]>
Sent: Friday, April 17, 2020 10:19 PM
To: [email protected]
Cc: [email protected]; Alan Dewar <[email protected]>; Singh, Jasvinder <[email protected]>
Subject: RE: [PATCH] sched: fix port time rounding error

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 48 lines

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Thursday, April 16, 2020 9:48 AM
To: Dumitrescu, Cristian <[email protected]>
Cc: [email protected]; Alan Dewar <[email protected]>
Subject: [PATCH] sched: fix port time rounding error

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..c656dba2d 100644

lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte 
 */
+	uint64_t cycles_per_byte;
 
lib/librte_sched/rte_sched.c +0 -0
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params

Quoted 16 lines
lib/librte_sched/rte_sched.c +1 -0
 *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +1 -1
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void 
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
Quoted 5 lines
Quoted 2 lines

lib/librte_sched/rte_sched.c static inline void · line 2683

Quoted 2 lines
lib/librte_sched/rte_sched.c +2 -0
+	if (cycles < port->time_cpu_cycles)
+		goto end;

Above check seems redundant as port->time_cpu_cycles will always be less than the current cycles due to roundoff in previous iteration.

This was to catch the condition where the cycles wraps back to zero (after 100+ years?? depending on clock speed). Rather than just going to end: the conditional should at least reset port->time_cpu_cycles back to zero. So there would be a very temporary glitch in accuracy once every 100+ years.

Quoted 24 lines
Quoted 24 lines
Quoted 21 lines
lib/librte_sched/rte_sched.c +4 -2
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << 
 RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >>
 RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
 
+end:
 	/* Reset pipe loop detection */
 	for (i = 0; i < port->n_subports_per_port; i++)
 		port->subports[i]->pipe_loop = RTE_SCHED_PIPE_INVALID;
--
Quoted 1 line

2.17.1

Adding Jasvinder.

#4 lore raw

Jasvinder, what is the conclusion of this patch?

21/04/2020 10:21, Dewar, Alan:

Quoted 68 lines

From: Singh, Jasvinder <[email protected]>

Quoted 58 lines
Quoted 58 lines

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 41 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..c656dba2d 100644

lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte 
 */
+	uint64_t cycles_per_byte;
 
lib/librte_sched/rte_sched.c +0 -0
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params

Quoted 16 lines
lib/librte_sched/rte_sched.c +1 -0
 *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
lib/librte_sched/rte_sched.c +1 -1
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void 
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
Quoted 5 lines
Quoted 2 lines

lib/librte_sched/rte_sched.c static inline void · line 2683

Quoted 2 lines
lib/librte_sched/rte_sched.c +2 -0
+	if (cycles < port->time_cpu_cycles)
+		goto end;

Above check seems redundant as port->time_cpu_cycles will always be less than the current cycles due to roundoff in previous iteration.

This was to catch the condition where the cycles wraps back to zero (after 100+ years?? depending on clock speed). Rather than just going to end: the conditional should at least reset port->time_cpu_cycles back to zero. So there would be a very temporary glitch in accuracy once every 100+ years.

#5 lore raw
Quoted 84 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Wednesday, June 24, 2020 11:50 PM
To: Singh, Jasvinder <[email protected]>
Cc: Dumitrescu, Cristian <[email protected]>; '[email protected]' <[email protected]>; [email protected]; 'Alan Dewar' <[email protected]>; Dewar, Alan <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] sched: fix port time rounding error

Jasvinder, what is the conclusion of this patch?

21/04/2020 10:21, Dewar, Alan:

Quoted 5 lines

From: Singh, Jasvinder <[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port

was

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per

byte.

Quoted 44 lines
Quoted 44 lines
Quoted 44 lines

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 26 lines

This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..c656dba2d 100644

lib/librte_sched/rte_sched.c +0 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per

byte */

lib/librte_sched/rte_sched.c +1 -0
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params

Quoted 17 lines
lib/librte_sched/rte_sched.c +1 -0
 *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte =
 rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
lib/librte_sched/rte_sched.c +1 -1
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,20 +2675,26 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

lib/librte_sched/rte_sched.c static inline void · line 2683

Quoted 2 lines
lib/librte_sched/rte_sched.c +2 -0
+	if (cycles < port->time_cpu_cycles)
+		goto end;

Above check seems redundant as port->time_cpu_cycles will always be

less than the current cycles due to roundoff in previous iteration.

Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 1 line

This was to catch the condition where the cycles wraps back to zero (after

100+ years?? depending on clock speed).

Quoted 3 lines

Rather than just going to end: the conditional should at least reset port- time_cpu_cycles back to zero.
So there would be a very temporary glitch in accuracy once every 100+

years.

Alan, Could you please resubmit the patch with above change? Other than that, patch looks good to me.

author
#6 lore raw

Okay will do.

Quoted 95 lines · Jasvinder
Quoted 85 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Wednesday, June 24, 2020 11:50 PM
To: Singh, Jasvinder <[email protected]>
Cc: Dumitrescu, Cristian <[email protected]>; '[email protected]' <[email protected]>; [email protected]; 'Alan Dewar' <[email protected]>; Dewar, Alan <[email protected]>
Subject: Re: [dpdk-dev] [PATCH] sched: fix port time rounding error

Jasvinder, what is the conclusion of this patch?

21/04/2020 10:21, Dewar, Alan:

Quoted 5 lines

From: Singh, Jasvinder <[email protected]>

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port

was

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per

byte.

Quoted 44 lines
Quoted 44 lines
Quoted 44 lines

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 26 lines

This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..c656dba2d 100644

lib/librte_sched/rte_sched.c +0 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
      uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes
 */
      uint64_t time;                /* Current NIC TX time measured in bytes */
      struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per

byte */

lib/librte_sched/rte_sched.c +1 -0
+     uint64_t cycles_per_byte;
 
      /* Grinders */
      struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params

Quoted 17 lines
lib/librte_sched/rte_sched.c +1 -0
 *params)
      cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
              / params->rate;
      port->inv_cycles_per_byte =
 rte_reciprocal_value(cycles_per_byte);
+     port->cycles_per_byte = cycles_per_byte;
 
      /* Grinders */
      port->pkts_out = NULL;
lib/librte_sched/rte_sched.c +1 -1
@@ -2673,20 +2675,26 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
      uint64_t cycles = rte_get_tsc_cycles();
-     uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+     uint64_t cycles_diff;
      uint64_t bytes_diff;
      uint32_t i;
 
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

lib/librte_sched/rte_sched.c static inline void · line 2683

Quoted 2 lines
lib/librte_sched/rte_sched.c +2 -0
+     if (cycles < port->time_cpu_cycles)
+             goto end;

Above check seems redundant as port->time_cpu_cycles will always be

less than the current cycles due to roundoff in previous iteration.

Quoted 2 lines
Quoted 1 line
Quoted 7 lines
Quoted 1 line

This was to catch the condition where the cycles wraps back to zero (after

100+ years?? depending on clock speed).

Quoted 3 lines

Rather than just going to end: the conditional should at least reset port- time_cpu_cycles back to zero.
So there would be a very temporary glitch in accuracy once every 100+

years.

Alan, Could you please resubmit the patch with above change? Other than that, patch looks good to me.

#7 lore raw

PATCH v2 1/1 sched: fix port time rounding error

In reply to [email protected]

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +9 -2
index c0983ddda..7c022cd61 100644
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
Signature · 1 line
2.17.1
#8 lore raw

Cristian, Jasvinder, please review.

25/06/2020 11:59, [email protected]:

Quoted 75 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +9 -2
index c0983ddda..7c022cd61 100644
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
#9 lore raw
Quoted 81 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Sunday, July 5, 2020 9:42 PM
To: Dumitrescu, Cristian <[email protected]>; Singh, Jasvinder <[email protected]>
Cc: [email protected]; Alan Dewar <[email protected]>; [email protected]
Subject: Re: [dpdk-dev] [PATCH v2] sched: fix port time rounding error

Cristian, Jasvinder, please review.

25/06/2020 11:59, [email protected]:

Quoted 35 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte.
This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/lib/librte_sched/rte_sched.c index c0983ddda..7c022cd61 100644

lib/librte_sched/rte_sched.c struct rte_sched_port { · line 222

Quoted 1 line
lib/librte_sched/rte_sched.c +0 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes

*/

lib/librte_sched/rte_sched.c rte_sched_port_config(struct

Quoted 7 lines
lib/librte_sched/rte_sched.c +1 -0
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct

rte_sched_port_params *params)

Quoted 11 lines
lib/librte_sched/rte_sched.c +1 -1
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)  {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;

lib/librte_sched/rte_sched.c static inline void · line 2686

Quoted 10 lines
lib/librte_sched/rte_sched.c +5 -0
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff <<

RTE_SCHED_TIME_SHIFT,

Quoted 12 lines

lib/librte_sched/rte_sched.c static inline void

Quoted 6 lines
lib/librte_sched/rte_sched.c +2 -1
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >>

RTE_SCHED_TIME_SHIFT;

Quoted 4 lines
lib/librte_sched/rte_sched.c +0 -0
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
 
#10 lore raw
Quoted 24 lines
Quoted 24 lines

25/06/2020 11:59, [email protected]:

Quoted 23 lines

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte.
This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Fixes de3cfa2c98 ("sched: initial import")

Applied, thanks

#11 lore raw
In reply to [email protected]

Hi,

Quoted 35 lines · On 25/06/2020 10:59, [email protected] wrote:

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +0 -0
index c0983ddda..7c022cd61 100644

lib/librte_sched/rte_sched.c struct rte_sched_port { · line 226

Quoted 5 lines
lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 

I was backporting this patch to 18.11. The older ABI checker complains about this structure change.

"cycles_per_byte has been added at the middle position of this structural type."

Isn't this an ABI break? Dropping from 18.11 for time being.

Quoted 35 lines
lib/librte_sched/rte_sched.c +8 -2
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
#12 lore raw
Quoted 47 lines · Kevin Traynor

Hi,

Quoted 35 lines · On 25/06/2020 10:59, [email protected] wrote:

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +0 -0
index c0983ddda..7c022cd61 100644

lib/librte_sched/rte_sched.c struct rte_sched_port { · line 226

Quoted 5 lines
lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 

I was backporting this patch to 18.11. The older ABI checker complains about this structure change.

Quoted 4 lines

"cycles_per_byte has been added at the middle position of this structural type."

Isn't this an ABI break? Dropping from 18.11 for time being.

So it looks like rte_sched_port * is an opaque pointers as it's contents are only known to rte_sched.c and not outside. To everyone else it is an opaque data structure, so structural changes to it would not be an ABI breakage.

Quoted 37 lines
Quoted 36 lines
lib/librte_sched/rte_sched.c +8 -2
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;
#13 lore raw
Quoted 50 lines · Ray
Quoted 47 lines · Kevin Traynor

Hi,

Quoted 35 lines · On 25/06/2020 10:59, [email protected] wrote:

From: Alan Dewar <[email protected]>

The QoS scheduler works off port time that is computed from the number of CPU cycles that have elapsed since the last time the port was
polled. It divides the number of elapsed cycles to calculate how
many bytes can be sent, however this division can generate rounding errors, where some fraction of a byte sent may be lost.

Lose enough of these fractional bytes and the QoS scheduler underperforms. The problem is worse with low bandwidths.

To compensate for this rounding error this fix doesn't advance the port's time_cpu_cycles by the number of cycles that have elapsed, but by multiplying the computed number of bytes that can be sent (which has been rounded down) by number of cycles per byte. This will mean that port's time_cpu_cycles will lag behind the CPU cycles momentarily. At the next poll, the lag will be taken into account.

v2:
If the cycles value wraps (100 year+) reset the port's cpu cycle back to zero.

Fixes de3cfa2c98 ("sched: initial import")
---
 lib/librte_sched/rte_sched.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
lib/librte_sched/rte_sched.c +0 -0
index c0983ddda..7c022cd61 100644

lib/librte_sched/rte_sched.c struct rte_sched_port { · line 226

Quoted 5 lines
lib/librte_sched/rte_sched.c +1 -0
@@ -222,6 +222,7 @@ struct rte_sched_port {
 	uint64_t time_cpu_bytes;      /* Current CPU time measured in bytes */
 	uint64_t time;                /* Current NIC TX time measured in bytes */
 	struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */
+	uint64_t cycles_per_byte;
 

I was backporting this patch to 18.11. The older ABI checker complains about this structure change.

Quoted 9 lines
Quoted 4 lines

"cycles_per_byte has been added at the middle position of this structural type."

Isn't this an ABI break? Dropping from 18.11 for time being.

So it looks like rte_sched_port * is an opaque pointers as it's contents are only known to rte_sched.c and not outside. To everyone else it is an opaque data structure, so structural changes to it would not be an ABI breakage.

Thanks Ray, makes sense. I've included the fix in 18.11.10-rc1.

Kevin.

Quoted 38 lines
Quoted 37 lines
Quoted 36 lines
lib/librte_sched/rte_sched.c +8 -2
 	/* Grinders */
 	struct rte_mbuf **pkts_out;
@@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params)
 	cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT)
 		/ params->rate;
 	port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte);
+	port->cycles_per_byte = cycles_per_byte;
 
 	/* Grinders */
 	port->pkts_out = NULL;
@@ -2673,16 +2675,21 @@ static inline void
 rte_sched_port_time_resync(struct rte_sched_port *port)
 {
 	uint64_t cycles = rte_get_tsc_cycles();
-	uint64_t cycles_diff = cycles - port->time_cpu_cycles;
+	uint64_t cycles_diff;
 	uint64_t bytes_diff;
 	uint32_t i;
 
+	if (cycles < port->time_cpu_cycles)
+		port->time_cpu_cycles = 0;
+
+	cycles_diff = cycles - port->time_cpu_cycles;
 	/* Compute elapsed time in bytes */
 	bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT,
 					   port->inv_cycles_per_byte);
 
 	/* Advance port time */
-	port->time_cpu_cycles = cycles;
+	port->time_cpu_cycles +=
+		(bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT;
 	port->time_cpu_bytes += bytes_diff;
 	if (port->time < port->time_cpu_bytes)
 		port->time = port->time_cpu_bytes;