ring: adding TPAUSE instruction to ring dequeue

From: David Coyle [mailto:[email protected]]
Sent: Wednesday, 3 May 2023 13.39

This is NOT for upstreaming. This is being submitted to allow early comparison testing with the preferred solution, which will add TAPUSE power management support to the ring library through the addition of callbacks. Initial stages of the preferred solution are available at
http://dpdk.org/patch/125454.

This patch adds functionality directly to rte_ring_dequeue functions to monitor the empty reads of the ring. When a configurable number of empty reads is reached, a TPAUSE instruction is triggered by using rte_power_pause() on supported architectures. rte_pause() is used on other architectures. The functionality can be included or excluded at compilation time using the RTE_RING_PMGMT flag. If included, the new API can be used to enable/disable the feature on a per-ring basis. Other related settings can also be configured using the API.

Quoted 9 lines

From: David Coyle [mailto:[email protected]]
Sent: Wednesday, 3 May 2023 13.39

This is NOT for upstreaming. This is being submitted to allow early comparison testing with the preferred solution, which will add TAPUSE power management support to the ring library through the addition of callbacks. Initial stages of the preferred solution are available at
http://dpdk.org/patch/125454.

Quoted 8 lines

This patch adds functionality directly to rte_ring_dequeue functions to monitor the empty reads of the ring. When a configurable number of empty reads is reached, a TPAUSE instruction is triggered by using rte_power_pause() on supported architectures. rte_pause() is used on other architectures. The functionality can be included or excluded at compilation time using the RTE_RING_PMGMT flag. If included, the new API can be used to enable/disable the feature on a per-ring basis. Other related settings can also be configured using the API.

I don't understand why DPDK developers keep spending time on trying to invent methods to determine application busyness based on entry/exit points in a variety of libraries, when the application is in a much better position to determine busyness. All of these "busyness measuring" library extensions have their own specific assumptions and weird limitations.

I do understand that the goal is power saving, which certainly is relevant! I only criticize the measuring methods.

For reference, we implemented something very simple in our application framework:
1. When each pipeline stage has completed a burst, it reports if it was busy or not.
2. If the pipeline busyness is low, we take a nap to save some power.

And here is the magic twist to this simple algorithm:
3. A pipeline stage is not considered busy unless it processed a full burst, and is ready to process more packets immediately. This interpretation of busyness has a significant impact on the percentage of time spent napping during the low-traffic hours.

This algorithm was very quickly implemented. It might not be perfect, and we do intend to improve it (also to determine CPU Utilization on a scale that the end user can translate to a linear interpretation of how busy the system is). But I seriously doubt that any of the proposed "busyness measuring" library extensions are any better.

So: The application knows better, please spend your precious time on something useful instead.

@David, my outburst is not directed at you specifically. Generally, I do appreciate experimenting as a good way of obtaining knowledge. So thank you for sharing your experiments with this audience!

PS: If cruft can be disabled at build time, I generally don't oppose to it.

-----Original Message-----
From: Morten Brørup <[email protected]>

Quoted 9 lines

From: David Coyle [mailto:[email protected]]
Sent: Wednesday, 3 May 2023 13.39

This is NOT for upstreaming. This is being submitted to allow early comparison testing with the preferred solution, which will add TAPUSE power management support to the ring library through the addition of callbacks. Initial stages of the preferred solution are available at
http://dpdk.org/patch/125454.

Quoted 8 lines

This patch adds functionality directly to rte_ring_dequeue functions to monitor the empty reads of the ring. When a configurable number of empty reads is reached, a TPAUSE instruction is triggered by using rte_power_pause() on supported architectures. rte_pause() is used on other architectures. The functionality can be included or excluded at compilation time using the RTE_RING_PMGMT flag. If included, the new API can be used to enable/disable the feature on a per-ring basis. Other related settings can also be configured using the API.

I don't understand why DPDK developers keep spending time on trying to invent methods to determine application busyness based on entry/exit points in a variety of libraries, when the application is in a much better position to determine busyness. All of these "busyness measuring" library extensions have their own specific assumptions and weird limitations.

I do understand that the goal is power saving, which certainly is relevant! I only criticize the measuring methods.

For reference, we implemented something very simple in our application framework:
1. When each pipeline stage has completed a burst, it reports if it was busy or not.
2. If the pipeline busyness is low, we take a nap to save some power.

And here is the magic twist to this simple algorithm:
3. A pipeline stage is not considered busy unless it processed a full burst, and is ready to process more packets immediately. This interpretation of busyness has a significant impact on the percentage of time spent napping during the low-traffic hours.

This algorithm was very quickly implemented. It might not be perfect, and we do intend to improve it (also to determine CPU Utilization on a scale that the end user can translate to a linear interpretation of how busy the system is). But I seriously doubt that any of the proposed "busyness measuring" library extensions are any better.

So: The application knows better, please spend your precious time on something useful instead.

@David, my outburst is not directed at you specifically. Generally, I do appreciate experimenting as a good way of obtaining knowledge. So thank you for sharing your experiments with this audience!

PS: If cruft can be disabled at build time, I generally don't oppose to it.

From: Coyle, David [mailto:[email protected]]
Sent: Wednesday, 3 May 2023 17.32

Hi Morten

Quoted 7 lines

From: Morten Brørup <[email protected]>

Quoted 5 lines

From: David Coyle [mailto:[email protected]]
Sent: Wednesday, 3 May 2023 13.39

This is NOT for upstreaming. This is being submitted to allow early comparison testing with the preferred solution, which will add

TAPUSE

Quoted 6 lines

Quoted 6 lines

power management support to the ring library through the addition of callbacks. Initial stages of the preferred solution are available at
http://dpdk.org/patch/125454.

This patch adds functionality directly to rte_ring_dequeue functions to monitor the empty reads of the ring. When a configurable number

of

Quoted 3 lines

Quoted 3 lines

empty reads is reached, a TPAUSE instruction is triggered by using rte_power_pause() on supported architectures. rte_pause() is used on other architectures. The functionality can be included or excluded

at

Quoted 7 lines

Quoted 3 lines

compilation time using the RTE_RING_PMGMT flag. If included, the new API can be used to enable/disable the feature on a per-ring basis. Other related settings can also be configured using the API.

I don't understand why DPDK developers keep spending time on trying to invent methods to determine application busyness based on entry/exit points in a variety of libraries, when the application is in a much

better

Quoted 1 line

position to determine busyness. All of these "busyness measuring"

library

Quoted 3 lines

extensions have their own specific assumptions and weird limitations.

I do understand that the goal is power saving, which certainly is

relevant! I

Quoted 5 lines

only criticize the measuring methods.

For reference, we implemented something very simple in our application framework:
1. When each pipeline stage has completed a burst, it reports if it

was busy or

Quoted 5 lines

not.
2. If the pipeline busyness is low, we take a nap to save some power.

And here is the magic twist to this simple algorithm:
3. A pipeline stage is not considered busy unless it processed a full

burst, and

Quoted 2 lines

is ready to process more packets immediately. This interpretation of busyness has a significant impact on the percentage of time spent

napping

Quoted 3 lines

during the low-traffic hours.

This algorithm was very quickly implemented. It might not be perfect,

and we

Quoted 1 line

do intend to improve it (also to determine CPU Utilization on a scale

that the

Quoted 1 line

end user can translate to a linear interpretation of how busy the

system is).

Quoted 1 line

But I seriously doubt that any of the proposed "busyness measuring"

library

Quoted 6 lines

extensions are any better.

So: The application knows better, please spend your precious time on something useful instead.

@David, my outburst is not directed at you specifically. Generally, I

do

Quoted 1 line

appreciate experimenting as a good way of obtaining knowledge. So

thank

Quoted 3 lines

you for sharing your experiments with this audience!

PS: If cruft can be disabled at build time, I generally don't oppose

to it.

[DC] Appreciate that feedback, and it is certainly another way of looking at
and tackling the problem that we are ultimately trying to solve (i.e power
saving)

The problem however is that we work with a large number of ISVs and operators,
each with their own workload architecture and implementation. That means we
would have to work individually with each of these to integrate this type of
pipeline-stage-busyness algorithm into their applications. And as these applications are usually commercial, non-open-source applications, that could
prove to be very difficult.

Also most ISVs and operators don't want to have to worry about changing their
application, especially their fast-path dataplane, in order to get power savings. They prefer for it to just happen without them caring about the finer
details.

For these reasons, consolidating the busyness algorithms down into the DPDK
libraries and PMDs is currently the preferred solution. As you say though, the
libraries and PMDs may not be in the best position to determine the busyness
of the pipeline, but it provides a good balance between achieving power savings
and ease of adoption.

It's also worth calling out again that this patch is only to allow early testing by some customers of the benefit of adding TPAUSE support to the ring
library. We don't intend on this patch being upstreamed. The preferred longer
term solution is to use callbacks from the ring library to initiate the pause
(either via the DPDK power management API or through functions that an ISV
may write themselves). This is mentioned in the commit message.

Also, the pipeline stage busyness algorithm that you have added to your pipeline - have you ever considered implementing this into DPDK as a generic
type library. This could certainly be of benefit to other DPDK application
developers, and having this mechanism in DPDK could again ease the adoption
and realisation of power savings for others. I understand though if this is your
own secret sauce and you want to keep it like that :)

-----Original Message-----
From: Morten Brørup <[email protected]>

Power saving is important for the environment (to save the planet and all that), so everyone should contribute, if they have a good solution. So even if our algorithm had a significant degree of innovation, we would probably choose to make it public anyway. Open sourcing it also makes it possible for chip vendors like Intel to fine tune it more than we can ourselves, which also comes back to benefit us. All products need some sort of power saving in to stay competitive, but power saving algorithms is not an area we want to pursue for competitive purposes in our products.

Our algorithm is too simple to make a library at this point, but I have been thinking about how we can make it a generic library when it has matured some more. I will take your information about the many customers' need to have it invisibly injected into consideration in this regard.

Our current algorithm works like this:

while (running) {
int more = 0;
more += stage1();
more += stage2();
more += stage3();
if (!more) sleep();
}

Each pipeline stage only returns 1 if it processed a full burst. Furthermore, if a pipeline stage processed a full burst, but happens to know that no more data is readily available for it, it returns 0 instead.

Obviously, the sleep() duration must be short enough to avoid that the NIC RX descriptor rings overflow before the ingress pipeline stage is serviced again.

Changing the algorithm to "more" (1 = more work expected by the pipeline stage) from "busy" (1 = some work done by the pipeline stage) has the consequence that sleep() is called more often, which has the follow-on consequence that the ingress stage is called less often, and thus more often has a full burst to process.

We know from our in-house profiler that processing a full burst provides *much* higher execution efficiency (cycles/packet) than processing a few packets. This is public knowledge - after all, this is the whole point of DPDK's vector packet processing design! Nonetheless, it might surprise some people how much the efficiency (cycles/packet) increases when processing a full burst compared to processing just a few packets. I will leave it up to the readers to make their own experiments. :-)

Our initial "busy" algorithm behaved like this:
Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, No packets to process (we are lucky this time!), sleep briefly, Repeat.

So we switched to our "more" algorithm, which behaves like this:
Process a few packets (at low efficiency), sleep briefly, Process a full burst of packets (at high efficiency), don't sleep, Repeat.

Instead of processing e.g. 8 small bursts per sleep, we now process only 2 bursts per sleep. And the big of the two bursts is processed at higher efficiency.

We can improve this algorithm in some areas...

E.g. some of our pipeline stages also know that they are not going to do anymore work for the next X amount of nanoseconds; but we don't use that information in our power management algorithm yet. The sleep duration could depend on this.

Also, we don't use the CPU power management states yet. I assume that doing some work for 20 us at half clock speed is more power conserving than doing the same work at full speed for 10 us and then sleeping for 10 us. That's another potential improvement.

What we need in generic a power management helper library are functions to feed it with the application's perception of how much work is being done, and functions to tell if we can sleep and/or if we should change the power management states of the individual CPU cores.

Such a unified power management helper (or "busyness") library could perhaps also be fed with data directly from the drivers and libraries to support the customer use cases you described.

Hi Morten

Quoted 3 lines

-----Original Message-----
From: Morten Brørup <[email protected]>

<snip>

Quoted 76 lines

Power saving is important for the environment (to save the planet and all that), so everyone should contribute, if they have a good solution. So even if our algorithm had a significant degree of innovation, we would probably choose to make it public anyway. Open sourcing it also makes it possible for chip vendors like Intel to fine tune it more than we can ourselves, which also comes back to benefit us. All products need some sort of power saving in to stay competitive, but power saving algorithms is not an area we want to pursue for competitive purposes in our products.

Our algorithm is too simple to make a library at this point, but I have been thinking about how we can make it a generic library when it has matured some more. I will take your information about the many customers' need to have it invisibly injected into consideration in this regard.

Our current algorithm works like this:

while (running) {
int more = 0;
more += stage1();
more += stage2();
more += stage3();
if (!more) sleep();
}

Each pipeline stage only returns 1 if it processed a full burst. Furthermore, if a pipeline stage processed a full burst, but happens to know that no more data is readily available for it, it returns 0 instead.

Obviously, the sleep() duration must be short enough to avoid that the NIC RX descriptor rings overflow before the ingress pipeline stage is serviced again.

Changing the algorithm to "more" (1 = more work expected by the pipeline stage) from "busy" (1 = some work done by the pipeline stage) has the consequence that sleep() is called more often, which has the follow-on consequence that the ingress stage is called less often, and thus more often has a full burst to process.

We know from our in-house profiler that processing a full burst provides *much* higher execution efficiency (cycles/packet) than processing a few packets. This is public knowledge - after all, this is the whole point of DPDK's vector packet processing design! Nonetheless, it might surprise some people how much the efficiency (cycles/packet) increases when processing a full burst compared to processing just a few packets. I will leave it up to the readers to make their own experiments. :-)

Our initial "busy" algorithm behaved like this:
Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, No packets to process (we are lucky this time!), sleep briefly, Repeat.

So we switched to our "more" algorithm, which behaves like this:
Process a few packets (at low efficiency), sleep briefly, Process a full burst of packets (at high efficiency), don't sleep, Repeat.

Instead of processing e.g. 8 small bursts per sleep, we now process only 2 bursts per sleep. And the big of the two bursts is processed at higher efficiency.

We can improve this algorithm in some areas...

E.g. some of our pipeline stages also know that they are not going to do anymore work for the next X amount of nanoseconds; but we don't use that information in our power management algorithm yet. The sleep duration could depend on this.

Also, we don't use the CPU power management states yet. I assume that doing some work for 20 us at half clock speed is more power conserving than doing the same work at full speed for 10 us and then sleeping for 10 us. That's another potential improvement.

Quoted 8 lines

What we need in generic a power management helper library are functions to feed it with the application's perception of how much work is being done, and functions to tell if we can sleep and/or if we should change the power management states of the individual CPU cores.

Such a unified power management helper (or "busyness") library could perhaps also be fed with data directly from the drivers and libraries to support the customer use cases you described.

[DC] Thank you for that detailed description, very interesting. There may well be merit in upstreaming such an algorithm as a library once it has matured as you said.

Configuration could include specifying what a "full burst"
actually is. Different stages of a pipeline may also have different definitions of busyness, so that may also need to considered:
- Some stages may perform an operation (e.g. running an acl rule check) on a burst of packets and then it is complete
- Other stages may be more asynchronous in nature e.g. enqueuing and dequeuing to/from a crypto device or a QoS scheduler. The dequeue might not dequeue any packets on a particular call of the dequeue API, but there may still be packets waiting inside the crypto device or scheduler. Those waiting packets would also need to be taken into account so as not to sleep for too long.

Using such an API would require a workload developer to update their datapath to report the pipeline stage busyness to the algorithm, but if those calls are kept to a minimum, then that shouldn't be too much of a problem

Thanks,
David

-----Original Message-----
From: Stephen Hemminger [mailto:[email protected]]

Quoted 10 lines · David

Hi Morten

Quoted 3 lines

-----Original Message-----
From: Morten Brørup <[email protected]>

<snip>

Quoted 2 lines

Power saving is important for the environment (to save the planet and all that), so everyone should contribute, if they have a good solution. So

even if

Quoted 2 lines

Quoted 2 lines

our algorithm had a significant degree of innovation, we would probably choose to make it public anyway. Open sourcing it also makes it possible

for

Quoted 1 line

Quoted 1 line

chip vendors like Intel to fine tune it more than we can ourselves, which

also

Quoted 1 line

Quoted 1 line

comes back to benefit us. All products need some sort of power saving in

to

Quoted 4 lines

Quoted 4 lines

stay competitive, but power saving algorithms is not an area we want to pursue for competitive purposes in our products.

Our algorithm is too simple to make a library at this point, but I have

been

Quoted 14 lines

Quoted 14 lines

thinking about how we can make it a generic library when it has matured some more. I will take your information about the many customers' need to have it invisibly injected into consideration in this regard.

Our current algorithm works like this:

while (running) {
int more = 0;
more += stage1();
more += stage2();
more += stage3();
if (!more) sleep();
}

Quoted 1 line

Quoted 1 line

Each pipeline stage only returns 1 if it processed a full burst.

Furthermore, if a

Quoted 1 line

Quoted 1 line

pipeline stage processed a full burst, but happens to know that no more

data

Quoted 10 lines

Quoted 10 lines

is readily available for it, it returns 0 instead.

Obviously, the sleep() duration must be short enough to avoid that the NIC RX descriptor rings overflow before the ingress pipeline stage is serviced again.

Changing the algorithm to "more" (1 = more work expected by the pipeline stage) from "busy" (1 = some work done by the pipeline stage) has the consequence that sleep() is called more often, which has the follow-on consequence that the ingress stage is called less often, and thus more

often

Quoted 5 lines

Quoted 5 lines

has a full burst to process.

We know from our in-house profiler that processing a full burst provides *much* higher execution efficiency (cycles/packet) than processing a few packets. This is public knowledge - after all, this is the whole point of

DPDK's

Quoted 1 line

Quoted 1 line

vector packet processing design! Nonetheless, it might surprise some

people

Quoted 6 lines

Quoted 6 lines

how much the efficiency (cycles/packet) increases when processing a full burst compared to processing just a few packets. I will leave it up to the readers to make their own experiments. :-)

Our initial "busy" algorithm behaved like this:
Process a few packets (at low efficiency), don't sleep, Process a few

packets

Quoted 1 line

Quoted 1 line

(at low efficiency), don't sleep, Process a few packets (at low

efficiency),

Quoted 1 line

Quoted 1 line

don't sleep, Process a few packets (at low efficiency), don't sleep,

Process a

Quoted 1 line

Quoted 1 line

few packets (at low efficiency), don't sleep, Process a few packets (at

low

Quoted 2 lines

Quoted 2 lines

efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, Process a few packets (at low efficiency), don't sleep, No packets

to

Quoted 4 lines

Quoted 4 lines

process (we are lucky this time!), sleep briefly, Repeat.

So we switched to our "more" algorithm, which behaves like this:
Process a few packets (at low efficiency), sleep briefly, Process a full

burst of

Quoted 8 lines

Quoted 8 lines

packets (at high efficiency), don't sleep, Repeat.

Instead of processing e.g. 8 small bursts per sleep, we now process only 2 bursts per sleep. And the big of the two bursts is processed at higher efficiency.

We can improve this algorithm in some areas...

Quoted 7 lines

Quoted 7 lines

E.g. some of our pipeline stages also know that they are not going to do anymore work for the next X amount of nanoseconds; but we don't use that information in our power management algorithm yet. The sleep duration could depend on this.

Also, we don't use the CPU power management states yet. I assume that doing some work for 20 us at half clock speed is more power conserving

than

Quoted 6 lines

Quoted 6 lines

doing the same work at full speed for 10 us and then sleeping for 10 us. That's another potential improvement.

What we need in generic a power management helper library are functions to feed it with the application's perception of how much work is being

done,

Quoted 7 lines

Quoted 6 lines

and functions to tell if we can sleep and/or if we should change the power management states of the individual CPU cores.

Such a unified power management helper (or "busyness") library could perhaps also be fed with data directly from the drivers and libraries to support the customer use cases you described.

Quoted 6 lines

[DC] Thank you for that detailed description, very interesting. There may well be merit in upstreaming such an algorithm as a library once it has matured as you said.

Configuration could include specifying what a "full burst" actually is. Different stages of a pipeline may also have different

definitions

Quoted 3 lines

of busyness, so that may also need to considered:
- Some stages may perform an operation (e.g. running an acl rule check) on a burst of packets and then it is complete

Quoted 4 lines

- Other stages may be more asynchronous in nature e.g. enqueuing and dequeuing to/from a crypto device or a QoS scheduler. The dequeue might not dequeue any packets on a particular call of the dequeue API, but there may still be packets waiting inside the crypto device or scheduler. Those

waiting

Quoted 1 line

packets would also need to be taken into account so as not to sleep for too

long.

Quoted 2 lines

Using such an API would require a workload developer to update their

datapath

Quoted 1 line

to report the pipeline stage busyness to the algorithm, but if those calls

are

Quoted 1 line

kept to a minimum, then that shouldn't be too much of a problem

Quoted 3 lines

Thanks,
David

I see two overlapping discussions here:

The first, is using some form of memory wait when or timed pause for the cases where it is spinning on contended region like lock or ring concurrency. There already is some of this available on Arm64 and having TPAUSE used on intel
makes sense. Using TPAUSE in rte_usleep is obvious good idea.

The other is having some overall indication of busyness. This would be how often things like rx_burst and ring_dequeue get data to work on. A mechanism for this must be lightweight (ie per-core and minimum data collection), and plumbed into the telemetry system.

It makes sense that this would
be a new DPDK EAL call that would be used in place of the sleep done by most applications in the main loop when not busy.

Any solution should be architecture independent

None of the designs presented so far seem complete and simple enough to be part of the main DPDK distribution. Keep working and experimenting.