KVM CPU frequency change hypercalls (resend)

Paolo: please comment on your objections and what should be done instead. Note the case "multiple vcpus on a given pcpu" is not part of the usecase in question.

Quoted 4 lines · Marcelo Tosatti

Paolo: please comment on your objections and what should be done instead. Note the case "multiple vcpus on a given pcpu" is not part of the usecase in question.

I would like to understand the intended usecase of cpufreq-userspace.

My understanding is that you would have a daemon handling a systemwide policy; examples are the historical (and now obsolete) users such as cpufreqd, cpudyn, powernowd, or cpuspeed.

The user alternatively can play the role of the daemon by writing to sysfs, but I've never seen userspace tasks talking to cpufreq-userspace to set their own running frequency.

Apparently DPDK does that, and I would like to know the opinion of the linux-pm folks;

one obvious downside is that any application that you
run after DPDK will have its CPU frequency hardcoded to something that is not appropriate.

This might be acceptable for DPDK, but it is worse for KVM which tries to provide isolation to its vCPU tasks.

Here are two possibilities that I could think of:

1) Introduce a mechanism that allows a task to override the governor's choice of CPU frequency. This could be a ioctl, a prctl, a cgroup-based mechanism or whatever else. As Marcelo pointed out in the original kvm@ thread, the latency and overhead of switching frequencies make it impractical to associate a desired CPU frequency with a task, because multiple tasks could be requesting a given frequency. One possibility could be to treat the per-task CPU frequency as advisory

and only obey
it in restricted cases---for example only if nohz_full is in effect.

From cpufreq documentation:

Quoted 3 lines · Paolo Bonzini

one obvious downside is that any application that you
run after DPDK will have its CPU frequency hardcoded to something that is not appropriate.

To isolate the CPU where DPDK runs it is already necessary to perform special procedures such as changing the cpumask of other tasks, changing cpumask of interrupt handlers (to remove the isolated CPU from that cpumask), etc. Changing the cpufreq governor to userspace is another step of that setup phase.

On shutdown (or CPU unpin), you can switch back the CPU to the previous governor, which can switch the frequency to whatever it finds suitable.

Quoted 9 lines

Here are two possibilities that I could think of:

1) Introduce a mechanism that allows a task to override the governor's choice of CPU frequency. This could be a ioctl, a prctl, a cgroup-based mechanism or whatever else. As Marcelo pointed out in the original kvm@ thread, the latency and overhead of switching frequencies make it impractical to associate a desired CPU frequency with a task, because multiple tasks could be requesting a given frequency. One possibility could be to treat the per-task CPU frequency as advisory

DPDK can't afford the frequency as advisory: failure in setting the processor frequency when requested means dropped packets (not dropping packets being a requirement).

Quoted 2 lines

and only obey
it in restricted cases---for example only if nohz_full is in effect.

From cpufreq documentation:

"On all other cpufreq implementations, these boundaries still need to be set. Then, a "governor" must be selected. Such a "governor" decides what speed the processor shall run within the boundaries. One such "governor" is the "userspace" governor. This one allows the user - or a yet-to-implement userspace program - to decide what specific speed the processor shall run at."

(it seems the cpufreq-hypercall+cpufreq-userspace combination is in accord with what cpufreq-userspace has been designed for).

Secondly, setting frequencies for multiple tasks is somewhat contradictory:

In the DPDK context, or in any context actually, it makes sense for a program to lower processor frequency when it decides the current frequency is sufficient to handle the job: that is lowering the frequency will still make it possible to handle the load.

With multiple applications sharing that processor, the percentage of time given to a certain application also interferes with the time it spends handling the job. So the other variable that affects "instructions per second" is timeslice given to the task by the scheduler, not only "frequency".

Having a task request for a particular frequency in that case becomes
ambiguous: you could be asking for "increased timeslice".

Quoted 5 lines · Marcelo Tosatti

Quoted 3 lines · Paolo Bonzini

one obvious downside is that any application that you
run after DPDK will have its CPU frequency hardcoded to something that is not appropriate.

Quoted 8 lines

To isolate the CPU where DPDK runs it is already necessary to perform special procedures such as changing the cpumask of other tasks, changing cpumask of interrupt handlers (to remove the isolated CPU from that cpumask), etc. Changing the cpufreq governor to userspace is another step of that setup phase.

On shutdown (or CPU unpin), you can switch back the CPU to the previous governor, which can switch the frequency to whatever it finds suitable.

But I thought that one of the reasons to do NFV is to simplify this setup. If you now have to do the same thing on virtual machines, things become more complicated to set up, and I don't think that NFV virtual machines are _that_ special.

In addition, in the list of setup steps above you forgot "chmod the sysfs files for cpufreq so that DPDK can access it". Doing that chmod is a very explicit act, and that's unlike the functionality of this patch.

By letting virtual machines do the same with a simple hypercall, you're giving powers to whoever opens /dev/kvm that they didn't have before (unless the userspace process also had access to sysfs). Worse, the effects last beyond the moment /dev/kvm is closed.

So, the question then is how to design the hypervisor so that these NFV virtual machines can play with cpufreq, but there are no adverse indefinite effects.

One possibility is to have some kind of per-task
cpufreq. Another is to do everything in userspace with virtual ACPI P-states and the userspace governor in the VM.

I was hoping to get more feedback from linux-pm.

Quoted 10 lines

Quoted 9 lines

Here are two possibilities that I could think of:

1) Introduce a mechanism that allows a task to override the governor's choice of CPU frequency. This could be a ioctl, a prctl, a cgroup-based mechanism or whatever else. As Marcelo pointed out in the original kvm@ thread, the latency and overhead of switching frequencies make it impractical to associate a desired CPU frequency with a task, because multiple tasks could be requesting a given frequency. One possibility could be to treat the per-task CPU frequency as advisory

Quoted 3 lines

DPDK can't afford the frequency as advisory: failure in setting the processor frequency when requested means dropped packets (not dropping packets being a requirement).

It can be advisory if you document a proper configuration where it's obeyed.

Paolo

Quoted 31 lines

Quoted 2 lines

and only obey
it in restricted cases---for example only if nohz_full is in effect.

From cpufreq documentation:

"On all other cpufreq implementations, these boundaries still need to be set. Then, a "governor" must be selected. Such a "governor" decides what speed the processor shall run within the boundaries. One such "governor" is the "userspace" governor. This one allows the user - or a yet-to-implement userspace program - to decide what specific speed the processor shall run at."

(it seems the cpufreq-hypercall+cpufreq-userspace combination is in accord with what cpufreq-userspace has been designed for).

Secondly, setting frequencies for multiple tasks is somewhat contradictory:

In the DPDK context, or in any context actually, it makes sense for a program to lower processor frequency when it decides the current frequency is sufficient to handle the job: that is lowering the frequency will still make it possible to handle the load.

With multiple applications sharing that processor, the percentage of time given to a certain application also interferes with the time it spends handling the job. So the other variable that affects "instructions per second" is timeslice given to the task by the scheduler, not only "frequency".

Having a task request for a particular frequency in that case becomes
ambiguous: you could be asking for "increased timeslice".

Quoted 3 lines

So, the question then is how to design the hypervisor so that these NFV virtual machines can play with cpufreq, but there are no adverse indefinite effects.

Ok, we can modify the cpufreq cgroups patch, to, from the hypercalls set the:

"The first three patches of this series introduces capacity_{min,max} tracking
in the core scheduler, as an extension of the CPU controller."

capacity_min == capacity_max values (which forces the CPU to run at that frequency, given there are no other tasks requesting frequency information on that CPU).

This is good enough DPDK.

Quoted 3 lines

One possibility is to have some kind of per-task
cpufreq. Another is to do everything in userspace with virtual ACPI P-states and the userspace governor in the VM.

Virtual ACPI P-state, that is an option. But why not make it in-kernel, the exit to userspace can be a significant fraction of the total if the frequency change time is small (say, 10us freq change and 5us for userspace exit).

Quoted 10 lines · Marcelo Tosatti

Quoted 3 lines

So, the question then is how to design the hypervisor so that these NFV virtual machines can play with cpufreq, but there are no adverse indefinite effects.

Ok, we can modify the cpufreq cgroups patch, to, from the hypercalls set the:

"The first three patches of this series introduces capacity_{min,max} tracking
in the core scheduler, as an extension of the CPU controller."

Quoted 5 lines

capacity_min == capacity_max values (which forces the CPU to run at that frequency, given there are no other tasks requesting frequency information on that CPU).

This is good enough DPDK.

So this sounds like a plan?

Quoted 9 lines

Quoted 4 lines

One possibility is to have some kind of per-task
cpufreq. Another is to do everything in userspace with virtual ACPI P-states and the userspace governor in the VM.

Virtual ACPI P-state, that is an option. But why not make it in-kernel, the exit to userspace can be a significant fraction of the total if the frequency change time is small (say, 10us freq change and 5us for userspace exit).

The advantage of doing it in userspace is that the sysfs chmod is a clear way to say "this VM should have the privilege of setting cpufreq.
In effect, userspace's file descriptor for the sysfs files represents
the capability to set cpufreq for the VM. You can even pass the file descriptor with SCM_RIGHTS if you wish to do so.

But of course that's only needed if the frequency change is global per physical CPU. if the CPU controller gains the ability to do per-task frequency switching, that's even better for KVM. Then the hypercalls are just fine and we can have a KVM-specific cpufreq controller.

Paolo