Skip to contentSkip to conversation

librte_power w/ intel_pstate cpufreq governor

DISCUSSION Matthew Hall 2 participants 7 messages Dec 6, 2015 – Jan 14, 2016

lore
Share & tools
lore
#0 lore raw

Hello all,

I wanted to ask some questions about librte_power and the great adaptive polling / IRQ mode example in l3fwd-power.

I am very interested in getting this to work in my project because it will make it much friendlier to attract new community developers if I am as cooperative as possible with system resources.

Let's discuss the init process for a moment. It has some problems on my system, and I need some help to figure out how to handle this right.

1. Begins with the call to rte_power_init.

2. Attempts to init ACPI cpufreq mode.

2.1. Sets lcore cpufreq governor to userspace mode.

2.2. Function power_get_available_freqs checks lcore CPU frequencies from:

/sys/devices/system/cpu/cpuX/cpufreq/scaling_available_frequencies

2.3. This fails with (cryptic) error "POWER: ERR: File not openned". I am planning to write a patch for this error a bit later.

My kernel is using the intel_pstate driver, so scaling_available_frequencies does not exist:

http://askubuntu.com/questions/544266/why-are-missing-the-frequency-options-on-cpufreq-utils-indicator

3. When power_get_available_freqs fails, rte_power_acpi_cpufreq_init fails.

4. rte_power_init will try rte_power_kvm_vm_init. That will fail because it's a physical Skylake system not some kind of VM.

5. Now rte_power_init totally fails, with error "POWER: ERR: Unable to set Power Management Environment for lcore 0".

So, I have a couple of questions to figure out from here:

1. It seems bad to switch the governor into userspace before verifying the frequencies available in scaling_available_frequencies. If there are no frequencies available, it seems like it should not be trying to take over control of an effectively uncontrollable value.

2. If the governor is switched to userspace, and then no governing is done, it seems like the clockrate will necessarily always be wrong also because nothing will be configuring it anymore, neither kernel, nor failed DPDK userspace code, since rte_power_freq_up / down function pointers will always be NULL. Is this true? This seems bad if so.

It seems that the librte_power code is basically out of date, as pstate has been present since Sandy Bridge, which is quite old by now for network processing. I am not sure how to make this work right now. So far I see a couple options but I really don't know much about this stuff:

1) skip rte_power_init completely, and let intel_pstate handle it using HWP mode

2) disable intel_pstate, switch to the legacy ACPI cpufreq (but people warned this old driver is mostly a no-op and the CPU ignores its frequency requests).

The Internet advice says it's possible, but not a very good idea, to switch from the modern intel_pstate driver to the legacy ACPI mode. Reading through the kernel docs (below) state that it's better to use HWP (Hardware P State) mode:

https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt

If none of this rte_power_init stuff works, are the other CPU conservation measures inside the l3fwd-power example enough to work right with HWP all by themselves with nothing additional?

Thanks,
Matthew.

#1 lore raw

Hello,

In about one month, I never received any response about all these major issues I was finding with librte_power and the intel_pstate based CPU clockrate control driver used in all the new Linux kernels.

From what I can tell, none of this librte_power code ever worked right in the first place on Sandy Bridge and newer, because the chip secretly ignores clockrate adjustments from outside.

Can anyone who is more expert about Intel Power Management please help me check this and point me to some documentation which explains how this is supposed to work?

I am kind of blocked on doing performance / production quality improvements on my code, without some kind of basic help understanding how this librte_power stuff should work.

Thanks,
Matthew.

Quoted 76 lines · Matthew Hall

Hello all,

I wanted to ask some questions about librte_power and the great adaptive polling / IRQ mode example in l3fwd-power.

I am very interested in getting this to work in my project because it will make it much friendlier to attract new community developers if I am as cooperative as possible with system resources.

Let's discuss the init process for a moment. It has some problems on my system, and I need some help to figure out how to handle this right.

1. Begins with the call to rte_power_init.

2. Attempts to init ACPI cpufreq mode.

2.1. Sets lcore cpufreq governor to userspace mode.

2.2. Function power_get_available_freqs checks lcore CPU frequencies from:

/sys/devices/system/cpu/cpuX/cpufreq/scaling_available_frequencies

2.3. This fails with (cryptic) error "POWER: ERR: File not openned". I am planning to write a patch for this error a bit later.

My kernel is using the intel_pstate driver, so scaling_available_frequencies does not exist:

http://askubuntu.com/questions/544266/why-are-missing-the-frequency-options-on-cpufreq-utils-indicator

3. When power_get_available_freqs fails, rte_power_acpi_cpufreq_init fails.

4. rte_power_init will try rte_power_kvm_vm_init. That will fail because it's a physical Skylake system not some kind of VM.

5. Now rte_power_init totally fails, with error "POWER: ERR: Unable to set Power Management Environment for lcore 0".

So, I have a couple of questions to figure out from here:

1. It seems bad to switch the governor into userspace before verifying the frequencies available in scaling_available_frequencies. If there are no frequencies available, it seems like it should not be trying to take over control of an effectively uncontrollable value.

2. If the governor is switched to userspace, and then no governing is done, it seems like the clockrate will necessarily always be wrong also because nothing will be configuring it anymore, neither kernel, nor failed DPDK userspace code, since rte_power_freq_up / down function pointers will always be NULL. Is this true? This seems bad if so.

It seems that the librte_power code is basically out of date, as pstate has been present since Sandy Bridge, which is quite old by now for network processing. I am not sure how to make this work right now. So far I see a couple options but I really don't know much about this stuff:

1) skip rte_power_init completely, and let intel_pstate handle it using HWP mode

2) disable intel_pstate, switch to the legacy ACPI cpufreq (but people warned this old driver is mostly a no-op and the CPU ignores its frequency requests).

The Internet advice says it's possible, but not a very good idea, to switch from the modern intel_pstate driver to the legacy ACPI mode. Reading through the kernel docs (below) state that it's better to use HWP (Hardware P State) mode:

https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt

If none of this rte_power_init stuff works, are the other CPU conservation measures inside the l3fwd-power example enough to work right with HWP all by themselves with nothing additional?

Thanks,
Matthew.

#2 lore raw

Hi Matthew

Yes, you have indicated out the key, the power management module has changed or upgraded. Could you help to try the legacy one to see if it still works, as indicated in your link?

Taking control of the governor from kernel to user space, might need one more checks before that. But it is actually not a big issue, as user can switch it back to anything via 'echo'.

Yes, it seems that librte_power is out of date for a while. It is not easy to track all the kernel versions.
Now we have good chance to do that, as you have reported issues. Let's have a look on the new power management mechanism and then see if we can do something.

Really thanks to your questions!

Regards,
Helin

Quoted 110 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Matthew Hall
Sent: Sunday, January 3, 2016 3:51 PM
To: [email protected]
Subject: Re: [dpdk-dev] librte_power w/ intel_pstate cpufreq governor

Hello,

In about one month, I never received any response about all these major issues I was finding with librte_power and the intel_pstate based CPU clockrate control driver used in all the new Linux kernels.

From what I can tell, none of this librte_power code ever worked right in the
first place on Sandy Bridge and newer, because the chip secretly ignores clockrate adjustments from outside.

Can anyone who is more expert about Intel Power Management please help me check this and point me to some documentation which explains how this is supposed to work?

I am kind of blocked on doing performance / production quality improvements on my code, without some kind of basic help understanding how this librte_power stuff should work.

Thanks,
Matthew.

Quoted 12 lines · Matthew Hall

Hello all,

I wanted to ask some questions about librte_power and the great adaptive polling / IRQ mode example in l3fwd-power.

I am very interested in getting this to work in my project because it will make it much friendlier to attract new community developers if I am as cooperative as possible with system resources.

Let's discuss the init process for a moment. It has some problems on my system, and I need some help to figure out how to handle this right.

Quoted 7 lines

1. Begins with the call to rte_power_init.

2. Attempts to init ACPI cpufreq mode.

2.1. Sets lcore cpufreq governor to userspace mode.

2.2. Function power_get_available_freqs checks lcore CPU frequencies

from:

Quoted 10 lines

/sys/devices/system/cpu/cpuX/cpufreq/scaling_available_frequencies

2.3. This fails with (cryptic) error "POWER: ERR: File not openned". I am planning to write a patch for this error a bit later.

My kernel is using the intel_pstate driver, so
scaling_available_frequencies does not exist:

http://askubuntu.com/questions/544266/why-are-missing-the-frequency-

op

Quoted 3 lines

tions-on-cpufreq-utils-indicator

3. When power_get_available_freqs fails, rte_power_acpi_cpufreq_init

fails.

Quoted 25 lines

4. rte_power_init will try rte_power_kvm_vm_init. That will fail because it's a physical Skylake system not some kind of VM.

5. Now rte_power_init totally fails, with error "POWER: ERR: Unable to set Power Management Environment for lcore 0".

So, I have a couple of questions to figure out from here:

1. It seems bad to switch the governor into userspace before verifying the frequencies available in scaling_available_frequencies. If there are no frequencies available, it seems like it should not be trying to take over control of an effectively uncontrollable value.

2. If the governor is switched to userspace, and then no governing is done, it seems like the clockrate will necessarily always be wrong also because nothing will be configuring it anymore, neither kernel, nor failed DPDK userspace code, since rte_power_freq_up / down function pointers will always be NULL. Is this true? This seems bad if so.

It seems that the librte_power code is basically out of date, as pstate has been present since Sandy Bridge, which is quite old by now for network processing. I am not sure how to make this work right now. So far I see a couple options but I really don't know much about this stuff:

Quoted 5 lines

1) skip rte_power_init completely, and let intel_pstate handle it using HWP mode

2) disable intel_pstate, switch to the legacy ACPI cpufreq (but people warned this old driver is mostly a no-op and the CPU ignores its frequency

requests).

Quoted 16 lines

The Internet advice says it's possible, but not a very good idea, to switch from the modern intel_pstate driver to the legacy ACPI mode. Reading through the kernel docs (below) state that it's better to use HWP (Hardware P State)
mode:

https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt

If none of this rte_power_init stuff works, are the other CPU conservation measures inside the l3fwd-power example enough to work right with HWP all by themselves with nothing additional?

Thanks,
Matthew.

#3 lore raw
Quoted 4 lines · Helin

Hi Matthew

Yes, you have indicated out the key, the power management module has changed or upgraded. Could you help to try the legacy one to see if it still works, as indicated in your link?

I can do this, but according to the documents I am reading, the old Power Management module is secretly stubbed out / no-opped inside of the Skylake CPU core, and the core manages its own clockrate internally every 1 msec instead of every 30 msec with input from the OS (Intel Speed Shift technology).

If this is true, then I suspect there is no point to getting it to work again with either the old frequency driver or the new driver, because the chip would not listen to it. So then it seems like it makes sense to skip the clock adjustment callbacks on Skylake and take extra stuff out of the fastpath code.

Quoted 3 lines

Taking control of the governor from kernel to user space, might need one more checks before that. But it is actually not a big issue, as user can switch it back to anything via 'echo'.

I think it's a bit bigger issue, as it leaves the chip in full-power mode without really warning anybody, instead of the standard default adaptive mode.

Quoted 4 lines

Yes, it seems that librte_power is out of date for a while. It is not easy to track all the kernel versions. Now we have good chance to do that, as you have reported issues. Let's have a look on the new power management mechanism and then see if we can do something.

Yes, let me know how I could help. I don't know very much yet. My machine is Skylake Core i7-6700k. Unfortunately I think I am in trouble here, because there is no whitepaper on the Intel website for Intel Speed Shift technology at all.

Quoted 1 line

Really thanks to your questions!

I am looking forward to getting some answers figured out together.

Quoted 2 lines

Regards,
Helin

Matthew.

#4 lore raw
Quoted 4 lines · Matthew Hall

Yes, let me know how I could help. I don't know very much yet. My machine is Skylake Core i7-6700k. Unfortunately I think I am in trouble here, because there is no whitepaper on the Intel website for Intel Speed Shift technology at all.

This is the closest thing I could find:

http://wccftech.com/idf15-intel-skylake-analysis-cpu-gpu-microarchitecture-ddr4-memory-impact/4/

Some copy of a presentation from Intel IDF15.

Can somebody at Intel help me to find more papers or the right instruction or architecture manuals for HWP (Hardware P-State) feature?

Matthew.

#5 lore raw
In reply to Matthew Hall
Quoted 23 lines

-----Original Message-----
From: Matthew Hall [mailto:[email protected]]
Sent: Thursday, January 14, 2016 3:04 PM
To: Zhang, Helin
Cc: [email protected]; Liang, Cunming; Zhou, Danny
Subject: Re: [dpdk-dev] librte_power w/ intel_pstate cpufreq governor

Quoted 3 lines · Helin

Hi Matthew

Yes, you have indicated out the key, the power management module has

changed or upgraded.

Quoted 1 line

Could you help to try the legacy one to see if it still works, as indicated in

your link?

I can do this, but according to the documents I am reading, the old Power Management module is secretly stubbed out / no-opped inside of the Skylake CPU core, and the core manages its own clockrate internally every 1 msec instead of every 30 msec with input from the OS (Intel Speed Shift technology).

Quoted 4 lines

If this is true, then I suspect there is no point to getting it to work again with either the old frequency driver or the new driver, because the chip would not listen to it. So then it seems like it makes sense to skip the clock adjustment callbacks on Skylake and take extra stuff out of the fastpath code.

That's disappointing if Skylake is like that. Let's have a learning first, and then check if we can fix that. But in addition, DPDK provide interrupt based packet receiving mechanism, can it be one of your choice?

For now, I am afraid that I don't have time on it, as we are all focusing on the next release development. If no objection, I will find time later (may be in a month) to investigate that.
Of cause, please try to investigate that from your side.

Quoted 7 lines
Quoted 4 lines

Taking control of the governor from kernel to user space, might need one more checks before that. But it is actually not a big issue, as user can switch it back to anything via 'echo'.

I think it's a bit bigger issue, as it leaves the chip in full-power mode without really warning anybody, instead of the standard default adaptive mode.

That's always there, for example, DPDK can exit accidently, without caring anything. Then you can have the similar issue again.

Quoted 10 lines
Quoted 5 lines

Yes, it seems that librte_power is out of date for a while. It is not easy to track all the kernel versions. Now we have good chance to do that, as you have reported issues. Let's have a look on the new power management mechanism and then see if we can do something.

Yes, let me know how I could help. I don't know very much yet. My machine is Skylake Core i7-6700k. Unfortunately I think I am in trouble here, because there is no whitepaper on the Intel website for Intel Speed Shift technology at all.

It seems that you are so important for Intel. :) I don't have Skylake in hand. :( Anyway, I will try to find time on that, and hopefully will find something or solution. Thank you very much for the great jobs!

Regards,
Helin

Quoted 9 lines
Quoted 2 lines

Really thanks to your questions!

I am looking forward to getting some answers figured out together.

Quoted 2 lines

Regards,
Helin

Matthew.

#6 lore raw
Quoted 3 lines · Helin

That's disappointing if Skylake is like that. Let's have a learning first, and then check if we can fix that. But in addition, DPDK provide interrupt based packet receiving mechanism, can it be one of your choice?

Maybe I am wrong. But I could not disprove what the Linux p_state driver Documentation file and other places claimed, which is that the clockrate control is no-opped, because the white papers on Intel HWP are not findable in the Intel website, or by using Google with the operator "site:intel.com".

The IRQ based part is still enabled and works quite well in a very trivial test so far... but the clockrate callback handlers are null and the governor setting gets corrupted, both due to failed init of librte_power. So I will have to rebuild DPDK with the librte_power ACPI + KVM init commented out and the fastpath clockrate callback functions commented out of course. It is minor so I can do it to see what will happen.

Quoted 2 lines

If no objection, I will find time later (may be in a month) to investigate that. Of cause, please try to investigate that from your side.

Agreed.

Quoted 2 lines

That's always there, for example, DPDK can exit accidently, without caring anything. Then you can have the similar issue again.

Of course, it could. But if there was some kind of shutdown function, at least then I could call it from the signal handler I already have which closes the ports (this prevents nasty port lockups on virtio-net port DMA memory zones which can happen on future runs otherwise).

Quoted 1 line

It seems that you are so important for Intel. :) I don't have Skylake in hand. :(

:) Hahaha... newegg.com to the rescue. I guess we need to be sure there is some program to test the stuff in DPDK for the new kernels and hardware. It appears we are pretty far behind now... I saw several threads about things that were behind just today.

Quoted 2 lines

Regards,
Helin

Matthew.