Skip to contentSkip to conversation

Powertop doesn't detects sleeping ARM cores

Powertop Igor Zhbanov 4 participants 12 messages Sep 17, 2012 – Sep 20, 2012

lore
Share & tools
lore

Igor Zhbanov

author
#0 lore raw

Hello!

When you start the PowerTOP on a "cold" ARM phone with several cores and low CPU average some cores could be powered down. It is very frequent situation on ARM platform to dynamically turn cores on/off to save power.

And when the PowerTOP scans /proc/cpuinfo file in enumerate_cpus() function, it will see only one working core, so PowerTOP calls handle_one_cpu(...) only once and sets number of CPUs to 1. So the PowerTOP will not register trace events for other cores.

Later when the system load (during the lifetime of PowerTOP) increases, the Linux kernel powers-up additional cores. But the PowerTOP will not catch events for them.

I thought about scanning the /sys/devices/system/cpu/ directory for finding out how many CPUs the system has. But the problem is that the directories for CPUs in offline mode doesn't provide needed subdirectories cpuidle/ and topology/ (also it misses some files in cpufreq/ directory).

The only way to enable these directories is to (temporarily?) turn the core on. This could be done e.g. by writing 1 to cpu/cpuX/online.
But I'm not sure how long the core will stay online before the kernel turns it off again because of low load. On my phone this happens in a less than a second.

Another possible way is to scan /sys/.../cpu/ for the number of CPUs in the system, register trace events for all that CPUs (by the way, is it possible to register a trace event for the core that is now off?), sleep for the needed amount of time as usual, and before the processing of data, temporarily turn every core on to find out information about it.

Or the PowerTOP could remember which cores was turned on before the testing, turn it all on, read needed info, turn some cores off to restore the state of the system and begin test.

Or it could scan /sys/.../cpu/ with some interval (1 second) instead of just sleeping, and grab the information about new cores brought online.

By the way, the PowerTOP scans /proc/ directory twice -- once before the start and once after to get information about processes (to provide more information from trace events). But if some process starts after the PowerTOP begins to sleep, and dies before the sleep ends, then the PowerTOP will be unable to get information about that process (e.g. command line). So the PowerTOP will know only its PID and shortened version of the command line (task->comm[16]).

I have a patch adding scanning of /proc/ capabilities to the PowerTOP. I use this /proc/ scanner in MALI supporting patch, because MALI provides only the PID of the process in their events, and that is not enough to know about the process. It needs more discussion so I will send the patch little later. But I want to notice, that it could be useful for the rest of the PowerTOP too to catch that processes. (The best solution will be to catch the process exit, but that event provides only short task->comm[16]).

Anyway, the PowerTOP could do more instead of just sleeping.

What do you think?

Thank you.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
author
#1 lore raw

Hi,

Here is some notes to my previous post.

I mean that only online cores are shown in /proc/cpuinfo. So the PowerTOP doesn't count currently offline ones. And you must run the PowerTOP on a heavy loaded system to let the PowerTOP see all CPU cores.

Therefore it is impossible to run the PowerTOP before starting of heavy application to get information about all CPU cores.

Probably the PowerTOP could cache CPU information in some file, so it will be possible to register trace events for all cores.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
#2 lore raw
Quoted 9 lines · Igor Zhbanov

Hi,

Here is some notes to my previous post.

I mean that only online cores are shown in /proc/cpuinfo. So the PowerTOP doesn't count currently offline ones. And you must run the PowerTOP on a heavy loaded system to let the PowerTOP see all CPU cores.

Quoted 6 lines

Therefore it is impossible to run the PowerTOP before starting of heavy application to get information about all CPU cores.

Probably the PowerTOP could cache CPU information in some file, so it will be possible to register trace events for all cores.

Off hand I don't have a good solution to help your ARM story. Your notes are interesting and I agree this is something that ARM is going to need. Do you have a complete solution you would like to purpose? I assume you have done some hacking to get around this, but if you have taken that and tuned it i suggest purposing the change. If you find a solution that doesn't digress the Architectures, and corrects the ARM story then please submit patches. I would be interested in seeing what you have come up with.

I will try and look for a general solution, are there any other ARM advocates on this list who would like to chime in?

-C

author
#3 lore raw
Quoted 9 lines · Chris Ferron
Quoted 8 lines · Igor Zhbanov

I mean that only online cores are shown in /proc/cpuinfo. So the PowerTOP doesn't count currently offline ones. And you must run the PowerTOP on a heavy loaded system to let the PowerTOP see all CPU cores.
Therefore it is impossible to run the PowerTOP before starting of heavy application to get information about all CPU cores. Probably the PowerTOP could cache CPU information in some file, so it will be possible to register trace events for all cores.

Quoted 10 lines

Off hand I don't have a good solution to help your ARM story. Your notes are interesting and I agree this is something that ARM is going to need. Do you have a complete solution you would like to purpose? I assume you have done some hacking to get around this, but if you have taken that and tuned it i suggest purposing the change. If you find a solution that doesn't digress the Architectures, and corrects the ARM story then please submit patches. I would be interested in seeing what you have come up with.
I will try and look for a general solution, are there any other ARM advocates on this list who would like to chime in?

Now I'm looking for solution. But I see now only four ways:

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.
2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
3) Ask the user to run PowerTOP on a heavy loaded system for the first time (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere in /var/cache/powertop and reuse it on later runs.
4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear. (Perhaps there could be something like D-Bus or netlink notification when a core brought online -- I don't know about it).

At your personal view, what way fits better in the PowerTOP project?

P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3. Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
#4 lore raw
Quoted 7 lines · Igor Zhbanov

Chris Ferron wrote:

Now I'm looking for solution. But I see now only four ways:

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.

procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for that purpose.

https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html

	-ss
Quoted 14 lines

2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
3) Ask the user to run PowerTOP on a heavy loaded system for the first time (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere in /var/cache/powertop and reuse it on later runs.
4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear. (Perhaps there could be something like D-Bus or netlink notification when a core brought online -- I don't know about it).

At your personal view, what way fits better in the PowerTOP project?

P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3. Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

author
#5 lore raw
Quoted 5 lines · Sergey Senozhatsky
Quoted 4 lines · Igor Zhbanov

Chris Ferron wrote:

Now I'm looking for solution. But I see now only four ways:

Quoted 7 lines
Quoted 3 lines

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.

procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for that purpose.

https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html

Counting of the CPUs is the least problem. (By the way, how sysconf() figures out the number? I suppose that it reads /proc/cpuinfo of scans /sys/. Is there any other way?).

The problem is that the PowerTOP needs to read information about cores from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings etc). And needed files in these directories are present only for cores which are currently online.

So the Powertop is unable
1) To count the number of available CPUs.
2) To read information about all CPUs.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
#6 lore raw
Quoted 13 lines · Igor Zhbanov
Quoted 5 lines · Igor Zhbanov
Quoted 4 lines

Chris Ferron wrote:

Now I'm looking for solution. But I see now only four ways:

Quoted 7 lines
Quoted 3 lines

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.

procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for that purpose.

https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html

Quoted 9 lines

Counting of the CPUs is the least problem. (By the way, how sysconf() figures out the number? I suppose that it reads /proc/cpuinfo of scans /sys/. Is there any other way?).

The problem is that the PowerTOP needs to read information about cores from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings etc). And needed files in these directories are present only for cores which are currently online.

right, I was thinking about that some time ago: and from the top of my head kind of a solution seemed to be to track number of CPUs before measure, compare with the previously known one, and "re-init" powertop (at least rebuild CPUs part), but it's not that often that I offline/online CPUs, so I decided to drop it. with ARM we have a different story.

on x86(_64) "dirty" CPU offline support looks "easy", just make get_msr() error non-fatal. online -- that's the problem, agree.

	-ss
Quoted 4 lines

So the Powertop is unable
1) To count the number of available CPUs.
2) To read information about all CPUs.

#8 lore raw
In reply to Igor Zhbanov
Quoted 35 lines · Igor Zhbanov
Quoted 9 lines · Chris Ferron
Quoted 8 lines · Igor Zhbanov

I mean that only online cores are shown in /proc/cpuinfo. So the PowerTOP doesn't count currently offline ones. And you must run the PowerTOP on a heavy loaded system to let the PowerTOP see all CPU cores.
Therefore it is impossible to run the PowerTOP before starting of heavy application to get information about all CPU cores. Probably the PowerTOP could cache CPU information in some file, so it will be possible to register trace events for all cores.

Quoted 10 lines

Off hand I don't have a good solution to help your ARM story. Your notes are interesting and I agree this is something that ARM is going to need. Do you have a complete solution you would like to purpose? I assume you have done some hacking to get around this, but if you have taken that and tuned it i suggest purposing the change. If you find a solution that doesn't digress the Architectures, and corrects the ARM story then please submit patches. I would be interested in seeing what you have come up with.
I will try and look for a general solution, are there any other ARM advocates on this list who would like to chime in?

Now I'm looking for solution. But I see now only four ways:

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.
2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
3) Ask the user to run PowerTOP on a heavy loaded system for the first time (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere in /var/cache/powertop and reuse it on later runs.
4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear. (Perhaps there could be something like D-Bus or netlink notification when a core brought online -- I don't know about it).

Quoted 1 line

At your personal view, what way fits better in the PowerTOP project?

it's messy. ARM in some ways is abusing CPU Offline for idle...

Quoted 4 lines

P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3. Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

on x86, you don't win any power at all by offlining CPUs (in fact, it's very power inefficient to do so, idle tends to be THAT good)

Quoted 1 line
author
#7 lore raw
In reply to Sergey Senozhatsky
Quoted 21 lines · Sergey Senozhatsky
Quoted 12 lines · Igor Zhbanov
Quoted 5 lines · Igor Zhbanov
Quoted 4 lines

Chris Ferron wrote:

Now I'm looking for solution. But I see now only four ways:

Quoted 7 lines
Quoted 3 lines

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.

procps (e.g. top utility) use sysconf _SC_NPROCESSORS_CONF/_SC_NPROCESSORS_ONLN for that purpose.

https://www.gnu.org/software/libc/manual/html_node/Processor-Resources.html

Quoted 8 lines

Counting of the CPUs is the least problem. (By the way, how sysconf() figures out the number? I suppose that it reads /proc/cpuinfo of scans /sys/. Is there any other way?).

The problem is that the PowerTOP needs to read information about cores from the files in /sys/.../cpu/cpuX/ directory (about C- and P-states, siblings etc). And needed files in these directories are present only for cores which are currently online.

Quoted 7 lines

right, I was thinking about that some time ago: and from the top of my head kind of a solution seemed to be to track number of CPUs before measure, compare with the previously known one, and "re-init" powertop (at least rebuild CPUs part), but it's not that often that I offline/online CPUs, so I decided to drop it. with ARM we have a different story.

on x86(_64) "dirty" CPU offline support looks "easy", just make get_msr() error non-fatal. online -- that's the problem, agree.

Perhaps PowerTOP could wake up every core in its calibration phase and story collected information in platform config file.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
author
#9 lore raw
In reply to Arjan van de Ven
Quoted 15 lines · Arjan van de Ven
Quoted 13 lines · Igor Zhbanov

Now I'm looking for solution. But I see now only four ways:

1) Develop another kernel mechanism for providing information about all available CPUs and cores. Or ask the kernel authors not to remove offline cores from /proc/cpuinfo and their information files from /sys/.../cpu/.
2) Scan /sys/.../cpu for all CPUs and temporarily wake it up to gather information.
3) Ask the user to run PowerTOP on a heavy loaded system for the first time (or wake up all CPUs by the PowerTOP) and store collected CPU information somewhere in /var/cache/powertop and reuse it on later runs.
4) Don't just sleep in a main cycle, but scan /sys/.../cpu for new CPUs to appear. (Perhaps there could be something like D-Bus or netlink notification when a core brought online -- I don't know about it).

Quoted 8 lines
Quoted 1 line

At your personal view, what way fits better in the PowerTOP project?

it's messy. ARM in some ways is abusing CPU Offline for idle...

Quoted 3 lines

P.S. This may be the case for x86 CPUs too. I have disable CPU2 by writing to /sys/devices/system/cpu2/online, and the PowerTOP sees only CPU0, CPU1, and CPU3. Although default PC's CPU governor as I know doesn't turn off CPUs frequently.

on x86, you don't win any power at all by offlining CPUs (in fact, it's very power inefficient to do so, idle tends to be THAT good)

Or it could be that kernel on ARM wrongly removes information from /proc/cpuinfo and /sys/ for offline cores. Or it wrongly considers them offline.

As I know many of the ARM CPUs has only two C-states. First is independent for each core:
WFI -- Wait For Interrupt. In this state the CPU just calls the halt instruction.

And the second state is ARM power down. This is the coupled state. As I understand, the whole package could enter this state only when all cores (except current) are in WFI state. So the last active core powers the CPU down.

But is it correct to consider ARM core to be offline when it is just in WFI state?

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation
#10 lore raw
Quoted 1 line · Igor Zhbanov

But is it correct to consider ARM core to be offline when it is just in WFI state?

nope

for all intents and purposes, the cpu is still there.

in an idle state, the cpu logically is there, and ready to resume execution if needed... that's the definition of idle ;-)

offline is where the administrator takes a cpu out of circulation.

neither specifies any physical hardware state... in fact offline is usually implemented as an idle state, at least on x86. the semantics for how to get back are different (system autonomous versus administrator)

now, quite possible some ARM platforms abuse this and don't implement something as "idle" but as "offline" instead. that sounds not very smart to me.

author
#11 lore raw
Quoted 9 lines · Arjan van de Ven
Quoted 1 line · Igor Zhbanov

But is it correct to consider ARM core to be offline when it is just in WFI state?

nope

for all intents and purposes, the cpu is still there.

in an idle state, the cpu logically is there, and ready to resume execution if needed... that's the definition of idle ;-)

Quoted 7 lines

offline is where the administrator takes a cpu out of circulation.

neither specifies any physical hardware state... in fact offline is usually implemented as an idle state, at least on x86. the semantics for how to get back are different (system autonomous versus administrator)

now, quite possible some ARM platforms abuse this and don't implement something as "idle" but as "offline" instead. that sounds not very smart to me.

Yes, it seems that on some ARM platforms offline and idle are mixed.
On my test device it is even hard to bring the core online. If I write 1 to "online" file, then I see other pseudo-files only for a less than a second, then the core will be brought offline again. It seems that kernel powers up the core, then sees that there no task for it, and powers it down again.

May be we should talk to ARM developers to do things more correctly. It seems that now impossible to administratively enable/disable some cores. The CPU governor will always consider all available cores to be possible to use (except of limit of nr_cpus boot parameter).

So I don't know what to do best.

Signature · 9 lines
Best regards,
Igor Zhbanov,
Expert Software Engineer,
phone: +7 (495) 797 25 00 ext 3806
e-mail: i.zhbanov(a)samsung.com

ASWG, Moscow R&D center, Samsung Electronics
12 Dvintsev street, building 1
127018, Moscow, Russian Federation