Skip to contentSkip to conversation

3.10-rcX: cpu governor ondemand doesn't scale well after s2ram

PATCH In review Toralf Förster 7 participants 44 messages Jun 18, 2013 – Jul 15, 2013 CPU frequency scaling framework

lore
Share & tools
lore
#0 lore raw

At a stable 32bit Gentoo Linux with a i5 processor (2.6 GHZ max, 3.1 GHz boost) I've to change the governor after a s2ram all 4 cores from "ondemand" to "performance" and back to "ondemand", otherwise cpu1 till cpu3 would stay after s2ram at 2.6 GHz (whilst cpu0 is at the expected 800 MHz if no load is at the system) - FWIW I do set ignore_nice to "1".

3.9.6 is ok, the bug was introduces somewhere in 3.10-rcX.

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#1 lore raw
Quoted 8 lines · Toralf Förster

At a stable 32bit Gentoo Linux with a i5 processor (2.6 GHZ max, 3.1 GHz boost) I've to change the governor after a s2ram all 4 cores from "ondemand" to "performance" and back to "ondemand", otherwise cpu1 till cpu3 would stay after s2ram at 2.6 GHz (whilst cpu0 is at the expected 800 MHz if no load is at the system) - FWIW I do set ignore_nice to "1".

3.9.6 is ok, the bug was introduces somewhere in 3.10-rcX.

FWIW the following commands circumvents that issue :

# for i in $(seq 1 3); do echo performance > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done
# for i in $(seq 1 3); do echo ondemand    > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done

cpu0 is unaffected

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#2 lore raw
In reply to Toralf Förster
Quoted 7 lines · Toralf Förster

At a stable 32bit Gentoo Linux with a i5 processor (2.6 GHZ max, 3.1 GHz boost) I've to change the governor after a s2ram all 4 cores from "ondemand" to "performance" and back to "ondemand", otherwise cpu1 till cpu3 would stay after s2ram at 2.6 GHz (whilst cpu0 is at the expected 800 MHz if no load is at the system) - FWIW I do set ignore_nice to "1".

3.9.6 is ok, the bug was introduces somewhere in 3.10-rcX.

Hmm. This looks like something unrelated to the previous reports.

How do you check the frequencies?

Does that happen every time? If so, any chance to bisect driver/cpufreq changes between 3.9 and now?

Does it work without ignore_nice?

Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
#3 lore raw
Quoted 1 line · Rafael J. Wysocki

Hmm. This looks like something unrelated to the previous reports.

yes

Quoted 1 line

How do you check the frequencies?

much more significant is the temperature :

$> cat /sys/devices/system/cpu/cpu\?/thermal_throttle/package_power_limit_count /proc/acpi/ibm/fan /sys/class/hwmon/hwmon\?/device/temp\?_input'

- it is around 49° if just 4 BOINC childs are running at nice level -19, but it is above 65°C if the issue exists. And in hat case I just do run

$> for g in performance ondemand; do for i in 0 1 2 3; do echo $g > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done; done; echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice

and the system is back to normal.

Quoted 2 lines

Does that happen every time? If so, any chance to bisect driver/cpufreq changes between 3.9 and now?

first bisect attempts failed. I realized that the issue only occurs if the power is plugged in. If the system is running on battery the issue doesn't occurred (I checked it more than once)

Quoted 1 line

Does it work without ignore_nice?

Seems not to be related to that.

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#4 lore raw
Quoted 17 lines · Toralf Förster
Quoted 1 line · Rafael J. Wysocki

Hmm. This looks like something unrelated to the previous reports.

yes

Quoted 1 line

How do you check the frequencies?

much more significant is the temperature :

$> cat /sys/devices/system/cpu/cpu\?/thermal_throttle/package_power_limit_count /proc/acpi/ibm/fan /sys/class/hwmon/hwmon\?/device/temp\?_input'

- it is around 49° if just 4 BOINC childs are running at nice level -19, but it is above 65°C if the issue exists. And in hat case I just do run

$> for g in performance ondemand; do for i in 0 1 2 3; do echo $g > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done; done; echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice

and the system is back to normal.

Quoted 7 lines
Quoted 2 lines

Does that happen every time? If so, any chance to bisect driver/cpufreq changes between 3.9 and now?

first bisect attempts failed. I realized that the issue only occurs if the power is plugged in. If the system is running on battery the issue doesn't occurred (I checked it more than once)

Quoted 1 line

Does it work without ignore_nice?

Seems not to be related to that.

Well, I'm not aware of any recent changes in cpufreq that would make things work differently depending on whether or not AC power is used.

Thanks,
Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
maintainer
#5 lore raw
Quoted 2 lines · Rafael J. Wysocki

Well, I'm not aware of any recent changes in cpufreq that would make things work differently depending on whether or not AC power is used.

Toralf,

It would be helpful if you can do git bisect and let us know which patch created this problem. Otherwise its very difficult to trace it.

#6 lore raw
Quoted 9 lines · Viresh Kumar
Quoted 2 lines · Rafael J. Wysocki

Well, I'm not aware of any recent changes in cpufreq that would make things work differently depending on whether or not AC power is used.

Toralf,

It would be helpful if you can do git bisect and let us know which patch created this problem. Otherwise its very difficult to trace it.

Yep, but 14 commits to bisect and about 20-30 min for each - will see, when I find a time slot.

Just FWIW thie following is now a good indicator for the issue - that's the picture after wakeup, where I'd expect always 800 MHz:

$ sudo cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq
800000
2601000
2601000
2601000
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
maintainer
#7 lore raw
Quoted 2 lines · Toralf Förster

Yep, but 14 commits to bisect and about 20-30 min for each - will see, when I find a time slot.

Hmm..

Quoted 8 lines

Just FWIW thie following is now a good indicator for the issue - that's the picture after wakeup, where I'd expect always 800 MHz:

$ sudo cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq
800000
2601000
2601000
2601000

Can you give dumps of complete cpufreq directories for all cpus before and after s2r. That may give us some hint about the problem.

#8 lore raw
Quoted 2 lines · Viresh Kumar

Can you give dumps of complete cpufreq directories for all cpus before and after s2r. That may give us some hint about the problem.

here it is:

$ sudo su -
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/scaling_cur_freq
800000
800000
800000
800000
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq
800000
800000
800000
800000
n22 ~ # find /sys/devices/system/cpu -type f | xargs tail -v > /tmp/before 2>/dev/null
n22 ~ # Fn4
 * Bringing down interface ppp0
 *   Running predown ...
 *   Stopping pppd on ppp0                                                                              [ ok ]
 * Bringing up interface ppp0
 *   Starting pppd in ppp0 ...                                                                          [ ok ]
 *   received address 92.224.127.7
 * WARNING: net.ppp0 has started, but is inactive
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/scaling_cur_freq
800000
800000
800000
800000
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq
800000
2601000
2601000
2601000
n22 ~ # find /sys/devices/system/cpu -type f | xargs tail -v > /tmp/after 2>/dev/null
n22 ~ # for g in performance ondemand; do for i in 1 2 3; do echo $g > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor; done; done; echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/scaling_cur_freq800000
800000
800000
800000
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq800000
800000
800000
800000
n22 ~ # chmod a+r /tmp/before /tmp/after
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

==> /sys/devices/system/cpu/cpu0/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus <== 0

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/related_cpus <== 0

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/time <== 19239

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/usage <== 55

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/time <== 924048

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/usage <== 7370

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/time <== 8659159

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/usage <== 18164

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/time <== 3258501

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/usage <== 4979

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/time <== 229632

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/usage <== 185

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/time <== 55353981

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/usage <== 34407

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu0/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu0/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu0/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu0/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu0/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu0/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu0/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu0/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu0/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu0/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu0/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu0/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu0/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu0/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu0/power/control <== auto

==> /sys/devices/system/cpu/cpu0/power/async <==
disabled

==> /sys/devices/system/cpu/cpu0/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu0/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu0/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu0/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu0/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu0/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu0/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu0/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu0/topology/core_id <== 0

==> /sys/devices/system/cpu/cpu0/topology/thread_siblings <== 03

==> /sys/devices/system/cpu/cpu0/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu0/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu0/topology/thread_siblings_list <== 0-1

==> /sys/devices/system/cpu/cpu0/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu1/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus <== 1

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/related_cpus <== 1

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/time <== 61773

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/usage <== 27

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/time <== 259855

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/usage <== 4464

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/time <== 7498088

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/usage <== 9108

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/time <== 3915900

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/usage <== 2583

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/time <== 163650

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/usage <== 73

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/time <== 56007989

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/usage <== 10807

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu1/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu1/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu1/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu1/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu1/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu1/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu1/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu1/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu1/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu1/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu1/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu1/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu1/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu1/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu1/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu1/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu1/power/control <== auto

==> /sys/devices/system/cpu/cpu1/power/async <==
disabled

==> /sys/devices/system/cpu/cpu1/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu1/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu1/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu1/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu1/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu1/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu1/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu1/online <==
1

==> /sys/devices/system/cpu/cpu1/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu1/topology/core_id <== 0

==> /sys/devices/system/cpu/cpu1/topology/thread_siblings <== 03

==> /sys/devices/system/cpu/cpu1/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu1/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu1/topology/thread_siblings_list <== 0-1

==> /sys/devices/system/cpu/cpu1/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu2/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_cur_freq <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/affected_cpus <== 2

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/related_cpus <== 2

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/time <== 15532

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/usage <== 36

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/time <== 425542

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/usage <== 7866

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/time <== 8230973

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/usage <== 8798

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/time <== 4179929

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/usage <== 2322

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/time <== 195865

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/usage <== 68

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/time <== 56796175

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/usage <== 10190

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu2/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu2/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu2/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu2/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu2/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu2/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu2/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu2/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu2/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu2/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu2/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu2/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu2/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu2/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu2/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu2/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu2/power/control <== auto

==> /sys/devices/system/cpu/cpu2/power/async <==
disabled

==> /sys/devices/system/cpu/cpu2/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu2/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu2/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu2/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu2/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu2/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu2/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu2/online <==
1

==> /sys/devices/system/cpu/cpu2/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu2/topology/core_id <== 1

==> /sys/devices/system/cpu/cpu2/topology/thread_siblings <== 0c

==> /sys/devices/system/cpu/cpu2/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu2/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu2/topology/thread_siblings_list <== 2-3

==> /sys/devices/system/cpu/cpu2/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu3/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_cur_freq <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/affected_cpus <== 3

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/related_cpus <== 3

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/time <== 2323

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/usage <== 28

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/time <== 456731

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/usage <== 7560

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/time <== 5585596

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/usage <== 6373

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/time <== 3177689

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/usage <== 1769

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/time <== 73846

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/usage <== 38

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/time <== 56910450

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/usage <== 9384

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu3/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu3/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu3/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu3/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu3/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu3/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu3/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu3/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu3/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu3/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu3/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu3/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu3/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu3/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu3/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu3/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu3/power/control <== auto

==> /sys/devices/system/cpu/cpu3/power/async <==
disabled

==> /sys/devices/system/cpu/cpu3/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu3/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu3/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu3/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu3/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu3/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu3/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu3/online <==
1

==> /sys/devices/system/cpu/cpu3/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu3/topology/core_id <== 1

==> /sys/devices/system/cpu/cpu3/topology/thread_siblings <== 0c

==> /sys/devices/system/cpu/cpu3/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu3/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu3/topology/thread_siblings_list <== 2-3

==> /sys/devices/system/cpu/cpu3/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpufreq/boost <==
1

==> /sys/devices/system/cpu/cpufreq/ondemand/powersave_bias <== 0

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate <== 10000

==> /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy <== 1

==> /sys/devices/system/cpu/cpufreq/ondemand/up_threshold <== 95

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min <== 10000

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor <== 1

==> /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice <== 1

==> /sys/devices/system/cpu/cpuidle/current_driver <== intel_idle

==> /sys/devices/system/cpu/cpuidle/current_governor_ro <== menu

==> /sys/devices/system/cpu/power/control <==
auto

==> /sys/devices/system/cpu/power/async <==
disabled

==> /sys/devices/system/cpu/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/power/runtime_usage <== 0

==> /sys/devices/system/cpu/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/modalias <== x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/kernel_max <==
7

==> /sys/devices/system/cpu/possible <==
0-7

==> /sys/devices/system/cpu/online <==
0-3

==> /sys/devices/system/cpu/offline <==
4-7

==> /sys/devices/system/cpu/uevent <==

==> /sys/devices/system/cpu/present <==
0-3

==> /sys/devices/system/cpu/cpu0/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu0/cpufreq/affected_cpus <== 0

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu0/cpufreq/related_cpus <== 0

==> /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/time <== 19233

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/usage <== 54

==> /sys/devices/system/cpu/cpu0/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/time <== 919026

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/usage <== 7243

==> /sys/devices/system/cpu/cpu0/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/time <== 8630346

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/usage <== 17935

==> /sys/devices/system/cpu/cpu0/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/time <== 3240741

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/usage <== 4955

==> /sys/devices/system/cpu/cpu0/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/time <== 226879

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/usage <== 184

==> /sys/devices/system/cpu/cpu0/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/time <== 52170100

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/usage <== 33875

==> /sys/devices/system/cpu/cpu0/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu0/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu0/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu0/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu0/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu0/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu0/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu0/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu0/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu0/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu0/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu0/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu0/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu0/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu0/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu0/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu0/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu0/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu0/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu0/power/control <== auto

==> /sys/devices/system/cpu/cpu0/power/async <==
disabled

==> /sys/devices/system/cpu/cpu0/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu0/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu0/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu0/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu0/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu0/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu0/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu0/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu0/topology/core_id <== 0

==> /sys/devices/system/cpu/cpu0/topology/thread_siblings <== 03

==> /sys/devices/system/cpu/cpu0/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu0/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu0/topology/thread_siblings_list <== 0-1

==> /sys/devices/system/cpu/cpu0/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu0/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu1/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu1/cpufreq/affected_cpus <== 1

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu1/cpufreq/related_cpus <== 1

==> /sys/devices/system/cpu/cpu1/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/time <== 61747

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/usage <== 26

==> /sys/devices/system/cpu/cpu1/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/time <== 256430

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/usage <== 4418

==> /sys/devices/system/cpu/cpu1/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/time <== 7489672

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/usage <== 9076

==> /sys/devices/system/cpu/cpu1/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/time <== 3911044

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/usage <== 2572

==> /sys/devices/system/cpu/cpu1/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/time <== 163650

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/usage <== 73

==> /sys/devices/system/cpu/cpu1/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/time <== 52737221

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/usage <== 10540

==> /sys/devices/system/cpu/cpu1/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu1/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu1/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu1/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu1/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu1/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu1/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu1/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu1/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu1/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu1/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map <== 03

==> /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_list <== 0-1

==> /sys/devices/system/cpu/cpu1/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu1/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu1/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu1/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu1/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu1/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu1/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu1/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu1/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu1/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu1/power/control <== auto

==> /sys/devices/system/cpu/cpu1/power/async <==
disabled

==> /sys/devices/system/cpu/cpu1/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu1/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu1/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu1/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu1/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu1/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu1/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu1/online <==
1

==> /sys/devices/system/cpu/cpu1/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu1/topology/core_id <== 0

==> /sys/devices/system/cpu/cpu1/topology/thread_siblings <== 03

==> /sys/devices/system/cpu/cpu1/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu1/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu1/topology/thread_siblings_list <== 0-1

==> /sys/devices/system/cpu/cpu1/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu1/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu2/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu2/cpufreq/affected_cpus <== 2

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu2/cpufreq/related_cpus <== 2

==> /sys/devices/system/cpu/cpu2/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/time <== 15532

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/usage <== 36

==> /sys/devices/system/cpu/cpu2/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/time <== 424554

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/usage <== 7838

==> /sys/devices/system/cpu/cpu2/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/time <== 8228801

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/usage <== 8796

==> /sys/devices/system/cpu/cpu2/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/time <== 4179929

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/usage <== 2322

==> /sys/devices/system/cpu/cpu2/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/time <== 184890

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/usage <== 67

==> /sys/devices/system/cpu/cpu2/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/time <== 53545208

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/usage <== 10083

==> /sys/devices/system/cpu/cpu2/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu2/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu2/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu2/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu2/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu2/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu2/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu2/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu2/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu2/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu2/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu2/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu2/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu2/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu2/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu2/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu2/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu2/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu2/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu2/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu2/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu2/power/control <== auto

==> /sys/devices/system/cpu/cpu2/power/async <==
disabled

==> /sys/devices/system/cpu/cpu2/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu2/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu2/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu2/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu2/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu2/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu2/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu2/online <==
1

==> /sys/devices/system/cpu/cpu2/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu2/topology/core_id <== 1

==> /sys/devices/system/cpu/cpu2/topology/thread_siblings <== 0c

==> /sys/devices/system/cpu/cpu2/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu2/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu2/topology/thread_siblings_list <== 2-3

==> /sys/devices/system/cpu/cpu2/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu2/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu3/cpufreq/bios_limit <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor <== ondemand

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_transition_latency <== 10000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_frequencies <== 2601000 2600000 2400000 2200000 2000000 1800000 1600000 1400000 1200000 1000000 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_driver <== acpi-cpufreq

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_available_governors <== ondemand performance

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/cpuinfo_min_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq <== 2601000

==> /sys/devices/system/cpu/cpu3/cpufreq/affected_cpus <== 3

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq <== 800000

==> /sys/devices/system/cpu/cpu3/cpufreq/related_cpus <== 3

==> /sys/devices/system/cpu/cpu3/cpufreq/scaling_setspeed <== <unsupported>

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/desc <== CPUIDLE CORE POLL IDLE

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/name <== POLL

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/time <== 2323

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/power <== 4294967295

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/usage <== 28

==> /sys/devices/system/cpu/cpu3/cpuidle/state0/latency <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/desc <== MWAIT 0x00

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/name <== C1-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/time <== 456227

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/usage <== 7556

==> /sys/devices/system/cpu/cpu3/cpuidle/state1/latency <== 2

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/desc <== MWAIT 0x01

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/name <== C1E-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/time <== 5578828

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/usage <== 6351

==> /sys/devices/system/cpu/cpu3/cpuidle/state2/latency <== 10

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/desc <== MWAIT 0x10

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/name <== C3-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/time <== 3177302

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/usage <== 1767

==> /sys/devices/system/cpu/cpu3/cpuidle/state3/latency <== 80

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/desc <== MWAIT 0x20

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/name <== C6-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/time <== 73846

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/usage <== 38

==> /sys/devices/system/cpu/cpu3/cpuidle/state4/latency <== 104

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/desc <== MWAIT 0x30

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/name <== C7-SNB

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/time <== 53452690

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/disable <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/power <== 0

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/usage <== 9184

==> /sys/devices/system/cpu/cpu3/cpuidle/state5/latency <== 109

==> /sys/devices/system/cpu/cpu3/cache/index0/size <== 32K

==> /sys/devices/system/cpu/cpu3/cache/index0/type <== Data

==> /sys/devices/system/cpu/cpu3/cache/index0/level <== 1

==> /sys/devices/system/cpu/cpu3/cache/index0/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index0/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index0/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index0/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index1/size <== 32K

==> /sys/devices/system/cpu/cpu3/cache/index1/type <== Instruction

==> /sys/devices/system/cpu/cpu3/cache/index1/level <== 1

==> /sys/devices/system/cpu/cpu3/cache/index1/number_of_sets <== 64

==> /sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index1/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index1/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index1/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index2/size <== 256K

==> /sys/devices/system/cpu/cpu3/cache/index2/type <== Unified

==> /sys/devices/system/cpu/cpu3/cache/index2/level <== 2

==> /sys/devices/system/cpu/cpu3/cache/index2/number_of_sets <== 512

==> /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map <== 0c

==> /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_list <== 2-3

==> /sys/devices/system/cpu/cpu3/cache/index2/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index2/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index2/ways_of_associativity <== 8

==> /sys/devices/system/cpu/cpu3/cache/index3/size <== 3072K

==> /sys/devices/system/cpu/cpu3/cache/index3/type <== Unified

==> /sys/devices/system/cpu/cpu3/cache/index3/level <== 3

==> /sys/devices/system/cpu/cpu3/cache/index3/number_of_sets <== 4096

==> /sys/devices/system/cpu/cpu3/cache/index3/shared_cpu_map <== 0f

==> /sys/devices/system/cpu/cpu3/cache/index3/shared_cpu_list <== 0-3

==> /sys/devices/system/cpu/cpu3/cache/index3/coherency_line_size <== 64

==> /sys/devices/system/cpu/cpu3/cache/index3/physical_line_partition <== 1

==> /sys/devices/system/cpu/cpu3/cache/index3/ways_of_associativity <== 12

==> /sys/devices/system/cpu/cpu3/power/control <== auto

==> /sys/devices/system/cpu/cpu3/power/async <==
disabled

==> /sys/devices/system/cpu/cpu3/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/cpu3/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/cpu3/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/cpu3/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/cpu3/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/cpu3/power/runtime_usage <== 0

==> /sys/devices/system/cpu/cpu3/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/cpu3/online <==
1

==> /sys/devices/system/cpu/cpu3/uevent <== MODALIAS=x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/cpu3/topology/core_id <== 1

==> /sys/devices/system/cpu/cpu3/topology/thread_siblings <== 0c

==> /sys/devices/system/cpu/cpu3/topology/core_siblings <== 0f

==> /sys/devices/system/cpu/cpu3/topology/core_siblings_list <== 0-3

==> /sys/devices/system/cpu/cpu3/topology/thread_siblings_list <== 2-3

==> /sys/devices/system/cpu/cpu3/topology/physical_package_id <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/core_power_limit_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/package_throttle_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/core_throttle_count <== 0

==> /sys/devices/system/cpu/cpu3/thermal_throttle/package_power_limit_count <== 0

==> /sys/devices/system/cpu/cpufreq/boost <==
1

==> /sys/devices/system/cpu/cpufreq/ondemand/powersave_bias <== 0

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate <== 10000

==> /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy <== 1

==> /sys/devices/system/cpu/cpufreq/ondemand/up_threshold <== 95

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate_min <== 10000

==> /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor <== 1

==> /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice <== 1

==> /sys/devices/system/cpu/cpuidle/current_driver <== intel_idle

==> /sys/devices/system/cpu/cpuidle/current_governor_ro <== menu

==> /sys/devices/system/cpu/power/control <==
auto

==> /sys/devices/system/cpu/power/async <==
disabled

==> /sys/devices/system/cpu/power/runtime_enabled <== disabled

==> /sys/devices/system/cpu/power/runtime_active_kids <== 0

==> /sys/devices/system/cpu/power/runtime_active_time <== 0

==> /sys/devices/system/cpu/power/autosuspend_delay_ms <==

==> /sys/devices/system/cpu/power/runtime_status <== unsupported

==> /sys/devices/system/cpu/power/runtime_usage <== 0

==> /sys/devices/system/cpu/power/runtime_suspended_time <== 0

==> /sys/devices/system/cpu/modalias <== x86cpu:vendor:0000:family:0006:model:002A:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0015,0016,0017,0018,0019,001A,001B,001C,001D,001F,0034,003B,003D,0066,0068,006B,006C,006D,0072,0076,0078,007C,007D,0080,0081,0082,0083,0084,0085,0086,0087,0088,0089,008D,008E,008F,0091,0093,0094,0095,0097,0098,0099,009A,009B,009C,00C0,00E0,00E1,00E3,00E4,00E5,00E6,00E7,0100,0101,0102,0103,0104

==> /sys/devices/system/cpu/kernel_max <==
7

==> /sys/devices/system/cpu/possible <==
0-7

==> /sys/devices/system/cpu/online <==
0-3

==> /sys/devices/system/cpu/offline <==
4-7

==> /sys/devices/system/cpu/uevent <==

==> /sys/devices/system/cpu/present <==
0-3

#41 lore raw
In reply to Viresh Kumar
Quoted 2 lines · Viresh Kumar

Can you give dumps of complete cpufreq directories for all cpus before and after s2r. That may give us some hint about the problem.

Because CPU0 is not affected I somehow arguing about nohz patches ...

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#43 lore raw
In reply to Rafael J. Wysocki
Quoted 12 lines · Rafael J. Wysocki
Quoted 5 lines · Toralf Förster
Quoted 2 lines · Viresh Kumar

Can you give dumps of complete cpufreq directories for all cpus before and after s2r. That may give us some hint about the problem.

Because CPU0 is not affected I somehow arguing about nohz patches ...

Do you have CONFIG_NO_HZ_FULL set by chance?

Rafael

no

$ zgrep  CONFIG_NO_HZ_FULL /proc/config.gz

I do just have :

$ zgrep  CONFIG_NO_HZ /proc/config.gz
CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ=y
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#42 lore raw
In reply to Toralf Förster
Quoted 5 lines · Toralf Förster
Quoted 2 lines · Viresh Kumar

Can you give dumps of complete cpufreq directories for all cpus before and after s2r. That may give us some hint about the problem.

Because CPU0 is not affected I somehow arguing about nohz patches ...

Do you have CONFIG_NO_HZ_FULL set by chance?

Rafael

maintainer
#9 lore raw
In reply to Toralf Förster
Quoted 10 lines · Toralf Förster

n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/scaling_cur_freq 800000
800000
800000
800000
n22 ~ # cat /sys/devices/system/cpu/cpu?/cpufreq/cpuinfo_cur_freq 800000
2601000
2601000
2601000

Hmm.. So this is what I make out of it.
scaling_cur_freq: is the freq which cpufreq core thinks is set. Which is correct. 800 MHz.

cpuinfo_cur_freq: is the actual freq in hardware, which is wrong. @Rafael: If I am not wrong we have seen this earlier too in last few months. There was some problem with bootloader or something else I can't remember.

author Acked-by Viresh Kumar
#10 lore raw

The latest bisect attempt gave :

commit a66b2e503fc79fff6632d02ef5a0ee47c1d2553d

Author: Srivatsa S. Bhat <[email protected]>
Date:   Wed May 15 21:47:17 2013 +0200

cpufreq: Preserve sysfs files across suspend/resume

The file permissions of cpufreq per-cpu sysfs files are not preserved across suspend/resume because we internally go through the CPU Hotplug path which reinitializes the file permissions on CPU online.

But the user is not supposed to know that we are using CPU hotplug internally within suspend/resume (IOW, the kernel should not silently wreck the user-set file permissions across a suspend cycle). Therefore, we need to preserve the file permissions as they are across suspend/resume.

The simplest way to achieve that is to just not touch the sysfs files at all - ie., just ignore the CPU hotplug notifications in the suspend/resume path (_FROZEN) in the cpufreq hotplug callback.

To get a more reliable bisect result I had to start BOINC before (4 childs each with nice -19 started)

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#11 lore raw
Quoted 30 lines · Toralf Förster

The latest bisect attempt gave :

commit a66b2e503fc79fff6632d02ef5a0ee47c1d2553d

Author: Srivatsa S. Bhat <[email protected]>
Date:   Wed May 15 21:47:17 2013 +0200

cpufreq: Preserve sysfs files across suspend/resume

The file permissions of cpufreq per-cpu sysfs files are not preserved across suspend/resume because we internally go through the CPU Hotplug path which reinitializes the file permissions on CPU online.

But the user is not supposed to know that we are using CPU hotplug internally within suspend/resume (IOW, the kernel should not silently wreck the user-set file permissions across a suspend cycle). Therefore, we need to preserve the file permissions as they are across suspend/resume.

The simplest way to achieve that is to just not touch the sysfs files at all - ie., just ignore the CPU hotplug notifications in the suspend/resume path (_FROZEN) in the cpufreq hotplug callback.

Quoted 2 lines

To get a more reliable bisect result I had to start BOINC before (4 childs each with nice -19 started)

Well, to be honest, I'm not really sure how the above commit can cause the problem you're seeing to happen ...

Srivatsa, Viresh, any ideas?

Rafael

Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
maintainer
#12 lore raw
Quoted 4 lines · Rafael J. Wysocki

Well, to be honest, I'm not really sure how the above commit can cause the problem you're seeing to happen ...

Srivatsa, Viresh, any ideas?

I don't. But its very simple to get this checked isn't it? Just revert this patch on top of your kernel (leave bisect) and see if gets fixed.

#13 lore raw
Quoted 10 lines · Viresh Kumar
Quoted 4 lines · Rafael J. Wysocki

Well, to be honest, I'm not really sure how the above commit can cause the problem you're seeing to happen ...

Srivatsa, Viresh, any ideas?

I don't. But its very simple to get this checked isn't it? Just revert this patch on top of your kernel (leave bisect) and see if gets fixed.

I do (mostly) check a bisect result before I post it to a mailing list to not blame the wrong commit to be guilty (b/c - erm - I did it in the past).

Said that I finished bisect, checked out commit a66b2e5, tested it (has the issue), did "git show a66b2e5 | patch -p1 -R" and checked that result (no issue).

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#14 lore raw
In reply to Viresh Kumar
Quoted 2 lines · Viresh Kumar

Just revert this patch on top of your kernel (leave bisect) and see if gets fixed.

negative,
reverting that commit on top of current -rc7+ tree breaks s2ram completely here (just sysrq+b works when the power led is blinking)

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#15 lore raw
In reply to Rafael J. Wysocki
Quoted 31 lines · Rafael J. Wysocki
Quoted 30 lines · Toralf Förster

The latest bisect attempt gave :

commit a66b2e503fc79fff6632d02ef5a0ee47c1d2553d

Author: Srivatsa S. Bhat <[email protected]>
Date:   Wed May 15 21:47:17 2013 +0200

cpufreq: Preserve sysfs files across suspend/resume

The file permissions of cpufreq per-cpu sysfs files are not preserved across suspend/resume because we internally go through the CPU Hotplug path which reinitializes the file permissions on CPU online.

But the user is not supposed to know that we are using CPU hotplug internally within suspend/resume (IOW, the kernel should not silently wreck the user-set file permissions across a suspend cycle). Therefore, we need to preserve the file permissions as they are across suspend/resume.

The simplest way to achieve that is to just not touch the sysfs files at all - ie., just ignore the CPU hotplug notifications in the suspend/resume path (_FROZEN) in the cpufreq hotplug callback.

Quoted 8 lines
Quoted 2 lines

To get a more reliable bisect result I had to start BOINC before (4 childs each with nice -19 started)

Well, to be honest, I'm not really sure how the above commit can cause the problem you're seeing to happen ...

Srivatsa, Viresh, any ideas?

I tried to look up what problem is being reported, but apart from the hint from the subject line, I couldn't find anything more. So, guessing that there is something wrong with cpufreq after an s3 cycle, Toralf, can you please try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

---

 drivers/cpufreq/cpufreq_stats.c |    1 +
 1 file changed, 1 insertion(+)
drivers/cpufreq/cpufreq_stats.c +1 -0
index fb65dec..591b6fb 100644
@@ -349,6 +349,7 @@ static int __cpuinit cpufreq_stat_cpu_callback(struct notifier_block *nfb,
 
 	switch (action) {
 	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
 		cpufreq_update_policy(cpu);
 		break;
 	case CPU_DOWN_PREPARE:
#16 lore raw
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Thx

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#17 lore raw

PATCH 1/1 cpufreq: Fix cpufreq regression after suspend/resume

Quoted 8 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Cool! So here is the proper patch, with changelog added.

Regards,
Srivatsa S. Bhat

-----------------------------------------------------------------------------

From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Toralf Förster reported that the cpufreq ondemand governor behaves erratically (doesn't scale well) after a suspend/resume cycle. The problem was that the cpufreq subsystem's idea of the cpu frequencies differed from the actual frequencies set in the hardware after a suspend/resume cycle. Toralf bisected the problem to commit a66b2e5 (cpufreq: Preserve sysfs files across suspend/resume).

Among other (harmless) things, that commit skipped the call to cpufreq_update_policy() in the resume path. But cpufreq_update_policy() plays an important role during resume, because it is responsible for checking if the BIOS changed the cpu frequencies behind our back and resynchronize the cpufreq subsystem's knowledge of the cpu frequencies, and update them accordingly.

So, restore the call to cpufreq_update_policy() in the resume path to fix the cpufreq regression.

---

 drivers/cpufreq/cpufreq_stats.c |    1 +
 1 file changed, 1 insertion(+)
drivers/cpufreq/cpufreq_stats.c +1 -0
index fb65dec..591b6fb 100644
@@ -349,6 +349,7 @@ static int __cpuinit cpufreq_stat_cpu_callback(struct notifier_block *nfb,
 
 	switch (action) {
 	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
 		cpufreq_update_policy(cpu);
 		break;
 	case CPU_DOWN_PREPARE:
#18 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Quoted 35 lines · Srivatsa S. Bhat
Quoted 8 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Cool! So here is the proper patch, with changelog added.

Regards,
Srivatsa S. Bhat

-----------------------------------------------------------------------------

From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Toralf Förster reported that the cpufreq ondemand governor behaves erratically (doesn't scale well) after a suspend/resume cycle. The problem was that the cpufreq subsystem's idea of the cpu frequencies differed from the actual frequencies set in the hardware after a suspend/resume cycle. Toralf bisected the problem to commit a66b2e5 (cpufreq: Preserve sysfs files across suspend/resume).

Among other (harmless) things, that commit skipped the call to cpufreq_update_policy() in the resume path. But cpufreq_update_policy() plays an important role during resume, because it is responsible for checking if the BIOS changed the cpu frequencies behind our back and resynchronize the cpufreq subsystem's knowledge of the cpu frequencies, and update them accordingly.

Quoted 6 lines

So, restore the call to cpufreq_update_policy() in the resume path to fix the cpufreq regression.

Thanks Srivatsa, I'll queue it up as 3.11 (and 3.10-stable) material.

Thanks,
Rafael

Quoted 19 lines

---

 drivers/cpufreq/cpufreq_stats.c |    1 +
 1 file changed, 1 insertion(+)
drivers/cpufreq/cpufreq_stats.c +1 -0
index fb65dec..591b6fb 100644
@@ -349,6 +349,7 @@ static int __cpuinit cpufreq_stat_cpu_callback(struct notifier_block *nfb,
 
 	switch (action) {
 	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
 		cpufreq_update_policy(cpu);
 		break;
 	case CPU_DOWN_PREPARE:
Signature · 2 lines
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.
#30 lore raw
In reply to Toralf Förster
Quoted 10 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Thx

But if I do apply that patch on top of kernel 3.10 then it has no effect
- 3.10 shows the same issue.

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#31 lore raw
Quoted 7 lines · Toralf Förster
Quoted 6 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

Quoted 7 lines
Quoted 4 lines

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Thx

But if I do apply that patch on top of kernel 3.10 then it has no effect
- 3.10 shows the same issue.

Weird. So I think something *else* got broken by some other patch, _after_ a66b2e5. Viresh, do you have any suspect commits in mind?

Regards,
Srivatsa S. Bhat

maintainer
#32 lore raw
Quoted 10 lines · Srivatsa S. Bhat
Quoted 9 lines · Toralf Förster
Quoted 8 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thx

But if I do apply that patch on top of kernel 3.10 then it has no effect
- 3.10 shows the same issue.

Weird. So I think something *else* got broken by some other patch, _after_ a66b2e5. Viresh, do you have any suspect commits in mind?

If I was God I could have. But as I am a poor human being I need more details (preferably with git bisect) :)

--
Viresh

#33 lore raw
Quoted 17 lines · Viresh Kumar
Quoted 16 lines · Srivatsa S. Bhat
Quoted 14 lines · Toralf Förster
Quoted 10 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Thx

But if I do apply that patch on top of kernel 3.10 then it has no effect
- 3.10 shows the same issue.

Quoted 6 lines
Quoted 2 lines

Weird. So I think something *else* got broken by some other patch, _after_ a66b2e5. Viresh, do you have any suspect commits in mind?

If I was God I could have. But as I am a poor human being I need more details (preferably with git bisect) :)

Haha :)

Coming to git bisect, I think it needs to be done between a66b2e5 + this-patch and 3.10 + this-patch, to ensure that we look for problematic commits in-between a66b2e5 and 3.10 (to avoid the bisect result ending up at a66b2e5 again).

Regards,
Srivatsa S. Bhat

maintainer
#34 lore raw
Quoted 3 lines · Srivatsa S. Bhat

Coming to git bisect, I think it needs to be done between a66b2e5 + this-patch and 3.10 + this-patch, to ensure that we look for problematic commits in-between a66b2e5 and 3.10 (to avoid the bisect result ending up at a66b2e5 again).

Exactly!!

maintainer
#36 lore raw
In reply to Srivatsa S. Bhat
Quoted 3 lines · Srivatsa S. Bhat

Coming to git bisect, I think it needs to be done between a66b2e5 + this-patch and 3.10 + this-patch, to ensure that we look for problematic commits in-between a66b2e5 and 3.10 (to avoid the bisect result ending up at a66b2e5 again).

Toralf,

Try reverting these patches in the order specified (revert both) 419e172145cf6c51d436a8bf4afcd17511f0ff79 c28375583b6471c1cb833a628ab6afb5b69079d0

They are touching ondemand governor and this may be creating some problems. But I am not sure about it, I must admit.

maintainer
#37 lore raw
Quoted 9 lines · Viresh Kumar
Quoted 3 lines · Srivatsa S. Bhat

Coming to git bisect, I think it needs to be done between a66b2e5 + this-patch and 3.10 + this-patch, to ensure that we look for problematic commits in-between a66b2e5 and 3.10 (to avoid the bisect result ending up at a66b2e5 again).

Toralf,

Try reverting these patches in the order specified (revert both) 419e172145cf6c51d436a8bf4afcd17511f0ff79

Ok, I did a mistake here (Thanks Srivatsa for telling me), this wasn't there in 3.10 but 3.11-rc1

Quoted 4 lines

c28375583b6471c1cb833a628ab6afb5b69079d0

They are touching ondemand governor and this may be creating some problems. But I am not sure about it, I must admit.

So just try reverting about commit.

In case you are using acpi-cpufreq driver, please try reverting this before above one:

8673b83bf2f013379453b4779047bf3c6ae387e4

It is playing with how frequencies are reported and may be the culprit.

#38 lore raw
Quoted 5 lines · Viresh Kumar

Ok, I did a mistake here (Thanks Srivatsa for telling me), this wasn't there in 3.10 but 3.11-rc1

Quoted 2 lines
Quoted 2 lines

c28375583b6471c1cb833a628ab6afb5b69079d0

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

They are touching ondemand governor and this may be creating some problems. But I am not sure about it, I must admit.

So just try reverting about commit.

In case you are using acpi-cpufreq driver, please try reverting this before above one:

8673b83bf2f013379453b4779047bf3c6ae387e4

I reverted 8673b83b on top of 3.10.0 - no success, then I reverted c2837558 too - no success.

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
maintainer
#39 lore raw
Quoted 8 lines · Toralf Förster
Quoted 7 lines · Viresh Kumar

Ok, I did a mistake here (Thanks Srivatsa for telling me), this wasn't there in 3.10 but 3.11-rc1

Quoted 4 lines
Quoted 4 lines

c28375583b6471c1cb833a628ab6afb5b69079d0

They are touching ondemand governor and this may be creating some problems. But I am not sure about it, I must admit.

Quoted 8 lines
Quoted 6 lines

So just try reverting about commit.

In case you are using acpi-cpufreq driver, please try reverting this before above one:

8673b83bf2f013379453b4779047bf3c6ae387e4

I reverted 8673b83b on top of 3.10.0 - no success, then I reverted c2837558 too - no success.

I assume that you have applied the patch given by Srivatsa earlier over these reverts?

In that case you can go for a revert I believe. There aren't many cpufreq patches between 3.10 and the commit where you tested successfully earlier.

But I now also suspect that this might have been caused by something outside cpufreq.

#40 lore raw
Quoted 2 lines · Viresh Kumar

I assume that you have applied the patch given by Srivatsa earlier over these reverts?

yes

Quoted 2 lines

But I now also suspect that this might have been caused by something outside cpufreq.

+1

At least the likelihood is big and the "saving" to narrow down bisecting just to the cpufreq patch is small compared to the "cost" of the risk of an unwanted side effect. OTOH not sure when I find a time frame for the bisect nightmare.
That's why I tweaked my s2ram script in this way:

+ if [[ "$1" = "mem" && "$(uname -r)" = "3.10.0" ]]; then
+ echo " tweak governor ..."
+ for g in performance ondemand; do
+ for i in 0 1 2 3; do
+ echo $g > /sys/devices/system/cpu/cpu$i/cpufreq/scaling_governor
+ done
+ done
+ echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice
+ fi

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#35 lore raw
In reply to Viresh Kumar
Quoted 7 lines · Viresh Kumar
Quoted 3 lines · Srivatsa S. Bhat

Coming to git bisect, I think it needs to be done between a66b2e5 + this-patch and 3.10 + this-patch, to ensure that we look for problematic commits in-between a66b2e5 and 3.10 (to avoid the bisect result ending up at a66b2e5 again).

Exactly!!

Problem : yesterday I wasn't even able to 100% validate the issue at commit a66b2e5 + f51e1eb just b/c there's another issue which erratically happens here: sometimes cpu0 is too (not only cpu 1-3). In that case the frequency are set to 2.6 GHz for each coret instead of 800 MHz for cpu0 and 2.601 GHz for cpu 1-3.

Furthermore I've the strong feeling, that the plugged in USB devices might have an effect. And in the mean while I would not even exclude the influence of the type of the 4 BOINC tasks (Asteorid@Home versus Einstein@Home / World Community Grid which I have to run with nice -19 in the back ground) at the bisect results.

So all my ongoing attempts to bisect weren't successful till now.

I just tested the latest git tree from today (v3.10-8615-g23e3a1d) - although f51e1eb is merged - the origin issue (cpu 1-3 set to 2.601 GHz after wakeup) is still here.

Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#19 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

In reply to Srivatsa S. Bhat

I tested the patch several times on top of a66b2e5 - the origin issue is fixed but - erratically another issue now appears : all 4 cores are runs after wakeup at 2.6 GHz.
The temporary hot fix is to switch between governor performance and ondemand for all 4 cores.

Quoted 61 lines · Srivatsa S. Bhat
Quoted 8 lines · Toralf Förster
Quoted 4 lines · Srivatsa S. Bhat

Toralf, can you please
try out the below patch and see if it improves anything? (Don't revert anything, just apply the below diff on a problematic kernel and see if it solves your issue).

applied on top of a66b2e5 - issue went away (either fixed or hidden now)

Cool! So here is the proper patch, with changelog added.

Regards,
Srivatsa S. Bhat

-----------------------------------------------------------------------------

From: Srivatsa S. Bhat <[email protected]>
Subject: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Toralf Förster reported that the cpufreq ondemand governor behaves erratically (doesn't scale well) after a suspend/resume cycle. The problem was that the cpufreq subsystem's idea of the cpu frequencies differed from the actual frequencies set in the hardware after a suspend/resume cycle. Toralf bisected the problem to commit a66b2e5 (cpufreq: Preserve sysfs files across suspend/resume).

Among other (harmless) things, that commit skipped the call to cpufreq_update_policy() in the resume path. But cpufreq_update_policy() plays an important role during resume, because it is responsible for checking if the BIOS changed the cpu frequencies behind our back and resynchronize the cpufreq subsystem's knowledge of the cpu frequencies, and update them accordingly.

So, restore the call to cpufreq_update_policy() in the resume path to fix the cpufreq regression.

---

 drivers/cpufreq/cpufreq_stats.c |    1 +
 1 file changed, 1 insertion(+)
drivers/cpufreq/cpufreq_stats.c +1 -0
index fb65dec..591b6fb 100644
@@ -349,6 +349,7 @@ static int __cpuinit cpufreq_stat_cpu_callback(struct notifier_block *nfb,
 
 	switch (action) {
 	case CPU_ONLINE:
+	case CPU_ONLINE_FROZEN:
 		cpufreq_update_policy(cpu);
 		break;
 	case CPU_DOWN_PREPARE:
Signature · 3 lines
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3
#20 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Hi Toralf,

Quoted 7 lines · Toralf Förster

I tested the patch several times on top of a66b2e5 - the origin issue is fixed but - erratically another issue now appears : all 4 cores are runs after wakeup at 2.6 GHz.
The temporary hot fix is to switch between governor performance and ondemand for all 4 cores.

Thanks for all your testing efforts. The commit a66b2e5 took a shortcut to achieve its goals but failed subtly in many aspects. Below is a patch (only compile-tested) which tries to achieve the goal (preserve sysfs files) in the proper way, by separating out the cpufreq-core bits from the sysfs bits. You might want to give it a try on current mainline and see if it improves anything.

Regards,
Srivatsa S. Bhat

(Applies on current mainline)
---

 drivers/cpufreq/cpufreq.c |  144 ++++++++++++++++++++++++++-------------------
 1 file changed, 82 insertions(+), 62 deletions(-)
drivers/cpufreq/cpufreq.c +82 -62
index 6a015ad..28c690f 100644
@@ -834,11 +834,8 @@ static int cpufreq_add_dev_interface(unsigned int cpu,
 				     struct cpufreq_policy *policy,
 				     struct device *dev)
 {
-	struct cpufreq_policy new_policy;
 	struct freq_attr **drv_attr;
-	unsigned long flags;
 	int ret = 0;
-	unsigned int j;
 
 	/* prepare interface data */
 	ret = kobject_init_and_add(&policy->kobj, &ktype_cpufreq,
@@ -870,31 +867,10 @@ static int cpufreq_add_dev_interface(unsigned int cpu,
 			goto err_out_kobj_put;
 	}
 
-	write_lock_irqsave(&cpufreq_driver_lock, flags);
-	for_each_cpu(j, policy->cpus) {
-		per_cpu(cpufreq_cpu_data, j) = policy;
-		per_cpu(cpufreq_policy_cpu, j) = policy->cpu;
-	}
-	write_unlock_irqrestore(&cpufreq_driver_lock, flags);
-
 	ret = cpufreq_add_dev_symlink(cpu, policy);
 	if (ret)
 		goto err_out_kobj_put;
 
-	memcpy(&new_policy, policy, sizeof(struct cpufreq_policy));
-	/* assure that the starting sequence is run in __cpufreq_set_policy */
-	policy->governor = NULL;
-
-	/* set default policy */
-	ret = __cpufreq_set_policy(policy, &new_policy);
-	policy->user_policy.policy = policy->policy;
-	policy->user_policy.governor = policy->governor;
-
-	if (ret) {
-		pr_debug("setting policy failed\n");
-		if (cpufreq_driver->exit)
-			cpufreq_driver->exit(policy);
-	}
 	return ret;
 
 err_out_kobj_put:
@@ -905,7 +881,7 @@ err_out_kobj_put:
 
 #ifdef CONFIG_HOTPLUG_CPU
 static int cpufreq_add_policy_cpu(unsigned int cpu, unsigned int sibling,
-				  struct device *dev)
+				  struct device *dev, bool init_sysfs)
 {
 	struct cpufreq_policy *policy;
 	int ret = 0, has_target = !!cpufreq_driver->target;
@@ -933,30 +909,25 @@ static int cpufreq_add_policy_cpu(unsigned int cpu, unsigned int sibling,
 		__cpufreq_governor(policy, CPUFREQ_GOV_LIMITS);
 	}
 
-	ret = sysfs_create_link(&dev->kobj, &policy->kobj, "cpufreq");
-	if (ret) {
-		cpufreq_cpu_put(policy);
-		return ret;
+	if (init_sysfs) {
+		ret = sysfs_create_link(&dev->kobj, &policy->kobj, "cpufreq");
+		if (ret) {
+			cpufreq_cpu_put(policy);
+			return ret;
+		}
 	}
 
 	return 0;
 }
 #endif
 
-/**
- * cpufreq_add_dev - add a CPU device
- *
- * Adds the cpufreq interface for a CPU device.
- *
- * The Oracle says: try running cpufreq registration/unregistration concurrently
- * with with cpu hotplugging and all hell will break loose. Tried to clean this
- * mess up, but more thorough testing is needed. - Mathieu
- */
-static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
+static int __cpufreq_add_dev(struct device *dev, struct subsys_interface *sif,
+			     bool init_sysfs)
 {
 	unsigned int j, cpu = dev->id;
 	int ret = -ENOMEM;
 	struct cpufreq_policy *policy;
+	struct cpufreq_policy new_policy;
 	unsigned long flags;
 #ifdef CONFIG_HOTPLUG_CPU
 	struct cpufreq_governor *gov;
@@ -984,7 +955,8 @@ static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
 		struct cpufreq_policy *cp = per_cpu(cpufreq_cpu_data, sibling);
 		if (cp && cpumask_test_cpu(cpu, cp->related_cpus)) {
 			read_unlock_irqrestore(&cpufreq_driver_lock, flags);
-			return cpufreq_add_policy_cpu(cpu, sibling, dev);
+			return cpufreq_add_policy_cpu(cpu, sibling, dev,
+						      init_sysfs);
 		}
 	}
 	read_unlock_irqrestore(&cpufreq_driver_lock, flags);
@@ -1049,9 +1021,33 @@ static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
 	}
 #endif
 
-	ret = cpufreq_add_dev_interface(cpu, policy, dev);
-	if (ret)
-		goto err_out_unregister;
+	write_lock_irqsave(&cpufreq_driver_lock, flags);
+	for_each_cpu(j, policy->cpus) {
+		per_cpu(cpufreq_cpu_data, j) = policy;
+		per_cpu(cpufreq_policy_cpu, j) = policy->cpu;
+	}
+	write_unlock_irqrestore(&cpufreq_driver_lock, flags);
+
+	if (init_sysfs) {
+		ret = cpufreq_add_dev_interface(cpu, policy, dev);
+		if (ret)
+			goto err_out_unregister;
+	}
+
+	memcpy(&new_policy, policy, sizeof(struct cpufreq_policy));
+	/* assure that the starting sequence is run in __cpufreq_set_policy */
+	policy->governor = NULL;
+
+	/* set default policy */
+	ret = __cpufreq_set_policy(policy, &new_policy);
+	policy->user_policy.policy = policy->policy;
+	policy->user_policy.governor = policy->governor;
+
+	if (ret) {
+		pr_debug("setting policy failed\n");
+		if (cpufreq_driver->exit)
+			cpufreq_driver->exit(policy);
+	}
 
 	kobject_uevent(&policy->kobj, KOBJ_ADD);
 	module_put(cpufreq_driver->owner);
@@ -1081,6 +1077,20 @@ module_out:
 	return ret;
 }
 
+/**
+ * cpufreq_add_dev - add a CPU device
+ *
+ * Adds the cpufreq interface for a CPU device.
+ *
+ * The Oracle says: try running cpufreq registration/unregistration concurrently
+ * with with cpu hotplugging and all hell will break loose. Tried to clean this
+ * mess up, but more thorough testing is needed. - Mathieu
+ */
+static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
+{
+	return __cpufreq_add_dev(dev, sif, true);
+}
+
 static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
 {
 	int j;
@@ -1106,7 +1116,7 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
  * This routine frees the rwsem before returning.
  */
 static int __cpufreq_remove_dev(struct device *dev,
-		struct subsys_interface *sif)
+				struct subsys_interface *sif, bool remove_sysfs)
 {
 	unsigned int cpu = dev->id, ret, cpus;
 	unsigned long flags;
@@ -1145,9 +1155,9 @@ static int __cpufreq_remove_dev(struct device *dev,
 		cpumask_clear_cpu(cpu, data->cpus);
 	unlock_policy_rwsem_write(cpu);
 
-	if (cpu != data->cpu) {
+	if (cpu != data->cpu && remove_sysfs) {
 		sysfs_remove_link(&dev->kobj, "cpufreq");
-	} else if (cpus > 1) {
+	} else if (cpus > 1 && remove_sysfs) {
 		/* first sibling now owns the new sysfs dir */
 		cpu_dev = get_cpu_device(cpumask_first(data->cpus));
 		sysfs_remove_link(&cpu_dev->kobj, "cpufreq");
@@ -1184,26 +1194,25 @@ static int __cpufreq_remove_dev(struct device *dev,
 
 	/* If cpu is last user of policy, free policy */
 	if (cpus == 1) {
-		lock_policy_rwsem_read(cpu);
-		kobj = &data->kobj;
-		cmp = &data->kobj_unregister;
-		unlock_policy_rwsem_read(cpu);
-		kobject_put(kobj);
-
-		/* we need to make sure that the underlying kobj is actually
-		 * not referenced anymore by anybody before we proceed with
-		 * unloading.
-		 */
-		pr_debug("waiting for dropping of refcount\n");
-		wait_for_completion(cmp);
-		pr_debug("wait complete\n");
-
 		if (cpufreq_driver->exit)
 			cpufreq_driver->exit(data);
 
 		free_cpumask_var(data->related_cpus);
 		free_cpumask_var(data->cpus);
 		kfree(data);
+
+		if (remove_sysfs) {
+			lock_policy_rwsem_read(cpu);
+			kobj = &data->kobj;
+			cmp = &data->kobj_unregister;
+			unlock_policy_rwsem_read(cpu);
+			kobject_put(kobj);
+
+			pr_debug("waiting for dropping of refcount\n");
+			wait_for_completion(cmp);
+			pr_debug("wait complete\n");
+		}
+
 	} else if (cpufreq_driver->target) {
 		__cpufreq_governor(data, CPUFREQ_GOV_START);
 		__cpufreq_governor(data, CPUFREQ_GOV_LIMITS);
@@ -1221,7 +1230,7 @@ static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif)
 	if (cpu_is_offline(cpu))
 		return 0;
 
-	retval = __cpufreq_remove_dev(dev, sif);
+	retval = __cpufreq_remove_dev(dev, sif, true);
 	return retval;
 }
 
@@ -1943,13 +1952,24 @@ static int __cpuinit cpufreq_cpu_callback(struct notifier_block *nfb,
 		case CPU_ONLINE:
 			cpufreq_add_dev(dev, NULL);
 			break;
+		case CPU_ONLINE_FROZEN:
+			__cpufreq_add_dev(dev, NULL, false);
+			break;
+
 		case CPU_DOWN_PREPARE:
 		case CPU_UP_CANCELED_FROZEN:
-			__cpufreq_remove_dev(dev, NULL);
+			__cpufreq_remove_dev(dev, NULL, true);
+			break;
+		case CPU_DOWN_PREPARE_FROZEN:
+			__cpufreq_remove_dev(dev, NULL, false);
 			break;
+
 		case CPU_DOWN_FAILED:
 			cpufreq_add_dev(dev, NULL);
 			break;
+		case CPU_DOWN_FAILED_FROZEN:
+			__cpufreq_add_dev(dev, NULL, false);
+			break;
 		}
 	}
 	return NOTIFY_OK;
#21 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

2013/7/11 Srivatsa S. Bhat <[email protected]>:

Quoted 234 lines

Hi Toralf,

Quoted 7 lines · Toralf Förster

I tested the patch several times on top of a66b2e5 - the origin issue is fixed but - erratically another issue now appears : all 4 cores are runs after wakeup at 2.6 GHz.
The temporary hot fix is to switch between governor performance and ondemand for all 4 cores.

Thanks for all your testing efforts. The commit a66b2e5 took a shortcut to achieve its goals but failed subtly in many aspects. Below is a patch (only compile-tested) which tries to achieve the goal (preserve sysfs files) in the proper way, by separating out the cpufreq-core bits from the sysfs bits. You might want to give it a try on current mainline and see if it improves anything.

Regards,
Srivatsa S. Bhat

(Applies on current mainline)
---

 drivers/cpufreq/cpufreq.c |  144 ++++++++++++++++++++++++++-------------------
 1 file changed, 82 insertions(+), 62 deletions(-)
drivers/cpufreq/cpufreq.c +56 -60
index 6a015ad..28c690f 100644
@@ -834,11 +834,8 @@ static int cpufreq_add_dev_interface(unsigned int cpu,
                                     struct cpufreq_policy *policy,
                                     struct device *dev)
 {
-       struct cpufreq_policy new_policy;
        struct freq_attr **drv_attr;
-       unsigned long flags;
        int ret = 0;
-       unsigned int j;
 
        /* prepare interface data */
        ret = kobject_init_and_add(&policy->kobj, &ktype_cpufreq,
@@ -870,31 +867,10 @@ static int cpufreq_add_dev_interface(unsigned int cpu,
                        goto err_out_kobj_put;
        }
 
-       write_lock_irqsave(&cpufreq_driver_lock, flags);
-       for_each_cpu(j, policy->cpus) {
-               per_cpu(cpufreq_cpu_data, j) = policy;
-               per_cpu(cpufreq_policy_cpu, j) = policy->cpu;
-       }
-       write_unlock_irqrestore(&cpufreq_driver_lock, flags);
-
        ret = cpufreq_add_dev_symlink(cpu, policy);
        if (ret)
                goto err_out_kobj_put;
 
-       memcpy(&new_policy, policy, sizeof(struct cpufreq_policy));
-       /* assure that the starting sequence is run in __cpufreq_set_policy */
-       policy->governor = NULL;
-
-       /* set default policy */
-       ret = __cpufreq_set_policy(policy, &new_policy);
-       policy->user_policy.policy = policy->policy;
-       policy->user_policy.governor = policy->governor;
-
-       if (ret) {
-               pr_debug("setting policy failed\n");
-               if (cpufreq_driver->exit)
-                       cpufreq_driver->exit(policy);
-       }
        return ret;
 
 err_out_kobj_put:
@@ -905,7 +881,7 @@ err_out_kobj_put:
 
 #ifdef CONFIG_HOTPLUG_CPU
 static int cpufreq_add_policy_cpu(unsigned int cpu, unsigned int sibling,
-                                 struct device *dev)
+                                 struct device *dev, bool init_sysfs)
 {
        struct cpufreq_policy *policy;
        int ret = 0, has_target = !!cpufreq_driver->target;
@@ -933,30 +909,25 @@ static int cpufreq_add_policy_cpu(unsigned int cpu, unsigned int sibling,
                __cpufreq_governor(policy, CPUFREQ_GOV_LIMITS);
        }
 
-       ret = sysfs_create_link(&dev->kobj, &policy->kobj, "cpufreq");
-       if (ret) {
-               cpufreq_cpu_put(policy);
-               return ret;
+       if (init_sysfs) {
+               ret = sysfs_create_link(&dev->kobj, &policy->kobj, "cpufreq");
+               if (ret) {
+                       cpufreq_cpu_put(policy);
+                       return ret;
+               }
        }
 
        return 0;
 }
 #endif
 
-/**
- * cpufreq_add_dev - add a CPU device
- *
- * Adds the cpufreq interface for a CPU device.
- *
- * The Oracle says: try running cpufreq registration/unregistration concurrently
- * with with cpu hotplugging and all hell will break loose. Tried to clean this
- * mess up, but more thorough testing is needed. - Mathieu
- */
-static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
+static int __cpufreq_add_dev(struct device *dev, struct subsys_interface *sif,
+                            bool init_sysfs)
 {
        unsigned int j, cpu = dev->id;
        int ret = -ENOMEM;
        struct cpufreq_policy *policy;
+       struct cpufreq_policy new_policy;
        unsigned long flags;
 #ifdef CONFIG_HOTPLUG_CPU
        struct cpufreq_governor *gov;
@@ -984,7 +955,8 @@ static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
                struct cpufreq_policy *cp = per_cpu(cpufreq_cpu_data, sibling);
                if (cp && cpumask_test_cpu(cpu, cp->related_cpus)) {
                        read_unlock_irqrestore(&cpufreq_driver_lock, flags);
-                       return cpufreq_add_policy_cpu(cpu, sibling, dev);
+                       return cpufreq_add_policy_cpu(cpu, sibling, dev,
+                                                     init_sysfs);
                }
        }
        read_unlock_irqrestore(&cpufreq_driver_lock, flags);
@@ -1049,9 +1021,33 @@ static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
        }
 #endif
 
-       ret = cpufreq_add_dev_interface(cpu, policy, dev);
-       if (ret)
-               goto err_out_unregister;
+       write_lock_irqsave(&cpufreq_driver_lock, flags);
+       for_each_cpu(j, policy->cpus) {
+               per_cpu(cpufreq_cpu_data, j) = policy;
+               per_cpu(cpufreq_policy_cpu, j) = policy->cpu;
+       }
+       write_unlock_irqrestore(&cpufreq_driver_lock, flags);
+
+       if (init_sysfs) {
+               ret = cpufreq_add_dev_interface(cpu, policy, dev);
+               if (ret)
+                       goto err_out_unregister;
+       }
+
+       memcpy(&new_policy, policy, sizeof(struct cpufreq_policy));
+       /* assure that the starting sequence is run in __cpufreq_set_policy */
+       policy->governor = NULL;
+
+       /* set default policy */
+       ret = __cpufreq_set_policy(policy, &new_policy);
+       policy->user_policy.policy = policy->policy;
+       policy->user_policy.governor = policy->governor;
+
+       if (ret) {
+               pr_debug("setting policy failed\n");
+               if (cpufreq_driver->exit)
+                       cpufreq_driver->exit(policy);
+       }
 
        kobject_uevent(&policy->kobj, KOBJ_ADD);
        module_put(cpufreq_driver->owner);
@@ -1081,6 +1077,20 @@ module_out:
        return ret;
 }
 
+/**
+ * cpufreq_add_dev - add a CPU device
+ *
+ * Adds the cpufreq interface for a CPU device.
+ *
+ * The Oracle says: try running cpufreq registration/unregistration concurrently
+ * with with cpu hotplugging and all hell will break loose. Tried to clean this
+ * mess up, but more thorough testing is needed. - Mathieu
+ */
+static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
+{
+       return __cpufreq_add_dev(dev, sif, true);
+}
+
 static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
 {
        int j;
@@ -1106,7 +1116,7 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
  * This routine frees the rwsem before returning.
  */
 static int __cpufreq_remove_dev(struct device *dev,
-               struct subsys_interface *sif)
+                               struct subsys_interface *sif, bool remove_sysfs)
 {
        unsigned int cpu = dev->id, ret, cpus;
        unsigned long flags;
@@ -1145,9 +1155,9 @@ static int __cpufreq_remove_dev(struct device *dev,
                cpumask_clear_cpu(cpu, data->cpus);
        unlock_policy_rwsem_write(cpu);
 
-       if (cpu != data->cpu) {
+       if (cpu != data->cpu && remove_sysfs) {
                sysfs_remove_link(&dev->kobj, "cpufreq");
-       } else if (cpus > 1) {
+       } else if (cpus > 1 && remove_sysfs) {
                /* first sibling now owns the new sysfs dir */
                cpu_dev = get_cpu_device(cpumask_first(data->cpus));
                sysfs_remove_link(&cpu_dev->kobj, "cpufreq");
@@ -1184,26 +1194,25 @@ static int __cpufreq_remove_dev(struct device *dev,
 
        /* If cpu is last user of policy, free policy */
        if (cpus == 1) {
-               lock_policy_rwsem_read(cpu);
-               kobj = &data->kobj;
-               cmp = &data->kobj_unregister;
-               unlock_policy_rwsem_read(cpu);
-               kobject_put(kobj);
-
-               /* we need to make sure that the underlying kobj is actually
-                * not referenced anymore by anybody before we proceed with
-                * unloading.
-                */
-               pr_debug("waiting for dropping of refcount\n");
-               wait_for_completion(cmp);
-               pr_debug("wait complete\n");
-
                if (cpufreq_driver->exit)

drivers/cpufreq/cpufreq.c static int __cpufreq_remove_dev(struct device *dev, · line 1207

Quoted 10 lines
drivers/cpufreq/cpufreq.c +5 -0
                        cpufreq_driver->exit(data);
 
                free_cpumask_var(data->related_cpus);
                free_cpumask_var(data->cpus);
                kfree(data);
+
+               if (remove_sysfs) {
+                       lock_policy_rwsem_read(cpu);
+                       kobj = &data->kobj;
+                       cmp = &data->kobj_unregister;

This looks no right. "data" has already been freed but data-> kobj still is referenced.

Quoted 51 lines
drivers/cpufreq/cpufreq.c +21 -3
+                       unlock_policy_rwsem_read(cpu);
+                       kobject_put(kobj);
+
+                       pr_debug("waiting for dropping of refcount\n");
+                       wait_for_completion(cmp);
+                       pr_debug("wait complete\n");
+               }
+
        } else if (cpufreq_driver->target) {
                __cpufreq_governor(data, CPUFREQ_GOV_START);
                __cpufreq_governor(data, CPUFREQ_GOV_LIMITS);
@@ -1221,7 +1230,7 @@ static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif)
        if (cpu_is_offline(cpu))
                return 0;
 
-       retval = __cpufreq_remove_dev(dev, sif);
+       retval = __cpufreq_remove_dev(dev, sif, true);
        return retval;
 }
 
@@ -1943,13 +1952,24 @@ static int __cpuinit cpufreq_cpu_callback(struct notifier_block *nfb,
                case CPU_ONLINE:
                        cpufreq_add_dev(dev, NULL);
                        break;
+               case CPU_ONLINE_FROZEN:
+                       __cpufreq_add_dev(dev, NULL, false);
+                       break;
+
                case CPU_DOWN_PREPARE:
                case CPU_UP_CANCELED_FROZEN:
-                       __cpufreq_remove_dev(dev, NULL);
+                       __cpufreq_remove_dev(dev, NULL, true);
+                       break;
+               case CPU_DOWN_PREPARE_FROZEN:
+                       __cpufreq_remove_dev(dev, NULL, false);
                        break;
+
                case CPU_DOWN_FAILED:
                        cpufreq_add_dev(dev, NULL);
                        break;
+               case CPU_DOWN_FAILED_FROZEN:
+                       __cpufreq_add_dev(dev, NULL, false);
+                       break;
                }
        }
        return NOTIFY_OK;
 
--

To unsubscribe from this list: send the line "unsubscribe cpufreq" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Best regards
Tianyu Lan
#22 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Quoted 29 lines · Lan Tianyu

2013/7/11 Srivatsa S. Bhat <[email protected]>:

Quoted 28 lines

Hi Toralf,

Quoted 7 lines · Toralf Förster

I tested the patch several times on top of a66b2e5 - the origin issue is fixed but - erratically another issue now appears : all 4 cores are runs after wakeup at 2.6 GHz.
The temporary hot fix is to switch between governor performance and ondemand for all 4 cores.

Thanks for all your testing efforts. The commit a66b2e5 took a shortcut to achieve its goals but failed subtly in many aspects. Below is a patch (only compile-tested) which tries to achieve the goal (preserve sysfs files) in the proper way, by separating out the cpufreq-core bits from the sysfs bits. You might want to give it a try on current mainline and see if it improves anything.

Regards,
Srivatsa S. Bhat

(Applies on current mainline)
---

 drivers/cpufreq/cpufreq.c |  144 ++++++++++++++++++++++++++-------------------
 1 file changed, 82 insertions(+), 62 deletions(-)

[...]

Quoted 59 lines
Quoted 59 lines
drivers/cpufreq/cpufreq.c +17 -17
+/**
+ * cpufreq_add_dev - add a CPU device
+ *
+ * Adds the cpufreq interface for a CPU device.
+ *
+ * The Oracle says: try running cpufreq registration/unregistration concurrently
+ * with with cpu hotplugging and all hell will break loose. Tried to clean this
+ * mess up, but more thorough testing is needed. - Mathieu
+ */
+static int cpufreq_add_dev(struct device *dev, struct subsys_interface *sif)
+{
+       return __cpufreq_add_dev(dev, sif, true);
+}
+
 static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
 {
        int j;
@@ -1106,7 +1116,7 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu)
  * This routine frees the rwsem before returning.
  */
 static int __cpufreq_remove_dev(struct device *dev,
-               struct subsys_interface *sif)
+                               struct subsys_interface *sif, bool remove_sysfs)
 {
        unsigned int cpu = dev->id, ret, cpus;
        unsigned long flags;
@@ -1145,9 +1155,9 @@ static int __cpufreq_remove_dev(struct device *dev,
                cpumask_clear_cpu(cpu, data->cpus);
        unlock_policy_rwsem_write(cpu);
 
-       if (cpu != data->cpu) {
+       if (cpu != data->cpu && remove_sysfs) {
                sysfs_remove_link(&dev->kobj, "cpufreq");
-       } else if (cpus > 1) {
+       } else if (cpus > 1 && remove_sysfs) {
                /* first sibling now owns the new sysfs dir */
                cpu_dev = get_cpu_device(cpumask_first(data->cpus));
                sysfs_remove_link(&cpu_dev->kobj, "cpufreq");
@@ -1184,26 +1194,25 @@ static int __cpufreq_remove_dev(struct device *dev,
 
        /* If cpu is last user of policy, free policy */
        if (cpus == 1) {
-               lock_policy_rwsem_read(cpu);
-               kobj = &data->kobj;
-               cmp = &data->kobj_unregister;
-               unlock_policy_rwsem_read(cpu);
-               kobject_put(kobj);
-
-               /* we need to make sure that the underlying kobj is actually
-                * not referenced anymore by anybody before we proceed with
-                * unloading.
-                */
-               pr_debug("waiting for dropping of refcount\n");
-               wait_for_completion(cmp);
-               pr_debug("wait complete\n");
-
                if (cpufreq_driver->exit)
                        cpufreq_driver->exit(data);
 
Quoted 12 lines

drivers/cpufreq/cpufreq.c static int __cpufreq_remove_dev(struct device *dev, · line 1207

Quoted 8 lines
drivers/cpufreq/cpufreq.c +5 -0
                free_cpumask_var(data->related_cpus);
                free_cpumask_var(data->cpus);
                kfree(data);
+
+               if (remove_sysfs) {
+                       lock_policy_rwsem_read(cpu);
+                       kobj = &data->kobj;
+                       cmp = &data->kobj_unregister;

This looks no right. "data" has already been freed but data-> kobj still is referenced.

Oops! You are right. Hmm, this looks quite difficult to get right :( There are multiple challenges here:

1. The sysfs files must not be removed during cpu_down, and not initialized

during cpu_up. That would help us preserve the file permissions.
2. But we should ensure that we really do the cpufreq-core parts of the cpu initialization during cpu_up. If we fail to free some of the data-structures during cpu_down, the cpu_up callback will think that a full-init is not required and not do its job. That will make cpufreq behave erratically after suspend/resume and take us back to square one.

3. A full re-init in the cpu_up callback also involves memory allocations. So if we don't release the memory in the cpu_down callback, we'll end up in a memory leak.

I tried to address all these in this patch, but you found yet another serious loop-hole. I guess I'm out of ideas now... if anybody has any thoughts on how to get this right, then I'm all ears. Else, we'll just revert the original commit like Rafael suggested and leave it upto userspace to save and restore the permissions across suspend/resume if it wants ;-(

Quoted 15 lines
Quoted 15 lines
drivers/cpufreq/cpufreq.c +8 -0
+                       unlock_policy_rwsem_read(cpu);
+                       kobject_put(kobj);
+
+                       pr_debug("waiting for dropping of refcount\n");
+                       wait_for_completion(cmp);
+                       pr_debug("wait complete\n");
+               }
+
        } else if (cpufreq_driver->target) {
                __cpufreq_governor(data, CPUFREQ_GOV_START);
                __cpufreq_governor(data, CPUFREQ_GOV_LIMITS);
@@ -1221,7 +1230,7 @@ static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif)
        if (cpu_is_offline(cpu))
                return 0;
 
Quoted 4 lines

drivers/cpufreq/cpufreq.c static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif) · line 1235

Quoted 4 lines
drivers/cpufreq/cpufreq.c +1 -1
-       retval = __cpufreq_remove_dev(dev, sif);
+       retval = __cpufreq_remove_dev(dev, sif, true);
        return retval;
 }

Regards,
Srivatsa S. Bhat

#23 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

2013/7/11 Srivatsa S. Bhat <[email protected]>:

Quoted 4 lines · Lan Tianyu
Quoted 1 line

2013/7/11 Srivatsa S. Bhat <[email protected]>:

Oops! You are right. Hmm, this looks quite difficult to get right :( There are multiple challenges here:

If I understand correctly, the most concern is how to deal with per cpus' cpufreq_policy structure. If something wrong , please correct me.

Quoted 4 lines

1. The sysfs files must not be removed during cpu_down, and not initialized

during cpu_up. That would help us preserve the file permissions.

For this case, cpufreq_policy must be reserved since all related cpufreq data and kobj is also store into it. We can't release it.

Quoted 6 lines

2. But we should ensure that we really do the cpufreq-core parts of the cpu initialization during cpu_up. If we fail to free some of the data-structures during cpu_down, the cpu_up callback will think that a full-init is not required and not do its job. That will make cpufreq behave erratically after suspend/resume and take us back to square one.

cpufreq_add_dev() checks whether the cpu has been attached cpufreq_cpu_data and associated kobj has been created. If yes, it would try to release it by cpufreq_cpu_put() and return directly. This seems to be conflicted with the above.

Quoted 4 lines

3. A full re-init in the cpu_up callback also involves memory allocations. So if we don't release the memory in the cpu_down callback, we'll end up in a memory leak.

Even we remove the previous check, cpu scaling driver's init() also will change some data of struct cpufreq_policy and this is also what we don't like to see.

Quoted 6 lines

I tried to address all these in this patch, but you found yet another serious loop-hole. I guess I'm out of ideas now... if anybody has any thoughts on how to get this right, then I'm all ears. Else, we'll just revert the original commit like Rafael suggested and leave it upto userspace to save and restore the permissions across suspend/resume if it wants ;-(

How about implement scaling driver's suspend/resume callback()? Although this needs to be dealt with case by case. If one's callbacks hasn't been implemented, it would have to follow current rule.

Quoted 23 lines
Quoted 15 lines
Quoted 15 lines
drivers/cpufreq/cpufreq.c +8 -0
+                       unlock_policy_rwsem_read(cpu);
+                       kobject_put(kobj);
+
+                       pr_debug("waiting for dropping of refcount\n");
+                       wait_for_completion(cmp);
+                       pr_debug("wait complete\n");
+               }
+
        } else if (cpufreq_driver->target) {
                __cpufreq_governor(data, CPUFREQ_GOV_START);
                __cpufreq_governor(data, CPUFREQ_GOV_LIMITS);
@@ -1221,7 +1230,7 @@ static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif)
        if (cpu_is_offline(cpu))
                return 0;
 
Quoted 4 lines

drivers/cpufreq/cpufreq.c static int cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif) · line 1235

Quoted 4 lines
drivers/cpufreq/cpufreq.c +1 -1
-       retval = __cpufreq_remove_dev(dev, sif);
+       retval = __cpufreq_remove_dev(dev, sif, true);
        return retval;
 }

Regards,
Srivatsa S. Bhat

Signature · 2 lines
Best regards
Tianyu Lan
#25 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

In reply to Srivatsa S. Bhat
Quoted 16 lines · Srivatsa S. Bhat

Oops! You are right. Hmm, this looks quite difficult to get right :( There are multiple challenges here:

1. The sysfs files must not be removed during cpu_down, and not initialized

during cpu_up. That would help us preserve the file permissions.
2. But we should ensure that we really do the cpufreq-core parts of the cpu initialization during cpu_up. If we fail to free some of the data-structures during cpu_down, the cpu_up callback will think that a full-init is not required and not do its job. That will make cpufreq behave erratically after suspend/resume and take us back to square one.

3. A full re-init in the cpu_up callback also involves memory allocations. So if we don't release the memory in the cpu_down callback, we'll end up in a memory leak.

Quoted 5 lines

I tried to address all these in this patch, but you found yet another serious loop-hole. I guess I'm out of ideas now... if anybody has any thoughts on how to get this right, then I'm all ears. Else, we'll just revert the original commit like Rafael suggested and leave it upto userspace to save and restore the permissions across suspend/resume if it wants ;-(

Asking as a naive outsider who is completely unfamiliar with the code, why are any of these things at all troublesome?

Can't cpu_up tell the difference between activating a brand-new CPU and reactivating one that was present before but was temporarily disabled?

Doesn't cpu_up know which data structures get freed when an active CPU is temporarily deactivated?

Doesn't cpu_down know what memory gets allocated in cpu_up? Can't it deallocate just the right parts for the type of transition it is doing?

It sounds like you're really asking how to make sure that cpu_up and cpu_down both know what the other is doing, so that each can do the opposite of the other. That doesn't sound hard.

Alan Stern

#24 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

In reply to Lan Tianyu
Quoted 9 lines · Lan Tianyu

2013/7/11 Srivatsa S. Bhat <[email protected]>:

Quoted 6 lines

I tried to address all these in this patch, but you found yet another serious loop-hole. I guess I'm out of ideas now... if anybody has any thoughts on how to get this right, then I'm all ears. Else, we'll just revert the original commit like Rafael suggested and leave it upto userspace to save and restore the permissions across suspend/resume if it wants ;-(

Quoted 4 lines

How about implement scaling driver's suspend/resume callback()? Although this needs to be dealt with case by case. If one's callbacks hasn't been implemented, it would have to follow current rule.

Well, I'm now trying a slightly different approach at reorganizing the code, and so far I think I'll be able to get it right this time. Let's see how it goes.

Regards,
Srivatsa S. Bhat

#26 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

In reply to Alan Stern
Quoted 34 lines · Alan Stern
Quoted 16 lines · Srivatsa S. Bhat

Oops! You are right. Hmm, this looks quite difficult to get right :( There are multiple challenges here:

1. The sysfs files must not be removed during cpu_down, and not initialized

during cpu_up. That would help us preserve the file permissions.
2. But we should ensure that we really do the cpufreq-core parts of the cpu initialization during cpu_up. If we fail to free some of the data-structures during cpu_down, the cpu_up callback will think that a full-init is not required and not do its job. That will make cpufreq behave erratically after suspend/resume and take us back to square one.

3. A full re-init in the cpu_up callback also involves memory allocations. So if we don't release the memory in the cpu_down callback, we'll end up in a memory leak.

Quoted 5 lines

I tried to address all these in this patch, but you found yet another serious loop-hole. I guess I'm out of ideas now... if anybody has any thoughts on how to get this right, then I'm all ears. Else, we'll just revert the original commit like Rafael suggested and leave it upto userspace to save and restore the permissions across suspend/resume if it wants ;-(

Asking as a naive outsider who is completely unfamiliar with the code, why are any of these things at all troublesome?

Can't cpu_up tell the difference between activating a brand-new CPU and reactivating one that was present before but was temporarily disabled?

Doesn't cpu_up know which data structures get freed when an active CPU is temporarily deactivated?

Quoted 8 lines

Doesn't cpu_down know what memory gets allocated in cpu_up? Can't it deallocate just the right parts for the type of transition it is doing?

It sounds like you're really asking how to make sure that cpu_up and cpu_down both know what the other is doing, so that each can do the opposite of the other. That doesn't sound hard.

It would not have been hard if the code had been already structured that way. Currently, the code is quite a bit entangled, and it doesn't distinguish the "temporarily deactivated" and the "fully torn-down" cases. To add to the mess, the code is generously sprinkled with notifiers that are invoked every now and then (which depend on previous init steps etc).

But the bottomline is that this is purely a code-reorganization issue, nothing more than that. And hopefully we'll be able to separate out the "temporarily deactivated" and the "fully down" cases reasonably well, such that we can preserve the sysfs file permissions easily across suspend/resume. That's the code reorganization I'm working on at the moment; I'll post it out as soon as I'm done.

Regards,
Srivatsa S. Bhat

#27 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

In reply to Toralf Förster
Quoted 2 lines · Toralf Förster

I tested the patch several times on top of a66b2e5 - the origin issue is fixed

Srivatsa's patch became commit f51e1eb63d ("cpufreq: Fix cpufreq regression after suspend/resume").

Quoted 2 lines

but - erratically another issue now appears : all 4 cores are runs
after wakeup at 2.6 GHz.

Well, a laptop I use seems to run into something related: the second of its two cores can get stuck at a fixed frequency after resume. Often it's its maximum frequency. That makes the CPU run hot, without actually doing much work. But it can also get stuck at its lowest frequency.

Please note that commit f51e1eb63d, which is part of v3.10.1-rc1, doesn't seem to cause this behavior. This issue can already be seen when running v3.10.0. So perhaps that commit just made this issue more noticeable to Toralf. But it's not clear to me what Toralf's origin(al?) issue actually was.

Anyhow, I suspect that the stuck frequency is a regression introduced in v3.10.0. But I have been unable to pinpoint it to a commit added in the v3.10 cycle through bisecting. I must have marked a commit good that was actually bad, just because a few suspend and resume cycles didn't trigger this issue. To be continued...

Quoted 2 lines

The temporary hot fix is to switch between governor performance and ondemand for all 4 cores.

That workaround works here too. I switch the core that is stuck at a particular frequency to some other governor and then back to ondemand.

Thanks,

Paul Bolle

#28 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Quoted 1 line · Paul Bolle

I suspect that the stuck frequency is a regression introduced in v3.10.0.

The culprit apparently is commit a66b2e503f ("cpufreq: Preserve sysfs files across suspend/resume"). Srivatsa submitted a patch to revert that commit (see https://lkml.org/lkml/2013/7/11/661 ). That revert seems to fix this issue too.

Paul Bolle

#29 lore raw

Re: [PATCH] cpufreq: Fix cpufreq regression after suspend/resume

Quoted 9 lines · Paul Bolle
Quoted 1 line · Paul Bolle

I suspect that the stuck frequency is a regression introduced in v3.10.0.

The culprit apparently is commit a66b2e503f ("cpufreq: Preserve sysfs files across suspend/resume"). Srivatsa submitted a patch to revert that commit (see https://lkml.org/lkml/2013/7/11/661 ). That revert seems to fix this issue too.

Thanks a lot for your tests and for confirming that the complete revert fixes your cpufreq issue.

Regards,
Srivatsa S. Bhat